Effectiveness of Vision Transformer for Fast and Accurate Single-Stage Pedestrian Detection
Authors: Jing Yuan
Barmpoutis Panagiotis
Tania Stathaki
Venue: NeurIPS 2022 Accept (2022)

Abstract: Vision transformers have demonstrated remarkable performance on a variety of computer vision tasks. In this paper, we illustrate the effectiveness of the deformable vision transformer for single-stage pedestrian detection and propose a spatial and multi-scale feature enhancement module, which aims to achieve the optimal balance between speed and accuracy. Performance improvement with vision transformers on various commonly used single-stage structures is demonstrated. The design of the proposed architecture is investigated in depth. Comprehensive comparisons with state-of-the-art single- and two-stage detectors on different pedestrian datasets are performed. The proposed detector achieves leading performance on Caltech and Citypersons datasets among single- and two-stage methods using fewer parameters than the baseline. The log-average miss rates for Reasonable and Heavy are decreased to 2.6% and 28.0% on the Caltech test set, and 10.9% and 38.6% on the Citypersons validation set, respectively. The proposed method outperforms SOTA two-stage detectors in the Heavy subset on the Citypersons validation set with considerably faster inference speed.

Effectiveness of Vision Transformer for Fast and Accurate Single-Stage Pedestrian Detection
Jing Yuan
Panagiotis Barmpoutis Department of Computer Science
Department of Electric and Electronic Engineering Imperial College London yyan200immerial. ac.uk
University College London
barnpoutisuucl. ac uk
Tania Stathaki
Department of Electric and Electronic Engineering Imperial College London
t. Serathaak@@mpppria ac. uk
Abstract
Vision transformers have demonstrated remarkable performance variety of computer vision tasks. this paper, we illustrate the effectiveness the de- formable vision transformer for single-stage pedestrian detection and propose spatial and multi-scale feature enhancement module, which aims to achieve the op- timal balance between speed and accuracy. Performance improvement with vision transformers on various commonly used single-stage structures is demonstrated. The design of the proposed architecture is investigated in depth. Comprehensive comparisons with tfte-offthe-art single- and two-stage detectors on different pedes- trian datasets performed. The proposed detector achieves leading performance on Caltech and Citypersons datasets among single- and two-stage methods using fewer parameters than the baseline. The log-average miss rates for Reasonable and Heavy are decreased to 2.6% and 28.0% on the Caltech test set, and 10.9% and 38.6% the Citypersons validation set, respectively. The proposed method outperforms SOTA two-stage detectors in the Heavy subset on the Citypersons validation with considerably faster inference speed.
Introduction
Pedestrian detection is popular task subordinate to object detection in computer vision. This task aims to locate and classify pedestrians images or videos accurately. Pedestrian detection is very important it serves the prerequisite of various vision tasks [1), such human-centric tasks (person re-identification
2 3, person search 44, human pose estimation S etc.) and more generic multi-object tracking ( has been applied to autonomous driving [7) video surveillance (9] and
action tracking. In this paper, we focus on the detection based on RGB images.
Pedestrian detection suffers from significant occlusion and varying scales. Intra- and inter-class occlusion occur when a pedestrian occluded by other pedestrians or objects like cars, bicycles etc. Both significantly reduce the discriminative features and destroy the regular shape of pedestrians. For varying scales, large pedestrians tend to have more informative features. Still, they are difficult to fully extract from vast region, while features of small targets compact but relatively ambiguous with less preserved details. In summary, the fluctuating amount and varying shape of effective features the core problems. They challenge the capabilities of feature extraction modules, which act long-standing bottleneck pedestrian detection.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

To deal with these problems, tto-stage methods [10 based on Fast [16] and Faster R-CNN [66 have pervaded pedestrian detection tasks owing to the high detection accuracy. These methods first make coarse predictions of targets via the Region Proposal Network (RPN), then refine the bounding boxes and predict the final scores based on the features inside the proposal regions. In addition to methods for general object detection, pedestrian detectors take advantage of unique characteristics of targets, such as the mask of visible parts [[7 [0] and key points of human bodies [88]. However, the inference speed of these methods is limited by the repeated predictions which makes them hard to be applied to real-world scenes.
To achieve faster inference, single-stage approaches, which only make one round prediction, are developed [19] and applied to pedestrian detection [20)24]. However, they suffer from decreased detection accuracy. The miss rates of two-stage methods in the Reasonable subset on Caltech are reduced to less than 4% [1 TO 25 18 while those for single-stage methods are larger than 4.5% [22]. For the Citypersons validation set, the miss rates for the former are less than 40% in the Heavy subset, which much lower than the latter (42% (24). In this paper, we aim to improve the detection accuracy, especially in Reasonable and Heavy subsets, and to narrow the gap between single- and two-stage methods with fast inference.
Typical single-stage detectors use anchors (SSD (26] or are anchor-free (CSP 22
The former generates rectangular bounding boxes with different aspect ratios and scales centered at each pixel of the feature maps at certain levels. Anchor scales are designed to be smaller at lower levels to facilitate the detection of small objects. These methods predict the offsets w.r.t. the upper left position, height and width of the anchor. Taking CSP as an example, the latter method only predicts the logarithm height and offsets w.r.t. centers of each pixel. For anchor-based single-stage detectors, ALFNet (21 refines anchors progressively with stacked prediction blocks to remedy the lack of proposal regions.
In the past year, most research has focused on the fusion of representative features 32] to improve single-stage methods. For example, (30] enhances features via increasing semantic information at a low level and enriches the localization information a high level. Similarly, [3] fuses the feature maps with different scales in adequate proportions. [29 31 also explore new strategies to aggregate multi-level features. These feature enhancements are mainly performed along the dimension of feature level due to the intrinsically unbalanced feature information between shallow and deep feature levels. However, this unhalance also exists in two-stage detectors. As such, this not the particular reason for the poorer accuracy of single-stage methods.
The general architecture of single- and two-stage detectors compared in Figure 1 Assuming that the training strategies and the detection head are the same for both methods, the difference in the architecture lies in the information fed into the detection head. For two-stage methods, both positions and features of the proposal regions are fed into the detection head. These proposals contain potential pedestrians. Thereby, the detection head classifies spatially target-focused features with fewer background interruptions and refines the bounding boxes by predicting small offsets from the proposal positions. For single-stage methods, each pixel in the feature map serves as the 'proposal region" with no pre-estimated positions. The receptive fields of these pixels share the same size, which may be too small to include sufficient information for large targets or so large that the background information overwhelms the useful features. This is more challenging for the classifier compared to two-stage methods. Additionally, single-stage methods have to regress from scratch, which is more difficult than simple refinement. Thus, the lack of spatially target-focused feature representation and the prediction of bounding boxes from scratch are the two key bottlenecks hindering the improvement of single-stage detectors.
To make the features fed into the detection head concentrate on the targets or other helpful information automatically without the assistance of proposal regions, we take advantage of vision transformers in this paper. Vision transformers describe the pairwise dependency of each entity in the feature map with attention weights. The output weighted averaged feature is the adaptive aggregation of important entities (with higher attention weights) while the disturbing information (with lower attention weights) is suppressed. Using such attention mechanism on top of the backbone enables the single-stage detector to supplement spatially filtered features easily for subsequent classification and regression. In this case, the modified detector makes the best use of the fast inference originating from single-stage methods and more effective features. Our main contributions are as follows:
Demonstrate the effectiveness of the deformable vision transformer in improving the accu- racy of commonly used single-stage detectors on pedestrian datasets.

---

Extend the application of vision transformers on top of the backbone in pedestrian detection tasks.
Achieve the best performance among single-stage detectors the Caltech test set and Citypersons validation set while maintaining fast inference and reducing the number of parameters.
Narrow the gap of detection accuracy between the single- and two-stage methods in pedes- trian detection.
Sactbone
Derection
head
Bounding
bnoos
Single stage
latt-slage
steps
Festure at one piacl
Figure 1:
Generic detection pipeline of the single- and two-stage detectors. The detection head consists of two parallel branches for classification and regression.
2 Related works
Currently, vision transformers are used to establish general general-purposs purpose backbones (ViT [33 and Swin Transformer (34) stack top of the backbone (DETR [35]). Since focus pedestrian detection, this paper explores the latter case. DETR consists of the convolution backbone, six encoders and decoders and the prediction head. It is inspiring end-to-end detector but is memory-consuminng DETR requires massive memory to store the self-attention weights within each Multi-Head Self- Attention (MHSA) layer. The memory cost is linear to the number of attention heads and is square to the number of pixels in the down-sampled feature map. Additionally, first and second-order momentums optimization introduce further memory cost in the training procedure. In pedestrian detection, more attention heads and relatively large down-sampled feature maps are preferred to enhance detection accuracy, especially for small targets. This results memory explosion using DETR. To this end, deformable DETR [36 is proposed. It only needs the attention weights at several sampling locations rather than each pixel in the feature map. The memory cost of the attention weights is linear to the number of pixels, which makes training with high resolutions possible. Experiments show that the deformable DETR outperforms the Faster R-CNN [37 and DETR on COCO 2017 validation set [38]
So far, vision transformers show great potential, but they have rarely been applied pedestrian detection in the form of DETR or its variants. This is because it has been observed that they perform worse than the commonly used Faster R-CNN on CrowdHuman dataset (39] and require tenfold training time [40]. Although 40 proposed using dense object queries and the rectified attention field enhance scale-adas adaptive feature extraction in the decoding phase, the modified deformable DETR still shows a limited advantage over the traditional Faster R-CNN. This implies that rigidly putting the whole six encoder-decoder pairs into the pedestrian detector may not be cost-effective.. As an example, Bo Tee [4 only substitutes the convolution layers in residual bottleneck blocks in the last stage of ResNet 42 for MHSA layers, but produces strong performance on ImageNet validation set [43]. Inspired by this and the limitation above, our work only utilizes single encoder of the deformable vision transformer as an adaptive feature extractor and applies commonly used single-stage detectors for better detection accuracy and fast inference.
3 Method
Deformable Vision Transformer Encoder: The deformable vision transformer encoder (Figure
2 takes the L feature maps {x1) J==1 with height Hi and width Wi at scale ( and reference points,
3

---

Reference points
Value
Out
Lireer
Lineer
Softmax
Sansling location
Linear
Interpolation
Linear
18568
Weight
Multiplication
vafe oPery
points
maps
Deformable vision
MSDA
FFN
transformer encoder
Figure 2: Left: The structure of the deformable vision transformer encoder. It is the same as the encoder in DETR except that the key tensor is removed while reference points required. Mid: The MSDA averages the features at sparse sampling locations across different scales with weights computed from the query tensor. Right: Feed-Forward Network (FFN).
which are the positions of grid centers of the feature maps, as inputs. It outputs the enhanced feature maps with the same resolution as the input. The input feature maps are first added by fixed encoded positional [35] and learnable level information (36] to disambiguate spatial and scale positions, then projected to the query feature map 2g via linear layer. The feature maps also generate the value feature maps 2u with linear layer but without encoding. The query feature map 2g. value feature map & together with the pre-generated reference points are sent to the Multi-Scale Deformable Attention (MSDA) layer to enhance spatially adaptive features, followed by a Feed-Forward Network (FFN). In summary, the encoder supplements the semantic information of input feature maps via the embedded attention layer and FFN.
The MSDA layer sums the selected entities at sampling locations in the value feature map 2, with predicted attention weights by each corresponding query entity. These attention weights W are the linear projection of query features =q followed by softmax operator along scale and sampling point dimensions. For single query entity, only needs NN attention weights, representing the significance of selected value features at different attention heads, scales, and points. Selections are decided by the sampling locations which are the summation of reference points p and sampling offsets Ap which is the embedding of the query features. At each float sampling location, the selected value feature is bilinear interpolated for accuracy and training offset predictor. With weight, sampling locations and selected value features prepared, the q-th element of the separate output feature z00h Xc, (N₂ Σi= HiW,, Gu the number of channels ) at attention head h (total N2, heads) is
Multi-Scale Deformable A Attention:
Nº
80,8
W Wplhggu Applia
1=1
where p. g. l and h index the sampling offsets, clements of the deformable attention feature 2a. the scale of value and attention head. Wythg is a value from the weight W E NN,,,N xLxN, are the reference points and sampling offsets. Pat and
p E N and Ap E M1,,xN,xxxx00 Ap ppphig denote the position of a single reference point and one of its corresponding sampling offsets respectively. The separate output features from N₁, attention heads are projected to the q-th element of the final output deformable attention feature x" by linear layer expressed as
Z
W 1 s0.h
where WA Rexc. denotes the learnable weight for the h-th attention head.
Proposed Feature Enhancement Module: The features are enhanced owing to the self-attention mechanism to supplement the spatially adaptive features across multiple scales. The proposed module

---

Conv
Group Norm
Deformable
Deconv
encoder
Decony
L2Norm
1
Group
Input feature maps
Enhanced feature maps
Figure 3: The overall architecture of the proposed feature enhancement module.
simply consists of convolutionddeconvlluiin and normalization layer pairs ahead of and after the deformable encoder and final feature fusion step illustrated in Figure 3 In this module, input feature maps from the backbone are first upsampled with deconvolution layers or encoded with
the convolution layer to generate multi-scale feature maps { [11 1
|=1' They are followed by group normalization to prevent the Internal Covariate Shift (ICS) that might be inducted by subsequent linear operations in the deformable encoder. The encoder yields enhanced multi-scale feature maps. Enhanced features are upsampled to keep the resolution as (H/ W/4) for accurate detection. They are normalized via 1.2Norm 22 before concatenation along the channel dimension. This makes the features different scales contribute equally the final feature representation fed into the detection head. Concatenated features are compressed along the channel dimension to reduce the network parameters. The output feature maps can be fed into the detection head used SSD, CSP etc. Except for this standard structure, the use of convolution/deconolliii layers can be adjusted according to the resolution of input, and the concatenation step can be removed if predictions are made at separate levels.
Training: For anchor-free cases, namely CSP, the loss function follows [22] The overall loss consists of three parts as Equation(3) where Lee Ia and T. stand for the center heatmap loss, height map loss and offset loss. Weights for each loss Åc. X, and X are set as 0.01, and 0.1 [22]. For anchor-based cases, namely SSD or ALFNet, the multi-task loss function is formulated with two objectives as Equation
where Adt experimentally set as 0.01 in the following experiments.
Laf AeLLe Ashh AoLeo
Lab Adtoees Lioc
(4)
Inference:
For anchor-free single-stage methods, the predicted width is the height multiplied by the uniform aspect ratio 0.41 [22] If not specified, bounding boxes with scores above 0.01 are kept and merged by Non-Maximum Suppression (NMS) with the IoU threshold of 0.5.
4
Experiments
4.1
Settings
Datasets:
The proposed detector is evaluated on two commonly used public pedestrian datasets:
Caltech 44 and Citypersons (45) The Caltech dataset is an approximately 10 hours of 480x640
video taken in single urban city. The standard training set contains about 250k frames with 350k
bounding boxes. In our experiments, the training data augmented by 10 folds containing 42782
images with 13674 persons and the standard test set containing 4024 images with corresponding new annotations [46] and fixed aspect-ratio for bounding boxes [47 are used. Citypersons training set recorded across 18 different cities, seasons and various weather conditions with 19654 persons in 2975 high resolution (1024x2048) images. The validation set contains 500 images across citics.
Training Details: If not specified, the ResNet50/VGG pre-trained on ImageNet, Adam with moving average weights [48] and step learning rate schedule are applied. Data augmentation techniques including random horizontal flips with probability of 0.5 and scaling are applied. For Caltech, additional random color distortion and cropping are implemented. The input images are rescaled to 336x448 and 640x 1280 for Caltech and Citypersons datasets respectively. The detectors are trained with a single NVIDIA GeForce RTX 3090 GPU for 10 and 75 epochs with batch size 16 and on Caltech and Citypersons respectively using the anchor-free CSP detection head. The base learning rate 0.5e-4 and decreased by a factor 0.5 after and epochs respectively. Initialization performed with randomly chosen and fixed seed. Tools provided by [47] are used in the experiments.
5

---

Log-average Miss Rate (denoted as MR or miss rate in this paper) over False Positive Per Image (FPPI) in the range [10-2 10 calculated over Reasonable, Small, Heavy, and All subsets defined in Table 1 The lower the MR the better.
Metrics:
Table I: The experimental settings of four subsets.
Subset
Reasonable
Small
Heavy
All
Height
[50, inf] [0.65, inf]
[50, 75] [0.65, inf]
[50, inf]
[20, inf]
Visibility
[0.2, 0.65]
[0.2, inf]
4.2 Ablation Experiments
Effectiveness of Enhanced Feature Maps: For generality, the proposed module is applied to three commonly used single-stage structures pedestrian detection: anchor-based, progressive refineeen,, and anchor-free.
For anchor-based single-stage methods, like SSD300 (26). Table 2 shows that the proposed module shows stable and significant improvement in all three subsets with an IoU threshold of 0.5 in NMS.
For refined anchors, append two Convolutional Predictor Blocks [1] after the feature maps at the last three stages the backbone and an extra stage. The second block refines the coarse anchors predicted by the first block. Table demonstrates that even though the anchors progressively to remedy the lack of proposal regions, the separate enhancement at each level can bring improvement in certain subsets (level 0 improves Reasonable and All subsets, level 1, improves Heavy subset). The combination of multi-level inputs brings in stable improvement in all the subsets.
refined
For anchor-fres methods, We evaluated the influence of the proposed module the baseline CSP [2 detector on Caltech (Table D and Citypersons (Table Results from both datasets indicate that with the enhanced feature maps, the miss rates decreased significantly in Reasonable, Heavy and All subsets, particular, the Heavy subset witnesses a decrease of up to 7.9%, and the Reasonable subset up 3.1% %. The additional enhancement module increases the inference time, however, they contain fewer learnable parameters as shown in Table 6 As the combination of CSP achieves the best results, subsequent experiments follow this implementation.
According the above, the proposed module effective for general single-stage detectors on pedestrian datasets, owing to the multi-scale deformable self-attention mechanism to enhance spatially adaptive features across levels.
Table 2: Comparison of the SSD300 detector w/o the proposed feature enhancement module on Caltech test set with input size 300x300.
Enhanced feature levels
Methods
loU
Reasonable
Heavy
All
2
3
4
0.5
28.6
70.7
72.3
SSD300
0.25
27.1
71.3
71.2
0.5
24.9
69.3
71.0
0.25
26.2
70.2
71.3
0.5
25.1
68,4
71.3
0.25
26.9
69.1
71.6
0.5
22.6
68.5
71.0
+ Enhancement module
0.25
24.9
67.7
71.2
0.5
23.9
68.4
72.2
0.25
25.6
72.6
0.5
23.5
68.7
71.4
0.25
24.4
71.0
71.8
Feature Maps Scales: Different combinations of multi-scale feature maps {=()²_ with three downsampling ratios (1/4, 1/8 and 1/16) are compared in Table 7 In this comparison, only 23

---

Table 3: Comparison of the progressive refinement detector w/o the proposed feature enhancement module on Caltech test set with input size 480x640.
Enhanced feature levels
Reasonable
Heavy
All
2
3
Progressive refinement
13.4
61.3
61.9
/
12.3
60.8
61.2
13.9
59.2
61.7
Enhancement module
13.5
60.0
62.7
12.4
61.5
59.8
Table 4: Comparison of the baseline CSP detector and the proposed detectors with the enhancement module on Caltech test set. FPS stands for frames per second evaluated during test process with input size 480x640.
Reasonable
Heavy
All
FPS
CSP
6,8 3.7 (3.14)
50.7
62.3
39.2
Enhancement module
42.8 (7.94)
56.97 (5.34)
29.5
which has the same resolution as 2 upsampled to (H/4, W/4), and fed into the detection head without intermediate concatenation and I.2Norm. Results show that feature maps with ratios 1 /4, 1/8, and 1/6 for each scale produce the best performance, which yielded by upsampling e₁ and C2 by two times while maintaining the scale of C3-
Ji=1 1/4, 1/8, and 1/16, feed different collections of multi-scale enhanced feature maps {2()] 1=1 to the detection head. Note that the deformable encoder keeps the resolutions the input feature maps, the downsampling ratios for { are 1/4, 1/8 and 1/6 respectively. All the enhanced feature maps are first upsampled to (H/4, W/4) if needed, followed by normalization when multiple feature maps are utilized. They are then processed three ways: Cat: Concatenate them along the channel dimension followed by compression layer reduce the number of channels to 256. 2. Add: Implement element- wise sum of the enhanced feature maps. 3. Sep: No fusion operations across multi-scales; send them separately to the detection head which doubles the number of predictions. Table & compares various strategies to fuse the multi-scale enhanced feature maps. shows that concatenation followed by 1.2Norm produces the overall best results in both Reasonable and Heavy subsets the Caltech test set.
Enhanced Feature Map Scales: Fix the downsampling ratios of {21 {21)²
Choice of the Normalization Method Applied {c') As Table 8 shows, GN performs best in the Reasonable subset while L2N in the Heavy subset. Considering thaithe miss rate of GN is 11.6% lower in the Reasonable subset and only 1.2% higher in the Heavy subset compared to I.2N, GN is utilized in our experiments before the encoder as presented in Figure33
Number of Encoders: Based on the settings in the last part, different numbers of encoders are tested. These encoders are connected in series. Table higher-level semantic information, the best result observed when only single encoder is applied. This phenomenon supports the design of BotNet 4 to some extent, which indicates that following the whole set of six encoders and decoders in (deformable) DETR may not suitable for specific
9 presents that although more encoders can provide
Table Comparison of the baseline CSP detector and the proposed detectors with the enhancement module on Citypersons validation set. FPS stands for frames per second. FPS stands for frames per second evaluated during test process with input size 1024x2048.
Reasonable
Small
Heavy
All
FPS
CSP
11.7
14.4
41.8
38.2
8.2
+ Enhancement module
10.9 (0.8))
13.7 (0.71)
38.6 (3.24)
37.2 (1.0L)
6.8
7

---

Table 6: The number of learnable parameters in the baseline and proposed detectors.
Backbone
Neck
Head
Total Param
CSP
23.51 M
14.68 M
1.77 M 1.03 K
39.96 M
+ Enhancement module
23.51 M
8.73 M
32.24 M
Table 7: Combinations of different resolutions of input feature maps fed into the encoder.
x
Reasonable
Heavy
1/4
1/8
1/16
4.1
44.7
1/4
1/16
1/16
4.2
45.9
1/8
1/8
1/16
4.6
48.5
1/4
1/8
1/8
5.9
47.8
object detection tasks. A single encoder can also work effectively with the least number of learnable parameters, which prevents overfitting.
4.3 Comparison with the state-of-the-ard
Table 10 and Table Table[[ssoww show that the proposed module achieves the lowest miss rates Reasonable and Heavy subsets and leading performance in the All subset both datasets among presented single-stage detectors.
For the Caltech dataset, the lowest miss rate (3.7%) of the proposed detector in the Reasonable subset 0.2% smaller than that of the two-stage KGSNet presented the upper part of Table [O] With pretraining on the Citypersons dataset, the miss rates all three subsets of the Caltech dataset reduced significantly and reach the lowest compared to other pretrained detectors as shown in the bottom part of Table TO For the Citypersons dataset, the proposed two detectors even outperform the competitive two-stage detectors in the Reasonable and especially Heavy subset with a miss rate of 38.6% which is 1.1% lower than that of the best of two-stage detectors.
Overall, with the enhanced spatially adaptive and multi-scale features, the gap between single- and two-stage detectors in Reasonable and Heavy subsets on different detectors has been narrowed. Surprisingly, the proposed single-stage method outperforms the accurate two-stage methods on certain pedestrian datasets, such as the Citypersons dataset. It should be noted that two-stage methods usually produce overall better accuracy than single-stage methods with the advantage of the region proposal network and refined bounding boxes and at the expense of inference time. Apart from accuracy, fast inference also matters for pedestrian detection in practical scenes. The combination of CSP and the proposed module has simple structure, which is easy to perform and effective. On the contrary, the leading two-stage KGSNet, for example, takes advantage of the additional proposal
Table Combinations of enhanced feature maps with different scales in different ways followed by three Normalization Methods (NM). L,N, LN and GN stand for L2Norm, Layer Normalization and Group Normalization respectively. Train boldly without any learning rate schedules. Normalization methods applied to the enhanced feature maps after deconvolution. Normalization methods applied to the feature maps produced by the backbone before being sent to the encoder.
22
23
Cat
Add
Sep
2
L2N
L2N
GN
LN!
L2N
L2N
L2N
L2N
L2N
GN
GN
GN
GN
GN
GN
GN
GN
GN
GN
GN
L2N
BN
GN
Reasonable
4.3
5.7
4.1
7.0
3.7
4.2
5.9
6.4
4.0
3.9
3.7
4.3
4.5
5.3
Heavy
44.5
48.8
44.7
51.3
45.5
48.1
50.7
44.5
42.8
44.6
44.1

---

Table 9: Using to 4 encoders in the detection neck.
Number of encoders
3
4
Reasonable Heavy
4.9
4.8
3.3
42.8
46.6
43.1
43.1
Table 10; Comparison with the state-of-the-art single- and two-stage pedestrian detectors on Caltech test set. Results reported by [22). Bottom part: Pretrained on Citypersons and tested Caltech.
The best and second-best results are bold and underlined with red for two-stage methods (upper part) and black for single-stage methods.
Method
Backbone
Stage
Reasonable
Heavy
All
Faster R-CNN
37
ResNet50
2
8.7
53.1
62.6
AL.FNet 21
ResNet50
1
8.1
51.0
59.1
RPN+BF
VGG16
2
7.3
54.6
59.9
RepLoss
14
ResNet50
2
5.0
47.9
59.0
CSP 22
ResNer50
1
4.5
45.8
36.9
KGSNet [18
ResNer50
2
3.9
34.2
42.2
JointDet 50
ResNer50
2
3.0
PedHunter 10
ResNet50
2
2.3
CSP+Proposed module
ResNet50
42.8
56.97
ALFNet" 21
ResNet50
1
4.5
43.4
56.8
RepLoss 14
ResNet50
2
4.0
41.8
58.6
CSP [22]
ResNet50
1
3,8
36.5
54.4
CSP+Proposed module
ResNet50
1
2.6
28.0
53.9
generation network, the refined bounding boxes, the key-point detector and the super-resolution network. With these components. the inference speed KGSNet is 5.9 FPS and 3.2 FPS (Titan X GPU, not including the time of using ALFNet to generate the candidate proposals) on Caltech and Citypersons datasets [8] while ours achieves 29.5 FPS and 6.8 FPS (RTX 3090 GPU) respectively. Therefore, the enhanced single-stage pedestrian detector is cost-effective with fast inference and competitive accuracy compared complicated two-stage methods.
Table 11: Comparison with the state-of-the-us pedestrian detectors Citypersons validation set. Results reported by [22) Less than 65% visibility instead of 20-65% visibility. The best and second- best results are in bold and underlined with red for two-stage methods and black for single-stage methods.
Method
Backbone
Stage
Reasonable
Heavy
All
FRCNN
451
VGG16
15.4
FRCNN+Seg
45
VGG16
2
14.8
TLL+MRF
23
ResNet50
14.4
52.0
OR-CNN
VGG16
2
12.8
55.71
RepLoss
ResNet50
13.2
56,9
ALFNet 21
ResNet50
12.0
51.9
CSP [22
ResNet50
11.0
49.3
PRNet 24
ResNet50
10.8
42.0
MGAN+ I7
VGG16
11.0
39.7
KGSNet
[8
ResNe150
2
11.0
39,7
36.2
CSP+Proposed module
ResNet50
10.9
38.6
37.2

---

Conclusion
The paper proposes module enhance spatial and multi-scale features based on a single encoder of the deformable vision transformer to improve the detection accuracy of single-stage pedestrian detectors with fast inference. This module effective on commonly used single-stage structures, including (progressively refined) anchor-based and anchor-free cases. With the CSP detection head, more than 40% parameters are reduced the detection neck compared to CSP; however, this combination still achieves the best results in Reasonable and Heavy subsets among presented single-stage detectors on both Caltech and Citypersons datasets. Utilising pre-training. the miss rates for these subsets can be decreased to 2.6% and 28.0%, respectively, which are far better than other single-stage methods and comparable to two-stage methods. The proposed method outperforms SOTA two-stage detectors in the Heavy subset by 1.1% on Citypersons with slightly decreased inference speed. This demonstrates that single-stage detectors can be improved if spatially adaptive and multi-scale features are jointly adopted, making them cost-effective and promising. It should be mentioned that false positives appear if the attention points of negative reference point extend to the target areas. Although the performance has been improved with the current module, the generation of attention weights and sampling locations can be carefully designed to suppress the false positives for further improvements.
10

---

References
[1] Jiale Cao, Yanwei Pang, Jin Xie, Fahad Shahbaz Khan, and Ling Shao. From handcrafted to deep features for pedestrian detection: survey, IEEE transactions pattern analysis and machine intelligence, 2021.
[2] Wei Li, Xiatian Zhu, and Shaogang Gong. Harmonious attention network for person re- identification. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pages 2285-2294, 2018.
[3] Yulin Li, Jianfeng He, Tianzhu Zhang, Xiang Liu, Yongdong Zhang, and Feng Wu. Diverse part discovery: Occluded person re-identification with part-aware transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2898-2900 2021.
[4] Yichao Yan, Jinpeng Li, Jie Qin, Song Bai, Shengeai Liao, Liu, Fan Zhu, and Ling Shao. Anchor-free person search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7690-7699, 2021.
[5] Yanjie Li, Shoukui Zhang, Zhicheng Wang, Sen Yang, Wankou Yang, Shu-Tao Xia, and Erjin Zhou. Tokenpose: Learning keypoint tokens for human pose estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11313-11322, 2021.
[6] Lianghua Huang, Xin Zhao, and Kaiqi Huang. Bridging the gap between detection and tracking:
A unified approach. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3999-4009, 2019.
[7] Zhilu Chen and Xinming Huang. Pedestrian detection for autonomous vehicle using multi- spectral cameras. IEEE Transactions Intelligent Vehicles, 4(2):211-219; 2019.
[8] Antonin Vobecky, Michal Uricár, David Hurych, and Radoslav Skoviera. Advanced pedestrian dataset augmentation for autonomous driving. Proceedings the Conference Computer Vision Workshops, 2019.
the IEEE/CVF International
[9] Quanzheng Mou, Longsheng Wei, Conghao Wang. Dapeng Luo, Songze He, Jing Zhang, Huimin Xu, Chen Luo, and Changxin Gao. Unsupervised domain-adaptive scene-specific pedestrian detection for static video surveillance. Pattern Recognition, 118: 108038, 2021.
[10] Cheng Chi, Shifeng Zhang, Junliang Xing, Zhen Lei, Stan Z Li, and Xudong Zou. Pedhunter: Occlusion robust pedestrian detector in crowded scenes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10639-10646,, 2020.
[11] Chunze Lin, Jiwen Lu, and Jie Zhou. Multi-graines deep feature learning for robust pedestrian detection. IEEE Transactions on Circuits and Systems for Video Technology, 29(12):3608-36 3621, 2018.
[12] Liu, W. Luo, L. Ma, J. Huang, Stathaki, and Dai. Coupled network for robust pedestrian detection with gated multi-layer feature extraction and deformable occlusion handling. IEEE Trunsactions on Image Processing, 30:754-766, 2021.
[13] Yanwei Pang, Jin Xic, Muhammad Haris Khan, Rao Muhammad Anwer, Fahad Shahbaz Khan, and Ling Shao. Mask-guided attention network for occluded pedestrian detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4967-4975, 2019.
[14] Xinlong Wang, Tete Xiao, Yuning Jiang, Shuai Shao, Jian Sun, and Chunhua Shen. Repulsion loss: Detecting pedestrians in crowd. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7774-7783, 2018.
[15] Shifeng Zhang, Longyin Wen, Xiao Bian, Zhen Lei, and Stan Z Li. Occlusion-aware r-cmn: detecting pedestrians in crowd. In Proceedings of the European Conference on Computer Vision (ECCV), pages 637-653, 2018.
[16] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 1440-1448, 2015.

---

[17] Jin Xie, Yanwei Pang, Muhammad Haris Khan, Rao Muhammad Anwer, Fahad Shahbaz Khan, and Ling Shao. Mask-guided attention network and occlusion-sensee hard example mining for occluded pedestrian detection. IEEE Transactions on Image Processing, 50:3872-3884, 2020.
[18] Y. Zhang, Y. Bai, M. Ding, S. Xu, and B. Ghanem. Kgsnet: Key-point-guided super-resolution network for pedestrian detection in the wild. IEEE Transactions on Neural Networks and Learning Systems, 32(5):2251-2265, 2021.
[19] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint 2018.
[20] Chunze Lin, Jiwen Lu, Gang Wang, and Jie Zhou. Graininess-aware deep feature learning for pedestrian detection. In Proceedings of the European conference computer vision (ECCV), pages 732-747, 2018.
[21] Wei Liu, Shengeai Liao, Weidong Hu, Xuezhi Liang, and Xiao Chen. Learning efficient single- stage pedestrian detectors by asymptotic localization fitting. In Proceedings of the European Conference on Computer Vision (ECCV), pages 618-634, 2018
[22] Wei Liu, Shengcai Liao, Weiqiang Ren, Weidong Hu, and Yinan Yu. High-level semantic feature detection: A new perspective for pedestrian detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5187-5196, 2019.
[23] Tao Song, Leiyu Sun, Di Xie, Haiming Sun, and Shiliang Pu. Small-scale pedestrian detection hased on topological line localization and temporal feature aggregation. In Proceedings of the European Conference on Computer Vision (ECCV), pages 536-551, 2018.
2018.
[24] Xiaolin Song, Kaili Zhao, Wen-Sheng Chu, Honggang Zhang, and Jun Guo. Progressive refinement network for occluded pedestrian detection. In European Conference on Computer Vision, pages 32-48. 22.48. Springer, 2020.
[25] Hong-hui Xu, Xin-qing Wang, Dong Wang, Bao-guo Duan, and Ting Rui. Object detection in crowded scenes via joint prediction. Defence Technology, 2021.
[26] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Fu, and Alexander C Berg, Ssd: Single shot multibox detector. In European conference on computer vision (ECCV), pages 21-37. Springer, 2016.
theng-Yang Yang
[27] Jun Ma, Honglin Wan, Junxia Wang, Hao Xia, and Chengjie Bai. An improved one-stage pedestrian detection method based on multi-scale attention feature extraction. Journal of Real-Time Image Processing,
2021.
[28] Jun Ma, Honglin Wan, Junxia Wang, Hao Xia, and Chengjie Bai. An improved scheme of deep dilated feature extraction on pedestrian detection. Signal, Image and Video Processing, 15(2):231-25 239, 2021.
[29] Binjie Ruan and Chongyang Zhang. Occluded pedestrian detection combined with semantic features. IET Image Process, 15(10):2292-232 23000, 2021.
[30] Yuzhi Tan, Hongxun Yao, Haoran Li, Xiusheng Lu, and Haozhe Xie. Prf-ped: Multi-scale pedestrian detector with brsor-based receptive field. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 6059-6064 IEEE, 2021.
[31] Hao Xia, Honglin Wan, Jiayu Ou, Jun Ma, Xinyao Lv, and Chengjie Bai. Mafa-net: pedestrian detection network based on multi-scale attention feature aggregation. Applied Intelligence, 52(7):768--769, 2022.
[32] Yao Xu and Qin Yu. Adaptive weighted multi-level fusion of multi-scale features: A new approach to pedestrian detection. Future Internet, 13(2):38, 2021.
[33] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, and Sylvain Gelly. An image is worth 16x16 words: Transformers for image recognition at scale, arXiv preprint 20X0:2010.111299 2020.
12

---

[34] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[35] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pages 213 219-229. Springer, 2020.
[36] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
[37] S. Ren, K. He, R. Girshick, and Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 19(6):137-1149, 2017.
[38] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer. 2014.
[39] Shuai Shao, Zijian Zhao, Boxun Li, Tete Xiao, Gang Yu, Xiangyu Zhang, and Jian Sun. Crowdhuman: A benchmark for detecting human in a crowd. arXiv preprint 1xXiv:8805.0022: 2018.
[40] Matthicu Lin, Chuming Li, Xingyuan Bu, Ming Sun, Chen Lin, Junjie Yan, Wanli Ouyang, and Zhidong Deng. Detr for crowd pedestrian detection. arXiv preprint arXiv::07785, 2020.
[41] Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Pieter Abbeel, and Ashish Vaswani. Bottleneck transformers for visual recognition. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 16519-16529, 2021.
[42] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[43] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, and Michael Bernstein. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-255 2015.
[44] Piotr Dollar, Christian Wojek, Bernt Schiele, and Pietro Perona. Pedestrian detection: An evaluation the state of the art. IEEE transactions on pattern analysis and machine intelligence, 34(4):743-761, 2011.
[45] Shanshan Zhang, Rodrigo Benenson, and Bernt Schiele. Citypersons: A diverse dataset for pedestrian detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3221, 2017.
[46] Shanshan Zhang, Rodrigo Benenson, Mohamed Omran, Jan Hosang, and Bernt Schiele. How far are we from solving pedestrian detection? In Proceedings of the iEEE conference on computer vision and pattern recognition, pages 1269-1267. 2016.
[47] Irtiza Hasan, Shengcai Liao, Jinpeng Li, Saad Ullah Akram, and Ling Shao. Generalizable pedestrian detection: The elephant in the room. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11328-11332 2021.
[48] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged con- sistency targets improve cemi-superiieed deep learning results. Advances in neural information processing systems, 30, 2017.
[49] Liliang Zhang. Liang Lin, Xiaodan Liang, and Kaiming He. Is faster r-cnn doing well for pedestrian detection? In European conference on computer vision, pages 443-457. Springer, 2016.
[50] Cheng Chi, Shifeng Zhang, Junliang Xing, Zhen Lei, Stan Z Li, and Xudong Zou. Relational learning for joint head and human detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 10647-10654, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [No]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] The data are available online.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "N1IcKwIu12_", "reviewer": "Reviewer_w34n", "summary": "The paper presents a deformable vision transformer for pedestrian detection. The vision transformer introduces multi-scale deformable attention [33] to enhance features for pedestrian classification and localization. The experiments on Caltech and CityPersons demonstrate that the enhanced features by the proposed deformable vision transformer help improve the pedestrian detection performance. ", "questions": "The paper needs careful proofreading. It contains some grammatical errors. For example, Line 3 the latter categorys --> the latter categories or the latter category.\n  ", "limitations": "No. See weaknesses. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strenghts:\n1. The introduced multi-scale deformable attention enhances features for pedestrian detection in one-stage pdestrian detector. The results in Tables 5 and 6 show that it brings performance improvements on both Caltech and CityPersons datasets.\n\n2. Overall, the paper is easy to follow. \n\nWeaknesses:\n1. The novelty and contributions of the paper are limited. It simply adopts multi-scale deformable attention [33] to enhance features for pedestrian detection. \n2. The performance of the proposed method is not impressive enough. On CityPersons, its peformance is only on par with state-of-the-art.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Abrh9GWEoEV", "reviewer": "Reviewer_s2Bv", "summary": "The paper addresses the crucial problem in computer vision: pedestrian detection. The issue of faster inference (i..e real-time performance) with competitive accuracy is still a major problem as stated by the author(s). To solve this major issue/problem, the author(s) proposes a single-stage anchor-free pedestrian detector with the deformable vision transformer to balance the issue of faster inference and high accuracy. Together with strong experiments, solid baselines, and evaluation criteria(s), the author(s) achieves competitive results on well-known datasets of Caltech and Citypersons. ", "questions": "There is a major concern, I have on this manuscript's contribution that although the trade-off between higher accuracy and faster inference has been seriously taken by the author(s), there is little-to-less information has been provided on the occlusion part. Well, in general, if we narrow it down to common scenarios, we see an occluded person(s) in lot more cases and should be a concern to raise, whereas except for related work, I wasn't able to see major information on this issue. I appreciate the author's efforts in providing very clear information on their contribution, could I please ask the authors to also provide some insight on this case. ", "limitations": "There's a significant limitation in terms of providing insight on the occlusions/occluded person(s). However, I do not see any major limitations except the mentioned question, the proposed solution for a trade-off between high accuracy and faster inference using Vision Transformer is a pretty solid contribution, in my view to the computer vision community. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "++ Novelty\n\n- The task formulation is concise, convincing, solid, and novel. A seemingly reasonable approach has been conducted in this paper. Compared to the existing single/two-stage detectors, the proposed single-stage detector based on the feature extractor and CSP strategy achieves competitive results.\n- To the best of my knowledge, this simple strategy of incorporating the vision transformers on top of the backbone of pedestrian detection tasks is a solid contribution to the computer vision community. \n\n++ Evaluation\n\n- The experiments are strong, sufficient, and very well presented. The analysis presented here showcases the author's efforts on how the small things have been noted down while performing such extensive experiments with solid evaluation criteria(s).\n- The experimental evaluations demonstrate the effectiveness of the complete approach and showcase its practical value. The detailed analyses clarify the contribution of each component.\n\n++ Clarity\n\n - The proposed solution by providing a multi-scale deformable feature extractor based on a deformable vision transformer is theoretically sound and well-written clearly describes the improvements and adequately contextualizes the contributions.\n- The manuscript also provides a good description of related work and background, motivating the problem. Additionally, it provides a thorough description of the part of the pipeline it deals with.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "78rIPkzQbLA", "reviewer": "Reviewer_6hn5", "summary": "Due to the lack of spatially adaptive features, in this paper, the authors propose deformable vision transformer aiming to achieve the balance between speed and accuracy.", "questions": "1.\tIn page 4, line 153, what is the original work?\n2.\tIn fig2’s caption, why don’t we need the key tensor?\n3.\tCan you explain the difference to [1]?\n[A] Xia, Zhuofan, et al. \"Vision transformer with deformable attention.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.", "limitations": "They should address their limitation in conclusion or discussion section.\n", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1.\tR an HO are not famous terminologies, but they explain these terms in Section 4.1. It hurts the readability.\n2.\tThe parameters in loss function are not well-justified by experiments. The same issue is also for inference stage.\n3.\tLack of simulations results in popular detection benchmarks. \n4.\tWe already have deformable vision transformer in CVPR2022 [A].\n[A] Xia, Zhuofan, et al. \"Vision transformer with deformable attention.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n5. The performance is not very impressive.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "2NyGCdTbd6V", "reviewer": "Reviewer_pH8n", "summary": "Pedestrian detection is crucial in many applications, but the fluctuation of information from the images makes it difficult to do it accurately. CNN-based two-stage detectors are used for this task but are slow at inference and are difficult to train. Single-stage detectors solve these issues but have a higher miss rate, especially in the presence of occlusion. The authors suggest that these problems stem from information fed to the detection head: single-stage detectors get insufficient information compared to two-stage detectors. \n\nTo solve the above-mentioned problems, the authors proposed to use a deformable vision transformer. The transformer enhances multi-scale features extracted from a backbone, thereby providing better information to the detection head. They also proposed to use center-and-scale prediction (CSP). The proposed method uses multi-scale features extracted from a CNN. These features are passed through a detection neck. These features are then passed through a deformable transformer to enhance them. These enhanced features are then used to find three outputs that are relevant to CSP. Comprehensive experiments are performed to show the performance of the method on two pedestrian datasets. A set of ablation studies are also conducted to show the effect of different components of the method.", "questions": "## Questions / Weaknesses\n\n1. What is the main contribution of the paper. More specifically, what parts of the method are proposed. Or the main contribution is to show the effectiveness of vision transformers in overcoming the issues in single-stage detectors? Both contributions (proposing a new method or showing the effectiveness of existing ones) are equally important, but this should be highlighted clearly.\n\n2. If I understand it correctly, the main proposed components are the use of deformable transformers to enhance features and the use of CSP.\n\n3. If I'm not mistaken, similar architectures are also used in general object detection. It would be interesting to highlight the difference between general object detection and how the proposed method is different.\n\n4. In the relevant work, it is noted that DETRs don't perform better than two-stage-detectors. Is this because DETR produces a certain number of encoded outputs? It would be great if the authors can comment on it. ", "limitations": "The limitations of the work are not highlighted clearly.  It would be interesting to see some cases where the proposed method fails. ", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n1. The paper is mostly well written and easy to follow.\n\n2. The motivation, to bring the speed of inference in single-stage detectors by using features enhanced by transformers, is interesting. \n\n3. The well-motivated problem is then solved in a way that makes sense. \n\n4. Results (in Table 5, 6) show that enhanced feature maps do improve results, especially in the presence of occlusion.\n\n5. Comprehensive experiments are performed to show not only the SOTA results but also the effect of individual components. \n\nWeaknesses are pointed out in the next section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept