Title: Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanism
Venue: NeurIPS 2023
Accepted: yes
Keywords: YOLO

object detection

computer vision
Abstract: In the past years, YOLO-series models have emerged as the leading approaches in the area of real-time object detection. Many studies pushed up the baseline to a higher level by modifying the architecture, augmenting data and designing new losses. However, we find previous models still suffer from information fusion problem, although Feature Pyramid Network (FPN) and Path Aggregation Network (PANet) have alleviated this. Therefore, this study provides an advanced Gatherand-Distribute mechanism (GD) mechanism, which is realized with convolution and self-attention operations. This new designed model named as Gold-YOLO, which boosts the multi-scale feature fusion capabilities and achieves an ideal balance between latency and accuracy across all model scales. Additionally, we implement MAE-style pretraining in the YOLO-series for the first time, allowing YOLOseries models could be to benefit from unsupervised pretraining. Gold-YOLO-N attains an outstanding 39.9% AP on the COCO val2017 datasets and 1030 FPS on a T4 GPU, which outperforms the previous SOTA model YOLOv6-3.0-N with similar FPS by +2.4%. The PyTorch code is available at https://github.com/huawei-noah/Efficient-Computing/tree/master/Detection/Gold-YOLO, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/Gold_YOLO.

Gold-YOLO: Efficient Object Detector via Gather-and-Distribute Mechanism

Abstract

In the past years, YOLO-series models have emerged as the leading approaches in the area of real-time object detection. Many studies pushed up the baseline to a higher level by modifying the architecture, augmenting data and designing new losses. However, we find previous models still suffer from information fusion problem, although Feature Pyramid Network (FPN) and Path Aggregation Network (PANet) have alleviated this. Therefore, this study provides an advanced Gatherand-Distribute mechanism (GD) mechanism, which is realized with convolution and self-attention operations. This new designed model named as Gold-YOLO, which boosts the multi-scale feature fusion capabilities and achieves an ideal balance between latency and accuracy across all model scales. Additionally, we implement MAE-style pretraining in the YOLO-series for the first time, allowing YOLO-series models could be to benefit from unsupervised pretraining. Gold-YOLO-N attains an outstanding 39.9% AP on the COCO val2017 datasets and 1030 FPS on a T4 GPU, which outperforms the previous SOTA model YOLOv6-3.0-N with similar FPS by +2.4%. The PyTorch code is available at https://github.com/huawei-noah/EfficientComputing/tree/master/Detection/Gold-YOLO, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/Gold_YOLO.

1 Introduction

Object detection as a fundamental vision task that aims to recognize the categories and locate the positions of objects. It can be widely used in a wide range of applications, such as intelligent security, autonomous driving, robot navigation, and medical diagnosis. High-performance and low-latency object detector is receiving increasing attention for deployment on the edge devices.

Over the past few years, researchers have extensive research on CNN-based detection networks, gradually evolving the object detection framework from two-stage ( e.g. , Faster RCNN [43] and Mask RCNN [25]) to one-stage ( e.g. , YOLO [40]), and from anchor-based ( e.g. , YOLOv3 [42] and YOLOv4 [2]) to anchor-free ( e.g. , CenterNet [10], FCOS [47] and YOLOX [11]). [12, 7, 17] studied the optimal network structure through NAS for object detection task, and [16, 23, 19] explore another way to improve the performance of the model by distillation. Single-stage detection models, especially YOLO series models, have been widely welcomed in the industry due to their simple structure and balance between speed and accuracy.

Improvement of backbone is also an important research direction in the field of vision. As described in the survey [20], [26, 27, 60, 21, 62] has achieved a balance between precision and speed, while [9, 36, 22, 18] has shown strong performance in precision. These backbones have improved the performance of the original model in different visual tasks, ranging from high-level tasks like object

200

400

600

800

1000

1200

Throughput(FPS) on T4 (bs=32, TensorRT8)

30

35

40

45

50

COCO AP(%)

Gold-YOLO-L

Gold-YOLO-M

Gold-YOLO-S

Gold-YOLO-N

Gold-YOLO

YOLOv5

YOLOv6-3.0

YOLOv7

YOLOv8

PPYOLOE

YOLOX

200

400

600

800

1000

1200

1400

Throughput(FPS) on T4 (bs=32, TensorRT8)

30

35

40

45

50

COCO AP(%)

Gold-YOLO-L

Gold-YOLO-M

Gold-YOLO-S

Gold-YOLO-N

Gold-YOLO

YOLOv5

YOLOv6-3.0

YOLOv7

YOLOv8

PPYOLOE

YOLOX

(a) TensorRT 7, FP16 Throughput (FPS), BS=32

(b) TensorRT 8, FP16 Throughput (FPS), BS=32

detection to low-level tasks like image restoration. By using the encoder-decoder structure with the transformer, researchers have constructed a series of DETR-like object detection models, such as DETR [3] and DINO [57]. These models can capture long-range dependency between objects, enabling transformer-based detectors to achieve comparable or superior performance with most refined classical detectors. Despite the notable performance of transformer-based detectors,they fall short when compared to the speed of CNN-based models. Small-scale object detection models based on CNN still dominate the speed-accuracy trade-off, such as YOLOX [11] and YOLOv6-v3 [33, 49, 14].

We focus on the real-time object detection models, especially YOLO series for mobile deployment. Mainstream real-time object detectors consist of three parts: backbone, neck, and head. The backbone architecture has been widely investigated [42, 44, 9, 36] and the head architecture is typically straight forward, consisting of several convolutional or fully-connected layers. The necks in YOLO series usually use Feature Pyramid Network (FPN) and its variants to fuse multi-level features. These neck modules basically follow the architecture shown in Fig. 3. However, the current approach to information fusion has a notable flaw: when there is a need to integrate information across layers ( e.g. , level-1 and level-3 are fused), the conventional FPN-like structure fails to transmit information without loss, which hinders YOLOs from better information fusion.

Built upon the concept of global information fusion, TopFormer [59] has achieved remarkable results in semantic segmentation tasks. In this paper, we expanding on the foundation of TopFormer's theory, propose a novel Gather-and-Distribute mechanism (GD) for efficient information exchanging in YOLOs by globally fusing multi-level features and injecting the global information into higher levels. This significantly enhances the information fusion capability of the neck without significantly increasing the latency, improving the model's performance across varying object sizes. Specifically, GD mechanism comprises two branches: a shallow gather-and-distribute branch and a deep gatherand-distribute branch, which extract and fuse feature information via a convolution-based block and an attention-based block, respectively. To further facilitate information flow, we introduce a lightweight adjacent-layer fusion module which combines features from neighboring levels on a local scale. Our Gold-YOLO architectures surpasses the existing YOLO series, effectively demonstrating the effectiveness of our proposed approach.

To further improve the accuracy of the model, we also introduce a pre-training method, where we pre-train the backbone on ImageNet 1K using the MAE method, which significantly improves the convergence speed and accuracy of the model. For example, our Gold-YOLO-S with pre-training achieves 46.4% AP, which outperforms the previous SOTA YOLOv6-3.0-S with 45.0% AP at similar speed.

2 Related works

2.1 Real-time object detectors

After years of development, the YOLO-series model has become popular in the real-time object detection area. YOLOv1-v3 [40, 41, 42] constructs the initial YOLOs, identifies a single-stage

Backbone

Head

Neck

B2

B3

B4

B5

N3

N4

N5

Low-IFM

Low-FAM

Inject

Inject

Inject

High-IFM

High-FAM

Inject

P5

P4

P3

Low-GD

High-GD

detection structure consisting of three parts, backbone-neck-head, predicts objects of different sizes through multi-scale branches, become a representative single-stage object detection model. YOLOv4 [2] optimizes the previously used darknet backbone structure and propose a series of improvements, like the Mish activation function, PANet and data augmentation methods. YOLOv5 [13] inheriting the YOLOv4 [2] scheme with improved data augmentation strategy and a greater variety of model variants. YOLOX[11] incorporates Multi positives, Anchor-free, and Decoupled Head into the model structure, setting a new paradigm for YOLO-model design. YOLOv6 [33, 32] brings the reparameterization method to YOLO-series models for the first time, proposing EfficientRep Backbone and Rep-PAN Neck. YOLOv7 [49] focuses on analyzing the effect of gradient paths on the model performance and proposes the E-ELAN structure to enhance the model capability without destroying the original gradient paths. The YOLOv8 [14] takes the strengths of previous YOLO models and integrates them to achieve the SOTA of the current YOLO family.

2.2 Transformer-base object detection

Vision Transformer (ViT) emerged as a competitive alternative to convolutional neural networks (CNNs) that are widely used for different image recognition tasks. DETR [3] applies the transformer structure to the object detection task, reconstructing the detection pipeline and eliminating many handdesigned parts and NMS components to simplify the model design and overall process. Combining the sparse sampling capability of deformable convolution with the global relationship modeling capability of transformer, Deformable DETR [63] improve convergence speed while improve model speed and accuracy. DINO [57] first time introduced Contrastive denoising, Mix query selection and a look forward twice scheme. The recent RT-DETR [37] improved the encoder-decoder structure to solve the slow DETR-like model problem, outperforming YOLO-L/X in both accuracy and speed. However, the limitations of the DETR-like structure prevent it from showing sufficient dominance in the small model region, where YOLOs remain the SOTA of accuracy and velocity balance.

2.3 Multi-scale features for object detection

Traditionally, features at different levels carry positional information about objects of various sizes. Larger features encompass low-dimensional texture details and positions of smaller objects. In contrast, smaller features contain high-dimensional information and positions of larger objects. The original idea behind Feature Pyramid Networks (FPN) proposed by [35] is that these diverse pieces of information can enhance network performance through mutual assistance. FPN provides an efficient architectural design for fusing multi-scale features through cross-scale connections and information exchange, thereby boosting the detection accuracy of objects of varied sizes.

Based on FPN, the Path Aggregation Network (PANet) [50] incorporates a bottom-up path to make information fusion between different levels more adequate.Similarly, EfficientDet [45] presents a new repeatable module (BiFPN) to increase the efficiency of information fusion between different levels. M2Det [61] introduced an efficient MLFPN architecture with U-shape and Feature Fusion Modules. Ping-Yang Chen [5] improved interaction between deep and shallow layers using bidirectional fusion modules. Unlike these inter-layer works, [38] explored individual feature information using the Centralized Feature Pyramid (CFP) method. Additionally, [54] extended FPN with the Asymptotic Feature Pyramid Network (AFPN) to interact across non-adjacent layers. In response to FPN's limitations in detecting large objects, [31] proposed a refined FPN structure. YOLO-F [6] achieved

level-3

level-2

level-1

level-3

level-2

level-1

fuse

fuse

fuse

fuse

(a) traditional neck structure

(b) traditional neck

(c) our proposed neck

state-of-the-art performance with single-level features. SFNet [34] aligns different level features with semantic flow to improves FPN performance in model. SAFNet [30] introduced Adaptive Feature Fusion and Self-Enhanced Modules. [4] presented a parallel FPN structure for object detection with bi-directional fusion.However, due to the excessive number of paths and indirect interaction methods in the network, the previous FPN-based fusion structures still have drawbacks in low speed, cross-level information exchange and information loss.

However, due to the excessive number of paths and indirect interaction methods in the network, the previous FPN-based fusion structures still have drawbacks in low speed, cross-level information exchange and information loss.

3 Method

3.1 Preliminaries

The YOLO series neck structure, as depicted in Fig.3, employs a traditional FPN structure, which comprises multiple branches for multi-scale feature fusion. However, it only fully fuse features from neighboring levels, for other layers information it can only be obtained indirectly ' recursively '. In Fig.3, it shows the information fusion structure of the conventional FPN: where existing level-1, 2, and 3 are arranged from top to bottom. FPN is used for fusion between different levels. There are two distinct scenarios when level-1 get information from the other two levels:

1)

If level-1 seeks to utilize information from level-2, it can directly access and fuse this information.

If level-1 wants to use level-3 information, level-1 should recursively calling the information fusion module of the adjacent layer. Specifically, the level-2 and level-3 information must be fused first, then level-1 can indirectly obtain level-3 information by combining level-2 information.

This transfer mode can result in a significant loss of information during calculation. Information interactions between layers can only exchange information that is selected by intermediate layers, and not selected information is discarded during transmission. This leads to a situation where information at a certain level can only adequately assist neighboring layers and weaken the assistance provided to other global layers. As a result, the overall effectiveness of the information fusion may be limited.

To avoid information loss in the transmission process of traditional FPN structures, we abandon the original recursive approach and construct a novel gather-and-distribute mechanism (GD). By using a unified module to gather and fuse information from all levels and subsequently distribute it to different levels, we not only avoid the loss of information inherent in the traditional FPN structure but also enhance the neck's partial information fusion capabilities without significantly increasing latency. Our approach thus allows for more effective leveraging of the features extracted by the backbone, and can be easily integrated into any existing backbone-neck-head structure.

In our implementation, the process gather and distribute correspond to three modules: Feature Alignment Module (FAM), Information Fusion Module (IFM), and Information Injection Module (Inject).

The gather process involves two steps. Firstly, the FAM collects and aligns features from various levels. Secondly, IFM fuses the aligned features to generate global information.

Upon obtaining the fused global information from the gather process, the inject module distribute this information across each level and injects it using simple attention operations, subsequently enhancing the branch's detection capability.

To enhance the model's ability to detect objects of varying sizes, we developed two branches: lowstage gather-and-distribute branch (Low-GD) and high-stage gather-and-distribute branch (High-GD). These branches extract and fuse large and small size feature maps, respectively. Further details are provided in Sections 4.1 and 4.2. As shown in Fig. 2, the neck's input comprises the feature maps B 2 , B 3 , B 4 , B 5 extracted by the backbone, where B i ∈ R N × C Bi × R Bi . The batch size is denoted by N , the channels by C , and the dimensions by R = H × W . Moreover, the dimensions of R B 2 , R B 3 , R B 4 , and R B 5 are R, 1 2 R, 1 4 R , and 1 8 R , respectively.

3.2 Low-stage gather-and-distribute branch

In this branch, the output B 2 , B 3 , B 4 , B 5 features from the backbone are selected for fusion to obtain high resolution features that retain small target information. The structure show in Fig.4(a)

Low-stage feature alignment module. In low-stage feature alignment module (Low-FAM), we employ the average pooling (AvgPool) operation to down-sample input features and achieve a unified size. By resizing the features to the smallest feature size of the group ( R B 4 = 1 4 R ), we obtain F align . The Low-FAM technique ensures efficient aggregation of information while minimizing the computational complexity for subsequent processing through the transformer module.

The target alignment size is chosen based on two conflicting considerations: (1) To retain more low-level information, larger feature sizes are preferable; however, (2) as the feature size increases, the computational latency of subsequent blocks also increases. To control the latency in the neck part, it is necessary to maintain a smaller feature size.

Therefore, we choose the R B 4 as the target size of feature alignment to achieve a balance between speed and accuracy.

Low-stage information fusion module. The low-stage information fusion module (Low-IFM) design comprises multi-layer reparameterized convolutional blocks (RepBlock) and a split operation. Specifically, RepBlock takes F align (channel = sum ( C B 2 , C B 3 , C B 4 , C B 5 ) ) as input and produces F fuse (channel = C B 4 + C B 5 ). The middle channel is an adjustable value ( e.g. , 256) to accommodate varying model sizes. The features generated by the RepBlock are subsequently split in the channel dimension into F inj _ P 3 and F inj _ P 4 , which are then fused with the different level's feature.

The formula is as follows:

Information injection module. In order to inject global information more efficiently into the different levels, we draw inspiration from the segmentation experience [48] and employ attention operations to fuse the information, as illustrated in Fig. 5. Specifically, we input both local information (which refers to the feature of the current level) and global inject information (generated by IFM), denoted as F local and F inj , respectively. We use two different Convs with F inj for calculation, resulting in F global _ embed and F act . While F local _ embed is calculated with F local using Conv. The fused feature F out is then computed through attention. Due to the size differences between F local and F global , we employ average pooling or bilinear interpolation to scale F global _ embed and F act according to the size of F inj , ensuring proper alignment. At the end of each attention fusion, we add the RepBlock to further extract and fuse the information.

C

Low-

FAM

Low-IFM

Conv

RepConv-

blocks

Conv

Split

B5

B4

B3

B2

Low-GD

avgpool

avgpool

Bilinear

Inject_P3

Inject_P4

(a) low-stage gather-and-distribute branch

Multi-Head

Attention

Feed-

Forward

Network

Split

High-IFM

x L

C

High-

FAM

+

+

P5

P4

P3

High-GD

avgpool

avgpool

Inject_N5

Inject_N4

(b) high-stage gather-and-distribute branch

In low stage, F local is equal to Bi , so the formula is as follows:

3.3 High-stage gather-and-distribute branch

The High-GD fuses the features { P 3 , P 4 , P 5 } that are generated by the Low-GD, as shown in Fig.4(b)

High-stage feature alignment module. The high-stage feature alignment module (High-FAM) consists of avgpool , which is utilized to reduce the dimension of input features to a uniform size. Specifically, when the size of the input feature is { R P 3 , R P 4 , R P 5 } , avgpool reduces the feature size to the smallest size within the group of features ( R P 5 = 1 8 R ). Since the transformer module extracts high-level information, the pooling operation facilitates information aggregation while decreasing the computational requirements for the subsequent step in the Transformer module.

High-stage information fusion module. The high-stage information fusion module (High-IFM) comprises the transformer block (explained in greater detail below) and a splitting operation, which involves a three-step process: (1) the F align , derived from the High-FAM, are combined using the transformer block to obtain the F fuse . (2) The F fuse channel is reduced to sum ( C P 4 , C P 5 ) via a Conv 1 × 1 operation. (3) The F fuse is partitioned into F inj _ N 4 and F inj _ N 5 along the channel dimension through a splitting operation, which is subsequently employed for fusion with the current level feature.

The formula is as follows:

The transformer fusion module in Eq. 8 comprises several stacked transformers, with the number of transformer blocks denoted by L . Each transformer block includes a multi-head attention block, a Feed-Forward Network (FFN), and residual connections. To configure the multi-head attention block, we adopt the same settings as LeViT [15], assigning head dimensions of keys K and queries Q to D (e.g., 16) channels, and V = 2 D (e.g., 32) channels. In order to accelerate inference, we substitute the velocity-unfriendly operator, Layer Normalization, with Batch Normalization for each convolution, and replace all GELU activations with ReLU. This minimizes the impact of the transformer module on the model's speed. To establish our Feed-Forward Network, we follow the methodologies presented in [28, 56] for constructing the FFN block. To enhance the local connections of the transformer block, we introduce a depth-wise convolution layer between the two 1x1 convolution layers. We also set the expansion factor of the FFN to 2, aiming to balance speed and computational cost.

Conv 1x1

Conv 1x1

Conv 1x1

Sigmoid

×

x_local

x_global

+

avgpool /

bilinear

avgpool /

bilinear

Inject

RepConv-

blocks

Conv 1x1

Conv 1x1

Conv 1x1

Sigmoid

×

x_local

x_global

+

Inject-LAF

RepConv-

blocks

LAF

avgpool /

bilinear

avgpool /

bilinear

avgpool

Conv 1x1

bilinear

Conv 1x1

C

B_n

B_n-1

B_n+1

LAF

Low-stage

avgpool

Conv 1x1

C

P_n

P_n-1

LAF

High-stage

(a) Information injection module

(b) lightweight adjacent layer fusion module

Information injection module. The information injection module in High-GD is exactly the same as in Low-GD. In high stage, F local is equal to Pi , so the formula is as follows:

3.4 Enhanced cross-layer information flow

We have achieved better performance than existing methods using only a global information fusion structure. To further enhance the performance, we drew inspiration from the PAFPN module in YOLOv6 [32] and introduced an Inject-LAF module. This module is an enhancement of the injection module and includes a lightweight adjacent layer fusion (LAF) module that is added to the input position of the injection module.

To achieve a balance between speed and accuracy, we designed two LAF models: LAF low-level model and LAF high-level model, which are respectively used for low-level injection (merging features from adjacent two layers) and high-level injection (merging features from adjacent one layer). There structure is shown in Fig. 5 (b).

To ensure that feature maps from different levels are aligned with the target size, the two LAF models in our implementation utilize only three operators: bilinear interpolation to up-sample features that are too small, average pooling to down-sample features that are too large, and 1x1 convolution to adjust features that differ from the target channel.

The combination of the LAF module with the information injection module in our model effectively balances the between accuracy and speed. By using simplified operations, we are able to increase the number of information flow paths between different levels, resulting in improved performance without significantly increased the latency.

3.5 Masked image modeling pre-training

Recent methods such as BEiT [1], MAE [24] and SimMIM [52], have demonstrated the effectiveness of masked image modeling (MIM) for vision tasks. However, these methods are not specifically tailored for convolutional networks (convnets). SparK [46] and ConvNeXt-V2 [51] are pioneers in exploring the potential of masked image modeling for convnets.

In this study, we adopt MIM Pre-training following the SparK's [46] methodology, which successfully identifies and overcomes two key obstacles in extending the success of MAE-style pretraining to

convolutional networks (convnets). These challenges include the convolutional operations' inability to handle irregular and randomly masked input images, as well as the inconsistency between the single-scale nature of BERT pretraining and the hierarchical structure of convnets.

To address the first issue, unmasked pixels are treated as sparse voxels of 3D point clouds and employ sparse convolution for encoding. For the latter issue, a hierarchical decoder is developed to reconstruct images from multi-scale encoded features. The framework adopts a UNet-style architecture to decode multi-scale sparse feature maps, where all spatial positions are filled with embedded masks. We pretrain our model's backbone on ImageNet 1K for multiple Gold-YOLO models, and results in notable improvements

4 Experiment

4.1 Setups

Datasets. We perform extensive experiments on the Microsoft COCO datasets to validate the proposed detector. For the ablation study, we train on COCO train2017 and validate on COCO val2017 datasets. We use the standard COCO AP metric with a single scale image as input, and report the standard mean average precision (AP) result under different IoU thresholds and object scales.

Implementation details. We followed the setup of YOLOv6-3.0 [32] use the same structure (except for neck) and training configurations. The backbone of the network was implemented with

the EfficientRep Backbone, while the head utilized the Efficient Decoupled Head. The optimizer learning schedule and other setting also same as YOLOv6, i.e. stochastic gradient descent (SGD) with momentum and cosine decay on learning rate. Warm-up, grouped weight decay strategy and the exponential moving average (EMA) are utilized. Self-distillation and anchor-aided training (AAT) also be used in training. The strong data augmentations we adopt Mosaic [2, 13] and Mixup [58].

We conducted MIM unsupervised pretraining on the backbone using the 1.28 million ImageNet-1K datasets [8]. Following the experiment settings in Spark [46], we employed a LAMB optimizer [55] and cosine-annealing learning rate strategy, with a masking ratio of 60 % and a mask patch size of 32. For the Gold-YOLO-L models, we employed a batch size of 1024, while for the Gold-YOLO-M models, a batch size of 1152 was used. MIM pretraining was not employed for Gold-YOLO-N due to the limited capacity of its small backbone.

All our models are trained on 8 NVIDIA A100 GPUs, and the speed performance is measured on an NVIDIA Tesla T4 GPU with TensorRT.

4.2 Comparisons

Our focus is primarily on evaluating the speed performance of our models after deployment. Specifically, we measure throughput (frames per second at a batch size of 1 or 32) and GPU latency, rather than FLOPs or the number of parameters. To compare our Gold-YOLO with other state-of-the-art detectors in the YOLO series, such as YOLOv5 [13], YOLOX [11], PPYOLOE [53], YOLOv7 [49], YOLOv8 [14] and YOLOv6-3.0 [32], we test the speed performance of all official models with FP16-precision on the same Tesla T4 GPU with TensorRT.

Gold-YOLO-N demonstrates notable advancements, achieving an improvement of 2.6%/2.4%/6.6% compared to YOLOv8-N, YOLOv6-3.0-N, and YOLOv7-Tiny (input size=416), respectively, while offering comparable or superior performance in terms of throughput and latency. When compared to YOLOX-S and PPYOLOE-S, Gold-YOLO-S demonstrates a notable increase in AP by 5.9%/3.1%, while operating at a faster speed of 50/27 FPS (with a batch size of 32).

Gold-YOLO-M outperforms YOLOv6-3.0-M, YOLOX-M and PPYOLOE-M by achieving 1.1%, 4.2% and 2.1% higher AP with a comparable speed. Additionally, it achieves 5.7% and 0.9% higher AP than YOLOv5-M and YOLOv8-M, respectively, while achieving a higher speed. Gold-YOLO-M outperforms YOLOv7 with a significant improvement of 98FPS (batch size = 32), while maintaining the same AP. Gold-YOLO-L also achieves a higher accuracy compared to YOLOv8-L and YOLOv63.0-L, with a noticeable accuracy advantage of 0.4% and 0.5% respectively, while maintaining similar FPS at a batch size of 32.

4.3 Ablation study

4.3.1 Ablation study on GD structure

To verify the validity of our analysis concerning the FPN and to assess the efficacy of the proposed gather-and-distribute mechanism, we examined each module in GD independently, focusing on AP, number of parameters, and latency on the T4 GPU. The Low-GD predominantly targets small and medium-sized objects, whereas the High-GD primarily detect large-sized objects, and the LAF module bolsters both branches. The experimental results are displayed in Table 2 .

4.3.2 Ablation study on LAF

In this ablation study, we conducted experiments to compare the effects of different module designs within the LAF framework and evaluate the influence of varying model sizes on accuracy. The

results of our study provide evidence to support the assertion that the existing LAF structure is indeed optimal. The difference between model-1 and model-2 is whether LAF uses add or concat, and model-3 increase the model size basis of model-2. The model-4 is based on model-3 but discards LAF. The experimental results are displayed in Table 3 .

4.3.3 Ablation study on other model and task

The GD mechanism is a general concept and can be applied beyond YOLOs. We have extend GD mechanism to other models and obtain significant improvement.

On Instance Segmentation task, we replace different necks in Mask R-CNN and train/test on the COCO instance datasets. The result as shown in the Table 4.

On Semantic Segmentation task, we replace different necks in PointRend and train/test on the Cityscapes datasets. The result as shown in the Table 5.

On object detection task, we replace different necks in EfficientDet and train/test on the COCO datasets. The result as shown in the Table 6.

5 Conclusion

In this paper, we revisit the traditional Feature Pyramid Network (FPN) architecture and critically analyze its constraints in terms of information transmission. Following this, we subsequently developed the Gold-YOLO series models for object detection tasks, achieving state-of-the-art results. In Gold-YOLO we introduce an innovative gather-and-distribute mechanism, strategically designed to enhance the efficacy and efficiency of information fusion and transmission, avoid unnecessary losses, thereby significantly improving the model's detection capabilities. We truly hope that our work will prove valuable in addressing real-world problems and may also ignite fresh ideas for researchers in this field.

References

Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers, 2022.

Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.10934 , 2020.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16 , pages 213-229. Springer, 2020.

Ping-Yang Chen, Ming-Ching Chang, Jun-Wei Hsieh, and Yong-Sheng Chen. Parallel residual bi-fusion feature pyramid network for accurate single-shot object detection. IEEE Transactions on Image Processing , 30:9099-9111, 2021.

Ping-Yang Chen, Jun-Wei Hsieh, Chien-Yao Wang, and Hong-Yuan Mark Liao. Recursive hybrid fusion pyramid network for real-time small object detection on embedded devices. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops , pages 402-403, 2020.

Qiang Chen, Yingming Wang, Tong Yang, Xiangyu Zhang, Jian Cheng, and Jian Sun. You only look one-level feature. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 13039-13048, 2021.

Yukang Chen, Tong Yang, Xiangyu Zhang, Gaofeng Meng, Xinyu Xiao, and Jian Sun. Detnas: Backbone search for object detection. Advances in Neural Information Processing Systems , 32, 2019.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi, Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6569-6578, 2019.

Zheng Ge, Songtao Liu, Feng Wang, Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430 , 2021.

Golnaz Ghiasi, Tsung-Yi Lin, and Quoc V Le. Nas-fpn: Learning scalable feature pyramid architecture for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7036-7045, 2019.

Jocher Glenn. Yolov5 release v6.1. https://github.com/ultralytics/yolov5/ releases/tag/v6.1 , 2022.

Jocher Glenn. Ultralytics yolov8. https://github.com/ultralytics/ultralytics , 2023.

Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEE/CVF international conference on computer vision , pages 1225912269, 2021.

Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu. Distilling object detectors via decoupled features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2154-2164, 2021.

Jianyuan Guo, Kai Han, Yunhe Wang, Chao Zhang, Zhaohui Yang, Han Wu, Xinghao Chen, and Chang Xu. Hit-detector: Hierarchical trinity architecture search for object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11405-11414, 2020.

Jianyuan Guo, Kai Han, Han Wu, Yehui Tang, Xinghao Chen, Yunhe Wang, and Chang Xu. Cmt: Convolutional neural networks meet vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12175-12185, 2022.

Jianyuan Guo, Kai Han, Han Wu, Chao Zhang, Xinghao Chen, Chunjing Xu, Chang Xu, and Yunhe Wang. Positive-unlabeled data purification in the wild for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2653-2662, 2021.

Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. A survey on vision transformer. IEEE transactions on pattern analysis and machine intelligence , 45(1):87-110, 2022.

Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1580-1589, 2020.

Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Neural Information Processing Systems , 34:15908-15919, 2021.

Zhiwei Hao, Jianyuan Guo, Ding Jia, Kai Han, Yehui Tang, Chao Zhang, Han Hu, and Yunhe Wang. Learning efficient vision transformers via fine-grained manifold distillation. In Advances in Neural Information Processing Systems , 2022.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021.

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 2961-2969, 2017.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017.

Zilong Huang, Youcheng Ben, Guozhong Luo, Pei Cheng, Gang Yu, and Bin Fu. Shuffle transformer: Rethinking spatial shuffle for vision transformer. arXiv preprint arXiv:2106.03650 , 2021.

Huawei. Mindspore. https://www.mindspore.cn/ , 2020.

Zhenchao Jin, Bin Liu, Qi Chu, and Nenghai Yu. Safnet: A semi-anchor-free network with enhanced feature pyramid for object detection. IEEE Transactions on Image Processing , 29:9445-9457, 2020.

Zhenchao Jin, Dongdong Yu, Luchuan Song, Zehuan Yuan, and Lequan Yu. You should look at all objects. In European Conference on Computer Vision , pages 332-349. Springer, 2022.

Chuyi Li, Lulu Li, Yifei Geng, Hongliang Jiang, Meng Cheng, Bo Zhang, Zaidan Ke, Xiaoming Xu, and Xiangxiang Chu. Yolov6 v3. 0: A full-scale reloading. arXiv preprint arXiv:2301.05586 , 2023.

Chuyi Li, Lulu Li, Hongliang Jiang, Kaiheng Weng, Yifei Geng, Liang Li, Zaidan Ke, Qingyuan Li, Meng Cheng, Weiqiang Nie, et al. Yolov6: A single-stage object detection framework for industrial applications. arXiv preprint arXiv:2209.02976 , 2022.

Xiangtai Li, Ansheng You, Zhen Zhu, Houlong Zhao, Maoke Yang, Kuiyuan Yang, Shaohua Tan, and Yunhai Tong. Semantic flow for fast and accurate scene parsing. In Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16 , pages 775-793. Springer, 2020.

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2117-2125, 2017.

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision , pages 10012-10022, 2021.

Wenyu Lv, Shangliang Xu, Yian Zhao, Guanzhong Wang, Jinman Wei, Cheng Cui, Yuning Du, Qingqing Dang, and Yi Liu. Detrs beat yolos on real-time object detection. arXiv preprint arXiv:2304.08069 , 2023.

Y Quan, D Zhang, L Zhang, and J Tang. Centralized feature pyramid for object detection. arxiv 2022. arXiv preprint arXiv:2210.02093 , 41.

Harish Guruprasad Ramaswamy et al. Ablation-cam: Visual explanations for deep convolutional network via gradient-free localization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 983-991, 2020.

Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 779-788, 2016.

Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7263-7271, 2017.

Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 , 2018.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems , 28, 2015.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

Mingxing Tan, Ruoming Pang, and Quoc V Le. Efficientdet: Scalable and efficient object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10781-10790, 2020.

Keyu Tian, Yi Jiang, Qishuai Diao, Chen Lin, Liwei Wang, and Zehuan Yuan. Designing bert for convolutional networks: Sparse and hierarchical masked modeling. arXiv preprint arXiv:2301.03580 , 2023.

Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9627-9636, 2019.

Qiang Wan, Zilong Huang, Jiachen Lu, Gang Yu, and Li Zhang. Seaformer: Squeeze-enhanced axial transformer for mobile semantic segmentation. arXiv preprint arXiv:2301.13156 , 2023.

Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao. Yolov7: Trainable bag-offreebies sets new state-of-the-art for real-time object detectors. arXiv preprint arXiv:2207.02696 , 2022.

Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In proceedings of the IEEE/CVF international conference on computer vision , pages 9197-9206, 2019.

Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Convnext v2: Co-designing and scaling convnets with masked autoencoders, 2023.

Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling, 2022.

Shangliang Xu, Xinxin Wang, Wenyu Lv, Qinyao Chang, Cheng Cui, Kaipeng Deng, Guanzhong Wang, Qingqing Dang, Shengyu Wei, Yuning Du, et al. Pp-yoloe: An evolved version of yolo. arXiv preprint arXiv:2203.16250 , 2022.

Guoyu Yang, Jie Lei, Zhikuan Zhu, Siyu Cheng, Zunlei Feng, and Ronghua Liang. Afpn: Asymptotic feature pyramid network for object detection. arXiv preprint arXiv:2306.15988 , 2023.

Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes, 2020.

Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 579-588, 2021.

Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and HeungYeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv preprint arXiv:2203.03605 , 2022.

Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412 , 2017.

Wenqiang Zhang, Zilong Huang, Guozhong Luo, Tao Chen, Xinggang Wang, Wenyu Liu, Gang Yu, and Chunhua Shen. Topformer: Token pyramid transformer for mobile semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12083-12093, 2022.

Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 6848-6856, 2018.

Qijie Zhao, Tao Sheng, Yongtao Wang, Zhi Tang, Ying Chen, Ling Cai, and Haibin Ling. M2det: A single-shot object detector based on multi-level feature pyramid network. In Proceedings of the AAAI conference on artificial intelligence , volume 33, pages 9259-9266, 2019.

Dehua Zheng, Wenhui Dong, Hailin Hu, Xinghao Chen, and Yunhe Wang. Less is more: Focus attention for efficient detr. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6674-6683, 2023.

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 , 2020.

A Additional experiment

A.1 More detailed accuracy and speed data for Gold-YOLO

In this section, we report the test performace of our Gold-YOLO with or without LAF module and pre-training. FPS and latency are measured in FP16-precision on a Tesla T4 in the same environment with TensorRT 7. Both the accuracy and the speed performance of our models are evaluated with the input resolution of 640x640. The result shown in Table 7 .

A.2 MIMpre-training ablation experiment

Wealso compared the Gold-YOLO-S on COCO 2017 validation results for different MIM pre-training epochs without self-distillation. The result shown in Table 8 .

B Comprehensive Latency and Throughput Benchmark

B.1 Model Latency and Throughput on T4 GPU with TensorRT 8

Comparisons with other YOLO-series detectors on COCO 2017 val. FPS and latency are measured in FP16-precision on Tesla T4 in the same environment with TensorRT 8.2. The result shown in Table 9.

B.2 Model Latency and Throughput on V100 GPU with TensorRT 7

Comparisons with other YOLO-series detectors on COCO 2017 val. FPS and latency are measured in FP16-precision on Tesla V100 in the same environment with TensorRT 7.2. The result shown in Table 10 .

C Broader impacts and limitations

Broader impacts. The YOLO model can be widely applied in fields such as healthcare and intelligent transportation. In the healthcare domain, the YOLO series models can improve the early diagnosis rates of certain diseases and reduce the cost of initial diagnosis, thereby saving more lives. In the field of intelligent transportation, the YOLO model can assist in autonomous driving of vehicles, enhancing traffic safety and efficiency. However, there are also risks associated with the military application of the YOLO model, such as target recognition for drones and assisting military reconnaissance. We will make every effort to prevent the use of our model for military purposes.

Limitations. Generally, making finer adjustments on the structure will help further improve the model's performance, but this requires a significant amount of computational resources. Additionally, due to our algorithm's heavy usage of attention operations, it may not be as friendly to some earlier hardware support.

D CAMvisualization

Below are the CAM visualization results of the neck for YOLOv5, YOLOv6, YOLOv7, YOLOv8, and our Gold-YOLO, shown in Fig. 6. It can be observed that our model assigns higher weights to the detected regions of the targets.

And we compare the neck CAM visualization between Gold-YOLO and YOLOv6, shown in Fig. 7.

COCO

YOLOv5-N

YOLOv6-N

YOLOv7-T

YOLOv8-N

Gold-YOLO-N

P4

P3

N3

N4

GD-Neck

FPN-Neck

E Discussion

E.1 The differences between feature alignment module of Gold-YOLO and other similar works.

Both M2Det and RHF-Net have incorporated additional information fusion modules within their alignment modules. In M2Det, the SFAM module includes an SE block, while in RHF-Net, the Spatial Pyramid Pooling block is augmented with a Bottleneck layer. In contrast to M2Det and RHF-Net, Gold-YOLO leans towards functional separation among modules, segregating feature alignment and feature fusion into distinct modules. Specifically, the FAM module in GoldD-YOLO focuses solely on feature alignment. This ensures computational efficiency within the FAM module. And the LAF module efficiently merges features from various levels with minimal computational cost, leaving a greater portion of fusion and injection functionalities to other modules.

Based on the GD mechanism, achieving SOTA performance for the YOLO model can be accomplished using simple and easily accessible operators. This strongly demonstrates the effectiveness of the approach we propose. Additionally, during the network construction process, we intentionally opted for simple and thoroughly validated structures. This choice serves to prevent potential development and performance issues arising from certain operators being unsupported by deployment devices. As a result, it guarantees the usability and portability of the entire mechanism.

E.2 Simple calculation operation

In the process of network construction, we have drawn from and built upon the experiences of previous researchers. Rather than focusing solely on performance improvements achieved by enhancing specific operators or local structures, our emphasis lies on the conceptual shift brought about by the GD mechanism in comparison to the traditional FPN structure. Through the GD mechanism, achieving SOTA performance for the YOLO model is attainable using simple and easily applicable operators. This serves as strong evidence for the effectiveness of the proposed approach.

Additionally, during the network construction process, we intentionally opted for simple and thoroughly validated structures. This choice serves to prevent potential development and performance issues arising from certain operators being unsupported by deployment devices. As a result, it guarantees the usability and portability of the entire mechanism. Moreover, this decision also creates opportunities for future performance enhancements.

The GD mechanism is a general concept and can be applied beyond YOLOs. We have extend GD mechanism to other models and obtain significant improvement. The experiments demonstrate

that our proposed GD mechanism exhibits robust adaptability and generalization. This mechanism consistently brings about performance improvements across different tasks and models.

Peer reviews:
comments: SUMMARY:
In this paper, the authors proposed a Gather-and-Distribute mechanism (GD) for efficient information exchange in YOLOs by globally fusing multi-level features and injecting the global information into higher levels. The proposed GD-YOLO architectures show good results compared with the existing YOLO series. The authors also presented a pre-training method, where we pre-train the backbone on ImageNet 1K using the MAE method, which improves the convergence speed and accuracy of the model.

STRENGTHS:
1. In this paper, the authors proposed a gather-and-distribute mechanism to replace the traditional FPN structure. By using this unified module to gather and fuse information from all levels and subsequently distribute it to different levels, they can avoid the loss of information inherent in the traditional FPN structure and also enhance the neck’s partial information fusion capabilities without significantly increasing latency. 
2. The paper is clear and organized, and easy to follow.
3. Authors have provided a comprehensive comparison between the proposed model and YOLOs.

WEAKNESSES:
1. The motivation is a little bit unclear. The advantage of this GD mechanism compared with traditional FPN are not very clear.  
2. Based on the results, the tradeoff between accuracy and latency is pretty similar compared with baselines (YOLOv8).

QUESTIONS:
If the best trade-off between latency and accuracy is the main goal of this method, isn't fairer to compare with YOLOv8 instead of YOLOv6, since YOLOv8 has the closest latency with GDYolo?

LIMITATIONS:
1. The idea is pretty interesting but the advantage of this proposed method doesn't show a very convincing result.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this research, the authors propose a novel Gather-and-Distribute (GD) mechanism implemented through convolution and self-attention operations. This mechanism, incorporated into the GD-YOLO model, significantly enhances multi-scale feature fusion capabilities and achieves a remarkable balance between latency and accuracy across all model scales. Notably, the researchers introduce MAE-style pretraining to the YOLO-series models for the first time, enabling unsupervised pretraining benefits. The GD-YOLO-N variant achieves exceptional results, with a 39.9% Average Precision (AP) on the COCO val2017 dataset and a remarkable 1030 Frames Per Second (FPS) on a T4 GPU. These results surpass the previous state-of-the-art model, YOLOv6-3.0-N, with a similar FPS by 2.4%.\n\nSTRENGTHS:\nGD-YOLO is a quite impressive work, I think as it globalized the features from local and can retrieve features without tagging other layers feature with it.\n\nWEAKNESSES:\n1. While GD-YOLO-N cannot use the proposed backbone for limited capacity, what will be the approaches for mobile deployment as your main objective is YOLO for mobile deployment?\n2. GD-YOLO-N implementation is not quite understandable while it misses a few parts like MIM and seems pretty same as YOLOv6 with EWA and AAT as stated.\n3. Comparison with the Transformer based model could add a bit more value to this paper.\n4. Abbreviations should be added after first introduction and then using the abbreviation is the ideal practice. For instance, in line 46, it should have been “... use Feature Pyramid Network (FPN) and…” instead of just FPN.\n5. Line 45 to 54: This para basically represents the contribution part, which can be presented in bullet terms. \n6. Line 58: “...SOTA YOLOv6…” : As per my knowledge, at this point, YOLOv8 is the SOTA. I would request for a checkup and update of this from your end. \n7. Line 74-75: What strengths YOLOv8 takes from their predecessor, it is not mentioned. \n8. Line 94: “In this study, we will…..” - it is recommended to use present tense, instead of future tense. \n9. Table 1: Best values should be bolded. \n\nMinor:\n1. Line 32: “...detectors.Despite…, Line 138: “...alignment module(Low-FAM)...” - Spacing issue. \n2. Line 257: Double comma issue.\n\nQUESTIONS:\n1. Why need to use LAMB Optimizer while already using SGD and manual scheduled Learning rate?\n2. If you use EMA, then why again, LAMB is needed?\n\nLIMITATIONS:\nLimitations of GD-YOLO not added; however, I would suggest adding it.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

comments: SUMMARY:
This paper studies the problem of efficient object detector and proposes the Gather-and-Distribute mechanism (GD) mechanism to alleviate the information fusion problem. The experiments on the COCO dataset demonstrate the effectiveness of the proposed method.

STRENGTHS:
+ This paper studies an important topic, efficient object detection, and achieves a great balance between accuracy and speed in its results.
+ The ablation studies have been provided to verify the effectiveness of the proposed module.

WEAKNESSES:
- The structural design of this paper is quite confusing in some aspects, such as the choice of where to inject information. For example, in Low-GD, semantic information is only injected into P3 and P4, while one would expect that global semantic information could also benefit the P5 branch. Similarly, in High-GD, information is only injected into N4 and N5. The authors should provide the rationale and advantages of this design choice to address these doubts.

- In line #157, the authors mention they were inspired by [28]. However, the injection design is more closely related to Topformer, which bears greater relevance to the structure of the current paper. The authors should consider discussing and citing Topformer as another source of inspiration or relevant related work.

[a] TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation, CVPR 2022.

QUESTIONS:
Please refer to the Weaknesses.

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposed an efficient object detection network to make a new trade-off between efficiency and effectivity. In the framework, a lightweight adjacent-layer fusion module termed as gather-and-distribute (GD) mechanism is proposed to take place the conventional neck module in general detectors. Experimental results on COCO built-on the YOLO-series off-the-peg detectors demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n-The bilateral interaction of information in  multi-layer layers is interesting, and I like the work that simple changes can bring significant improvement for foundational tasks.\n\n-The experimental setup of ablation studies on COCO is helpful for follow-up work in the future.\n\nWEAKNESSES:\nThe main weakness of this paper is the limited technical novelty and the relatively inadequate experiments. First, although the authors try to explain that the proposed neck module of the GD mechanism can improve the detection accuracy of the detection task and benefit the computational efficiency, the novelty of the proposed module can be seen as a compromise similar to the stack of the existing technologies. Additionally, the experiment is carried out only on YOLO-series methods, and there is a lack of testing and verification on other datasets.\n\nThis paper needs further modification in terms of layout design, e.g., the fonts in the Fig.4 are generally small and unclear, which is very difficult for review.\n\nPage 3 Line 82-83, Improving ->can improve, introduce -> introduced.\n\nQUESTIONS:\nThe innovation of the proposed method needs to be improved comprehensively, and there is a lack of more substantial analysis and understanding in terms of innovation in technology. Then, it is necessary to verify the scalability and robustness of the proposed method in more experiments and even more tasks. In addition, the writing and layout concept of the article still needs to be improved.\n\nLIMITATIONS:\nApplicable", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a real-time object detection method for the YOLO series, introducing a 'Gather-and-Distribute' (GD) mechanism. Despite achieving good results on the COCO dataset, the paper lacks significant novelty and doesn't significantly advance multi-scale feature fusion or FPN-based methods. A deeper comparison with prior work could strengthen its scientific contribution.\n\nSTRENGTHS:\n1. The paper exhibits commendable clarity with well-written content and lucidly presented figures, making it easy to understand.\n\n2. Performance-wise, the proposed contribution demonstrates impressive results on the COCO dataset in terms of both accuracy and computational efficiency.\n\n3. The introduction of the 'Gather-and-Distribute' (GD) mechanism is particularly striking. This method enhances multi-scale feature fusion capabilities, striking an excellent balance between latency and accuracy across different model scales.\n\nWEAKNESSES:\nWeaknesses:\n\n1. Despite achieving strong results on the COCO dataset in terms of accuracy and efficiency, the paper lacks substantial scientific novelty. The method, while technically sound, doesn't significantly advance the field.\n\n2. The manuscript's focus is on real-time object detection and multi-scale features for the YOLO-Series. However, the related work section needs to delve more into multi-scale features.\n\n3. The concepts of Low/High-FAM and the lightweight adjacent layer fusion (LAF) module are not new in the field, having been discussed in M2Det [1] and [2] respectively.\n\nSuggestions:\n\n1. The authors need to restructure the related work section to better represent and compare with multi-scale features[1] or FPN-based methods [2-8].\n\n2. Adding more comparative analyses and surveys related to multi-scale features and FPN-based methods will establish their work as more than a minor modification of previous works.\n\n3. Despite YOLO-Series being known for speed and efficiency, GD-YOLO appears to be slower than the baseline (YOLOV6: v3). The authors should address this discrepancy.\n\n4. To reiterate, the work seems to be more application-focused with minimal contribution to the field. I recommend the authors address the points above to enhance their scientific contribution, which may change the review score. Otherwise, the work might not meet the standards of a top-tier conference.\n\n\n[1] Zhao, Q., Sheng, T., Wang, Y., Tang, Z., Chen, Y., Cai, L., & Ling, H. (2019, July). M2det: A single-shot object detector based on multi-level feature pyramid network. In Proceedings of the AAAI conference on artificial intelligence (Vol. 33, No. 01, pp. 9259-9266).\n\n[2] Chen, P. Y., Hsieh, J. W., Wang, C. Y., & Liao, H. Y. M. (2020). Recursive hybrid fusion pyramid network for real-time small object detection on embedded devices. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (pp. 402-403).\n\n[3] Quan, Y., Zhang, D., Zhang, L., & Tang, J. (2023). Centralized feature pyramid for object detection. IEEE Transactions on Image Processing.\n\n[4]Yang, G., Lei, J., Zhu, Z., Cheng, S., Feng, Z., & Liang, R. (2023). AFPN: Asymptotic Feature Pyramid Network for Object Detection. arXiv preprint arXiv:2306.15988.\n\n[5] Jin, Z., Yu, D., Song, L., Yuan, Z., & Yu, L. (2022, October). You should look at all objects. In European Conference on Computer Vision (pp. 332-349). Cham: Springer Nature Switzerland.\n\n[6] Chen, Q., Wang, Y., Yang, T., Zhang, X., Cheng, J., & Sun, J. (2021). You only look one-level feature. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 13039-13048).\n\n[7] Jin, Z., Liu, B., Chu, Q., & Yu, N. (2020). SAFNet: A semi-anchor-free network with enhanced feature pyramid for object detection. IEEE Transactions on Image Processing, 29, 9445-9457.\n\n[8] Chen, P. Y., Chang, M. C., Hsieh, J. W., & Chen, Y. S. (2021). Parallel residual bi-fusion feature pyramid network for accurate single-shot object detection. IEEE Transactions on Image Processing, 30, 9099-9111.\n\nQUESTIONS:\n1. Could you elaborate on why YOLOv7-E6E has been chosen as the \"L\" model for comparison, given that its size is 1280? It seems that YOLOv7-X would be a more suitable choice for a fair comparison.\n\n2. The relevance of the section \"Masked image modeling pre-training\" isn't clear, as it doesn't appear to directly relate to multi-scale features and FPN-based methods. Additionally, the results from this section don't seem to add significant value. Would it be more beneficial to include additional comparisons or ablation studies for multi-scale features in lieu of this section?\n\nLIMITATIONS:\nI want to highlight the author's commendable acknowledgment of the potential military applications of their model. They clearly state their commitment to prevent such uses. This level of social impact consideration is laudable and sets a positive precedent for responsible research conduct.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-08-02"}