SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Chengcheng Wang Wei He Ying Nie Jianyuan Guo Chuanjian Liu Kai Han* Yunhe Wang*
{wangchengcheng11,heweil42,ying.nie,jiangyuan.guo,liuchuanjian,
kai.han,yunhe.wang}@huawei.com
In the past years, YOLO-series models have emerged as the leading approaches in the area of real-time object detection. Many studies pushed up the baseline to a higher level by modifying the architecture, augmenting data and designing new losses. However, we find previous models still suffer from information fusion problem, although Feature Pyramid Network (FPN) and Path Aggregation Network (PANet) have alleviated this. Therefore, this study provides an advanced Gather-and-Distribute mechanism (GD) mechanism, which is realized with convolution and self-attention operations. This new designed model named as Gold-YOLO, which boosts the multi-scale feature fusion capabilities and achieves an ideal balance between latency and accuracy across all model scales. Additionally, we implement MAE-style pretraining in the YOLO-series for the first time, allowing YOLO-series models could be to benefit from unsupervised pretraining. Gold-YOLO-N attains an outstanding  $39.9\%$  AP on the COCO val2017 datasets and 1030 FPS on a T4 GPU, which outperforms the previous SOTA model YOLOv6-3.0-N with similar FPS by  $+2.4\%$ . The PyTorch code is available at https://github.com/huawei-noah/EfficientComputing/tree/master/Detection/Gold-YOLO, and the MindSpore code is available at https://gitee.com/mindspore/models/tree/master/research/cv/Gold_YOLO.
Object detection as a fundamental vision task that aims to recognize the categories and locate the positions of objects. It can be widely used in a wide range of applications, such as intelligent security, autonomous driving, robot navigation, and medical diagnosis. High-performance and low-latency object detector is receiving increasing attention for deployment on the edge devices.
Over the past few years, researchers have extensive research on CNN-based detection networks, gradually evolving the object detection framework from two-stage (e.g., Faster RCNN [43] and Mask RCNN [25]) to one-stage (e.g.,YOLO [40]), and from anchor-based (e.g.,YOLOv3 [42] and YOLOv4 [2]) to anchor-free (e.g., CenterNet [10], FCOS [47] and YOLOX [11]). [12, 7, 17] studied the optimal network structure through NAS for object detection task, and [16, 23, 19] explore another way to improve the performance of the model by distillation. Single-stage detection models, especiallyYOLO series models, have been widely welcomed in the industry due to their simple structure and balance between speed and accuracy.
Improvement of backbone is also an important research direction in the field of vision. As described in the survey [20], [26, 27, 60, 21, 62] has achieved a balance between precision and speed, while [9, 36, 22, 18] has shown strong performance in precision. These backbones have improved the performance of the original model in different visual tasks, ranging from high-level tasks like object
(a) TensorRT 7, FP16 Throughput (FPS), BS=32
(b) TensorRT 8, FP16 Throughput (FPS), BS=32
Figure 1: Comparison of state-of-the-art efficient object detectors in Tesla T4 GPU. Both latency and throughput (batch size of 32) are given for a handy reference. (a) and (b) test with TensorRT 7 and 8, respectively.
detection to low-level tasks like image restoration. By using the encoder-decoder structure with the transformer, researchers have constructed a series of DETR-like object detection models, such as DETR [3] and DINO [57]. These models can capture long-range dependency between objects, enabling transformer-based detectors to achieve comparable or superior performance with most refined classical detectors. Despite the notable performance of transformer-based detectors, they fall short when compared to the speed of CNN-based models. Small-scale object detection models based on CNN still dominate the speed-accuracy trade-off, such as YOLOX [11] and YOLOv6-v3 [33, 49, 14].
We focus on the real-time object detection models, especially YOLO series for mobile deployment. Mainstream real-time object detectors consist of three parts: backbone, neck, and head. The backbone architecture has been widely investigated [42, 44, 9, 36] and the head architecture is typically straight forward, consisting of several convolutional or fully-connected layers. The necks in YOLO series usually use Feature Pyramid Network (FPN) and its variants to fuse multi-level features. These neck modules basically follow the architecture shown in Fig. 3. However, the current approach to information fusion has a notable flaw: when there is a need to integrate information across layers (e.g., level-1 and level-3 are fused), the conventional FPN-like structure fails to transmit information without loss, which hinders YOLOs from better information fusion.
Built upon the concept of global information fusion, TopFormer [59] has achieved remarkable results in semantic segmentation tasks. In this paper, we expanding on the foundation of TopFormer's theory, propose a novel Gather-and-Distribute mechanism (GD) for efficient information exchanging in YOLOs by globally fusing multi-level features and injecting the global information into higher levels. This significantly enhances the information fusion capability of the neck without significantly increasing the latency, improving the model's performance across varying object sizes. Specifically, GD mechanism comprises two branches: a shallow gather-and-distribute branch and a deep gather-and-distribute branch, which extract and fuse feature information via a convolution-based block and an attention-based block, respectively. To further facilitate information flow, we introduce a lightweight adjacent-layer fusion module which combines features from neighboring levels on a local scale. Our Gold-YOLO architectures surpasses the existing YOLO series, effectively demonstrating the effectiveness of our proposed approach.
To further improve the accuracy of the model, we also introduce a pre-training method, where we pre-train the backbone on ImageNet 1K using the MAE method, which significantly improves the convergence speed and accuracy of the model. For example, our Gold-YOLO-S with pre-training achieves  $46.4\%$  AP, which outperforms the previous SOTA YOLOv6-3.0-S with  $45.0\%$  AP at similar speed.
2.1 Real-time object detectors
After years of development, the YOLO-series model has become popular in the real-time object detection area. YOLOv1-v3 [40, 41, 42] constructs the initial YOLOs, identifies a single-stage
Figure 2: The architecture of the proposed Gold-YOLO.
detection structure consisting of three parts, backbone-neck-head, predicts objects of different sizes through multi-scale branches, become a representative single-stage object detection model. YOLOv4 [2] optimizes the previously used darknet backbone structure and propose a series of improvements, like the Mish activation function, PANet and data augmentation methods. YOLOv5 [13] inheriting the YOLOv4 [2] scheme with improved data augmentation strategy and a greater variety of model variants. YOLOX [11] incorporates Multi positives, Anchor-free, and Decoupled Head into the model structure, setting a new paradigm for YOLO-model design. YOLOv6 [33, 32] brings the reparameterization method to YOLO-series models for the first time, proposing EfficientRep Backbone and Rep-PAN Neck. YOLOv7 [49] focuses on analyzing the effect of gradient paths on the model performance and proposes the E-ELAN structure to enhance the model capability without destroying the original gradient paths. The YOLOv8 [14] takes the strengths of previous YOLO models and integrates them to achieve the SOTA of the current YOLO family.
2.2 Transformer-base object detection
Vision Transformer (ViT) emerged as a competitive alternative to convolutional neural networks (CNNs) that are widely used for different image recognition tasks. DETR [3] applies the transformer structure to the object detection task, reconstructing the detection pipeline and eliminating many hand-designed parts and NMS components to simplify the model design and overall process. Combining the sparse sampling capability of deformable convolution with the global relationship modeling capability of transformer, Deformable DETR [63] improve convergence speed while improve model speed and accuracy. DINO [57] first time introduced Contrastive denoising, Mix query selection and a look forward twice scheme. The recent RT-DETR [37] improved the encoder-decoder structure to solve the slow DETR-like model problem, outperforming YOLO-L/X in both accuracy and speed. However, the limitations of the DETR-like structure prevent it from showing sufficient dominance in the small model region, where YOLOs remain the SOTA of accuracy and velocity balance.
2.3 Multi-scale features for object detection
Traditionally, features at different levels carry positional information about objects of various sizes. Larger features encompass low-dimensional texture details and positions of smaller objects. In contrast, smaller features contain high-dimensional information and positions of larger objects. The original idea behind Feature Pyramid Networks (FPN) proposed by [35] is that these diverse pieces of information can enhance network performance through mutual assistance. FPN provides an efficient architectural design for fusing multi-scale features through cross-scale connections and information exchange, thereby boosting the detection accuracy of objects of varied sizes.
Based on FPN, the Path Aggregation Network (PANet) [50] incorporates a bottom-up path to make information fusion between different levels more adequate. Similarly, EfficientDet [45] presents a new repeatable module (BiFPN) to increase the efficiency of information fusion between different levels. M2Det [61] introduced an efficient MLFPN architecture with U-shape and Feature Fusion Modules. Ping-Yang Chen [5] improved interaction between deep and shallow layers using bidirectional fusion modules. Unlike these inter-layer works, [38] explored individual feature information using the Centralized Feature Pyramid (CFP) method. Additionally, [54] extended FPN with the Asymptotic Feature Pyramid Network (AFPN) to interact across non-adjacent layers. In response to FPN's limitations in detecting large objects, [31] proposed a refined FPN structure. YOLO-F [6] achieved
(a) traditional neck structure
(b) traditional neck
Figure 3: (a) is example diagram of traditional neck information fusion structure. (b) and (c) is AblationCAM [39] visualization
(c) our proposed neck
state-of-the-art performance with single-level features. SFNet [34] aligns different level features with semantic flow to improves FPN performance in model. SAFNet [30] introduced Adaptive Feature Fusion and Self-Enhanced Modules. [4] presented a parallel FPN structure for object detection with bi-directional fusion. However, due to the excessive number of paths and indirect interaction methods in the network, the previous FPN-based fusion structures still have drawbacks in low speed, cross-level information exchange and information loss.
However, due to the excessive number of paths and indirect interaction methods in the network, the previous FPN-based fusion structures still have drawbacks in low speed, cross-level information exchange and information loss.
The YOLO series neck structure, as depicted in Fig.3, employs a traditional FPN structure, which comprises multiple branches for multi-scale feature fusion. However, it only fully fuse features from neighboring levels, for other layers information it can only be obtained indirectly 'recursively'. In Fig.3, it shows the information fusion structure of the conventional FPN: where existing level-1, 2, and 3 are arranged from top to bottom. FPN is used for fusion between different levels. There are two distinct scenarios when level-1 get information from the other two levels:
1) If level-1 seeks to utilize information from level-2, it can directly access and fuse this information.
2) If level-1 wants to use level-3 information, level-1 should recursively calling the information fusion module of the adjacent layer. Specifically, the level-2 and level-3 information must be fused first, then level-1 can indirectly obtain level-3 information by combining level-2 information.
This transfer mode can result in a significant loss of information during calculation. Information interactions between layers can only exchange information that is selected by intermediate layers, and not selected information is discarded during transmission. This leads to a situation where information at a certain level can only adequately assist neighboring layers and weaken the assistance provided to other global layers. As a result, the overall effectiveness of the information fusion may be limited.
To avoid information loss in the transmission process of traditional FPN structures, we abandon the original recursive approach and construct a novel gather-and-distribute mechanism (GD). By using a unified module to gather and fuse information from all levels and subsequently distribute it to different levels, we not only avoid the loss of information inherent in the traditional FPN structure but also enhance the neck's partial information fusion capabilities without significantly increasing latency. Our approach thus allows for more effective leveraging of the features extracted by the backbone, and can be easily integrated into any existing backbone-neck-head structure.
In our implementation, the process gather and distribute correspond to three modules: Feature Alignment Module (FAM), Information Fusion Module (IFM), and Information Injection Module (Inject).
- The gather process involves two steps. Firstly, the FAM collects and aligns features from various levels. Secondly, IFM fuses the aligned features to generate global information.
- Upon obtaining the fused global information from the gather process, the inject module distribute this information across each level and injects it using simple attention operations, subsequently enhancing the branch's detection capability.
To enhance the model's ability to detect objects of varying sizes, we developed two branches: low-stage gather-and-distribute branch (Low-GD) and high-stage gather-and-distribute branch (High-GD). These branches extract and fuse large and small size feature maps, respectively. Further details are provided in Sections 4.1 and 4.2. As shown in Fig. 2, the neck's input comprises the feature maps  $B2$ ,  $B3$ ,  $B4$ ,  $B5$  extracted by the backbone, where  $B_i \in \mathbb{R}^{N \times C_{Bi} \times R_{Bi}}$ . The batch size is denoted by  $N$ , the channels by  $C$ , and the dimensions by  $R = H \times W$ . Moreover, the dimensions of  $R_{B2}$ ,  $R_{B3}$ ,  $R_{B4}$ , and  $R_{B5}$  are  $R$ ,  $\frac{1}{2} R$ ,  $\frac{1}{4} R$ , and  $\frac{1}{8} R$ , respectively.
3.2 Low-stage gather-and-distribute branch
In this branch, the output  $B2, B3, B4, B5$  features from the backbone are selected for fusion to obtain high resolution features that retain small target information. The structure show in Fig.4(a)
Low-stage feature alignment module. In low-stage feature alignment module (Low-FAM), we employ the average pooling (AvgPool) operation to down-sample input features and achieve a unified size. By resizing the features to the smallest feature size of the group  $(R_{B4} = \frac{1}{4} R)$ , we obtain  $F_{\text{align}}$ . The Low-FAM technique ensures efficient aggregation of information while minimizing the computational complexity for subsequent processing through the transformer module.
The target alignment size is chosen based on two conflicting considerations: (1) To retain more low-level information, larger feature sizes are preferable; however, (2) as the feature size increases, the computational latency of subsequent blocks also increases. To control the latency in the neck part, it is necessary to maintain a smaller feature size.
Therefore, we choose the  $R_{B4}$  as the target size of feature alignment to achieve a balance between speed and accuracy.
Low-stage information fusion module. The low-stage information fusion module (Low-IFM) design comprises multi-layer reparameterized convolutional blocks (RepBlock) and a split operation. Specifically, RepBlock takes  $F_{\text{align}}$  (channel = sum( $C_{B2}, C_{B3}, C_{B4}, C_{B5}$ ) as input and produces  $F_{\text{fuse}}$  (channel =  $C_{B4} + C_{B5}$ ). The middle channel is an adjustable value (e.g., 256) to accommodate varying model sizes. The features generated by the RepBlock are subsequently split in the channel dimension into  $F_{\text{inj\_P3}}$  and  $F_{\text{inj\_P4}}$ , which are then fused with the different level's feature.
The formula is as follows:
$$
F _ {\text {a l i g n}} = \operatorname {L o w} _ {-} F A M ([ B 2, B 3, B 4, B 5 ]), \tag {1}
$$
$$
F _ {\text {f u s e}} = \operatorname {R e p B l o c k} \left(F _ {\text {a l i g n}}\right), \tag {2}
$$
$$
F _ {\text {i n j} \_ \mathrm {P} 3}, F _ {\text {i n j} \_ \mathrm {P} 4} = S p l i t \left(F _ {\text {f u s e}}\right). \tag {3}
$$
Information injection module. In order to inject global information more efficiently into the different levels, we draw inspiration from the segmentation experience [48] and employ attention operations to fuse the information, as illustrated in Fig. 5. Specifically, we input both local information (which refers to the feature of the current level) and global inject information (generated by IFM), denoted as  $F_{local}$  and  $F_{inj}$ , respectively. We use two different Convs with  $F_{inj}$  for calculation, resulting in  $F_{global\_embed}$  and  $F_{act}$ . While  $F_{local\_embed}$  is calculated with  $F_{local}$  using Conv. The fused feature  $F_{out}$  is then computed through attention. Due to the size differences between  $F_{local}$  and  $F_{global}$ , we employ average pooling or bilinear interpolation to scale  $F_{global\_embed}$  and  $F_{act}$  according to the size of  $F_{inj}$ , ensuring proper alignment. At the end of each attention fusion, we add the RepBlock to further extract and fuse the information.
(a) low-stage gather-and-distribute branch
(b) high-stage gather-and-distribute branch
Figure 4: Gather-and-Distribute structure. In (a), the Low-FAM and Low-IFM is low-stage feature alignment module and low-stage information fusion module in low-stage branch, respectively. In (b), the High-FAM and High-IFM is high-stage feature alignment module and high-stage information fusion module, respectively.
In low stage,  $F_{local}$  is equal to  $Bi$ , so the formula is as follows:
$$
F _ {\text {g l o b a l} \_ \text {a c t} \_ \text {P i}} = \operatorname {r e s i z e} \left(\operatorname {S i g m o i d} \left(\operatorname {C o n v} _ {\text {a c t}} \left(F _ {\text {i n j} \_ \text {P i}}\right)\right)\right), \tag {4}
$$
$$
F _ {\text {g l o b a l} \_ \text {e m b e d} \_ \text {P i}} = \operatorname {r e s i z e} \left(\operatorname {C o n v} _ {\text {g l o b a l} \_ \text {e m b e d} \_ \text {P i}} \left(F _ {\text {i n j} \_ \text {P i}}\right)\right), \tag {5}
$$
$$
F _ {\text {a t t - f u s e - P i}} = \operatorname {C o n v} _ {\text {l o c a l - e m b e d - P i}} (B i) * F _ {\text {i n g - a c t - P i}} + F _ {\text {g l o b a l - e m b e d - P i}}, \tag {6}
$$
$$
P i = \operatorname {R e p B l o c k} \left(F _ {\text {a t t - f u s e - P i}}\right). \tag {7}
$$
3.3 High-stage gather-and-distribute branch
The High-GD fuses the features  $\{P3, P4, P5\}$  that are generated by the Low-GD, as shown in Fig.4(b)
High-stage feature alignment module. The high-stage feature alignment module (High-FAM) consists of avgpool, which is utilized to reduce the dimension of input features to a uniform size. Specifically, when the size of the input feature is  $\{R_{P3}, R_{P4}, R_{P5}\}$ , avgpool reduces the feature size to the smallest size within the group of features  $(R_{P5} = \frac{1}{8} R)$ . Since the transformer module extracts high-level information, the pooling operation facilitates information aggregation while decreasing the computational requirements for the subsequent step in the Transformer module.
High-stage information fusion module. The high-stage information fusion module (High-IFM) comprises the transformer block (explained in greater detail below) and a splitting operation, which involves a three-step process: (1) the  $F_{\text{align}}$ , derived from the High-FAM, are combined using the transformer block to obtain the  $F_{\text{fuse}}$ . (2) The  $F_{\text{fuse}}$  channel is reduced to  $\text{sum}(C_{P4}, C_{P5})$  via a  $\text{Conv}1 \times 1$  operation. (3) The  $F_{\text{fuse}}$  is partitioned into  $F_{\text{inj\_N4}}$  and  $F_{\text{inj\_N5}}$  along the channel dimension through a splitting operation, which is subsequently employed for fusion with the current level feature.
$$
F _ {\text {a l i g n}} = \operatorname {H i g h} _ {-} F A M ([ P 3, P 4, P 5 ]), \tag {8}
$$
$$
F _ {\text {f u s e}} = \text {T r a n s f o r m e r} \left(F _ {\text {a l i g n}}\right), \tag {9}
$$
$$
F _ {\text {i n j} - \mathrm {N} 4}, F _ {\text {i n j} - \mathrm {N} 5} = S p l i t (C o n v 1 \times 1 \left(F _ {\text {f u s e}}\right)). \tag {10}
$$
The transformer fusion module in Eq. 8 comprises several stacked transformers, with the number of transformer blocks denoted by  $L$ . Each transformer block includes a multi-head attention block, a Feed-Forward Network (FFN), and residual connections. To configure the multi-head attention block, we adopt the same settings as LeViT [15], assigning head dimensions of keys  $K$  and queries  $Q$  to  $D$  (e.g., 16) channels, and  $V = 2D$  (e.g., 32) channels. In order to accelerate inference, we substitute the velocity-unfriendly operator, Layer Normalization, with Batch Normalization for each convolution, and replace all GELU activations with ReLU. This minimizes the impact of the transformer module on the model's speed. To establish our Feed-Forward Network, we follow the methodologies presented in [28, 56] for constructing the FFN block. To enhance the local connections of the transformer block, we introduce a depth-wise convolution layer between the two 1x1 convolution layers. We also set the expansion factor of the FFN to 2, aiming to balance speed and computational cost.
(a) Information injection module
(b) lightweight adjacent layer fusion module
Figure 5: The information injection module and lightweight adjacent layer fusion(LAF) module
Information injection module. The information injection module in High-GD is exactly the same as in Low-GD. In high stage,  $F_{local}$  is equal to  $Pi$ , so the formula is as follows:
$$
F _ {\text {g l o b a l} \_ \text {a c t} \_ \text {N i}} = \operatorname {r e s i z e} \left(\operatorname {S i g m o i d} \left(\operatorname {C o n v} _ {\text {a c t}} \left(F _ {\text {i n j} \_ \text {N i}}\right)\right)\right), \tag {11}
$$
$$
F _ {\text {g l o b a l} \_ \text {e m b e d} \_ \mathrm {N i}} = \operatorname {r e s i z e} \left(\operatorname {C o n v} _ {\text {g l o b a l} \_ \text {e m b e d} \_ \mathrm {N i}} \left(F _ {\text {i n j} \_ \mathrm {N i}}\right)\right), \tag {12}
$$
$$
F _ {\text {a t t - f u s e - N i}} = \operatorname {C o n v} _ {\text {l o c a l - e m b e d - N i}} (P i) * F _ {\text {i n g - a c t - N i}} + F _ {\text {g l o b a l - e m b e d - N i}}, \tag {13}
$$
$$
N i = \operatorname {R e p B l o c k} \left(F _ {\text {a t t - f u s e - N i}}\right). \tag {14}
$$
3.4 Enhanced cross-layer information flow
We have achieved better performance than existing methods using only a global information fusion structure. To further enhance the performance, we drew inspiration from the PAFPN module in YOLOv6 [32] and introduced an Inject-LAF module. This module is an enhancement of the injection module and includes a lightweight adjacent layer fusion (LAF) module that is added to the input position of the injection module.
To achieve a balance between speed and accuracy, we designed two LAF models: LAF low-level model and LAF high-level model, which are respectively used for low-level injection (merging features from adjacent two layers) and high-level injection (merging features from adjacent one layer). There structure is shown in Fig. 5 (b).