# FH-YOLOv4 with Constrained Aspect Ratio Loss for Video Face Detection and Public Safety

Authors: Yue Wang; Liang Hong; Dewen Gu; Pingping Fu
Journal: Discrete Dynamics in Nature and Society
 (2022)

---

## Abstract

Video face detection is a crucial first step in many facial recognition and face analysis systems. It should serve postprocessing steps as much as possible while satisfying high-accuracy real-time detection. In this paper, we first introduce the constrained aspect ratio loss (CARLoss) for better facial boxes regression and incorporate it into the modified FH-YOLOv4, then the IoU Tracker-based video face image deduplication algorithm is proposed on the detection level. Extensive experiments and comparative tests show the effectiveness of our method.


---

## Body

## 1. Introduction

Face detection aims at estimating face bounding boxes in a digital image without any scale and position priors, and its landing applications (long-distance automatic body temperature measuring device, digital camera auto-focus function, etc.) have affected all aspects of people’s daily life. Importantly, face detection is also a prerequisite for tasks such as facial identify recognition, facial attribute analysis, face editing, and face tracking, and its performance directly affects the effectiveness of these tasks [1, 2]. Therefore, whether it is to satisfy the user’s experience or serve the postprocessing steps, higher requirements are placed on the accuracy and real-time video face detection.Benefiting from the development of generic object detection methods, face detection has also made significant progress. The idea of early face detection methods was to first extract hand-crafted features from sliding windows on the image and then feed them into a classifier to detect possible face regions. One of the most iconic works is Histogram of Oriented Gradients (HOG) followed by SVM [3]. After all, the accuracy of such methods is limited. With the improvement of computing and storage capabilities, deep learning-based face detection methods have surpassed traditional methods in terms of speed, accuracy, and portability.Existing object detectors can be broadly divided into two categories: two-stage and one-stage methods. For two-stage detectors, R-CNN series [4–7] generate object proposals in the first stage for classification as well as bounding box refinement in the second stage. In particular, the Faster R-CNN [6] architecture uses a regional proposal network (RPN) rather than the selective search method to propose bounding boxes, making object detection much faster. Mask R-CNN [7] can generate high-quality segmentation masks for each instance while efficiently performing object detection. Unlike two-stage proposal-classification detectors, YOLO [8] (you look only once) is a one-step regression method proposed by Redmon et al., whose main contribution is real-time detection of full images and webcams. The YOLO pipeline first divides the input image into S×S non-overlapping grid cells, then each cell is responsible for detecting those objects whose center points fall within that cell. YOLO network runs at 45 frames per second with no batch processing on a Titan X GPU as compared to Faster R-CNN at 7 fps. However, the experiments showed that YOLO was not good at accurate localization. Soon, several follow-up works [9–14] adopt a series of design decisions from past works with novel concepts to improve YOLOs’ speed and precision. For instance, the mature detection framework YOLOv4 [14] uses the architecture of CSPDarkNet53 with an SPP layer as backbone, PANet as Neck and YOLO detection head. The detection of occluded targets, small targets, etc. has been significantly improved. Another classic one-stage detection framework, SSD [15], introduces a detection method based on pyramidal feature hierarchy to predict objects on feature maps of different receptive fields. Furthermore, CornerNet [16] directly detects an object bounding box as pairs of keypoints, i.e. top-left corner and the bottom-right corner, which triggers the emergence of series of anchor-free detection methods [17, 18].As well, there are also algorithms [19–24] that are specially tailored for face detection. MTCNN [19] cascades the three networks of P-Net, R-Net, and O-Net, which can simultaneously detect faces and five facial landmarks. Also, Cai at el. [20] still adopt a multitask cascaded CNN-based framework for simultaneous face detection, dense face alignment and fine head pose estimation. The lightweight anchor-free face detector CenterFace [21] can run in real time on a single CPU core and 200 fps using NVIDIA 2080TI for VGA-resolution images but produces more false positives. RetinaFace [22], a generalized face localization method, its architecture consists of three main parts: feature pyramid network, context module, and cascade regression. Also, it utilizes a multi-task learning strategy that combines extra-supervision and self-supervision to achieve stable face detection, accurate 2D face alignment, and robust 3D face reconstruction.In this paper, we focus on improving a state-of-the-art object detection method to make it suitable for the real-world video face detection task. More and more scenarios in today’s society involve video face detection, but there are still some inherent challenges in the development of this technology. On the one hand, not only high precision but also real-time speed is required in video processing. On the other hand, there is a lot of redundant information between adjacent frames in the video data. If all detected faces are output, there will be multiple face images of the same person, resulting in a lot of repetitive work in postprocessing stage. So face deduplication at the detection level is also necessary. In summary, our key contributions are(i)
Due to smaller variations in the facial boxes’ aspect ratio, CARLoss (constrained aspect ratio loss) is proposed as a new feedback mechanism.(ii)
Adding a prediction head on the original YOLOv4 for tiny faces, so the modified FH-YOLOv4 is more suitable for large-scale variations.(iii)
A video face image deduplication algorithm based on IoU Tracker is proposed to serve postprocessing tasks.The remainder of this paper is organized as follows: Section2 reviews several popular loss functions for bounding box regression. In Section 3, we propose CARLosses and four-head architecture FH-YOLOv4 for better face detection. In Section 4, we propose a video face image deduplication algorithm based on IoU Tracker. Extensive experiments are conducted in Section 5. Section 6 presents our conclusions.


## 2. Analysis of Traditional Bounding Box Regression Loss Functions

Generally, the loss function of the object detection task consists of two components, classification loss and bounding box regression loss. In this section, we focus on the evolution of bounding box regression loss, and analyze several of the most representative loss functions. There are many formats for the labeling of the bounding box, the common ones include Pascal VOC, COCO, and YOLO, which can be converted into each other. We follow the label format of the YOLO dataset and the box is parameterized by the coordinate of its center point, the width and the height. For convenience, we denote the region proposal and ground truth asB=x,y,w,h and Bgt=xgt,ygt,wgt,hgt, respectively.


### 2.1.ln-Norm Loss

Theln-norm loss functions are widely employed in bounding box regression. l1 loss function stands for least absolute deviations, and l2 loss function stands for least square errors. When the gradient descent algorithm is applied, the derivative function of l1 loss is piecewise constant, so l1 loss is insensitive to outliers but tends to fluctuate near the stable value in the later training period. l2 loss is continuously differentiable, but due to its amplification effect on outliers, it is easy to cause gradient explosion in the early training stage. l1-smooth loss [5] is exactly the integration of the advantages of l1 loss and l2 loss. However, the general representation of the location loss based on the ln-norm is as follows:(1)Lln=∑i∈x,y,w,hlnBigt−Bi,which ignores the correlation between the four parameters of the bounding box.


### 2.2. IoU and GIoU Loss

Intersection over union (IoU) ofBgt and B,(2)IoU=Bgt∩BBgt∪B.The evaluation criterion of positioning accuracy is used to de-redundant region proposals or determine positive and negative samples. In [25], for the first time, it was used as a measure of the distance between the candidate box and the ground truth to construct the loss function.(3)LIoU=−lnIoU,LIoU=1−IoU.IoU loss not only treats the bounding box as a unit, but the metric is also scale-invariant.If the anchor box and the target box have no overlapping area, IoU loss can neither reflect how far apart the two boxes are nor guide the movement of the anchor box. To address this issue, GIoU loss is proposed in [26],(4)GIoU=IoU−E∖Bgt∪BE,LGIoU=1−GIoU,where E is the smallest rectangular enclosing both Bgt and B. Empirically, this generalization tends to first increase the size of the proposal box to make it overlaps with the target box, and then degenerate into the IoU evaluation-feedback mechanism [27]. There are still problems such as slow convergence speed and inaccurate alignment. Similar to GIoU loss, DIoU loss ([28]) can still provide the direction of movement for the bounding box when it does not overlap with the target box.


### 2.3. Loss Functions That Consider Both IoU and Parametric Representation

As mentioned above, on the one hand, each box can be uniquely represented by a set of variables. For example, if the coordinates of two nonadjacent vertices are known, or the coordinates of the centroid and the width and height are given, then the rectangle can be positioned and drawn. On the other hand, IoU is an important indicator to judge the similarity of two boxes. Therefore, various positioning losses that take into account the overlap area of the two boxes and the normalized distance between the parameters are emerging one after another as illustrated in Figure1.Figure 1
Illustrations of several loss functions that take into account the overlap area of two boxes and the normalized distance between parameters. Green denotes the anchor box. Black denotes the target box. Gray denotes the smallest enclosing box covering the two boxes.In [27], the Complete-IoU (CIoU) loss is proposed,(5)LCIoU=1−IoU+b−bgt22c2+α⋅vwh,wgthgt,where b=x,y and bgt=xgt,ygt are the centroids of B and Bgt, respectively, c is the diagonal length of the smallest enclosing rectangle covering the two boxes, v=2arctanw/h−arctanwgt/hgt2/π2 measures the difference in the aspect ratio between the two boxes, and α=v/1−IoU+v is a control parameter. From formula (5), it is obvious that in the process of oriented boxes regression, the deep network first tries to pull the center point of the generating box towards the center point of the target box until the two boxes intersect, and then pay more attention to adjusting the aspect ratio later.Efficient-IoU (EIoU) loss [29], a revised version of CIoU loss, directly minimizes the gap between the width and height of the two boxes instead of the aspect ratio. Its definition is as follows:(6)LEIoU=1−IoU+b−bgt22c2+w−wgt2Cw2+h−hgt2Ch2,where Cw and Ch are the width and height of the smallest enclosing box covering the two boxes. In addition, Control Distance-IoU (CDIoU) loss [30] considers the regression of the four vertices of the box. Starting from the upper left point of the rectangle, denote the four vertices of B and Bgt clockwise as bi and bigti=1,2,3,4. The CDIoU loss is defined as follows:(7)LCDIoU=1−IoU−λ1−∑i∈1,2,3,4bi−bigt24c.Compared with the previous loss functions, they have greatly improved the convergence speed and detection accuracy. Since the physical description of the distance between boxes is diverse, there is still a lot of room for optimization.


## 2.1.ln-Norm Loss

Theln-norm loss functions are widely employed in bounding box regression. l1 loss function stands for least absolute deviations, and l2 loss function stands for least square errors. When the gradient descent algorithm is applied, the derivative function of l1 loss is piecewise constant, so l1 loss is insensitive to outliers but tends to fluctuate near the stable value in the later training period. l2 loss is continuously differentiable, but due to its amplification effect on outliers, it is easy to cause gradient explosion in the early training stage. l1-smooth loss [5] is exactly the integration of the advantages of l1 loss and l2 loss. However, the general representation of the location loss based on the ln-norm is as follows:(1)Lln=∑i∈x,y,w,hlnBigt−Bi,which ignores the correlation between the four parameters of the bounding box.


## 2.2. IoU and GIoU Loss

Intersection over union (IoU) ofBgt and B,(2)IoU=Bgt∩BBgt∪B.The evaluation criterion of positioning accuracy is used to de-redundant region proposals or determine positive and negative samples. In [25], for the first time, it was used as a measure of the distance between the candidate box and the ground truth to construct the loss function.(3)LIoU=−lnIoU,LIoU=1−IoU.IoU loss not only treats the bounding box as a unit, but the metric is also scale-invariant.If the anchor box and the target box have no overlapping area, IoU loss can neither reflect how far apart the two boxes are nor guide the movement of the anchor box. To address this issue, GIoU loss is proposed in [26],(4)GIoU=IoU−E∖Bgt∪BE,LGIoU=1−GIoU,where E is the smallest rectangular enclosing both Bgt and B. Empirically, this generalization tends to first increase the size of the proposal box to make it overlaps with the target box, and then degenerate into the IoU evaluation-feedback mechanism [27]. There are still problems such as slow convergence speed and inaccurate alignment. Similar to GIoU loss, DIoU loss ([28]) can still provide the direction of movement for the bounding box when it does not overlap with the target box.


## 2.3. Loss Functions That Consider Both IoU and Parametric Representation

As mentioned above, on the one hand, each box can be uniquely represented by a set of variables. For example, if the coordinates of two nonadjacent vertices are known, or the coordinates of the centroid and the width and height are given, then the rectangle can be positioned and drawn. On the other hand, IoU is an important indicator to judge the similarity of two boxes. Therefore, various positioning losses that take into account the overlap area of the two boxes and the normalized distance between the parameters are emerging one after another as illustrated in Figure1.Figure 1
Illustrations of several loss functions that take into account the overlap area of two boxes and the normalized distance between parameters. Green denotes the anchor box. Black denotes the target box. Gray denotes the smallest enclosing box covering the two boxes.In [27], the Complete-IoU (CIoU) loss is proposed,(5)LCIoU=1−IoU+b−bgt22c2+α⋅vwh,wgthgt,where b=x,y and bgt=xgt,ygt are the centroids of B and Bgt, respectively, c is the diagonal length of the smallest enclosing rectangle covering the two boxes, v=2arctanw/h−arctanwgt/hgt2/π2 measures the difference in the aspect ratio between the two boxes, and α=v/1−IoU+v is a control parameter. From formula (5), it is obvious that in the process of oriented boxes regression, the deep network first tries to pull the center point of the generating box towards the center point of the target box until the two boxes intersect, and then pay more attention to adjusting the aspect ratio later.Efficient-IoU (EIoU) loss [29], a revised version of CIoU loss, directly minimizes the gap between the width and height of the two boxes instead of the aspect ratio. Its definition is as follows:(6)LEIoU=1−IoU+b−bgt22c2+w−wgt2Cw2+h−hgt2Ch2,where Cw and Ch are the width and height of the smallest enclosing box covering the two boxes. In addition, Control Distance-IoU (CDIoU) loss [30] considers the regression of the four vertices of the box. Starting from the upper left point of the rectangle, denote the four vertices of B and Bgt clockwise as bi and bigti=1,2,3,4. The CDIoU loss is defined as follows:(7)LCDIoU=1−IoU−λ1−∑i∈1,2,3,4bi−bigt24c.Compared with the previous loss functions, they have greatly improved the convergence speed and detection accuracy. Since the physical description of the distance between boxes is diverse, there is still a lot of room for optimization.


## 3. FH-YOLOv4 with Constrained Aspect Ratio Loss for Better Face Detection

As a special case of object detection, face detection is also featured by the limited aspect ratio of the facial box (ranging from 1 : 1 to 1 : 1.5) and large-scale variation (occupying several pixels or even thousands of pixels) [22]. These properties open up opportunities for us to adjust the loss function and network structure of advanced general object detection methods to yield more promising and faster facial box inference.


### 3.1. Constrained Aspect Ratio Losses for Better Facial Box Regressing

#### 3.1.1. Limitation of CIoU Loss for Face Detection

Reviewing the definition of CIoU loss in formula (5), we might as well write its three key components as I1=1−IoU, I2=b−bgt22/c2, and I3=α⋅vw/h,wgt/hgt, where I1∈0,1, I2∈0,1. And then we analyze the range of I3. Typically, the width-to-height ratio (of the facial box Bgt) wgt/hgt∈2/3,1. In Figure 2, the graph of the function v is drawn in blue for the independent variable w/h∈0.08,4.5 when wgt/hgt is fixed at 0.6, 0.8, 1.0, and 1.2, respectively. In addition, since the coefficient function α is monotonically increasing with respect to IoU, a larger IoU (fixed IoU = 0.9) is selected here to explore the upper bound of I3, and the graph of α is plotted in green. So far, we can obtain the graph of I3=α⋅v (orange curve) and find that the range of I3 is about [0,0.1) in the process of facial box regression. Therefore, compared with I1 and I2, the contribution of I3 to CIoU loss is very small.Figure 2
Exploration of the range ofI3 in CIoU loss. The x-axis label is w/h. Set the facial box width-to-height ratio wgt/hgt to 0.6, 0.8, 1.0, 1.2 in sequence, and fix IoU = 0.9. Then, the graphs of functions v, α, and I3=α⋅v are shown.


#### 3.1.2. Constrained Aspect Ratio Loss

To enhance the contribution of the shape parameterw/h to the facial box regression loss, we revise the CIoU loss and propose a series of more efficient versions, i.e., constrained aspect ratio losses (CARLosses). For the sake of brevity, CARLosses can be unified as expression (10):(8)LRiB,Bgt=1−IoU+b−bgt22c2+Riwh,wgthgt.Here, in view of the constraint that the aspect ratio of the facial box is limited, several penalty termsRiw/h,wgt/hgti=1,2,3,4 are designed to improve the convergence speed and positioning accuracy.Specifically, we define the functionv˜=1−arctanw/h/arctanwgt/hgt2, and introduce the piecewise function f⋅ and the hyperbolic tangent function g⋅:(9)fx=0,ifx≤0x,if0<x<11,ifx≥1,gx=ex−e−xex+e−x.Then, the following loss functionsLRi are proposed.Case 1.
(10)LR1=1−IoU+b−bgt22c2+α1˜fv˜,where the trade-off parameterα1˜=fv˜/1−IoU+fv˜.Case 2.
(11)LR2=1−IoU+b−bgt22c2+α2˜gv˜,where the balance parameterα2˜=gv˜/1−IoU+gv˜.Case 3.
(12)LR3=1−IoU+b−bgt22c2+βfv˜,where the weighting coefficient β=eIoU−1.Case 4.
(13)LR4=1−IoU+b−bgt22c2+βgv˜,where the coefficient β is the same as the one in expression (12).
First, rough visualization of componentsRi in LRi shows that the contribution of the shape parameter w/h to facial box regression loss is indeed greatly increased, see Figure 3. Furthermore, the performance evaluation of the proposed CARLosses on face detection is presented in Section 5.Figure 3
Description of the range ofRii=1,2,3,4 in CARLosses. Set the facial box width-to-height ratio wgt/hgt to 0.8. The x-axis label is w/h.


### 3.2. Improved YOLOv4 by Adding a Prediction Head for Tiny Faces

#### 3.2.1. Four-Head Structure (FH-YOLOv4) for Large-Scale Variations

Since the facial boxes in the WIDER FACE dataset vary dramatically in scale (from a few pixels to tens of thousands of pixels), and almost half of them are small instances (occupying less than 200 pixels), we add on the original YOLOv4 framework a prediction head to facilitates correct detection of tiny faces. As shown in Figure4, the prediction head (the red branch) we add is generated from a low-level feature map with a small receptive field, which is more sensitive to tiny faces. Therefore, FH-YOLOv4 contains a total of 4 detection heads, which are used to detect tiny, small, medium and large faces, respectively.Figure 4
The architecture of the FH-YOLOv4. Compared with the original YOLOv4 structure, one more head (Red) is introduced for tiny face detection.


#### 3.2.2. Predefined Anchors

We still usek-means clustering to mine the facial bounding box priors. Cluster the width and height in the annotation information of the benchmark dataset WIDER FACE [31] for face detection. In the case where the size of the input image is 608×608, we employ 12-mean clustering to get predefined anchors’ size. The 12 clusters centroids are (2.36,3.96), (3.56,6.33), (4.75,8.75), (5.34,12.67), (6.53,10.68), (8.31,15.11), (10.69,20.77), (15.44,28.49), (22.56,40.95), (34.44,59.09), (59.38,97.92), and (139.53,208.18). Also, the prior settings in YOLOv4 are adjusted accordingly to make it more suitable for face detection tasks.


## 3.1. Constrained Aspect Ratio Losses for Better Facial Box Regressing

### 3.1.1. Limitation of CIoU Loss for Face Detection

Reviewing the definition of CIoU loss in formula (5), we might as well write its three key components as I1=1−IoU, I2=b−bgt22/c2, and I3=α⋅vw/h,wgt/hgt, where I1∈0,1, I2∈0,1. And then we analyze the range of I3. Typically, the width-to-height ratio (of the facial box Bgt) wgt/hgt∈2/3,1. In Figure 2, the graph of the function v is drawn in blue for the independent variable w/h∈0.08,4.5 when wgt/hgt is fixed at 0.6, 0.8, 1.0, and 1.2, respectively. In addition, since the coefficient function α is monotonically increasing with respect to IoU, a larger IoU (fixed IoU = 0.9) is selected here to explore the upper bound of I3, and the graph of α is plotted in green. So far, we can obtain the graph of I3=α⋅v (orange curve) and find that the range of I3 is about [0,0.1) in the process of facial box regression. Therefore, compared with I1 and I2, the contribution of I3 to CIoU loss is very small.Figure 2
Exploration of the range ofI3 in CIoU loss. The x-axis label is w/h. Set the facial box width-to-height ratio wgt/hgt to 0.6, 0.8, 1.0, 1.2 in sequence, and fix IoU = 0.9. Then, the graphs of functions v, α, and I3=α⋅v are shown.


### 3.1.2. Constrained Aspect Ratio Loss

To enhance the contribution of the shape parameterw/h to the facial box regression loss, we revise the CIoU loss and propose a series of more efficient versions, i.e., constrained aspect ratio losses (CARLosses). For the sake of brevity, CARLosses can be unified as expression (10):(8)LRiB,Bgt=1−IoU+b−bgt22c2+Riwh,wgthgt.Here, in view of the constraint that the aspect ratio of the facial box is limited, several penalty termsRiw/h,wgt/hgti=1,2,3,4 are designed to improve the convergence speed and positioning accuracy.Specifically, we define the functionv˜=1−arctanw/h/arctanwgt/hgt2, and introduce the piecewise function f⋅ and the hyperbolic tangent function g⋅:(9)fx=0,ifx≤0x,if0<x<11,ifx≥1,gx=ex−e−xex+e−x.Then, the following loss functionsLRi are proposed.Case 1.
(10)LR1=1−IoU+b−bgt22c2+α1˜fv˜,where the trade-off parameterα1˜=fv˜/1−IoU+fv˜.Case 2.
(11)LR2=1−IoU+b−bgt22c2+α2˜gv˜,where the balance parameterα2˜=gv˜/1−IoU+gv˜.Case 3.
(12)LR3=1−IoU+b−bgt22c2+βfv˜,where the weighting coefficient β=eIoU−1.Case 4.
(13)LR4=1−IoU+b−bgt22c2+βgv˜,where the coefficient β is the same as the one in expression (12).
First, rough visualization of componentsRi in LRi shows that the contribution of the shape parameter w/h to facial box regression loss is indeed greatly increased, see Figure 3. Furthermore, the performance evaluation of the proposed CARLosses on face detection is presented in Section 5.Figure 3
Description of the range ofRii=1,2,3,4 in CARLosses. Set the facial box width-to-height ratio wgt/hgt to 0.8. The x-axis label is w/h.


## 3.1.1. Limitation of CIoU Loss for Face Detection

Reviewing the definition of CIoU loss in formula (5), we might as well write its three key components as I1=1−IoU, I2=b−bgt22/c2, and I3=α⋅vw/h,wgt/hgt, where I1∈0,1, I2∈0,1. And then we analyze the range of I3. Typically, the width-to-height ratio (of the facial box Bgt) wgt/hgt∈2/3,1. In Figure 2, the graph of the function v is drawn in blue for the independent variable w/h∈0.08,4.5 when wgt/hgt is fixed at 0.6, 0.8, 1.0, and 1.2, respectively. In addition, since the coefficient function α is monotonically increasing with respect to IoU, a larger IoU (fixed IoU = 0.9) is selected here to explore the upper bound of I3, and the graph of α is plotted in green. So far, we can obtain the graph of I3=α⋅v (orange curve) and find that the range of I3 is about [0,0.1) in the process of facial box regression. Therefore, compared with I1 and I2, the contribution of I3 to CIoU loss is very small.Figure 2
Exploration of the range ofI3 in CIoU loss. The x-axis label is w/h. Set the facial box width-to-height ratio wgt/hgt to 0.6, 0.8, 1.0, 1.2 in sequence, and fix IoU = 0.9. Then, the graphs of functions v, α, and I3=α⋅v are shown.


## 3.1.2. Constrained Aspect Ratio Loss

To enhance the contribution of the shape parameterw/h to the facial box regression loss, we revise the CIoU loss and propose a series of more efficient versions, i.e., constrained aspect ratio losses (CARLosses). For the sake of brevity, CARLosses can be unified as expression (10):(8)LRiB,Bgt=1−IoU+b−bgt22c2+Riwh,wgthgt.Here, in view of the constraint that the aspect ratio of the facial box is limited, several penalty termsRiw/h,wgt/hgti=1,2,3,4 are designed to improve the convergence speed and positioning accuracy.Specifically, we define the functionv˜=1−arctanw/h/arctanwgt/hgt2, and introduce the piecewise function f⋅ and the hyperbolic tangent function g⋅:(9)fx=0,ifx≤0x,if0<x<11,ifx≥1,gx=ex−e−xex+e−x.Then, the following loss functionsLRi are proposed.Case 1.
(10)LR1=1−IoU+b−bgt22c2+α1˜fv˜,where the trade-off parameterα1˜=fv˜/1−IoU+fv˜.Case 2.
(11)LR2=1−IoU+b−bgt22c2+α2˜gv˜,where the balance parameterα2˜=gv˜/1−IoU+gv˜.Case 3.
(12)LR3=1−IoU+b−bgt22c2+βfv˜,where the weighting coefficient β=eIoU−1.Case 4.
(13)LR4=1−IoU+b−bgt22c2+βgv˜,where the coefficient β is the same as the one in expression (12).
First, rough visualization of componentsRi in LRi shows that the contribution of the shape parameter w/h to facial box regression loss is indeed greatly increased, see Figure 3. Furthermore, the performance evaluation of the proposed CARLosses on face detection is presented in Section 5.Figure 3
Description of the range ofRii=1,2,3,4 in CARLosses. Set the facial box width-to-height ratio wgt/hgt to 0.8. The x-axis label is w/h.


## 3.2. Improved YOLOv4 by Adding a Prediction Head for Tiny Faces

### 3.2.1. Four-Head Structure (FH-YOLOv4) for Large-Scale Variations

Since the facial boxes in the WIDER FACE dataset vary dramatically in scale (from a few pixels to tens of thousands of pixels), and almost half of them are small instances (occupying less than 200 pixels), we add on the original YOLOv4 framework a prediction head to facilitates correct detection of tiny faces. As shown in Figure4, the prediction head (the red branch) we add is generated from a low-level feature map with a small receptive field, which is more sensitive to tiny faces. Therefore, FH-YOLOv4 contains a total of 4 detection heads, which are used to detect tiny, small, medium and large faces, respectively.Figure 4
The architecture of the FH-YOLOv4. Compared with the original YOLOv4 structure, one more head (Red) is introduced for tiny face detection.


### 3.2.2. Predefined Anchors

We still usek-means clustering to mine the facial bounding box priors. Cluster the width and height in the annotation information of the benchmark dataset WIDER FACE [31] for face detection. In the case where the size of the input image is 608×608, we employ 12-mean clustering to get predefined anchors’ size. The 12 clusters centroids are (2.36,3.96), (3.56,6.33), (4.75,8.75), (5.34,12.67), (6.53,10.68), (8.31,15.11), (10.69,20.77), (15.44,28.49), (22.56,40.95), (34.44,59.09), (59.38,97.92), and (139.53,208.18). Also, the prior settings in YOLOv4 are adjusted accordingly to make it more suitable for face detection tasks.


## 3.2.1. Four-Head Structure (FH-YOLOv4) for Large-Scale Variations

Since the facial boxes in the WIDER FACE dataset vary dramatically in scale (from a few pixels to tens of thousands of pixels), and almost half of them are small instances (occupying less than 200 pixels), we add on the original YOLOv4 framework a prediction head to facilitates correct detection of tiny faces. As shown in Figure4, the prediction head (the red branch) we add is generated from a low-level feature map with a small receptive field, which is more sensitive to tiny faces. Therefore, FH-YOLOv4 contains a total of 4 detection heads, which are used to detect tiny, small, medium and large faces, respectively.Figure 4
The architecture of the FH-YOLOv4. Compared with the original YOLOv4 structure, one more head (Red) is introduced for tiny face detection.


## 3.2.2. Predefined Anchors

We still usek-means clustering to mine the facial bounding box priors. Cluster the width and height in the annotation information of the benchmark dataset WIDER FACE [31] for face detection. In the case where the size of the input image is 608×608, we employ 12-mean clustering to get predefined anchors’ size. The 12 clusters centroids are (2.36,3.96), (3.56,6.33), (4.75,8.75), (5.34,12.67), (6.53,10.68), (8.31,15.11), (10.69,20.77), (15.44,28.49), (22.56,40.95), (34.44,59.09), (59.38,97.92), and (139.53,208.18). Also, the prior settings in YOLOv4 are adjusted accordingly to make it more suitable for face detection tasks.


## 4. The Application of FH-YOLOv4 for Video Face Detection: IoU Tracker-Based Face Image Deduplication

Because the detected faces may be within the range of video surveillance for a long time, a large number of repeated detection faces are filled between consecutive frames. It is obviously unreasonable to do a follow-up1:N face recognition for all repeating faces. So, in this subsection, we study the face deduplication algorithm, which deduplicates the detected faces to reduce the number of recognitions of the same person.Inspired by the idea of IoU Tracker [32] in target tracking, we simplify and apply it to video face deduplication tasks. Different from multiobjective tracking [33], the deduplication algorithm proposed in this paper does not need to store a large amount of historical information, which reduces the storage cost.When the time interval between frames is short and the person moves slowly, analyzing the position of all facial boxes in the current frame and the previous frame will find that the facial boxes of the same person will partially overlap (please see Figure5). Thus, we can use the IoU as a measurement indicator for face deduplication. A detailed description of the IoU Tracker-based video face deduplication algorithm is shown in Algorithm 1, where V represents the test video, a continuous image sequence containing F frames in total. Period T and IoU threshold p need to be set empirically, flag is a counter, temp, and temp_loc are used to store faces and their corresponding locations, respectively. The detections at frame i are recorded in Di and Li, dij is the jth face at frame i, lij is the location of dij. IoUlij,temp_loc means calculating the IoU between lij and all the facial boxes in the temp_loc, and the element in the temp_loc that makes the max_IoU valid is assigned to max_loc.Figure 5
Illustration of the facial boxes in consecutive frames. Blue, Orange, and Green represent the facial boxes of different people, respectively.Algorithm 1:The IoU tracker-based video face deduplication algorithm.
Input:V=vii=1,2,…,F, T, pInitialize:flag=0, temp=∅, temp_loc=∅ForviinV:ifflag==T:Save the faces stored in thetemp;flag=0, temp=∅, temp_loc=∅;Di=∅, Li=∅;Perform face detection on imagevi,add the facesdijj∈ℕ to Di,add the corresponding locationlijj∈ℕ to Li;ifDi≠∅:iftemp==∅:Adddi,j to temp, add li,j to temp_loc;flag+=1, continue;forlij in Li:max_loc,max_IoU=maxIoUlij,temp_loc;ifmax_IoU<p:Adddi,j to temp, add li,j to temp_loc.else:Replacemax_loc with li,j;flag+=1;Output: The faces saved after deduplication operation.Because no visual information about the frames is used, the overall complexity of the method is very low. So, it can be thought of as a simple filtering process at the detection level.


## 5. Experiments

### 5.1. Effect of CARLosses

To evaluate the performance of the proposed CARLosses on face detection, we train the traditional YOLOv4 usingLCIoU (baseline) and proposed LRii=1,2,3,4 on the WIDER FACE training set. The input image size is set to be 608×608. Also, the AP at epochs 50th, 100th, 150th and maximum AP on the WIDER FACE validation set are recorded in Table 1. The results show that training YOLOv4 using our LR2 and LR4 can not only promote the fast convergence of the model but also improve its performance compared to CIoU loss.Table 1
Quantitative comparison of traditional YOLOv4 trained usingLCIoU (baseline) and proposed LRi.
LCIoULR1LR2LR3LR4Epoch 50th66.14%65.39%66.46%66.79%66.58%Epoch 100th72.91%72.07%72.95%73.30%73.63%Epoch 150th74.08%74.01%74.55%73.96%74.26%Max AP74.55%74.28%74.63%74.10%74.57%


### 5.2. Ablation Studies

Aims to verify the effectiveness of FH-YOLOv4 and CARLoss in face detection, some experiments are carried out, and the results are recorded in Table2.Table 2
The performance when incorporating the CIoU loss and CARLosses with FH-YOLOv4.
MethodsYOLOv4+LCIoUFH-YOLOv4+LCIoUFH-YOLOv4+LR1FH-YOLOv4+LR2FH-YOLOv4+LR3FPS+LR4AP74.55%78.89%78.91%78.38%79.06%79.20%First, cooperating the CIoU lossLCIoU with traditional YOLOv4 and FH-YOLOv4, respectively, one can find that the added prediction head for tiny faces brings an astonishing performance gain of 4.34%. Second, when FH-YOLOv4 is trained with our proposed CARLosses, LR1, LR3, and LR4 can all improve its performance compared to CIoU loss. It is worth noting that, the results in Tables 1 and 2 show consistent improvements in the performance of YOLOv4 and FH-YOLOv4 when they are trained using LR4. Thus, in the following text, we will abbreviate the proposed method FH-YOLOv4+ LR4 as FH-YOLOv4.


### 5.3. Video Face Detection Speed

We apply FH-YOLOv4 to perform face detection on 7 test videos. The basic information of the videos (number of frames, duration), the time spent in the detection process, and the total running time are shown in Table3. Among them, the total running time refers to the time consumption of all links including the reading of video frames, face detection, and result saving. Also, FPS refers to the number of video frames detected per second.Table 3
The performance of FH-YOLOv4 on test videos.
Test videosFramesVideo lengthDetection timeTotal run timeFPSvideo1.mp4778300:05:1800:05:5500:07:2721.9video2.mp4628300:04:1800:04:4000:05:5622.4video3.mp4778300:05:1800:06:2100:07:5520.4video4.mp4778300:05:1800:06:0600:07:3821.3video5.mp4778300:05:1800:05:5800:07:3321.7video6.mp4778300:05:1800:05:5700:07:3421.8video7.mp4778300:05:1800:03:5900:04:1932.6With the exception of video7.mp4, the FSP of other videos is around 21, which does not show a considerable speed advantage. On the one hand, the newly added prediction head for tiny faces in FH-YOLOv4 increases the number of network layers, which inevitably leads to an increase in the amount of computation. However, it is worth sacrificing a small computational cost in exchange for a big boost to the AP. On the other hand, the above experiment is a frame-by-frame detection of video. In practical application scenarios, the restricted random sampling (RRS) method [34] is usually used to randomly sample the video frames first, and then only the sampled video frames are processed. This not only improves the efficiency of the program but also allows more time for subsequent face recognition tasks.


### 5.4. Face Deduplication

In the following experiments, the FH-YOLOv4 and IoU Tracker-based deduplication algorithms are combined to detect and deduplicate the faces in the video clips. Under the setting of the IoU thresholdp=0.8, the statistical results of the number of faces before and after deduplication are shown in Table 4. It can be seen that the IoU Tracker-based deduplication algorithm can effectively remove repeated faces without increasing a lot of computing time, which helps to relieve the pressure of subsequent face recognition.Table 4
Effectiveness of the IoU tracker-based face deduplication algorithm.
Test videosDetected facesTime for detectionReserved facesDetection + deduplicationDeduplication ratio (%)video1.mp4266000:00:515900:00:5497.782video2.mp4427600:00:498600:00:5297.989video3.mp467200:00:504800:00:5292.857video4.mp4175000:00:516500:00:5396.286video5.mp4157300:00:515600:00:5496.440


## 5.1. Effect of CARLosses

To evaluate the performance of the proposed CARLosses on face detection, we train the traditional YOLOv4 usingLCIoU (baseline) and proposed LRii=1,2,3,4 on the WIDER FACE training set. The input image size is set to be 608×608. Also, the AP at epochs 50th, 100th, 150th and maximum AP on the WIDER FACE validation set are recorded in Table 1. The results show that training YOLOv4 using our LR2 and LR4 can not only promote the fast convergence of the model but also improve its performance compared to CIoU loss.Table 1
Quantitative comparison of traditional YOLOv4 trained usingLCIoU (baseline) and proposed LRi.
LCIoULR1LR2LR3LR4Epoch 50th66.14%65.39%66.46%66.79%66.58%Epoch 100th72.91%72.07%72.95%73.30%73.63%Epoch 150th74.08%74.01%74.55%73.96%74.26%Max AP74.55%74.28%74.63%74.10%74.57%


## 5.2. Ablation Studies

Aims to verify the effectiveness of FH-YOLOv4 and CARLoss in face detection, some experiments are carried out, and the results are recorded in Table2.Table 2
The performance when incorporating the CIoU loss and CARLosses with FH-YOLOv4.
MethodsYOLOv4+LCIoUFH-YOLOv4+LCIoUFH-YOLOv4+LR1FH-YOLOv4+LR2FH-YOLOv4+LR3FPS+LR4AP74.55%78.89%78.91%78.38%79.06%79.20%First, cooperating the CIoU lossLCIoU with traditional YOLOv4 and FH-YOLOv4, respectively, one can find that the added prediction head for tiny faces brings an astonishing performance gain of 4.34%. Second, when FH-YOLOv4 is trained with our proposed CARLosses, LR1, LR3, and LR4 can all improve its performance compared to CIoU loss. It is worth noting that, the results in Tables 1 and 2 show consistent improvements in the performance of YOLOv4 and FH-YOLOv4 when they are trained using LR4. Thus, in the following text, we will abbreviate the proposed method FH-YOLOv4+ LR4 as FH-YOLOv4.


## 5.3. Video Face Detection Speed

We apply FH-YOLOv4 to perform face detection on 7 test videos. The basic information of the videos (number of frames, duration), the time spent in the detection process, and the total running time are shown in Table3. Among them, the total running time refers to the time consumption of all links including the reading of video frames, face detection, and result saving. Also, FPS refers to the number of video frames detected per second.Table 3
The performance of FH-YOLOv4 on test videos.
Test videosFramesVideo lengthDetection timeTotal run timeFPSvideo1.mp4778300:05:1800:05:5500:07:2721.9video2.mp4628300:04:1800:04:4000:05:5622.4video3.mp4778300:05:1800:06:2100:07:5520.4video4.mp4778300:05:1800:06:0600:07:3821.3video5.mp4778300:05:1800:05:5800:07:3321.7video6.mp4778300:05:1800:05:5700:07:3421.8video7.mp4778300:05:1800:03:5900:04:1932.6With the exception of video7.mp4, the FSP of other videos is around 21, which does not show a considerable speed advantage. On the one hand, the newly added prediction head for tiny faces in FH-YOLOv4 increases the number of network layers, which inevitably leads to an increase in the amount of computation. However, it is worth sacrificing a small computational cost in exchange for a big boost to the AP. On the other hand, the above experiment is a frame-by-frame detection of video. In practical application scenarios, the restricted random sampling (RRS) method [34] is usually used to randomly sample the video frames first, and then only the sampled video frames are processed. This not only improves the efficiency of the program but also allows more time for subsequent face recognition tasks.


## 5.4. Face Deduplication

In the following experiments, the FH-YOLOv4 and IoU Tracker-based deduplication algorithms are combined to detect and deduplicate the faces in the video clips. Under the setting of the IoU thresholdp=0.8, the statistical results of the number of faces before and after deduplication are shown in Table 4. It can be seen that the IoU Tracker-based deduplication algorithm can effectively remove repeated faces without increasing a lot of computing time, which helps to relieve the pressure of subsequent face recognition.Table 4
Effectiveness of the IoU tracker-based face deduplication algorithm.
Test videosDetected facesTime for detectionReserved facesDetection + deduplicationDeduplication ratio (%)video1.mp4266000:00:515900:00:5497.782video2.mp4427600:00:498600:00:5297.989video3.mp467200:00:504800:00:5292.857video4.mp4175000:00:516500:00:5396.286video5.mp4157300:00:515600:00:5496.440


## 6. Conclusion

In this paper, we first propose CARLosses for better facial boxes regression according to the fact that the width-to-height ratio of faces is roughly ranging from 1 : 1 and 1 : 1.5. Second, by clustering the width and height in the annotation information of the benchmark dataset WIDER FACE, we add a prediction head on the original YOLOv4 for tiny faces and obtain a modified network structure FH-YOLOv4. The FH-YOLOv4 guided by CARLoss can achieve a significant improvement in AP compared to traditional YOLOv4. Third, we propose the IoU Tracker-based face image deduplication algorithm, and the deduplication rate is over 95 for all test videos. Experiments demonstrate that our method can achieve real-time speed and high accuracy, making it an ideal alternative for most face detection and recognition applications.



---

*Source: 8355174-2022-08-09.xml*