An efficient anchor‐free method for pig detection

IET Image ProcessingVolume 17, Issue 2 p. 613-626 ORIGINAL RESEARCHOpen Access An efficient anchor-free method for pig detection Morann Mattina, Morann Mattina ENIB, UMR CNRS 6285 LabSTICC, Brest, France COOPERL INNOVATION S.A.S, Lamballe, France Contribution: Conceptualization, Data curation, Formal analysis, ​Investigation, Methodology, Resources, Software, Validation, Visualization, Writing - original draft, Writing - review & editingSearch for more papers by this authorAbdesslam Benzinou, Corresponding Author Abdesslam Benzinou [email protected] orcid.org/0000-0003-1859-3409 ENIB, UMR CNRS 6285 LabSTICC, Brest, France Correspondence Abdesslam Benzinou, ENIB, UMR CNRS 6285 LabSTICC, 29238, Brest, France. Email: [email protected] Contribution: Conceptualization, Data curation, Formal analysis, Funding acquisition, ​Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing - original draft, Writing - review & editingSearch for more papers by this authorKamal Nasreddine, Kamal Nasreddine orcid.org/0000-0002-9834-3399 ENIB, UMR CNRS 6285 LabSTICC, Brest, France Contribution: Conceptualization, Data curation, Formal analysis, ​Investigation, Methodology, Resources, Software, Supervision, Validation, Visualization, Writing - original draft, Writing - review & editingSearch for more papers by this authorFrancis Richard, Francis Richard COOPERL INNOVATION S.A.S, Lamballe, France Contribution: Conceptualization, Data curation, Formal analysis, ​Investigation, Methodology, Resources, Software, Supervision, Validation, Visualization, Writing - original draft, Writing - review & editingSearch for more papers by this author Morann Mattina, Morann Mattina ENIB, UMR CNRS 6285 LabSTICC, Brest, France COOPERL INNOVATION S.A.S, Lamballe, France Contribution: Conceptualization, Data curation, Formal analysis, ​Investigation, Methodology, Resources, Software, Validation, Visualization, Writing - original draft, Writing - review & editingSearch for more papers by this authorAbdesslam Benzinou, Corresponding Author Abdesslam Benzinou [email protected] orcid.org/0000-0003-1859-3409 ENIB, UMR CNRS 6285 LabSTICC, Brest, France Correspondence Abdesslam Benzinou, ENIB, UMR CNRS 6285 LabSTICC, 29238, Brest, France. Email: [email protected] Contribution: Conceptualization, Data curation, Formal analysis, Funding acquisition, ​Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing - original draft, Writing - review & editingSearch for more papers by this authorKamal Nasreddine, Kamal Nasreddine orcid.org/0000-0002-9834-3399 ENIB, UMR CNRS 6285 LabSTICC, Brest, France Contribution: Conceptualization, Data curation, Formal analysis, ​Investigation, Methodology, Resources, Software, Supervision, Validation, Visualization, Writing - original draft, Writing - review & editingSearch for more papers by this authorFrancis Richard, Francis Richard COOPERL INNOVATION S.A.S, Lamballe, France Contribution: Conceptualization, Data curation, Formal analysis, ​Investigation, Methodology, Resources, Software, Supervision, Validation, Visualization, Writing - original draft, Writing - review & editingSearch for more papers by this author First published: 24 November 2022 https://doi.org/10.1049/ipr2.12659AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract Given the rapid growth of commercial pig farms, the need to automatically monitor pig behaviour becomes more important in order to assist farmers. Recent advances in convolutional neural networks may pave the way for new solutions. However, the primary task of individual pig detection under real-world conditions is still a challenging task. Previous studies used anchor-based frameworks that are unsuitable for such crowded scenarios with extreme overlapping. Furthermore, most applications focus on specific levels of brightness, farm facilities, or pig species without considering generalization. To tackle these problems, an anchor-free pig detection method based on pig centre localization is first proposed. Then, a novel negative training data augmentation technique is introduced using examples from outside the training distribution. Furthermore, using the test time augmentation technique is proposed to improve the model performance. Experiments are conducted on two online pig detection datasets; the network surpasses state-of-the-art results for both datasets. It is also found that the proposed method outperforms the latest anchor-free techniques commonly used in crowded scenarios. The method can detect pigs individually, even if their bounding boxes overlap strongly or occlude each other. Moreover, the real-time system achieves an improvement of 10% in F measure $F_{\text{measure}}$ when testing in unconstrained real-world conditions. 1 INTRODUCTION At present, the monitoring of pig behaviour and activity is mainly done by visual observation and analysis by farmers. The farmer observes the animals in each pen, one by one, several times a day, which can be a very time-consuming process. The need for automated monitoring thus increases as farms grow in size. An automated system would simplify farmers' work and enable them to switch to continuous monitoring of the animals, thus resulting in more accurate observations. This would improve animal welfare, reduce production costs, and bring low-cost and high-quality products to the market. Due to strong limitations such as noisy images and fluctuating brightness, computer vision is not widely used in commercial pig farming. In addition, the highly gregarious instinct of pigs means that the animals spend most of their time in groups, making it difficult to separate them individually. To create an automated animal monitoring system, pig detection in an image is the most important step. In fact, the detection performance of a model greatly influences the performance of subsequent computer vision tasks such as tracking individual animals and classifying behaviours (standing, lying, feeding etc.). To accomplish this task, traditional image processing techniques have been employed by several authors such as Zhu et al. [1]. Despite the good performance of their datasets, these detection algorithms have a poor generalization capability. The use of convolutional neural networks (CNNs) for computer vision tasks for pig monitoring has become increasingly common with recent advances in technology and deep learning [2-12]. While this technology has shown satisfactory results for the detection of other animals and humans [13], the various constraints mentioned above prevent the development of a robust framework for pig detection under real-world conditions. It is difficult for the model to adapt a generic feature to identify the animals in images, because they bend, get dirty, turn in all directions, and hide from each other. Moreover, the farms where the animals live are all different and do not necessarily have the same pig species. Thus, the model's performance varies greatly depending on how similar the training and test distributions are. In most cases, pigs are detected on an image using an anchor-based detector. In addition to the feature extractor, two sibling networks assign a probability score and regress the localization of the candidate regions. A rectangular sliding-window algorithm is used to generate regions of interest (RoIs). Handcrafted anchors require many configurations such as the number of boxes, shapes, and overlap thresholds between them as well as a ground truth to assign positive or negative labels for training purposes. In addition, the sliding-window system requires a duplicate removal method such as non-maximum suppression (NMS) [14, 15], which keeps only one bounding box per pig. Our real-world experiments reveal that the algorithms work well when the animals are isolated, but when they are grouped together, many nearby animals are eliminated. Due to extreme overcrowding in pig farms, it is difficult to predict whether two adjacent boxes contain the same pig or whether two pigs overlap. Furthermore, a region with an occluded pig might not contain sufficient features to be classified as positive. A few studies suggest a bottom-up approach to detect pigs in an image [6, 7, 12]. They detect the individual animal parts and use an association algorithm to group them together. However, the labelling process requires additional time, and each part of the animal must be clearly visible, which makes this method unsuitable for occluded pigs. Additionally, it introduces a part association algorithm that requires more calculations and hyperparameters. In uncontrolled conditions, they also suffer from a sharp drop in performance, as demonstrated by Posta et al. [6]. In summary, both anchor-based and anchor-free keypoint-based approaches have limited performances under such extremely overlapping and crowded scenarios. To tackle this problem, in this study we propose a pig detection method that does not require sliding-windows or a part association phase. Rather than classifying RoIs, we use a fully convolutional network (FCN) [16] to locate the central point of each animal and regress its width and height. With bounding box labelling, the animal's centre, width, and height are directly obtained to train the model. Furthermore, we demonstrate that by combining our approach with effective negative data augmentation during training and test time augmentation, we achieve a good generalization ability. The experiments are conducted on two publicly available pig detection datasets: Bergamini et al. [11] and Psota et al [6]. Thus, we compare our proposed approach to an anchor-based framework and an anchor-free keypoint-based framework. The main contributions of our work can therefore be summarized as follows: We propose a new detector known as anchor-free centre-based (AFCB) that is designed for the overlapping and crowded scenario of pig detection. We propose a combination of novel negative data augmentation during training and test time augmentation to improve the robustness and generalization ability of our AFCB framework. The proposed method was tested on two pig detection datasets and shown to be more effective than state-of-the-art detectors. The remainder of this paper is structured as follows. Section 3.2 presents an overview of relevant research. Section 3 details the detection approach proposed here. Then, Section 4 describes the different databases used and compares the results obtained. Finally, Section 5 concludes the paper. 2 RELATED WORK In this section, we describe state-of-the-art research in the field of pig detection and related work. An initial comparison of these methods is reported in Table 1. Most authors use anchor-based frameworks to detect pigs, while only a few use an anchor-free keypoint-based approach. In this section, we will discuss the drawbacks of both approaches and the lack of generalization ability that is inherent in most CNN models. TABLE 1. CNN architectures used in the literature for pig detection since 2019 Anchor-based Anchor-free Authors two stage one stage key point-based centre-based Cownton et al. [2] X Nasirahmadi et al. [3] X Zhang et al. [4] X Li et al. [5] X Psota et al. [6, 7] X Seo et al. [8] X Liu et al. [9] X Riekert et al. [10] X Bergamini et al. [11] X Wutke et al. [12] X Ours X Anchor-based approaches Nowadays, the CNN architectures that achieve the best performances in object detection use anchor-based frameworks. They are composed of two parts. A CNN generates a feature map from an input image. Then, using a rectangular sliding-window algorithm, features are extracted and provided to two sibling networks that assign a probability score and regress localization for each candidate region. Approaches based on anchors can be divided into two categories: two-stage and one-stage detectors. Authors using two-stage detectors to detect pigs employ a variant of the faster region-based convolutional neural network (faster-RCNN) [2, 3, 5, 10]. A region proposal phase is carried out using a region proposal network (RPN) followed by the classification and regression of the proposed regions. Single-stage detection models have also been used to detect pigs [4, 8, 9, 11]. This approach does not use region proposals and directly predicts the classification and regression from anchors. In general, two-stage detectors are slower, although they have better performance than single-stage detectors, even though the difference tends to diminish with some datasets [17]. As these architectures have proven successful in detecting other objects in the literature [13], they are also widely applied to the detection of pigs. Indeed, by using a wide variety of locations, sizes, and ratios of the anchors, the model greatly increases the likelihood of locating every object in the image. However, these approaches have several drawbacks. The computation time must be balanced with the number of anchors used. In addition, their ratios and shapes should be adapted for the detection scenario. This leads to many configurations. A large number of anchors are required for effective detection, which leads to a poor real-time performance. Without varied negative examples in training, the model produces false positives with a high probability under uncontrolled conditions. Moreover, the use of sliding windows implies the detection of an object multiple times in the same image. Thus, a duplicate removal method such as NMS is used to keep only one bounding box per object. Using the intersection over union (IoU) algorithm, two candidate regions are compared in terms of their overlapping rate, and the region with the smaller score is eliminated. Based on our real-world experiments, we found that the algorithms work effectively when the animals are isolated, and their bounding boxes do not overlap. By contrast, when they are grouped together during feeding or resting, many nearby animals are eliminated. Due to the extremely crowded conditions, it is difficult to predict whether two adjacent boxes contain the same pig or whether two pigs overlap. When using these anchor-based frameworks, it is impossible to find an optimal overlapping threshold that eliminates duplicate detections while keeping the animals close together. To address this problem, improvements were proposed to detect objects in crowded environments (e.g. pedestrian detection, medical image analysis). Bodla et al. [14] propose reducing the confidence score in relation to IoU rather than eliminating potential regions with high IoU. However, our experiments show that this technique does not improve the performance of our application. Rukhovich et al. [18] perform a detection in several iterations by feeding the model with information from the previous iterations. Chu et al. [19] make multiple detections from a single proposed region. Xie et al. [20] suggest replacing traditional rectangular anchors with oriented anchors. Although they all improve performances, these approaches increase the image processing time and require additional hyperparameters that complicate the model architecture. Anchor-free approaches Few authors have proposed a CNN architecture to detect pigs without the use of anchors. They use an FCN, typically used in segmentation tasks, to detect high-level semantic features such as the head or tail of the pig. Psota et al. [6] suggest individually detecting the parts of the animal and then grouping them together using an association algorithm. Wutke et al. [12] also detect the lines connecting the tail and shoulder, left shoulder and ear, and right shoulder and ear of each animal. However, the labelling process requires additional time, because each part (i.e. left ear, right ear, shoulder, and tail) must be manually located. In addition, animal parts must be clearly visible, thus making this method unsuitable for occluded pigs. Additionally, this approach introduces a part association phase that requires more computations and hyperparameters. Finally, in uncontrolled conditions, they also suffer from a heavy drop in performance, as demonstrated by ref. [6]. Our detector is also an anchor-free approach, but unlike the aforementioned authors, we locate the pig in an image using only its single centre pixel and then generate a bounding box that contains it. This eliminates the need for a part association algorithm and allows for the detection of an occluded pig, even if its head or tail are not visible. Furthermore, we can use the most common bounding box labelling format that is less time-consuming. Motivated by the promising results of ref. [21] regarding centre and scale precision (CSP) in pedestrian and face detection, we added two heads to our backbone that are respectively responsible for the centre point detection and location regression. However, in dense scenarios such as pig detection, Hasan et al. [22] demonstrate that two-stage anchor-based detectors outperform CSP. In their view, a probable explanation is the benefit provided to feature alignment by two-stage detectors (ROI-alignment or ROI pooling). In addition, we argue that the poorer performance is attributable to the truncated classifier used for feature extraction, which involves a down-sampled feature map. Thus, the output centre heatmap has a low resolution, resulting in its poor localization and inability to detect small objects. Instead, we propose using an autoencoder network such as backbone, which provides a feature map with the same resolution as the input image. Furthermore, unlike CSP, we do not require an additional offset prediction branch to adjust the centre location. Due to the wide variations in pig bounding box shapes, we cannot predict a scale as CSP does, but instead we must separately predict width and height. Several anchor-free methods have been proposed lastly to perform well in crowded environments. DETR [23] and deformable DETR [24] are two recent detectors that directly predict a set of box predictions by combining a CNN and a transformer-based backbone. Sparse R-CNN [25] uses a feature pyramid network (FPN) based on ResNet architecture to extract multi-scale feature maps. A dynamic head outputs a set of object proposals. YOLOX [26] is an anchor-free version of YOLO series [27]. These methods achieve good results on the CrowdHuman dataset [28], a benchmark for detecting humans in a crowd. Howewer, they have never been used for the challenging task of pig detection. Compared to CrowdHuman dataset, where each individual image has a different background, these methods could be not effective in pig environments due to the lack of variety in the images. Generalization ability In computer vision tasks, CNN models are known to have performance drops under uncontrolled conditions. As shown by ref. [6], this also occurs in pig detection. This is due to the difficulty of the model to identify generic features, because the appearance of the animals varies greatly when they bend, become dirty, turn in all directions, or hide from one another. As the pig environment is often poorly visible in the background, the model cannot learn a diverse set of negative examples during training. There is a severe imbalance between these “poor” background samples and “varied” pig samples. During training, artificial data augmentation is typically used to reduce performance degradation under uncontrolled conditions [29]. Geometric and photometric transformations are used to increase the number of training images and force the model to learn more generic features during training. Nevertheless, the artificial images created are too similar to the original distribution and are therefore insufficient to drastically reduce the prediction failures. In other studies, hard negative mining has been employed to improve the generalization of models [30]. A pretrained model is used to detect false positives and add them as hard negative samples to the training set; the model is then retrained. However, these methods require another diversified dataset. Shrivastava et al. [31] suggest performing hard negative mining during training by employing an initial prediction and then only using training examples that cause high loss. The samples still originate from the same training base, although they are not sufficient to guarantee the performance of the model under uncontrolled usage conditions. Similar to Wang et al. [32], other authors employ generative adversarial networks (GANs) to generate synthetic images to train the model. Unfortunately, training a GAN requires a large amount of data. Cai et al. [33] and Cheng et al. [34] increase the number of classifiers to reduce false positives. However, in addition to increasing model complexity and processing time, these classifiers are also trained with images from the same distribution, and thus they potentially suffer from the same defects. In summary, anchor-based frameworks are still widely used to detect pigs despite their lack of effectiveness due to their rectangular sliding windows algorithm. Nowadays in dense scenarios, such as detecting a human in a crowd, researchers tend to use anchor-free frameworks. In recent years, some authors have proposed a bottom-up approach to pig detection by detecting and grouping the key points such as the centre of the pig's head and tail. However, even with traditional data augmentation techniques the performances of these approaches are severely degraded when evaluation data have not the same distribution as the training data. This study proposes an approach that addresses both detection in crowded scenarios and performance degradation in uncontrolled environments. 3 PROPOSED METHOD In this section, we present our proposed approach for pig detection, which consists of a combination of an anchor-free centre-based detector, an efficient negative data augmentation during training and the test time augmentation technique. Anchor-free detection 3.1.1 Overall architecture Our FCN shown in Figure 1 consists of three components: a feature extractor and two heads that are respectively responsible for detecting a central point and regressing the shape (width and height) of each animal. The feature extractor is a symmetric autoencoder that takes an image of 512 × 512 pixels as the input and uses a combination of 24 convolutional layers, four max pooling layers, four upsampling layers, and four concatenation layers to produce 32 feature maps. Feature maps are produced with the same resolution as the input image. In the feature extractor, all convolutional layers are activated using the ReLU function. At each upsampling layer, a skip connection is used to copy the output feature map from the symmetric encoder layer. This residual connection minimizes the loss of information during the reconstruction of the output image. To reduce overfitting, we use a dropout rate of 0.2 after each concatenation layer. The feature map generated by the autoencoder is used by both prediction heads. To allow each head to have its own high-level features, we add a convolutional layer activated by the ReLU function between the feature map and the final prediction layers. Both the centre head and the shape head use the same output shape as the input image. The centre and shape heads have a sigmoid activation and a linear activation, respectively. For the height and width prediction, the shape head has two output channels. FIGURE 1Open in figure viewerPowerPoint Proposed fully convolutional network for pig detection. An autoencoder extracts features. Two prediction heads locate the centre of each animal and regress its width and height 3.1.2 Training and optimization Our framework predicts the location of the centre of the pigs by producing a centre heatmap C ̂ ∈ [ 0 , 1 ] $\hat{C}\in [0,1]$ with the same size as the input image I ∈ R 3 $I\in {R}^{3}$ . It also produces a shape heatmap S ̂ ∈ [ 0 , 1 ] 2 $\hat{S}\in [0,1]^{2}$ that has two channels with a width and height prediction for every pixel. The ground truth heatmaps C and S are obtained directly from the bounding box labelling. A pixel C x y = 1 ${C}_{xy}=1$ when it matches the centre of an animal in the image. The height and width of the animal are then associated with the S x y ${S}_{xy}$ pixel of the shape map. Values are normalized to the dimensions of the image. The other pixels are zeros. The classification loss is obtained by calculating the cross-entropy between the prediction C ̂ $\hat{C}$ and the ground truth C. To reduce the ambiguity of the pixels close to the centres, around each positive pixel we apply p x y $p_{xy}$ as a Gaussian mask computed as: C x y = exp − ( x − p x ) 2 2 σ w 2 + ( y − p y ) 2 2 σ h 2 $$\begin{equation} C_{xy}=\exp {\left({-\frac{(x-p_x)^2}{2\sigma ^2_w}}+\frac{(y-p_y)^2}{2\sigma ^2_h}\right)} \end{equation}$$ (1) The variances σ w $\sigma _w$ and σ h $\sigma _h$ of the Gaussian depend on the width and height of the animal. An example of centreness and shape target is available in Figure 2. Finally, to compensate for the large number of negative pixels compared to positives, we use the focal loss [17]: L c = − 1 N ∑ x y ( 1 − C ̂ ) α log ( C ̂ ) if C x y = 1 ( 1 − C ) β ( C ̂ ) α log ( 1 − C ̂ ) otherwise $$\begin{equation} L_{c} = \frac{-1}{N} \sum _{xy} {\begin{cases} (1-\hat{C})^{\alpha }\log (\hat{C}) &\quad \text{if } C_{xy}=1\\[9pt] (1-{C})^{\beta }(\hat{C})^{\alpha }\text{log}(1-\hat{C}) &\quad \text{otherwise} \end{cases}} \end{equation}$$ (2) FIGURE 2Open in figure viewerPowerPoint Based on bounding box annotations of an input image, targets for centreness and shape are generated. In the centreness target, the colour represents a value between 0 and 1. The redder the colour, the closer it is to 1. In the shape target, only the pixels containing the centre of an animal are not null. Heights (green) and widths (blue) are calculated from bounding box of each animal In the above equation, α and β are the focal loss parameters. We use α = 2 $\alpha =2$ and β = 4 $\beta =4$ . We normalize the loss by the number N of positive pixels in C. Regression loss is only calculated when a pixel contains a centre using the following equation: L r = − 1 N ∑ x y | S ̂ x y − S ̂ x y | if S ̂ x y = 1 0 , otherwise $$\begin{equation} L_{r} = \frac{-1}{N} \sum _{xy} {\begin{cases} |{\skew4\hat{S}}_{xy}-{\skew4\hat{S}}_{xy}| &\quad \text{if } {\skew4\hat{S}}_{xy} = 1\\[9pt] 0, &\quad \text{otherwise} \end{cases}} \end{equation}$$ (3)The total loss for training is defined as: L total = L c + λ L r $$\begin{equation} L_{\text{total}} = L_c + \lambda L_r \end{equation}$$ (4)For the experiments, λ is fixed at 0.04. 3.1.3 Inference process During the inference, we set to 0 all pixels that do not have the best score among the 20 connected neighbours in C ̂ $\hat{C}$ to prevent false positives and duplicate detections. A minimal confidence score was also set for the centre heatmap to retain the pixels. For each selected pixel, the corresponding height and width predictions are obtained from S ̂ $\hat{S}$ . Our model requires 10 ms for one forward pass. Post-processing takes a further 5 ms. Our proposed method can process 65 images per second, thus making it suitable for real-time object detection. Data augmentation 3.2.1 During training Data augmentation techniques are used to train our model. The input image is randomly flipped left or right, rotated from 0° to 360°, zoomed in or out, and shifted XY from the ranges [ 0 : W 2 ] $[0:\frac{W}{2}]$ and [ 0 : H 2 ] $[0:\frac{H}{2}]$ . According to the augmentation method applied to the input image, we modify the output targets. These techniques artificially increase the number of images seen during training. Thus, overtraining is delayed, while the generalization ability of the model is increased. Nevertheless, the created artificial images are too close to the original distribution, thus making them insufficient to drastically reduce the prediction failures. We use the ImageNet dataset [35] as a source for negatives in our training process. Some data augmentation functions may result in black pixels in the image background during training (such as zooming out, shifting, and rotating). Our negative data augmentation is performed by replacing each black pixel in the input image with a random image from the ImageNet database. A few examples produced by NTDA are shown in Figure 3. Thus, the training process provides a more varied background for the model. During training, the variety and randomness of the ImageNet images make it more difficult to identify features that distinguish between positive and negative labels. With this NTDA strategy, we were able to generate more robust features along with a better generalization ability. FIGURE 3Open in figure viewerPowerPoint NTDA (negative training data augmentation) produces out-of-distribution back