# Multiscale Deep Network with Centerness-Aware Loss for Salient Object Detection

Authors: Liangliang Duan
Journal: Advances in Multimedia
 (2022)

---

## Abstract

Deep encoder-decoder networks have been adopted for saliency detection and achieved state-of-the-art performance. However, most existing saliency models usually fail to detect very small salient objects. In this paper, we propose a multitask architecture, M2Net, and a novel centerness-aware loss for salient object detection. The proposed M2Net aims to solve saliency prediction and centerness prediction simultaneously. Specifically, the network architecture is composed of a bottom-up encoder module, top-down decoder module, and centerness prediction module. In addition, different from binary cross entropy, the proposed centerness-aware loss can guide the proposed M2Net to uniformly highlight the entire salient regions with well-defined object boundaries. Experimental results on five benchmark saliency datasets demonstrate that M2Net outperforms state-of-the-art methods on different evaluation metrics.


---

## Body

## 1. Introduction

Salient object detection (SOD) [1–3] aims to extract the most visually distinctive objects in an image or video. During the past decades, it has become a hotspot in the research field of computer vision. Saliency detection results often serve as the first step for a variety of downstream computer vision tasks, including object recognition [4], visual tracking [5], image retrieval [6], no-reference synthetic image quality assessment [7], robot navigation [8] image and video compression [9, 10], and object discovery [11–13].Earlier SOD methods mostly rely on hand-crafted features (e.g., color, brightness, and texture) to produce saliency maps. However, these low-level features can hardly capture high-level semantic information and are not robust enough to various complex scenarios.Recently, convolutional neural networks (CNNs), especially fully convolutional neural networks (FCNs) [14], have pushed salient object detection to achieve very promising results on many popular public benchmark datasets. Encoder-decoder framework [3, 15–19] is frequently used to extract and combine enriched feature blocks and therefore can generate more accurate saliency maps. More recently, many researchers further improved the saliency model by incorporating domain-specific information from other tasks such as contour/edge detection [18, 20, 21], image classification [22, 23], and noise pattern modeling [24].These U-shape models [3, 21] have greatly refreshed the leaderboards on all commonly used datasets. However, existing saliency methods still hold many problems that are not solved totally and are worthy of further research. First, due to the repeated subsampling, a single-scale convolutional kernel has difficulty in accurately segmenting size-varying salient object. Two state-of-the-art methods cannot uniformly highlight small foreground object with well-defined boundaries, as is shown in Figures 1(d) and 1(e). This motivates some efforts to characterize the multiscale information from a single layer. Second, most of the existing saliency methods [15, 25] use binary cross entropy (BCE) loss to train the saliency networks. But these models with BCE loss usually have low confidence in making a distinction between foreground and background, leading to blurred boundaries. The recent survey [26] indicates that the elaborate design of loss function can help to train more effective saliency detection models. Some training losses, such as PPA loss [19], Intersection over Union (IoU) loss [17, 27], and F-measure loss [28], were proposed for improving model performance. In consequence, it is essential to design a mechanism to extract multiscale information from each layer and develop a novel training loss.Figure 1
Sample results of our approach (M2Net) compared to GateNet and MINet. (a) and (b) show the input images with small foreground object and the ground truth (GT), respectively. (c), (d), and (e) are saliency maps of ours (M2Net), GateNet, and MINet.
(a)(b)(c)(d)(e)To address the above challenges, we proposed a novel multiscale and multitask network, named M2Net, which can generate high-quality saliency maps with clear boundaries (see Figure1(c)). Firstly, in the bottom-up encoder module, we use two branches to extract robust feature blocks. The backbone branch is based on a common pretrained image classification network, while the transformation branch is based on the sequence of three operations, including convolution, batch normalization, and ReLU. Secondly, in the decoder module, we develop two units, including multiscale feature extraction unit and cross-layer feature block fusion unit, to generate the saliency maps. Multiscale feature extraction unit can extract multiscale contextual features, while cross-layer feature block fusion unit can continually fuse adjacent level feature blocks. Thirdly, to take full advantage of ground truth, we design a centerness-aware loss, which considers the location of salient objects. This loss can guide the proposed network to generate high-quality saliency maps.We conduct experiments on five benchmark saliency datasets and demonstrate the better performance of the proposed M2Net. In summary, our contributions are as follows:(i)
We propose a multiscale and multitask deep framework with a centerness-aware loss for salient object detection. The M2Net consists of encode module, decoder module, and centerness prediction module.(ii)
We develop a centerness-aware loss, which can help to generate high-quality saliency maps, and it can push the proposed M2Net to uniformly highlight the entire salient regions with clear boundaries.(iii)
Extensive experiments on five public SOD datasets show that our model M2Net outperforms state-of-the-art saliency methods on different evaluation metrics. In particular, the proposed model (M2Net) can achieve the best performance under different challenging situations.


## 2. Related Work

### 2.1. Salient Object Detection

Early SOD methods [2, 29, 30] are mainly based on hand-crafted features and some intrinsic cues, such as center prior, color contrast, and background prior. Recently, convolutional neural networks (CNNs) have been used to extract multilevel features from input images. CNNs-based methods treat patches/superpixels [31–33] and generic object proposals [34–37] as image processing units, and an MLP-classifier is used to train the network. Wang et al. [35] trained two different CNN models to generate a saliency map. DNN-L and DNN-G are used to extract local and global features, respectively. Particularly, fully convolutional networks (FCN) show their advantage and refresh the state-of-the-art records in saliency prediction task. The encoder-decoder framework is frequently used in the FCN-based saliency models [3, 15–19, 38–40]. Liu et al. [16] proposed a novel network to embed local and global pixelwise contextual attention modules into a U-shape network. Zhao et al. [3] proposed a simple and effective gated network architecture to control the meaningful message passing from encoder to decoder feature blocks. Almost all of the above methods try to develop more complicated modules and strategies to fuse feature blocks of different levels. Different from the methods mentioned above, we propose a simple and effective multitask architecture, which attempts to solve saliency tasks by adding an extra centerness prediction branch.


### 2.2. Multiscale Feature Extraction

The atrous spatial pyramid pooling (ASPP) module [41] is widely used in many computer vision tasks. The atrous convolution can expand the receptive field with fewer parameters to get large-scale and more comprehensive features. The pyramid pooling module (PPM) [42] is another choice for extracting multiscale features. Zhang et al. [43] insert five ASPP modules into the encoder feature blocks of five levels. The larger the atrous rate, the more the difficulty in capturing the changes of image details. To alleviate the above problem, Zhao et al. [3] designed a folded ASPP and achieved a local-in-local effect. Besides, the pyramid attention module [44] can generate multiscale attention maps to enhance saliency features. The above methods can extract multiscale features from images, but it is more sensitive to background noise. To improve the recall rate of saliency objects under complex background, we propose a multiscale feature extraction module and insert it into decoder feature blocks.


### 2.3. Multitask Learning

Multitask learning (MTL) has led to successes in many research fields, from computer vision and speech recognition to drug discovery and natural language processing. Multitask learning aims at simultaneous training using two or more related tasks. It is found that learning multiple tasks jointly can lead to better performance improvement compared with learning them individually. Recent multitask learning-based saliency methods have shown good results by jointly tackling multiple related tasks such as image classification, fixation prediction, and edge detection. Li et al. [23] and Wang et al. [22] proposed to apply image-level tags to assist the detection of the foreground object. Kruthiventi et al. [47] proposed a unified multitask learning framework to jointly solve salient object detection and fixation prediction. Zhao et al. [20] presented an edge guidance network to extract two complementary features, including salient object features and salient edge features. As we all know, location is the important information of an object. To the best of our knowledge, this information has never been directly used in saliency prediction tasks. In this paper, we investigate how to integrate the centerness prediction task into saliency detection.


## 2.1. Salient Object Detection

Early SOD methods [2, 29, 30] are mainly based on hand-crafted features and some intrinsic cues, such as center prior, color contrast, and background prior. Recently, convolutional neural networks (CNNs) have been used to extract multilevel features from input images. CNNs-based methods treat patches/superpixels [31–33] and generic object proposals [34–37] as image processing units, and an MLP-classifier is used to train the network. Wang et al. [35] trained two different CNN models to generate a saliency map. DNN-L and DNN-G are used to extract local and global features, respectively. Particularly, fully convolutional networks (FCN) show their advantage and refresh the state-of-the-art records in saliency prediction task. The encoder-decoder framework is frequently used in the FCN-based saliency models [3, 15–19, 38–40]. Liu et al. [16] proposed a novel network to embed local and global pixelwise contextual attention modules into a U-shape network. Zhao et al. [3] proposed a simple and effective gated network architecture to control the meaningful message passing from encoder to decoder feature blocks. Almost all of the above methods try to develop more complicated modules and strategies to fuse feature blocks of different levels. Different from the methods mentioned above, we propose a simple and effective multitask architecture, which attempts to solve saliency tasks by adding an extra centerness prediction branch.


## 2.2. Multiscale Feature Extraction

The atrous spatial pyramid pooling (ASPP) module [41] is widely used in many computer vision tasks. The atrous convolution can expand the receptive field with fewer parameters to get large-scale and more comprehensive features. The pyramid pooling module (PPM) [42] is another choice for extracting multiscale features. Zhang et al. [43] insert five ASPP modules into the encoder feature blocks of five levels. The larger the atrous rate, the more the difficulty in capturing the changes of image details. To alleviate the above problem, Zhao et al. [3] designed a folded ASPP and achieved a local-in-local effect. Besides, the pyramid attention module [44] can generate multiscale attention maps to enhance saliency features. The above methods can extract multiscale features from images, but it is more sensitive to background noise. To improve the recall rate of saliency objects under complex background, we propose a multiscale feature extraction module and insert it into decoder feature blocks.


## 2.3. Multitask Learning

Multitask learning (MTL) has led to successes in many research fields, from computer vision and speech recognition to drug discovery and natural language processing. Multitask learning aims at simultaneous training using two or more related tasks. It is found that learning multiple tasks jointly can lead to better performance improvement compared with learning them individually. Recent multitask learning-based saliency methods have shown good results by jointly tackling multiple related tasks such as image classification, fixation prediction, and edge detection. Li et al. [23] and Wang et al. [22] proposed to apply image-level tags to assist the detection of the foreground object. Kruthiventi et al. [47] proposed a unified multitask learning framework to jointly solve salient object detection and fixation prediction. Zhao et al. [20] presented an edge guidance network to extract two complementary features, including salient object features and salient edge features. As we all know, location is the important information of an object. To the best of our knowledge, this information has never been directly used in saliency prediction tasks. In this paper, we investigate how to integrate the centerness prediction task into saliency detection.


## 3. Proposed Method

In this paper, we propose a multitask and multiscale deep network for salient object detection. The overview of the proposed network consists of three related modules, as shown in Figure2. To guide the saliency network to uniformly highlight the entire object with different size, we propose a multiscale feature extraction approach. To further improve the detection accuracy, we introduce centerness-aware loss, which helps to reduce the impact of complex background.Figure 2
The overall framework of our proposed multiscale deep network (M2Net). M2Net is based on ResNet-50 [45] or ResNeXt-101 [46] with supervision from saliency map and object position. M2Net consists of the backbone branch, transformation branch, decoder branch, and centerness prediction branch. The backbone network is used to extract some important saliency features, the transformation part is used to generate robust saliency features, the decoder part is used to generate the final saliency maps, and the last part is used to predict object position in an image.


### 3.1. Network Overview

The encoder-decoder architecture has been widely used in the salient object detection task, and it has a strong ability to combine features from different network layers. Our method is built on the feature pyramid networks (FPN) [48] with the pretrained ResNeXt-101 [46] or ResNet-50 [45] as the backbone network, both of which can extract meaningful saliency features to build high-quality U-shape networks. To reduce network parameters, we discard all the fully connected layer of the pretrained backbone [45, 46]. The proposed M2Net consists of a bottom-up encoder module, top-down decoder module, and centerness prediction module. In the encoder, we use the pretrained backbone to extract multilevel saliency features from preprocessed images. To obtain robust saliency features, each feature block is processed by 1 × 1 convolutional layers followed by batch norm and ReLU (Figure 2). Next, in the decoder, we use a skip/concatenation connection scheme. To generate the final saliency maps, a novel multiscale feature extraction approach is proposed (Figure 3). Lastly, we design a centerness prediction module (Figure 2), which can help to generate high-quality saliency maps. We describe the structures of the three modules and explain their transformation in the following sections.Figure 3
Multiscale feature extraction unit; CBR is composed of convolution, batch normalization, and ReLU operation; AP denotes average pooling.


### 3.2. Encoder Module

In our M2Net, the encoder module is composed of a backbone branch and a transformation branch. The backbone branch is based on a common pretrained image classification network, for example, the VGG, ResNet-50 [45] or ResNeXt-101 [46]. In order to fit the saliency prediction task, similar to most previous saliency methods [3, 16, 20], we remove the last pooling layer and cast away all the fully connected layers of the ResNet or ResNeXt. Let I ∈ R320 × 320 ×C denote the input training image with ground truth labels Y ∈ R320 × 320 × 1 as is shown in Figure 2, where C denotes the channel of the input image. For a given input image with size H × W, the pretrained image classification network will extract its saliency features at five different levels, denoted as {Ei ∈ RH × W × C|i = 1, …, 5} with resolutions [H/(2i − 1), W/(2i − 1)], where C denotes the channel of the feature blocks. H and W are the height and width of different level feature blocks. In the transformation branch, the sequence of three operations is used to generate robust and meaningful saliency features, as is shown in Figure 2. The detailed parameters of the three different operators can be found in Table 1. After the processes above, we obtain five different feature blocks {Ti ∈ RH × W × C|i = 1, …, 5}, which are to be used in the decoder part of the network.Table 1
Detailed parameters of three different operators, including convolutional operator, batch normalization operator, and ReLU operator, respectively. The left two parameters of Conv2d are the number of input and output channels, and the right two parameters are the kernel size. The parameter of the BatchNorm2d operator is the number of feature channels, and the parameter of the ReLU operator is in place.
No.FeatureConv2dBatchNorm2dReLUResult1F12048, 512, 1 × 1512TrueT12F21024, 256, 1 × 1256TrueT23F3512, 128, 1 × 1128TrueT34F4256, 64, 1 × 164TrueT45F5128, 32, 1 × 132TrueT5


### 3.3. Decoder Module

In the encoder, different levels of feature blocks contain different information. The high-layer feature blocks encode the semantic information for category, and these layers do not care about local detail for image.The low-layer feature blocks contain more detailed information about the image, and these layers suffer from the problem of semantic ambiguity. The decoder module is designed to integrate these different feature blocks. The combination of these different level feature blocks can enhance the representation ability to complete the saliency prediction task. The decoder network comprises two main computing units: (i) Multiscale feature extraction unit, which can extract multiscale contextual features to facilitate saliency prediction models to extract discriminative features. (ii) Cross-layer feature block fusion unit, which continually fuses adjacent level feature blocks from {Ti|i = 1, …, 5}.Figure3 shows the details of the multiscale feature extraction unit. Given a feature block Fini∈RH×W×C, we first use average pooling to perform a downsampling operation. After that, we can obtain F˜ini∈RH/2×W/2×C. To obtain robust saliency features, the two branches are processed by combination operation, which is composed of convolution, batch normalization, and ReLU operation. After a series of above-mentioned processing, we obtain Fouti=CBRCBRFini and F˜outi=CBRCBRF˜ini. The output of the bottom branch F˜outi is upsampled to match the output of top branch Fouti. To extract the multiscale features, we integrate the two branches by using multiplication and addition operations. The multiscale feature extraction unit is formulated as follows:(1)Ffinali=0.5×Fouti+0.5×UpF˜outi+Fouti×UpF˜outi,where U⋅ denote upsampling operation.Figure4 shows the details of the cross-layer feature block fusion unit. This transformation unit is composed of four different kinds of operators, including convolution, upsampling, concatenation, and combinator. The first convolution layer can halve the number of channels for high-level feature block. To adapt to the low-level feature block, the transformed high-level feature block is processed by the second upsampling operator, which can increase the size of feature blocks by 2 times. Then, the concatenation operator is used to build one larger feature block. Finally, the combinator is composed of the two repeated cascaded structures of convolution operators, each of them followed by a batch normalization layer and a ReLU layer. After a series of above-mentioned processing, we can obtain four different feature blocks {Mi∈RH×W×C|i=1,…,4}, as shown in Figure 2.Figure 4
Cross-layer feature block fusion unit, which continually fuses adjacent level feature blocks.The cross-layer feature block fusion unit is formulated as follows:(2)Mi=CBRCatTi,UDi+1,where CBR⋅ and U⋅ represent the combined operation as mentioned above and the upsampling operation, respectively. Di denote the output of the decoder, and it is formulated as follows:(3)Di=MSFT5,i=5,MSFMi,i=1,2,3,4,where MSF⋅ denote multiscale feature extraction operation, which is defined in equation (1).


### 3.4. Centerness Prediction Module

Object location information can be very useful to improve the image classification task but seldom used in the saliency detection task.In this paper, we introduce the centerness to the saliency detection. We define centerness as a ratio between EO and EC, as is shown in Figure5. The location of node O represents the center of ground truth or saliency map, and it can be calculated as follows:(4)Ox,Oy=∑i=1H∑j=1Wxifij∑i=1H∑j=1Wfij,∑i=1H∑j=1Wyifij∑i=1H∑j=1Wfij,where fij∈0,1 denotes the gray value of ground truth or predicted saliency map.Figure 5
Illustration of the centerness calculation. We define centerness as the ratio between EO and EC.Centerness prediction module comprised two main components: (i) FCL (two fully connected layers), which directly maps high-dimensional feature space to 1-dimensional feature space; (ii) logistic function, which applies a sigmoid function to restrict the number from a large scale to within the range 0–1. Figure6 shows the details of the FCL. This component contains three fully connected layers, and each layer contains a different number of neural nodes.Figure 6
Illustration of the detailed FCL (fully connected layer). The first column contains 8192 nodes, the second column contains 100 nodes, and the third column contains only one node.


### 3.5. Deep Supervision

An effective loss function plays an important role in training more effective saliency models [26]. When the image contains complex background, the deep network with BCE loss will probably generate poor saliency results. To generate high-quality saliency maps with clear boundaries, we propose a centerness-aware loss, which is defined as follows:(5)LCAL=Lbce+Lloc+λ×Liou,where Lbce, Lloc, and Liou denote BCE loss [49], location loss, and IoU loss [17, 27], respectively. The parameter λ is a hyperparameter which is set to 0.5 in this paper.Binary cross entropy (BCE) is a widely used loss in saliency detection tasks, and it is defined as follows:(6)Lbce=−1B×M∑k=0k=B∑i=0i=Myik×logxik+1−yik×log1−xik,where xi and yi∈0,1 denote the prediction of the pixel i and ground truth. B is the batch size and M is the product of the height and width of a given image.The position of salient objects is very important information. Hence, we introduced it into our training loss;Lloc is defined as follows:(7)Lloc=1B∑k=0k=BCgk−Cpk2,where Cgk denotes the ground truth of the k-th image and Cpk is the result of centerness prediction.To uniformly highlight the whole salient region, we integrated IoU [17, 27] into our training loss. It is defined as(8)Liou=1B∑k=1B1−∑i=0Msikyik∑i=0Msik+yik−sikyik.where sik∈0,1 is the predicted probability of being the foreground object and yik is the ground truth of the pixel i.


## 3.1. Network Overview

The encoder-decoder architecture has been widely used in the salient object detection task, and it has a strong ability to combine features from different network layers. Our method is built on the feature pyramid networks (FPN) [48] with the pretrained ResNeXt-101 [46] or ResNet-50 [45] as the backbone network, both of which can extract meaningful saliency features to build high-quality U-shape networks. To reduce network parameters, we discard all the fully connected layer of the pretrained backbone [45, 46]. The proposed M2Net consists of a bottom-up encoder module, top-down decoder module, and centerness prediction module. In the encoder, we use the pretrained backbone to extract multilevel saliency features from preprocessed images. To obtain robust saliency features, each feature block is processed by 1 × 1 convolutional layers followed by batch norm and ReLU (Figure 2). Next, in the decoder, we use a skip/concatenation connection scheme. To generate the final saliency maps, a novel multiscale feature extraction approach is proposed (Figure 3). Lastly, we design a centerness prediction module (Figure 2), which can help to generate high-quality saliency maps. We describe the structures of the three modules and explain their transformation in the following sections.Figure 3
Multiscale feature extraction unit; CBR is composed of convolution, batch normalization, and ReLU operation; AP denotes average pooling.


## 3.2. Encoder Module

In our M2Net, the encoder module is composed of a backbone branch and a transformation branch. The backbone branch is based on a common pretrained image classification network, for example, the VGG, ResNet-50 [45] or ResNeXt-101 [46]. In order to fit the saliency prediction task, similar to most previous saliency methods [3, 16, 20], we remove the last pooling layer and cast away all the fully connected layers of the ResNet or ResNeXt. Let I ∈ R320 × 320 ×C denote the input training image with ground truth labels Y ∈ R320 × 320 × 1 as is shown in Figure 2, where C denotes the channel of the input image. For a given input image with size H × W, the pretrained image classification network will extract its saliency features at five different levels, denoted as {Ei ∈ RH × W × C|i = 1, …, 5} with resolutions [H/(2i − 1), W/(2i − 1)], where C denotes the channel of the feature blocks. H and W are the height and width of different level feature blocks. In the transformation branch, the sequence of three operations is used to generate robust and meaningful saliency features, as is shown in Figure 2. The detailed parameters of the three different operators can be found in Table 1. After the processes above, we obtain five different feature blocks {Ti ∈ RH × W × C|i = 1, …, 5}, which are to be used in the decoder part of the network.Table 1
Detailed parameters of three different operators, including convolutional operator, batch normalization operator, and ReLU operator, respectively. The left two parameters of Conv2d are the number of input and output channels, and the right two parameters are the kernel size. The parameter of the BatchNorm2d operator is the number of feature channels, and the parameter of the ReLU operator is in place.
No.FeatureConv2dBatchNorm2dReLUResult1F12048, 512, 1 × 1512TrueT12F21024, 256, 1 × 1256TrueT23F3512, 128, 1 × 1128TrueT34F4256, 64, 1 × 164TrueT45F5128, 32, 1 × 132TrueT5


## 3.3. Decoder Module

In the encoder, different levels of feature blocks contain different information. The high-layer feature blocks encode the semantic information for category, and these layers do not care about local detail for image.The low-layer feature blocks contain more detailed information about the image, and these layers suffer from the problem of semantic ambiguity. The decoder module is designed to integrate these different feature blocks. The combination of these different level feature blocks can enhance the representation ability to complete the saliency prediction task. The decoder network comprises two main computing units: (i) Multiscale feature extraction unit, which can extract multiscale contextual features to facilitate saliency prediction models to extract discriminative features. (ii) Cross-layer feature block fusion unit, which continually fuses adjacent level feature blocks from {Ti|i = 1, …, 5}.Figure3 shows the details of the multiscale feature extraction unit. Given a feature block Fini∈RH×W×C, we first use average pooling to perform a downsampling operation. After that, we can obtain F˜ini∈RH/2×W/2×C. To obtain robust saliency features, the two branches are processed by combination operation, which is composed of convolution, batch normalization, and ReLU operation. After a series of above-mentioned processing, we obtain Fouti=CBRCBRFini and F˜outi=CBRCBRF˜ini. The output of the bottom branch F˜outi is upsampled to match the output of top branch Fouti. To extract the multiscale features, we integrate the two branches by using multiplication and addition operations. The multiscale feature extraction unit is formulated as follows:(1)Ffinali=0.5×Fouti+0.5×UpF˜outi+Fouti×UpF˜outi,where U⋅ denote upsampling operation.Figure4 shows the details of the cross-layer feature block fusion unit. This transformation unit is composed of four different kinds of operators, including convolution, upsampling, concatenation, and combinator. The first convolution layer can halve the number of channels for high-level feature block. To adapt to the low-level feature block, the transformed high-level feature block is processed by the second upsampling operator, which can increase the size of feature blocks by 2 times. Then, the concatenation operator is used to build one larger feature block. Finally, the combinator is composed of the two repeated cascaded structures of convolution operators, each of them followed by a batch normalization layer and a ReLU layer. After a series of above-mentioned processing, we can obtain four different feature blocks {Mi∈RH×W×C|i=1,…,4}, as shown in Figure 2.Figure 4
Cross-layer feature block fusion unit, which continually fuses adjacent level feature blocks.The cross-layer feature block fusion unit is formulated as follows:(2)Mi=CBRCatTi,UDi+1,where CBR⋅ and U⋅ represent the combined operation as mentioned above and the upsampling operation, respectively. Di denote the output of the decoder, and it is formulated as follows:(3)Di=MSFT5,i=5,MSFMi,i=1,2,3,4,where MSF⋅ denote multiscale feature extraction operation, which is defined in equation (1).


## 3.4. Centerness Prediction Module

Object location information can be very useful to improve the image classification task but seldom used in the saliency detection task.In this paper, we introduce the centerness to the saliency detection. We define centerness as a ratio between EO and EC, as is shown in Figure5. The location of node O represents the center of ground truth or saliency map, and it can be calculated as follows:(4)Ox,Oy=∑i=1H∑j=1Wxifij∑i=1H∑j=1Wfij,∑i=1H∑j=1Wyifij∑i=1H∑j=1Wfij,where fij∈0,1 denotes the gray value of ground truth or predicted saliency map.Figure 5
Illustration of the centerness calculation. We define centerness as the ratio between EO and EC.Centerness prediction module comprised two main components: (i) FCL (two fully connected layers), which directly maps high-dimensional feature space to 1-dimensional feature space; (ii) logistic function, which applies a sigmoid function to restrict the number from a large scale to within the range 0–1. Figure6 shows the details of the FCL. This component contains three fully connected layers, and each layer contains a different number of neural nodes.Figure 6
Illustration of the detailed FCL (fully connected layer). The first column contains 8192 nodes, the second column contains 100 nodes, and the third column contains only one node.


## 3.5. Deep Supervision

An effective loss function plays an important role in training more effective saliency models [26]. When the image contains complex background, the deep network with BCE loss will probably generate poor saliency results. To generate high-quality saliency maps with clear boundaries, we propose a centerness-aware loss, which is defined as follows:(5)LCAL=Lbce+Lloc+λ×Liou,where Lbce, Lloc, and Liou denote BCE loss [49], location loss, and IoU loss [17, 27], respectively. The parameter λ is a hyperparameter which is set to 0.5 in this paper.Binary cross entropy (BCE) is a widely used loss in saliency detection tasks, and it is defined as follows:(6)Lbce=−1B×M∑k=0k=B∑i=0i=Myik×logxik+1−yik×log1−xik,where xi and yi∈0,1 denote the prediction of the pixel i and ground truth. B is the batch size and M is the product of the height and width of a given image.The position of salient objects is very important information. Hence, we introduced it into our training loss;Lloc is defined as follows:(7)Lloc=1B∑k=0k=BCgk−Cpk2,where Cgk denotes the ground truth of the k-th image and Cpk is the result of centerness prediction.To uniformly highlight the whole salient region, we integrated IoU [17, 27] into our training loss. It is defined as(8)Liou=1B∑k=1B1−∑i=0Msikyik∑i=0Msik+yik−sikyik.where sik∈0,1 is the predicted probability of being the foreground object and yik is the ground truth of the pixel i.


## 4. Experiments

### 4.1. Implementation Details

We train our saliency model on the DUTS-TR [22] dataset with 10553 images as followed by [3, 16]. For a fair comparison, we use ResNet and ResNeXt as backbone networks, respectively. For convenience, all the training and testing images are resized to 320 × 320. Our saliency model is implemented in PyTorch. The parameters of backbone networks are initialized with the models pretrained on the classification dataset. All the other parameters of M2Net are set by the default setting of PyTorch 1.2.0. The hyperparameters are set as follows: weight decay = 0.0005 and momentum = 0.9, and the initial learning rate is set to 0.005 for pretrained backbone networks [45, 46] and 0.05 for the rest parts. In this paper, we use the warm-up and linear decay methods to dynamically adjust the learning rate. During the training stage, random flip, random contrast, random saturation, and random brightness act as data augmentation techniques to avoid the overfitting problem. We apply a stochastic gradient descent algorithm to update all the parameters of the proposed M2Net. To ensure model convergence, M2Net is trained for 32 epochs with a minibatch of 15 on an NVIDIA GTX 2080 Ti GPU.


### 4.2. Datasets

The performance of M2Net is evaluated on five benchmark saliency datasets, including ECSSD [50], PASCAL-S [51], DUTS [22], DUT-OMRON [30], and HKU-IS [34]. ECSSD [50] contains 1000 meaningful semantic images with pixel-accurate annotations. The PASCAL-S [51] dataset is composed of 850 challenging images, which are carefully selected from the PASCAL VOC segmentation dataset. DUTS is the largest salient object detection (SOD) dataset. It contains 10553 images for training and the remaining 5019 images for testing. DUT-OMRON [30] is composed of 5168 high-quality but challenging images. Images in this dataset contain one or more salient objects with complex background. The HKU-IS [34] contains 4447 challenging images which have multiple disconnected salient objects.


### 4.3. Evaluation Criteria

To quantitatively evaluate the performance, four measurements, including Precision-Recall (PR) curve, F-measure, and Mean Absolute Error (MAE), and S-measure, are adopted in our experiments.Precision-Recall curve is a widely used graphical tool to evaluate the robustness of saliency maps. It can demonstrate the relation of precision and recall by thresholding the final saliency results from 0 to 255. The larger the area under the PR curve, the better the performance.TheF-measure is a weighted combination of precision and recall, which is defined as(9)Fβ=1+β2×precision×recallβ2×precision+recall,where β2 is set to 0.3 as done in most recent state-of-the-art saliency methods [3, 16, 19, 52–56] to emphasize the precision. The mean F-measure (Fm) of each benchmark dataset is reported in the paper.Mean Absolute Error (MAE) is a metric, which measures the pixelwise average absolute difference between saliency map and its corresponding ground truth. The MAE score is defined as follows:(10)MAE=1M∑m=1Mxm−ym,wherex and y are the prediction result and ground truth, respectively, and M indicates the total number of image pixels.S-measure is more sensitive to foreground structural information of saliency maps, which is closer to the human visual system. It considers the object-aware structural similarity So and the region-aware structural similarity Sr:(11)S=γ×So+1−γ×Sr,where γ is set to 0.5 as suggested in [3, 20, 53, 57].


### 4.4. Comparison with State of the Art

In this section, we compare our method with seventeen previous state-of-the-art saliency models, including NLDF [58], Amulet [15], R3Net [59], RAS [60], DGRL [61], C2SNet [54], PiCANet [16], BMPM [43], BASNet [17], AFNet [62], SCRN [63], CPD [64], EGNet [20], PoolNet [18], F3Net [19], MINet [53], and GateNet [3]. Note that all the saliency maps of above saliency methods are produced by running source codes or precomputed by the authors.


#### 4.4.1. Quantitative Evaluation

To fully compare the proposed saliency model with these state-of-the-art methods, the detailed experimental results in terms of three metrics are listed in Table2. For better comparison, we use the ResNet-50 and ResNeXt-101 as backbone networks for training our proposed M2Net. Specifically, our method achieves a great improvement in terms of the Fm compared to the most recent saliency model GateNet [3] on the challenging DUT-TE [22] (0.857 versus 0.816), DUT-OMRON [30] (0.791 versus 0.762), and PASCAL-S [51] (0.858 versus 0.827). In addition, we demonstrate the standard PR curves in Figures 7 and 8. Our method achieves the best performance on the ECSSD, HKU-IS, PASCAL-S, DUT-OMRON, and DUT-TE datasets.Table 2
Performance comparison with state-of-the-art methods on five popular saliency datasets. MAE (smaller is better), maxF-measure (larger is better), and E-measure (larger is better) are used to measure the model performance.
MethodDUT-OMRONDUTSECSSDPASCAL-SHKU-ISFmMAESmFmMAESmFmMAESmFmMAESmFmMAESmVGG backboneAmulet170.6470.0980.7810.6780.0850.8040.8680.0590.8940.7690.0990.8190.8410.0510.886NLDF170.6840.0800.770———0.8780.0630.8750.7800.1010.8010.8740.0480.879BMPM180.6920.0640.8090.7450.0490.8620.8680.0450.9110.7710.0750.8450.8710.0390.907C2SNet180.6830.0720.7980.7160.0630.8280.8640.0550.8930.7690.0830.8350.8510.0480.883RAS180.7130.0620.8140.7510.0590.8390.8890.0560.8930.7850.1060.7930.8710.0450.887PiCANet180.7100.0680.8260.7490.0540.8610.8850.0460.9140.8010.0790.8490.8700.0420.906CPD190.7450.0570.8180.8130.0430.8670.9150.0400.9100.8300.0750.8410.8960.0330.904MINet200.7410.0570.8210.8230.0390.8750.9220.0360.9190.8400.0660.8520.9040.0310.912ResNet backboneDGRL180.7330.0620.8060.7940.0500.8420.9060.0410.9030.8270.0730.8370.8900.0360.894PiCANet180.7170.0650.8320.7590.0510.8690.8860.0460.9170.8020.0780.8520.8700.0430.904BASNet190.7560.0570.8360.7910.0480.8660.8800.0370.9160.7770.0790.8340.8960.0320.909EGNet190.7560.0530.8410.8150.0390.8870.9200.0370.9250.8290.0760.8500.9010.0310.918PoolNet190.7470.0560.8360.8090.0400.8830.9150.0390.9210.8280.0760.8490.8990.0320.917CPD190.7470.0560.8250.8050.0430.8690.9170.0370.9180.8290.0740.8440.8910.0340.906SCRN190.7460.0560.8370.8090.0400.8850.9180.0380.9270.8370.0660.8650.8970.0340.916MINet200.7550.0550.8330.8280.0370.8840.9250.0340.9250.8400.0660.8540.9090.0290.919GateNet200.7460.0550.8380.8070.0400.8850.9160.0400.9200.8300.0710.8540.8990.0330.915Ours0.7740.0540.8460.8370.0380.8890.9260.0330.9270.8470.0640.8630.9130.0290.922ResNeXt backboneR3Net180.7470.0620.815———0.9140.0400.9100.8030.0950.8030.8940.0360.895GateNet200.7620.0510.8490.8160.0350.8970.9170.0350.9290.8270.0650.8650.9030.0300.925Ours0.7910.0490.8600.8570.0340.9000.9340.0320.9330.8580.0610.8700.9230.0250.929Bold, italics, and underline indicate the best, second best, and third best performance. “—” means that the author has not provided corresponding saliency maps.Figure 7
Precision-Recall curves on three saliency datasets, including DUT-OMRON, DUTS, and PASCAL-S.Figure 8
Precision-Recall curves on two common saliency datasets, including HKU-IS and ECSSD.


#### 4.4.2. Qualitative Evaluation

Some prediction results of the proposed M2Net and ten state-of-the-art saliency methods have been shown in Figure9. We observe that the proposed method M2Net not only uniformly highlights the correct salient object region clearly but also well suppresses the background clutter effectively. It excels in dealing with various challenging scenarios, including small objects (rows 2, 8, and 9), cluttered backgrounds (rows 1, 3, and 5), low contrast between the salient object and background region (rows 4 and 7), and image boundary (row 6). Compared with other state-of-the-art methods, the detected object boundaries of our saliency map are clear and sharper. Most importantly, the proposed saliency model M2Net achieves these results without any postprocessing.Figure 9
Visual comparison of our method with ten state-of-the-art methods.


### 4.5. Ablation Analysis

Before analyzing the influence of each saliency module, there is one hyperparameterλ to be determined. λ is used in CAL loss to balance different losses. Table 3 lists the scores of Fm, MAE, and Sm when λ gives four discrete values. As can be seen, when λ equals 0.50, these indicators reach the best results. To investigate the importance of different components in our proposed M2Net, we will conduct a detailed analysis next.Table 3
Comparison with differentλ. When λ = 0.50, the proposed model achieves the best results.
DUT-OMRONDUTSECSSDPASCAL-SHKU-ISFmMAESmFmMAESmFmMAESmFmMAESmFmMAESmλ = 0.250.7610.0540.8460.8250.0380.8920.9180.0360.9280.8370.0670.8620.9070.0300.923λ = 0.500.7910.0490.8600.8570.0340.9000.9340.0320.9330.8580.0610.8700.9230.0250.929λ = 0.750.7770.0510.8470.8390.0380.8870.9260.0340.9260.8490.0650.8610.9130.0290.920λ = 1.000.7780.0510.8450.8440.0360.8870.9270.0350.9240.8450.0690.8530.9170.0280.919


#### 4.5.1. Effectiveness of Backbones

In the saliency detection, VGG [65], ResNet [45], and ResNeXt [46] are widely used as the pretrained backbone. Table 2 demonstrates that ResNet-50 and ResNeXt-101 can achieve better performance compared with VGG in most cases. To demonstrate the effectiveness of ResNet-50 and ResNeXt-101, we also selected two widely used datasets for evaluation, and the comparison results are shown in Table 4. From Table 4, we can see that M2Net with ResNeXt-101 [46] can get better performance compared with ResNet-50 [45].Table 4
Ablation analysis on two challenge datasets. FPN-1: FPN [48] with ResNet-50 backbone; FPN-2: FPN [48] with ResNeXt-101 backbone; M2N-1: the proposed M2Net with ResNet-based backbone; M2N-2: the proposed M2Net with ResNeXt-based backbone B: binary cross entropy; C: centerness; I: Intersection over Union.
ModelLossDUT-OMRONPASCAL-SFmMAESmFmMAESmFPN-1+B0.7110.0640.8150.8140.0740.849M2N-1+B0.7420.0550.8400.8300.0690.861M2N-1+B + C0.7480.0550.8420.8360.0670.862M2N-1+B + C + I0.7740.0540.8460.8470.0640.863FPN-2+B0.7450.0560.8420.8350.0670.861M2N-2+B0.7610.0510.8510.8450.0630.864M2N-2+B + C0.7680.0500.8550.8520.0610.866M2N-2+B + C + I0.7910.0490.8600.8580.0610.870


#### 4.5.2. Effectiveness of Components

We take an FPN-like network as our baseline network. Then, we install the multiscale feature unit on the baseline network and evaluate its performance. The comparison results are shown in Table4. It can be seen that a multiscale feature unit can achieve significant improvement over the FPN-like network. We also quantitatively evaluate the effect of the centerness-aware loss in Table 4. Compared to “+B,” the proposed M2Net with the CAL achieves consistent performance enhancements in terms of three metrics. Visual comparison of saliency maps generated by BCE loss and our centerness-aware loss are shown in Figures 10(c), 10(d), and 10(e). To fully compare CAL and three other losses, including FLoss, PPA, and IoU loss, the detailed experimental results are listed in Table 5. As it can be seen, our proposed CAL loss can get the best results on two challenge saliency datasets.Figure 10
Visual comparisons for showing the proposed modules. M: the proposed M2Net with binary cross entropy loss; C: centerness; I: Intersection over Union. (a) Image. (b) GT. (c) M. (d) M + C. (e) M + C + I.
(a)(b)(c)(d)(e)Table 5
Comparison of the proposed loss and three other methods on two challenge datasets. M2N: the proposed M2Net with ResNeXt-based backbone CAL: centerness-aware loss; I: Intersection over Union loss [17, 27]; FLoss: F-measure based loss [28]; PPA: pixel position aware loss [19].
ModelDUT-OMRONPASCAL-SFmMAESmFmMAESmM2N + I0.7500.0480.8020.8130.0780.813M2N + FLoss0.7810.0510.8170.8500.0740.829M2N + PPA0.7700.0540.8410.8450.0660.857M2N + CAL0.7910.0490.8600.8580.0610.870


## 4.1. Implementation Details

We train our saliency model on the DUTS-TR [22] dataset with 10553 images as followed by [3, 16]. For a fair comparison, we use ResNet and ResNeXt as backbone networks, respectively. For convenience, all the training and testing images are resized to 320 × 320. Our saliency model is implemented in PyTorch. The parameters of backbone networks are initialized with the models pretrained on the classification dataset. All the other parameters of M2Net are set by the default setting of PyTorch 1.2.0. The hyperparameters are set as follows: weight decay = 0.0005 and momentum = 0.9, and the initial learning rate is set to 0.005 for pretrained backbone networks [45, 46] and 0.05 for the rest parts. In this paper, we use the warm-up and linear decay methods to dynamically adjust the learning rate. During the training stage, random flip, random contrast, random saturation, and random brightness act as data augmentation techniques to avoid the overfitting problem. We apply a stochastic gradient descent algorithm to update all the parameters of the proposed M2Net. To ensure model convergence, M2Net is trained for 32 epochs with a minibatch of 15 on an NVIDIA GTX 2080 Ti GPU.


## 4.2. Datasets

The performance of M2Net is evaluated on five benchmark saliency datasets, including ECSSD [50], PASCAL-S [51], DUTS [22], DUT-OMRON [30], and HKU-IS [34]. ECSSD [50] contains 1000 meaningful semantic images with pixel-accurate annotations. The PASCAL-S [51] dataset is composed of 850 challenging images, which are carefully selected from the PASCAL VOC segmentation dataset. DUTS is the largest salient object detection (SOD) dataset. It contains 10553 images for training and the remaining 5019 images for testing. DUT-OMRON [30] is composed of 5168 high-quality but challenging images. Images in this dataset contain one or more salient objects with complex background. The HKU-IS [34] contains 4447 challenging images which have multiple disconnected salient objects.


## 4.3. Evaluation Criteria

To quantitatively evaluate the performance, four measurements, including Precision-Recall (PR) curve, F-measure, and Mean Absolute Error (MAE), and S-measure, are adopted in our experiments.Precision-Recall curve is a widely used graphical tool to evaluate the robustness of saliency maps. It can demonstrate the relation of precision and recall by thresholding the final saliency results from 0 to 255. The larger the area under the PR curve, the better the performance.TheF-measure is a weighted combination of precision and recall, which is defined as(9)Fβ=1+β2×precision×recallβ2×precision+recall,where β2 is set to 0.3 as done in most recent state-of-the-art saliency methods [3, 16, 19, 52–56] to emphasize the precision. The mean F-measure (Fm) of each benchmark dataset is reported in the paper.Mean Absolute Error (MAE) is a metric, which measures the pixelwise average absolute difference between saliency map and its corresponding ground truth. The MAE score is defined as follows:(10)MAE=1M∑m=1Mxm−ym,wherex and y are the prediction result and ground truth, respectively, and M indicates the total number of image pixels.S-measure is more sensitive to foreground structural information of saliency maps, which is closer to the human visual system. It considers the object-aware structural similarity So and the region-aware structural similarity Sr:(11)S=γ×So+1−γ×Sr,where γ is set to 0.5 as suggested in [3, 20, 53, 57].


## 4.4. Comparison with State of the Art

In this section, we compare our method with seventeen previous state-of-the-art saliency models, including NLDF [58], Amulet [15], R3Net [59], RAS [60], DGRL [61], C2SNet [54], PiCANet [16], BMPM [43], BASNet [17], AFNet [62], SCRN [63], CPD [64], EGNet [20], PoolNet [18], F3Net [19], MINet [53], and GateNet [3]. Note that all the saliency maps of above saliency methods are produced by running source codes or precomputed by the authors.


### 4.4.1. Quantitative Evaluation

To fully compare the proposed saliency model with these state-of-the-art methods, the detailed experimental results in terms of three metrics are listed in Table2. For better comparison, we use the ResNet-50 and ResNeXt-101 as backbone networks for training our proposed M2Net. Specifically, our method achieves a great improvement in terms of the Fm compared to the most recent saliency model GateNet [3] on the challenging DUT-TE [22] (0.857 versus 0.816), DUT-OMRON [30] (0.791 versus 0.762), and PASCAL-S [51] (0.858 versus 0.827). In addition, we demonstrate the standard PR curves in Figures 7 and 8. Our method achieves the best performance on the ECSSD, HKU-IS, PASCAL-S, DUT-OMRON, and DUT-TE datasets.Table 2
Performance comparison with state-of-the-art methods on five popular saliency datasets. MAE (smaller is better), maxF-measure (larger is better), and E-measure (larger is better) are used to measure the model performance.
MethodDUT-OMRONDUTSECSSDPASCAL-SHKU-ISFmMAESmFmMAESmFmMAESmFmMAESmFmMAESmVGG backboneAmulet170.6470.0980.7810.6780.0850.8040.8680.0590.8940.7690.0990.8190.8410.0510.886NLDF170.6840.0800.770———0.8780.0630.8750.7800.1010.8010.8740.0480.879BMPM180.6920.0640.8090.7450.0490.8620.8680.0450.9110.7710.0750.8450.8710.0390.907C2SNet180.6830.0720.7980.7160.0630.8280.8640.0550.8930.7690.0830.8350.8510.0480.883RAS180.7130.0620.8140.7510.0590.8390.8890.0560.8930.7850.1060.7930.8710.0450.887PiCANet180.7100.0680.8260.7490.0540.8610.8850.0460.9140.8010.0790.8490.8700.0420.906CPD190.7450.0570.8180.8130.0430.8670.9150.0400.9100.8300.0750.8410.8960.0330.904MINet200.7410.0570.8210.8230.0390.8750.9220.0360.9190.8400.0660.8520.9040.0310.912ResNet backboneDGRL180.7330.0620.8060.7940.0500.8420.9060.0410.9030.8270.0730.8370.8900.0360.894PiCANet180.7170.0650.8320.7590.0510.8690.8860.0460.9170.8020.0780.8520.8700.0430.904BASNet190.7560.0570.8360.7910.0480.8660.8800.0370.9160.7770.0790.8340.8960.0320.909EGNet190.7560.0530.8410.8150.0390.8870.9200.0370.9250.8290.0760.8500.9010.0310.918PoolNet190.7470.0560.8360.8090.0400.8830.9150.0390.9210.8280.0760.8490.8990.0320.917CPD190.7470.0560.8250.8050.0430.8690.9170.0370.9180.8290.0740.8440.8910.0340.906SCRN190.7460.0560.8370.8090.0400.8850.9180.0380.9270.8370.0660.8650.8970.0340.916MINet200.7550.0550.8330.8280.0370.8840.9250.0340.9250.8400.0660.8540.9090.0290.919GateNet200.7460.0550.8380.8070.0400.8850.9160.0400.9200.8300.0710.8540.8990.0330.915Ours0.7740.0540.8460.8370.0380.8890.9260.0330.9270.8470.0640.8630.9130.0290.922ResNeXt backboneR3Net180.7470.0620.815———0.9140.0400.9100.8030.0950.8030.8940.0360.895GateNet200.7620.0510.8490.8160.0350.8970.9170.0350.9290.8270.0650.8650.9030.0300.925Ours0.7910.0490.8600.8570.0340.9000.9340.0320.9330.8580.0610.8700.9230.0250.929Bold, italics, and underline indicate the best, second best, and third best performance. “—” means that the author has not provided corresponding saliency maps.Figure 7
Precision-Recall curves on three saliency datasets, including DUT-OMRON, DUTS, and PASCAL-S.Figure 8
Precision-Recall curves on two common saliency datasets, including HKU-IS and ECSSD.


### 4.4.2. Qualitative Evaluation

Some prediction results of the proposed M2Net and ten state-of-the-art saliency methods have been shown in Figure9. We observe that the proposed method M2Net not only uniformly highlights the correct salient object region clearly but also well suppresses the background clutter effectively. It excels in dealing with various challenging scenarios, including small objects (rows 2, 8, and 9), cluttered backgrounds (rows 1, 3, and 5), low contrast between the salient object and background region (rows 4 and 7), and image boundary (row 6). Compared with other state-of-the-art methods, the detected object boundaries of our saliency map are clear and sharper. Most importantly, the proposed saliency model M2Net achieves these results without any postprocessing.Figure 9
Visual comparison of our method with ten state-of-the-art methods.


## 4.4.1. Quantitative Evaluation

To fully compare the proposed saliency model with these state-of-the-art methods, the detailed experimental results in terms of three metrics are listed in Table2. For better comparison, we use the ResNet-50 and ResNeXt-101 as backbone networks for training our proposed M2Net. Specifically, our method achieves a great improvement in terms of the Fm compared to the most recent saliency model GateNet [3] on the challenging DUT-TE [22] (0.857 versus 0.816), DUT-OMRON [30] (0.791 versus 0.762), and PASCAL-S [51] (0.858 versus 0.827). In addition, we demonstrate the standard PR curves in Figures 7 and 8. Our method achieves the best performance on the ECSSD, HKU-IS, PASCAL-S, DUT-OMRON, and DUT-TE datasets.Table 2
Performance comparison with state-of-the-art methods on five popular saliency datasets. MAE (smaller is better), maxF-measure (larger is better), and E-measure (larger is better) are used to measure the model performance.
MethodDUT-OMRONDUTSECSSDPASCAL-SHKU-ISFmMAESmFmMAESmFmMAESmFmMAESmFmMAESmVGG backboneAmulet170.6470.0980.7810.6780.0850.8040.8680.0590.8940.7690.0990.8190.8410.0510.886NLDF170.6840.0800.770———0.8780.0630.8750.7800.1010.8010.8740.0480.879BMPM180.6920.0640.8090.7450.0490.8620.8680.0450.9110.7710.0750.8450.8710.0390.907C2SNet180.6830.0720.7980.7160.0630.8280.8640.0550.8930.7690.0830.8350.8510.0480.883RAS180.7130.0620.8140.7510.0590.8390.8890.0560.8930.7850.1060.7930.8710.0450.887PiCANet180.7100.0680.8260.7490.0540.8610.8850.0460.9140.8010.0790.8490.8700.0420.906CPD190.7450.0570.8180.8130.0430.8670.9150.0400.9100.8300.0750.8410.8960.0330.904MINet200.7410.0570.8210.8230.0390.8750.9220.0360.9190.8400.0660.8520.9040.0310.912ResNet backboneDGRL180.7330.0620.8060.7940.0500.8420.9060.0410.9030.8270.0730.8370.8900.0360.894PiCANet180.7170.0650.8320.7590.0510.8690.8860.0460.9170.8020.0780.8520.8700.0430.904BASNet190.7560.0570.8360.7910.0480.8660.8800.0370.9160.7770.0790.8340.8960.0320.909EGNet190.7560.0530.8410.8150.0390.8870.9200.0370.9250.8290.0760.8500.9010.0310.918PoolNet190.7470.0560.8360.8090.0400.8830.9150.0390.9210.8280.0760.8490.8990.0320.917CPD190.7470.0560.8250.8050.0430.8690.9170.0370.9180.8290.0740.8440.8910.0340.906SCRN190.7460.0560.8370.8090.0400.8850.9180.0380.9270.8370.0660.8650.8970.0340.916MINet200.7550.0550.8330.8280.0370.8840.9250.0340.9250.8400.0660.8540.9090.0290.919GateNet200.7460.0550.8380.8070.0400.8850.9160.0400.9200.8300.0710.8540.8990.0330.915Ours0.7740.0540.8460.8370.0380.8890.9260.0330.9270.8470.0640.8630.9130.0290.922ResNeXt backboneR3Net180.7470.0620.815———0.9140.0400.9100.8030.0950.8030.8940.0360.895GateNet200.7620.0510.8490.8160.0350.8970.9170.0350.9290.8270.0650.8650.9030.0300.925Ours0.7910.0490.8600.8570.0340.9000.9340.0320.9330.8580.0610.8700.9230.0250.929Bold, italics, and underline indicate the best, second best, and third best performance. “—” means that the author has not provided corresponding saliency maps.Figure 7
Precision-Recall curves on three saliency datasets, including DUT-OMRON, DUTS, and PASCAL-S.Figure 8
Precision-Recall curves on two common saliency datasets, including HKU-IS and ECSSD.


## 4.4.2. Qualitative Evaluation

Some prediction results of the proposed M2Net and ten state-of-the-art saliency methods have been shown in Figure9. We observe that the proposed method M2Net not only uniformly highlights the correct salient object region clearly but also well suppresses the background clutter effectively. It excels in dealing with various challenging scenarios, including small objects (rows 2, 8, and 9), cluttered backgrounds (rows 1, 3, and 5), low contrast between the salient object and background region (rows 4 and 7), and image boundary (row 6). Compared with other state-of-the-art methods, the detected object boundaries of our saliency map are clear and sharper. Most importantly, the proposed saliency model M2Net achieves these results without any postprocessing.Figure 9
Visual comparison of our method with ten state-of-the-art methods.


## 4.5. Ablation Analysis

Before analyzing the influence of each saliency module, there is one hyperparameterλ to be determined. λ is used in CAL loss to balance different losses. Table 3 lists the scores of Fm, MAE, and Sm when λ gives four discrete values. As can be seen, when λ equals 0.50, these indicators reach the best results. To investigate the importance of different components in our proposed M2Net, we will conduct a detailed analysis next.Table 3
Comparison with differentλ. When λ = 0.50, the proposed model achieves the best results.
DUT-OMRONDUTSECSSDPASCAL-SHKU-ISFmMAESmFmMAESmFmMAESmFmMAESmFmMAESmλ = 0.250.7610.0540.8460.8250.0380.8920.9180.0360.9280.8370.0670.8620.9070.0300.923λ = 0.500.7910.0490.8600.8570.0340.9000.9340.0320.9330.8580.0610.8700.9230.0250.929λ = 0.750.7770.0510.8470.8390.0380.8870.9260.0340.9260.8490.0650.8610.9130.0290.920λ = 1.000.7780.0510.8450.8440.0360.8870.9270.0350.9240.8450.0690.8530.9170.0280.919


### 4.5.1. Effectiveness of Backbones

In the saliency detection, VGG [65], ResNet [45], and ResNeXt [46] are widely used as the pretrained backbone. Table 2 demonstrates that ResNet-50 and ResNeXt-101 can achieve better performance compared with VGG in most cases. To demonstrate the effectiveness of ResNet-50 and ResNeXt-101, we also selected two widely used datasets for evaluation, and the comparison results are shown in Table 4. From Table 4, we can see that M2Net with ResNeXt-101 [46] can get better performance compared with ResNet-50 [45].Table 4
Ablation analysis on two challenge datasets. FPN-1: FPN [48] with ResNet-50 backbone; FPN-2: FPN [48] with ResNeXt-101 backbone; M2N-1: the proposed M2Net with ResNet-based backbone; M2N-2: the proposed M2Net with ResNeXt-based backbone B: binary cross entropy; C: centerness; I: Intersection over Union.
ModelLossDUT-OMRONPASCAL-SFmMAESmFmMAESmFPN-1+B0.7110.0640.8150.8140.0740.849M2N-1+B0.7420.0550.8400.8300.0690.861M2N-1+B + C0.7480.0550.8420.8360.0670.862M2N-1+B + C + I0.7740.0540.8460.8470.0640.863FPN-2+B0.7450.0560.8420.8350.0670.861M2N-2+B0.7610.0510.8510.8450.0630.864M2N-2+B + C0.7680.0500.8550.8520.0610.866M2N-2+B + C + I0.7910.0490.8600.8580.0610.870


### 4.5.2. Effectiveness of Components

We take an FPN-like network as our baseline network. Then, we install the multiscale feature unit on the baseline network and evaluate its performance. The comparison results are shown in Table4. It can be seen that a multiscale feature unit can achieve significant improvement over the FPN-like network. We also quantitatively evaluate the effect of the centerness-aware loss in Table 4. Compared to “+B,” the proposed M2Net with the CAL achieves consistent performance enhancements in terms of three metrics. Visual comparison of saliency maps generated by BCE loss and our centerness-aware loss are shown in Figures 10(c), 10(d), and 10(e). To fully compare CAL and three other losses, including FLoss, PPA, and IoU loss, the detailed experimental results are listed in Table 5. As it can be seen, our proposed CAL loss can get the best results on two challenge saliency datasets.Figure 10
Visual comparisons for showing the proposed modules. M: the proposed M2Net with binary cross entropy loss; C: centerness; I: Intersection over Union. (a) Image. (b) GT. (c) M. (d) M + C. (e) M + C + I.
(a)(b)(c)(d)(e)Table 5
Comparison of the proposed loss and three other methods on two challenge datasets. M2N: the proposed M2Net with ResNeXt-based backbone CAL: centerness-aware loss; I: Intersection over Union loss [17, 27]; FLoss: F-measure based loss [28]; PPA: pixel position aware loss [19].
ModelDUT-OMRONPASCAL-SFmMAESmFmMAESmM2N + I0.7500.0480.8020.8130.0780.813M2N + FLoss0.7810.0510.8170.8500.0740.829M2N + PPA0.7700.0540.8410.8450.0660.857M2N + CAL0.7910.0490.8600.8580.0610.870


## 4.5.1. Effectiveness of Backbones

In the saliency detection, VGG [65], ResNet [45], and ResNeXt [46] are widely used as the pretrained backbone. Table 2 demonstrates that ResNet-50 and ResNeXt-101 can achieve better performance compared with VGG in most cases. To demonstrate the effectiveness of ResNet-50 and ResNeXt-101, we also selected two widely used datasets for evaluation, and the comparison results are shown in Table 4. From Table 4, we can see that M2Net with ResNeXt-101 [46] can get better performance compared with ResNet-50 [45].Table 4
Ablation analysis on two challenge datasets. FPN-1: FPN [48] with ResNet-50 backbone; FPN-2: FPN [48] with ResNeXt-101 backbone; M2N-1: the proposed M2Net with ResNet-based backbone; M2N-2: the proposed M2Net with ResNeXt-based backbone B: binary cross entropy; C: centerness; I: Intersection over Union.
ModelLossDUT-OMRONPASCAL-SFmMAESmFmMAESmFPN-1+B0.7110.0640.8150.8140.0740.849M2N-1+B0.7420.0550.8400.8300.0690.861M2N-1+B + C0.7480.0550.8420.8360.0670.862M2N-1+B + C + I0.7740.0540.8460.8470.0640.863FPN-2+B0.7450.0560.8420.8350.0670.861M2N-2+B0.7610.0510.8510.8450.0630.864M2N-2+B + C0.7680.0500.8550.8520.0610.866M2N-2+B + C + I0.7910.0490.8600.8580.0610.870


## 4.5.2. Effectiveness of Components

We take an FPN-like network as our baseline network. Then, we install the multiscale feature unit on the baseline network and evaluate its performance. The comparison results are shown in Table4. It can be seen that a multiscale feature unit can achieve significant improvement over the FPN-like network. We also quantitatively evaluate the effect of the centerness-aware loss in Table 4. Compared to “+B,” the proposed M2Net with the CAL achieves consistent performance enhancements in terms of three metrics. Visual comparison of saliency maps generated by BCE loss and our centerness-aware loss are shown in Figures 10(c), 10(d), and 10(e). To fully compare CAL and three other losses, including FLoss, PPA, and IoU loss, the detailed experimental results are listed in Table 5. As it can be seen, our proposed CAL loss can get the best results on two challenge saliency datasets.Figure 10
Visual comparisons for showing the proposed modules. M: the proposed M2Net with binary cross entropy loss; C: centerness; I: Intersection over Union. (a) Image. (b) GT. (c) M. (d) M + C. (e) M + C + I.
(a)(b)(c)(d)(e)Table 5
Comparison of the proposed loss and three other methods on two challenge datasets. M2N: the proposed M2Net with ResNeXt-based backbone CAL: centerness-aware loss; I: Intersection over Union loss [17, 27]; FLoss: F-measure based loss [28]; PPA: pixel position aware loss [19].
ModelDUT-OMRONPASCAL-SFmMAESmFmMAESmM2N + I0.7500.0480.8020.8130.0780.813M2N + FLoss0.7810.0510.8170.8500.0740.829M2N + PPA0.7700.0540.8410.8450.0660.857M2N + CAL0.7910.0490.8600.8580.0610.870


## 5. Conclusion

In this paper, we proposed a multiscale deep network with a centerness-aware loss for salient object detection. The proposed M2Net aims to solve saliency prediction and centerness prediction simultaneously. Our model consists of a bottom-up encoder module, top-down decoder module, and centerness prediction module. In the encoder, we use the pretrained backbone to extract multilevel saliency features from preprocessed images. Next, in the decoder module, we use a skip/concatenation connection scheme. To generate the final saliency maps, we proposed a novel multiscale feature extraction method. Lastly, we design a centerness prediction module, which can help to uniformly highlight the entire salient object. Extensive experimental results on five widely used datasets demonstrate that our method outperforms 17 state-of-the-art approaches under different evaluation metrics.



---

*Source: 2243927-2022-01-13.xml*