DPANet: Dual Pooling‐aggregated Attention Network for fish segmentation

IET Computer VisionVolume 16, Issue 1 p. 67-82 ORIGINAL RESEARCH PAPEROpen Access DPANet: Dual Pooling-aggregated Attention Network for fish segmentation Wenbo Zhang, Corresponding Author Wenbo Zhang zhangwenbo@bjut.edu.cn orcid.org/0000-0002-8601-802X The Faculty of Information Technology, Beijing University of Technology, Beijing, China Correspondence Wenbo Zhang, The Faculty of Information Technology, Beijing University of Technology, 100124 Beijing, China. Email: zhangwenbo@bjut.edu.cnSearch for more papers by this authorChaoyi Wu, Chaoyi Wu The Faculty of Information Technology, Beijing University of Technology, Beijing, ChinaSearch for more papers by this authorZhenshan Bao, Zhenshan Bao The Faculty of Information Technology, Beijing University of Technology, Beijing, ChinaSearch for more papers by this author Wenbo Zhang, Corresponding Author Wenbo Zhang zhangwenbo@bjut.edu.cn orcid.org/0000-0002-8601-802X The Faculty of Information Technology, Beijing University of Technology, Beijing, China Correspondence Wenbo Zhang, The Faculty of Information Technology, Beijing University of Technology, 100124 Beijing, China. Email: zhangwenbo@bjut.edu.cnSearch for more papers by this authorChaoyi Wu, Chaoyi Wu The Faculty of Information Technology, Beijing University of Technology, Beijing, ChinaSearch for more papers by this authorZhenshan Bao, Zhenshan Bao The Faculty of Information Technology, Beijing University of Technology, Beijing, ChinaSearch for more papers by this author First published: 10 September 2021 https://doi.org/10.1049/cvi2.12065AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat Abstract The sustainable development of marine fisheries depends on the accurate measurement of data on fish stocks. Semantic segmentation methods based on deep learning can be applied to automatically obtain segmentation masks of fish in images to obtain measurement data. However, general semantic segmentation methods cannot accurately segment fish objects in underwater images. In this study, a Dual Pooling-aggregated Attention Network (DPANet) to adaptively capture long-range dependencies through an efficient and computing-friendly manner to enhance feature representation and improve segmentation performance is proposed. Specifically, a novel pooling-aggregate position attention module and a pooling-aggregate channel attention module are designed to aggregate contexts in the spatial dimension and channel dimension, respectively. These two modules adopt pooling operations along the channel dimension and along the spatial dimension to aggregate information, respectively, thus reducing computational costs. In these modules, attention maps are generated by four different paths and are aggregated into one. The authors conduct extensive experiments to validate the effectiveness of the DPANet and achieve new state-of-the-art segmentation performance on the well-known fish image dataset DeepFish as well as on the underwater image dataset SUIM, achieving a Mean IoU score of 91.08% and 85.39% respectively, while significantly reducing FLOPs of attention modules by about 93%. 1 INTRODUCTION Monitoring fish habitats is necessary to protect the sustainability of commercial fisheries for marine fish, to prevent damage to marine fish ecosystems caused by overfishing, and to support scientific research on fish survival in natural underwater environments. Fish habitat monitoring includes measuring the size, shape and weight of fish, and the data obtained is used to determine the growth of fish stocks, guide feeding, harvesting and conservation, and to ensure the healthy development of fish stocks [1]. Therefore, obtaining the information on fish size and shape is important for the sustainable development of fisheries. However, traditional methods of measuring fish size usually involve image acquisition of fish in the water column by underwater cameras, followed by visual inspection or precise measurement with a ruler [2], and there are also fishermen who use echo sounders to detect the size of fish [3]. The drawbacks of these methods are obvious: they are time consuming and labour intensive. Automated measurement of image content using machine vision technology is a promising substitutive method. Semantic segmentation is a problem of great interest in machine vision, aiming to segment images into different regions according to semantic categories, that is, to classify every pixel in images, thus yielding a fine-grained understanding of the image information, as shown in Figure 1. In recent years, convolutional neural networks have been widely used in computer vision tasks, and semantic segmentation methods based on deep learning have shown very good performance in many tasks, such as autonomous driving tasks and medical image processing tasks. Deep neural network models for these methods learn from large labelled datasets, such as Pascal VOC 2012, CityScape [4], and BRATS [5]. These authoritative open-source datasets provide researchers with a consistent measurement platform to advance semantic segmentation techniques. FIGURE 1Open in figure viewerPowerPoint The goal of fish image segmentation systems is to identify whether each pixel belongs to the category of fish or to the background, so that the region where the fish is located can be segmented out of the image in detail However, for the task of underwater fish image segmentation, the existing prevailing semantic segmentation methods are not applicable. Since images of underwater environments differ significantly from those taken in terrestrial environments in terms of object categories, background features, and illumination conditions, those neural networks designed based on image datasets taken in terrestrial environments cannot be directly transferred to underwater environment tasks to obtain excellent results. Generally speaking, the images captured in the terrestrial environment mainly include object categories such as pedestrians, vehicles, buildings, trees, and roads etc., while the object categories in the underwater environment mainly include fish, reefs, and corals etc., which are very different from each other. The colours of artificial objects in terrestrial environmental images are usually designed by humans and have sharp differences, while natural organisms in underwater environmental images usually have protective colours that can blend into the environment. In addition, the background of images of terrestrial environments is usually the sky, while the background of underwater environments are the water body and the ocean floor. There is also a significant difference between the lighting conditions in terrestrial environments and underwater environments, where lighting conditions are usually inadequate compared to terrestrial environments. Neural networks designed for the task of fish segmentation in underwater images are proposed in some works. However, these methods are not advanced. For example, the ResNet-FCN proposed in work [6] is only based on the architecture of FCN using ResNet for feature extraction, the network proposed in work [7] is simply based on DeepLabv3+ with the addition of an unsupervised colour correction method module and two up sampling layers, and the method proposed in work [8] is also based on SegNet merely. Some methods that have been successful in semantic segmentation networks based on terrestrial image datasets have not been used in semantic segmentation networks for underwater image datasets, especially the attention mechanism. The studies [9-13] adopt attention modules to capture long-range dependencies and achieve excellent performance on terrestrial image datasets. Utilising the self-attention mechanism [14, 15], it is capable of making a single feature at any position to perceive the features at all other positions, thus generating a more robust feature representation. The DANet [16] proposed by Fu et al. is an excellent method for capturing rich contextual dependencies leveraging attention modules, the proposed position attention module and channel attention module capture semantic inter-dependencies in the spatial and channel dimensions, respectively. However, these methods require a large amount of computation to model the relationships of each pixel-pair or each channel map-pair in the process of generating attention maps. The complexity in time for generating attention maps by DANet's position attention module is O C ⋅ H ⋅ W 2 , and the complexity in time for generating attention maps by DANet's channel attention module is O C 2 ⋅ H ⋅ W , where C donates the channel amount of the input feature map and H⋅W donates the spatial dimension of the input feature map. As a result, these attention modules bring excellent performance while requiring a lot of additional computational overhead. Some studies attempt to solve the problem of computational cost in generating attention maps. EMANet [17] proposed by Li et al. adopts the EM algorithm to find a compact basis set from all pixels to generate attention maps. CCNet [18] proposed by Huang et al. captures the contextual information around each pixel on a criss-cross path through a novel cross-attention module to generate attention maps. DRANet [19] proposes a compact position attention module and a compact channel attention module based on the attention module of DANet, which associates each pixel or channel with a limited number of aggregation centres, and then captures long-range dependencies over these aggregation centres. However, these methods have a significant drawback: they harm the granularity of the information in the target dimension while reducing the scale of the number involved in the computation that generates the attention maps. Specifically, EMANet formulates the attention mechanism as an expectation-maximising approach that estimates a more compact set of bases in an iterative manner, and then computes the attention graph on top of that. Since the attention module of CCNet only harvests the contextual information of its surrounding pixels on the criss-cross path, CCNet has to use two these attention modules in a stack to make each pixel eventually available for long-distance dependence from all pixels. The compact channel attention module destroys the fine-grained inner-dependencies between channel maps when reducing the number of channels. To provide an advanced solution to the task of fish segmentation based on underwater image datasets, we propose in this study a framework for semantic segmentation called Dual Pooled-aggregate Attention Network (DPANet), an approach that incorporates two novel computing-friendly attention modules. Specifically, we propose a Pooling-aggregated Position Attention Module (PPAM) and a Pooling-aggregated Channel Attention Module (PCAM), appended in parallel on the top of dilated FCN. These two proposed attention modules capture long-range dependencies in an innovative way and are able to fully aggregate contextual information in underwater images to improve the fish segmentation performance of the network with only little computational overhead. The PPAM employs average-pooling operation and max-pooling operation along the channel dimension to reduce the number of channels of features to one. The information at each position is represented by both the average and the maximum of all channels at that position to ensure the completeness of the information. Then, the spatial dependencies between any two positions are captured on the feature descriptors generated by the average-pooling operation and the feature descriptors generated by the max-pooling operation, generating a total of four attention maps that each reflect the correlation of each position from different perspectives. That is, the values on the attention maps are determined by the feature similarities of two positions, and positions with similar features can contribute to each other for improvement. Spatial dependencies of features are subsequently aggregated by summing these four attention maps. Thereby, the PPAM reduces the computational scale of generating attention maps, which reduces the complexity in time to O H ⋅ W 2 without ignoring any pixels and without destroying the fine-grained relations between pixels. Similarly, the PCAM employs the general global average-pooling operation and the global max-pooling operation to aggregate features over the spatial dimension of each channel map, and then captures the channel dependencies between any two channel maps on the feature descriptors generated by the average-pooling operation and the feature descriptors generated by the max-pooling operation, which also generate four attention maps that reflect the channel correlations from different perspectives. These four attention maps are then aggregated into one and each channel map is updated with a weighted sum of all channel maps. Thus, the PCAM reduces the computational scale of generating attention maps, which reduces the complexity in time to O C 2 without destroying the fine-grained inner-dependencies between channel maps. Moreover, our PPAM and PCAM can be plugged into any fully convolutional neural network for learning segmentation in an end-to-end manner. In this study, we present the excellent performance of our proposed method on the task of fish segmentation. With the above design, our proposed DPANet efficiently captures rich contexts in a computing-friendly manner, thus segmenting fish perfectly from the background. Our approach achieves the highest performance on the popular fish segmentation dataset DeepFish [20]. In addition, we validated the superior performance of our method for fish segmentation on SUIM [21], a professional underwater image dataset. With the above design, our proposed DPANet efficiently captures rich contexts in a computing-friendly manner and achieves the highest performance on the popular fish segmentation dataset DeepFish [20] and the underwater image dataset SUIM [21], demonstrating the effectiveness of our approach for underwater image-based fish segmentation tasks. Our main contributions can be summarised as follows. We propose a DPANet to capture the context dependence of spatial and channel dimensions for the fish segmentation task of underwater images. We design a PPAM and a PCAM to bring superior performance improvement with lower computational cost. Our method achieves leading performance on the DeepFish benchmark and SUIM benchmark. The remainder of this paper is organised as follows: in Section 2, we mainly review related works about semantic segmentation, as well as review some methods about attention mechanisms. In Section 3, we will present our proposed DPANet, including the design details of the PPAM and PCAM and the way these two attention modules are integrated. In Section 4, we present the experimental results and perform the necessary analysis to validate the effectiveness of our work. Finally, in Section 5, we conclude our work. 2 RELATED WORK 2.1 Semantic segmentation Semantic segmentation is one of the key computer vision techniques, which is widely applied in real life [22]. Fully convolutional network [23] is the first work to use deep learning for semantic segmentation and is a milestone in the field of semantic segmentation. FCN consists entirely of convolutional layers without any fully connected layers, which enables it to produce segmentation maps of the same size for any size input image. Besides, researchers employed skip connections in the network architecture, enabling the network to combine semantic information from both low-level and high-level layers. The representation capability of the feature map can thus be improved, which in turn improves the performance of the network. Skip connection is very important in the field of semantic segmentation and is used in many well-known networks [24-26]. FCN provides the underlying network architecture for subsequent related work. However, FCN is also deficient in that it fails to enable feature maps to combine global contextual information in an efficient manner, which limits its performance. A lot of the subsequent works have explored making feature maps incorporate richer contextual information. Receptive fields have a direct impact on the performance of the network. Larger receptive fields can combine more contextual information and can be achieved by larger convolution kernels, however larger convolution kernels increase the computational cost and number of parameters as well, which can negatively affect the volume and real-time performance of the network. Instead of using large convolutional kernels directly, Global Convolutional Network [27] uses a combination of 1 × k + k × 1 and k × 1 + 1 × k convolutions. As a result, Global Convolutional Network achieves a receptive field of the same size as the k × k convolution kernel and effectively saves computational cost and number of parameters. In addition, using dilated convolution is an effective way to enlarge the receptive field without increasing the computational cost and number of parameters. The dilated convolution introduces the dilation rate to the original convolution kernel, which determines the distance between the weights in the convolution kernel. By setting the distances between the parameters of the convolution kernels, the dilated convolution can enlarge the receptive field without increasing the computational cost. Since expansion convolution has been proposed, many works [28-30] adopt this approach, the most famous of which is the DeepLab family [31, 32]. Encoder-Decoder architecture is very popular in networks for semantic segmentation. Based on the transposed convolution, the decoder is able to take the feature map as input and generate a map of pixel-wise class probabilities. The use of Encoder-Decoder architecture and skip connections enables the network to fuse contextual semantic information at different granularities. Learning deconvolution network [26] was one of the first semantic segmentation works using Encoder-Decoder architecture, and it achieved the best performance at the time without external data. SegNet [24] is also a network with an Encoder-Decoder architecture, which innovates the up-sampling approach so that it achieves a much smaller number of parameters than other network architectures. In addition, U-Net [33] is well known and widely used in the field of medical image segmentation. Many works [34-37] in this field are based on U-Net. The holistically guided decoder proposed in EfficientFCN [38] leverages multi-level feature maps from the last three blocks of the encoder to achieve high-level feature up-sampling with semantic-rich features. In addition, multi-scale pyramid network is also an architecture that works very well. Multi-scale analysis is a popular approach in the field of computer vision, for example, Feature Pyramid Network [39] has achieved a superior performance in the field of object detection. In the field of semantic segmentation, the Pyramid Scene Parsing Network [40] uses ResNet for feature extraction, then feeds the feature map into the pyramid pooling module, processes the feature map from four different scales, and then fuses the feature maps from the four scales for up-sampling. Moreover, a number of studies [25, 41] have also adopted the multi-scale pyramid architecture. While RNNs are commonly used to solve problems in the field of natural language processing, there are also some works [42-45] that attempt to use RNNs to solve semantic segmentation problems. With the development of GANs, researchers have used them in a variety of visual tasks, including semantic segmentation [46-48]. Other studies innovatively propose semantic segmentation methods from other perspectives. DecoupleSegNets [49] decouples the feature map into a body part and an edge part, and optimises the body features and residual edge features by explicitly modelling the pixels in these two parts, resulting in better consistency within the object and better object edge localisation. SNE-RoadSeg [50] introduces a surface normal estimator to infer surface normal information from dense depth images for freespace segmentation. SFNet [51] integrates a Flow Alignment Module into the feature pyramid structure for learning the Semantic Flow between adjacent level feature maps and efficiently broadcasting high-level features into high-resolution features. 2.2 Fish segmentation It is difficult to make an accurate determination of the area where an object is located when using traditional methods for underwater image processing, and the shape and location of the object are likely to be misjudged [52, 53]. General image segmentation techniques based on deep learning have been widely utilised, but these models are not applicable to the underwater environment. So underwater image segmentation techniques lag behind the traditional segmentation techniques. There are some studies in recent years that explored the underwater image segmentation techniques. Wang et al. [54] proposed an effective fuzzy Cmeans method based on particle swarm optimisation specifically for underwater optical segmentation. By modelling the effect of haze in underwater images, Chen et al. [53] proposed a new method for underwater object segmentation where edge-level transmission features and region-level significant cues were blended together and computed using a level set formulation. Zhang et al. [52], on the other hand, combined the fractal theory with the characteristics of underwater images and proposed a Brownian random field method that can effectively help in the segmentation of underwater images. In addition, Wang et al. [55] designed a deep neural network framework that divides underwater images into regions using a gradient optimisation algorithm. Even less work has been conducted on fish segmentation based on underwater images. The little work that has been done to study this problem has used methods that are not sufficiently advanced. Early work [56] based on double local-thresholded images using the histogram backprojection procedure to overcome the very low luminance contrast between the fish and their underwater background. In [6], a basic FCN architecture is employed to solve the fish segmentation problem, using 152 layers of ResNet for feature extraction to improve the network robustness to changing background and illumination variations. Obviously, this simple network design is outdated and ineffective, and requires very large computational resources. The work [21] proposes an optional residual skip block consisting of three convolutional layers with batch normalisation and ReLU non-linearity after each convolutional layer. Furthermore, they embed this residual skip block module in an end-to-end architecture to extract multi-scale features. Liu et al. [57] introduced an unsupervised colour correction method module for the segmentation of underwater images based on Deeplabv3+ to improve the quality of the images. This is still a basic CNN architecture with limited receptive fields. Even the very recent work [58] on fish segmentation is only using a basic semantic segmentation network like SegNet. 2.3 Attention model Recently, researchers have thoroughly explored attention mechanisms, and attention modules have been widely used in a variety of computer vision tasks, including semantic segmentation [10, 59-61]. Attention modules can usually be divided into channel attention modules and spatial attention modules. The channel attention module can enhance the representation of the network by modelling the relationships between channels of the feature map [62], and the spatial attention module can enhance the representation of the network by modelling the relationship of each pixel to a large or even global context [18]. GANet [63] proposes a Spatial Gated Attention module that generates corresponding attention masks for the feature maps extracted from different levels of the encoder, and adaptively reweights the features at different levels through a multi-scale feature interaction mechanism. FocusNet [64] employs two parallel branches of the encoder-decoder structure, one of which is devoted to attention. The decoder output of the attention branch skip-connects to the encoder of the other branch used to generate the segmentation mask to promote better gradient flow. Multi-Modality Fusion Network [65] employs four channel-independent encoding paths to extract the features from four modalities independently, and an attention mechanism-based fusion block is designed to fuse the four features. GSANet [66] introduces a selective attention module to fuse the multi-scale contextual information extracted by the feature extraction network and gives appropriate relative importance to different scales at different spatial locations. There are also some works exploring segmentation methods for RGB-D images. For example, the work [67] proposes a Bilateral Attention Network that introduces a novel Bilateral Attention Module. Its foreground-first attention mechanism focusses on the foreground region in a progressive manner, while its background-first attention mechanism recovers potentially useful salient information in the background region to capture more meaningful foreground and background cues. Work [68] proposes a parallel reverse attention network addressing the task of polyp segmentation based on colonoscopic images, which fully exploits the boundary cues through the reverse attention module to establish the relationship between object areas and boundary cues. Work [15] proposes a method that generates attention maps by calculating the correlation matrix between each spatial point in the feature map, which is based on self-attention mechanisms, and many subsequent studies have improved it. DANet [16] proposes a position attention module and a channel attention module to collect contextual information in the spatial dimension and the channel dimension using this self-attention mechanism. Concretely, the position attention module captures the spatial dependencies between any two pixels of the feature maps while the channel attention module captures the channel dependencies between any two channel maps of the feature maps. This method is effective in capturing long-distance dependencies, but generating attention maps brings a large computational burden. Many studies try to reduce the computational cost of the self-attention module. A2-Net [69] designs the double attention module in two steps, first using second-order attention pooling to aggregate features from the entire space into a compact set, and then adaptively selecting and distributing features to each location through another attention. EMANet [17] employs the EM algorithm to obtain a compact basis set from all pixels for generating attention maps. CCNet [18] proposes a cross-cross attention module that collects only long-range contextual information in the horizontal and vertical directions and stacks two cross-cross attention modules in series. DRANet [19] proposes a compact position attention module and a compact channel attention module based on DANet. The aggregation centres are obtained using spatial pyramid pooling in the compact position attention module for generating spatial attention maps, while the aggregation centres are obtained using dimension-reduction convolution operations in the compact channel attention module for generating channel attention maps. However, these methods harm the information granularity of the target dimension while reducing the computational cost, which is disadvantageous for capturing long-range dependencies. In this work, we design a PPAM and PCAMe to aggregate features along the channel dimension by pooling operations when capturing spatial long-range dependencies, and to aggregate features in the spatial dimension by pooling operations when capturing channel long-range dependencies, in order to achieve a reduction in computational cost without damaging the granularity of information in the target dimension. 3 METHOD In this section, we give the details of the proposed DPANet for semantic segmentation. First, we will present a general framework of our network. Then, we present the two attention modules, which can capture long-range contextual information in spatial and channel dimensions efficiently. Finally, we introduce how to aggregate these attention modules together. 3.1 Overview As shown in Figure 2, we adopt a dilated residual network to extract features, and design two types of attention modules to draw global context information over features generated by the residual network, thus acquiring robust feature representations. Following these, we use a simple decoder for pixel-wise prediction. Specifically, following previous studies, we abandon the down-sampling operations and employ dilated convolutions in the last two ResNet blocks, which can include more details without any extra parameters. The network thus enlarges the size of the final feature map to 1 8 of the input image. Then, to obtain better feature representations, we feed the features generated by the dilated residual network into two parallel attention modules, which can, respectively, capture the global contextual dependencies in spatial and channel dimensions by a more efficient way.