# Multithreshold Microbial Image Segmentation Using Improved Deep Reinforcement Learning

Authors: Minghui Zhou
Journal: Mathematical Problems in Engineering
 (2022)

---

## Abstract

Image segmentation technology can effectively extract the foreground target in the image. However, the microbial image is easily disturbed by noise, its greyscale has the characteristics of nonuniform distribution, and several microorganisms with diverse forms exist in the same image, resulting in insufficient accuracy of microbial image segmentation. Therefore, a multithreshold microbial image segmentation algorithm using improved deep reinforcement learning is proposed. The wavelet transform method is used to remove the noise of the microbial image, the threshold number of the microbial image after denoising is determined by calculating the number of peaks of the grey histogram, and the foreground target of the microbial image is enhanced by the mean iterative threshold segmentation method, the preliminary segmentation of the microbial image is realized, the multithreshold microbial image segmentation model based on ResNet-Unet is constructed, and the cavity convolution and dualQ network mechanism are introduced to improve the segmentation model. The preliminary segmented microbial image is input into the improved segmentation model to realize the segmentation of the multithreshold microbial image. The results show that the proposed algorithm can effectively remove the noise of microbial images. With the increase in the number of thresholds, the peak signal-to-noise ratio, structural similarity, and feature similarity show an upward trend, and the loss rate of the model is less than 0.05%. The minimum running time of the algorithm is 3.804 s. It can effectively and quickly segment multithreshold microbial images and has important application value in the field of microbial recognition.


---

## Body

## 1. Introduction

In people’s daily life, microorganisms are everywhere, in the recognition of microorganism images, how to accurately obtain the features of microorganism images is the core problem that needs to be addressed [1]. Extracting image features and obtaining microbial targets by image segmentation is a commonly used effective method. However, the microbial image segmentation is often susceptible to various factors, resulting in poor microbial image target extraction [2, 3]. First, microorganisms are diverse, most of them have nonrigid features, their morphology shows nonregularity, their shapes are very different and their sizes vary. Second, in the microbial images, the difference degree between the image target and background greyscale is small. For multitarget microbial images, the spatial distribution of microorganisms is crossed and overlapping, and conventional segmentation techniques may produce a pseudocontour phenomenon [4]. In addition, noise is another important factor affecting the imaging effect of microbial images. Excessive noise will cause the lack of detailed information in microbial images, especially when there is a large number of bright signals, the microbial target will be submerged in the background, resulting in image quality degradation [5]. These factors can adversely affect image segmentation and lead to the degradation of image segmentation quality and efficiency. Therefore, it is important to study a new multi-threshold microbial image segmentation method.To address the important research topic of multi-threshold microbial image segmentation, Li et al. [6] implemented the segmentation process of greyscale image thresholding by constructing a quantum circuit, which can complete the acquisition of the histogram of pixel distribution and determine the image threshold by obtaining the maximum interclass variance of each greyscale level, to achieve image segmentation. However, the single threshold value determined by this method is not suitable for the segmentation of a wide variety of microbial images. Li et al. [7] proposed to improve the traditional fuzzy C-mean algorithm for poor image segmentation by using a parallel Lévy-grey-wolf optimization algorithm to improve the image segmentation by obtaining the best initial centre. However, this method still results in a serious loss of image detail information. Cai et al. [8] proposed the use of an improved particle swarm optimization algorithm to achieve segmentation processing of thresholded images, and the method obtained a small number of image thresholds, which has limitations in applying it to the segmentation of a variety of microbial images. Jia et al. [9] proposed to improve the multiverse optimization algorithm using the Leyland flight mechanism to improve the accuracy of image segmentation by accurately determining the number of image thresholds. However, this method fails to achieve accurate learning of image information due to premature convergence. Sun et al. [10] proposed an image small target recognition method based on a deep reinforcement learning method using a multilayer convolutional neural network to achieve image target segmentation. However, the convolutional neural network constructed by this method suffers from gradient loss, which reduces the image segmentation performance and thus has a serious impact on the accurate recognition of image targets. To improve the quality and efficiency of multithreshold microbial image segmentation, a new multithreshold microbial image segmentation algorithm using improved deep reinforcement learning is proposed, and the contributions of this paper are as follows:(1)
The wavelet transform method is used to remove the microbial image noise, which improves the image quality and reduces the interference of noise to the subsequent image segmentation(2)
The foreground target of microbial images is enhanced to achieve the initial segmentation of microbial images and lay the foundation for the subsequent multithreshold microbial image segmentation(3)
To improve and introduce a reinforcement learning mechanism to build the relevant segmentation model to achieve the ultimate aim of image segmentation quality and efficiency


## 2. Methodology

### 2.1. Multithreshold Microbial Image Segmentation

The process of multithreshold microbial image segmentation algorithm using improved deep reinforcement learning is shown in Figure1.Figure 1
A multithreshold microbial image segmentation process.Analysis of Figure1 shows that the microbial image is preprocessed to remove the noise contained in the image. The calculation of the number of image thresholds is completed by analyzing the number of wave peaks in the grey histogram to determine the number of thresholds. The foreground target of the microbial image is enhanced to achieve the initial segmentation of the background and foreground. The image segmentation model based on ResNet-Unet is constructed, and the improvement of the segmentation model is completed by introducing the null convolution and double Q network mechanisms, and the initial segmentation results are input to the improved model to obtain the segmentation results of the relevant multithreshold microbial images.


### 2.2. Microbial Image Denoising Based on Wavelet Transform

In this paper, the wavelet transform method is used to realize the noise reduction of microbial image. Its principle is as follows: setfi,j,i,j=1,2,⋯N as the original microbial image, in which the value of N is among the integer power of 2, i,j is the pixel point of the image, and the grey value of the pixel is expressed as fi,j. When a large amount of noise is covered in the original multithreshold microbial image, the noisy microbial image can be described by the following equation [11]:(1)vi,j=fi,j+bi,j,where bi,j is the noise in the microbial image, which meets the Gaussian distribution, the mean value is 0 is, σ2 is the variance which is independent just as fi,j. It is processed using the wavelet method, which is described by the following equation:(2)Zvi,j=Zfi,j+Zbi,j,where for noisy microbial images, Zvi,j is the wavelet coefficient. For the original microbial image, Zfi,j is wavelet coefficient. The noise in the microbial image, Zbi,j is wavelet coefficient.The multithreshold microbial image noise reduction process based on wavelet transform is as follows:Step 1: wavelet transform is used to process noisy microbial imagesvi,j. After s layers of orthogonal redundancy decomposition, the wavelet coefficient can be determined, expressed as Wvi,js,j, and j=1,2,⋯s.Step 2: the estimation of noise varianceσ2 is realized from different decomposition levels and directions [12]. The following calculation equation is as follows:(3)σ^bi,js,j=medianWvi,js,j0.6634,Step 3: the parameters of the noise threshold are determined. For the original microbial image,σfi,j is the variance of Zfi,j. A satisfies a Gaussian distribution whose variance Equation can be described by the following equation. Zvi,j satisfies Gaussian distribution, the calculation equation is as follows:(4)σ^vi,j2s,j=∑i,j=1NjZvi,j2s,jN2j.According toσvi,j2=σfi,j2+σbi,j2, it can be determined that:(5)σ^fi,js,j=maxσ^vi,j2s,j−σ^bi,j2s,j1/2.Step 4: determine the threshold coefficient, expressed asβ. Through β, it can update the thresholds of different high-frequency sub-bands in different decomposition layers. The calculation Equation of β is expressed as follows:(6)β=logLk/j1/24.08,where for wavelet coefficient afterj layers of decomposition, Lk is the length of layer k.Step 5: determine the threshold calculation Equation, described as follows:(7)Ts,j=βσ^bi,j2s,jσ^fi,js,j−1.The wavelet coefficient of the denoised image is determined by the wavelet soft threshold processing method, and its equation is described as follows:(8)Z˜vi,js,j=WSTZvi,js,j.whereWST⋅ is the soft threshold processing function. The method to obtain the original microbial image fi,j is to adopt the inverse wavelet transform based on determining Z˜vi,js,j.Based on the above-given process, the noise reduction processing of microbial images can be obtained and realized.


### 2.3. Determination of Multithreshold for Microbial Images

In the multithreshold segmentation of the microbial image, the selection of threshold number has a direct impact on the segmentation effect of the microbial image [13]. The accurate threshold number can effectively avoid the misclassification of false peaks in the histogram of the microbial image and improve the segmentation effect of the microbial image. For the microbial image, obtain its grey histogram, expressed as h. Suppose U is the number of grey levels contained, i is the grayscale, and i∈0,U−1. The mean value of frequency is expressed as have whose solution can be obtained by have=meanh. In this paper, the threshold amount of microbial image is determined by the following steps:Step 1: the denoise-colour or microbial image is obtained, perform the grey operation on it, and obtain the grey histogram.Step 2: according to the obtained grey histogram, the frequency comparison relative to the grey value is completed in corresponding order. For any grey value, when its frequency is greater than that of the adjacent grey value, the determination of the wave crest can be realized, that is, the frequency relative to the grey value.Step 3: the histogram peak can be obtained through step 2, but there are some pseudopeaks in it. To ensure that the false peak misclassification problem does not occur, setD as the peak spacing, and consider that there is only one peak in D area.Step 4: most of the pixels in the microbial image are covered by each target. To accurately determine the wave crest [14], it is necessary to further screen the wave crest, and the screening rule is set that its grey value should be higher than have.Step 5: setAi as the wave peak, which can be expressed by the frequency of the grey value of i, and An refers to the number of wave peaks. The solution Equation is described as follows:(9)An=countAik.whereAik−D<Aik<Aik+D and Aik>have. k refers to the frequency, and k∈1,2,…,n. Aik refers to the wave peak with the frequency as k.Step 6: thus, the threshold number of microbial images can be obtained, expressed asm. The following solution equation is described as follows:(10)m=An−1.


### 2.4. Foreground Target Enhancement of Microbial Image Using Mean Iterative Threshold Segmentation

The mean iterative threshold segmentation method is widely used in multithreshold image segmentation because of its low computational difficulty [15, 16]. Therefore, this paper uses this algorithm to enhance the foreground target of multithreshold microbial images. Inputting the enhanced microbial image into the image segmentation model of improved deep reinforcement learning can effectively eliminate the characteristics of worthless background and avoid the influence of image background on multithreshold microbial image target segmentation. The algorithm can reduce the complexity of image segmentation and reduce the parameters of the multithreshold microbial image segmentation model and the number of training samples.The principle of this method is to divide the microbial image into two categories, namely, foreground and background, according to the different grey attributes of microbial image foreground and background. The grey levels of each category show different characteristics. The classification of image pixels is realized by threshold T. This method can distinguish the foreground and background of microbial images. The specific steps are as follows:Step 1: set the original threshold value T. The average grey value of the microbial image is taken as its initial value.Step 2: the foreground and background of the microbial image are distinguished by threshold valueT, which are expressed as E1, E2. When the grey value of the pixel is higher than the threshold value T, it is classified into E1. Otherwise, it is classified as E2.Step 3: get the solution of the mean grey value of all the pixels ofE1, E2, which are expressed as μ1 and μ2.Step 4: according to the threshold numberm of microbial image m determined in Section 2.2, the threshold value T is adjusted by the following equation:(11)T=2m−1μ1+μ2.Step 5: repeat steps 2, 3, and 4, and the algorithm ends when the average change of the last two times is lower than the set value.Step 6: thus, the new threshold valueT′ can be obtained. Through T′, the distinction between the foreground and background of the microbial image is realized.


### 2.5. Multithreshold Microbial Image Segmentation Model Using ResNet-UNet

Resnet-34 is a residual convolution neural network. Its structure has 34 layers and includes three parts: independent convolution module, residual convolution module, mean pooling, and full connection module [17]. The independent convolution module consists of a size of 7 × 64 channel convolution kernel of 7 and 3 × 3. The former can realize the convolution operation of the input image, and the latter is used to perform the maximum pool processing of the image. The residual convolution part contains four residual convolution blocks with differences. For any residual convolution block, it is realized by twice convolution, batch regularization, and ReLu. Taking the residual convolution block as a processing unit, complete the residual processing of the residual convolution module with 3, 4, 6, and 3 different times, respectively, and then “smooth” the feature vector. Through mean pooling and full connection, the microbial image is segmented and the segmented image is obtained, to reduce the impact of network gradient disappearance on network performance [18].The UNet network is composed of the front and rear parts, in which the former is used to obtain microbial image features, and the latter can realize upsampling of image information [19]. According to the dimension difference of the feature map, it can be divided into 5 layers from top to bottom, and 3 layers × 3. The convolution and ReLu adjustable linear unit are used as the structure of each layer. For the adjacent two layers, the feature extraction structure reduces the size of the feature image and improves the feature dimension through feature cutting and downsampling. The upsampling structure enlarges the feature size of the microbial image through feature difference and upsampling and reduces its dimension. For the same layer, the front and back structure of the net network is completed by feature replication and splicing. Then, after the last processing of the upper sampling structure, execute 1 × 1. Convolution and ReLu operation can realize the segmentation of microbial image and obtain the target segmentation image. The ResNet-UNet designed in this paper takes ResNet-34 residual convolution neural network as the feature image extraction network of microbial image. Based on the basic principle of the UNet network, the resolution restoration of microbial images is realized through upsampling, feature replication, and splicing.The specific steps are as follows: take the microbial image processed in Section2.3 as the input of the ResNet-UNet network, and its size is 512 × 512. The characteristic map of the microbial image is obtained by ResNet -34 network, but its network structure needs to be adjusted, its independent convolution and residual convolution modules are retained, and the mean pooling and full connection modules are deleted. To ensure the tight connection between ResNet -34 network and the U-Net network, the number of convolution channels of the fourth residual convolution module of the ResNet-UNet network needs to be designed to have the same number of convolution channels as the convolution at the end layer of ResNet -34 network. According to the principle of the UNet network, the difference processing of the image is realized through upsampling to achieve the purpose of image resolution restoration. When there are features with the same size in upsampling and extracting the features of the image, they need to be spliced and fused, which can be realized through feature replication and splicing processing. After that, three tests were carried out twice 3 × 3 convolutions, batch regularization, and ReLu processing. The sigmoid function is applied to the last layer convolution of the upsampling structure to ensure that the ResNet-UNet network has a strong learning ability and improves the classification accuracy. Then the nonlocal operation is used to realize the integration of microbial image information. Through 1 × 1 convolution and upsampling processing, the initial segmentation of the multithreshold microbial image is realized.


### 2.6. Improvement of Multithreshold Microbial Image Segmentation Model

#### 2.6.1. Void Convolution

To ensure that the ResNet-UNet network has a strong feature extraction ability and make the obtained microbial image feature map contain complete image details, this paper improves the ResNet-UNet network, introduces void convolution, and replaces the original convolution based on the original mean pooling and the deletion of the full connection module. While ensuring that the network parameters do not increase, the receptive field of the image is improved so that the details of the microbial image are not lost. For void convolution, it can be regarded as porous convolution, i.e., inserting pixels with a 0-pixel value between the pixels of the original convolution core, to increase the size of the convolution core and improve the image receptive field [20, 21].


#### 2.6.2. DualQ Network Mechanism

The improved ResNet-UNet network realizes the panoramic perception of multithreshold microbial image features, obtains the microbial image feature map, takes it as the training data set of reinforcement learning, fully considers the dualQ network mechanism, and introduces the target Net as a network copy of the depth Q value. The target value function is determined through the ResNet-UNet network and provided to the depth Q-value network [22]. While continuously adjusting it, the target Net, give play to the decision-making performance of the deep Q-value network and complete the feature extraction of the microbial map and the collaborative operation of parameters. Through the positive feedback mechanism, the error of the collaborative operation process is adjusted, to establish the best segmentation mechanism for the multithreshold microbial image. The objective function gradient function equation is as follows:(12)∇θJπθ=∫AS∫ASρπs∇θπθs,aQπs,adads=Es⋅ρπ,a⋅πθ∇θlogπθa|sQπs,a,where state information and its collection are represented as s and S, the set of action a is A; the strategy is π, the network parameter is θ. Qπs,a is used to describe the expected reward of choosing an action a when the status is s, the probability distribution function is expressed as ρ. According to its value, the determination of the optimal action in the current state can be realized, E is the reward for the best action in the current status.According to the objective function gradient function of equation (12), the deterministic strategy expression is obtained, use function μ and according to the action information, actions can be obtained. μ is regarded as the optimal action strategy at=μst|θμ. The quantitative analysis of multi-threshold microbial image segmentation can be realized by the following equation:(13)Jμθ=∫ASρμs∫ASrs,μθsds=Es⋅ρμrs,μθs.Under competitive conditions, equation (13) has poor stationarity. Therefore, this paper performs the first-order derivation. At this time, the deterministic strategy gradient can be described by the following equation:(14)∇θJμθ=∫ASρμs∫ASQμs,a∇θμθs|a=μθds=Es⋅ρπQμs,a∇θμθs|a=μθ.The compatibility of equation (14) is outstanding. It can complete the segmentation of various multithreshold microbial images in continuous adaptive learning and obtain the best-segmented image. Due to the variety and diversity of microorganisms, there are many characteristics of microorganisms [23]. Therefore, in this paper, the strategy network μ is regarded as the actor. The function s,a is fitted through the value network and regarded as critical, to complete the recognition and segmentation of different multithreshold microbial images. The expression of the objective function is described by the following equation:(15)Jθμ=Eθμr1+Eθμγr2+Eθμγ2r3+…Eθμγ2rn.


### 2.7. Data Sets and Evaluation Index

Experimental data sets were used Earth Microbiome Project, Human gut microbiomes, and Human Microbiome Project. Earth Microbiome Project: EMP, founded in 2010, is a large-scale crowdsourcing work aimed at analyzing the global microbial community. The general premise is to study from the perspective of the microbial community itself. The researchers used amplicon sequencing, metagenomics, and metabolomics to analyze 200000 samples from these communities, including 100000 microbial images, which is very suitable for multithreshold microbial image segmentation. Human gut microbiomes: human gut microbiomes refer to all microorganisms inhabiting the human gastrointestinal tract. The different roles of intestinal microbiota in human health and disease have been rerecognized. Metagenomic research has changed our understanding of the taxonomy and functional diversity of human microbiota, such as promoting the maturation of the immune system and providing a direct barrier against pathogen colonization. The data set includes a series of studies on human intestinal microbiota, including images, papers, and patents. In this paper, two thousand microbial images are used for experimental tests to verify the application effect of different algorithms. Human Microbiome Project: the overall mission of the HMP is to generate resources to facilitate the characterization of the human microbiota to further our understanding of how the microbiome impacts human health and disease. The initial phase of the project, HMP1, established in 2008, characterized the microbial communities of 300 healthy individuals, across several different sites on the human body: nasal passages, oral cavity, skin, gastrointestinal tract, and urogenital tract. 16S rRNA sequencing was performed to characterize the complexity of microbial communities at each body site and to begin to ask to investigate whether there is a core healthy microbiome. Metagenomic whole genome shotgun sequencing provided insights into the functions and pathways present in the human microbiome.A total of five thousand microbial images were selected in the earth microbiome projects data set, human gut microbiomes data set, and Human microbiome project data set, the discount coefficient is 0.95, and the learning rate is 10−3. The value of strategy reward is [−1, 1], and the reward of action is −1. The parameters of the multithreshold microbial image segmentation model are adaptively selected to achieve a better effect of model training. Among them, four thousand images are used to train the segmentation model, and the other images are used for image testing. Different algorithms are applied to the segmentation of microbial images in the data set to analyze the image segmentation effect of this algorithm.(1)
Image denoising ability: due to the interference of noise factors when collecting microbial images, the visual effect of the obtained microbial images is poor. Therefore, the clearer the image after denoising, the stronger the denoising ability.(2)
Peak signal-to-noise ratio (PSNR): it is an index for evaluating image quality. The calculation Equation of this index is as follows:(16)PSNR=10×log102n−12MSE,whereMSE refers to the mean square error between an original image and a denoised image, and n refers to the number of bits of the sample value.(3)
Structural similarity (SSIM) is an important index to judge the contrast of segmented multithreshold microbial images and whether the structure is suitable and complete.(17)SSIMx,y=lx,yαcx,yβsx,yγ,wherex and y refer to the two input microbial images, lx,y refers to the brightness comparison function, cx,y refers to contrast comparison function, sx,y refers to structure comparison function, and α,β,γ are different constants.(4)
After using the algorithm to segment the microbial image, the similarity of the segmented image features compared with the original microbial image can be reflected by feature similarity (FSIM). The value range of FSIM is [0, 1], and the closer its value is to 1, the closer their characteristics are. The calculation Equation of this index is as follows:(18)FSIM=∑x∈ΩSLxPCmx∑x∈ΩPCmx,wherePCmx refers to phase consistency judgment function, and SLx refers to coupling functions of different terms.(5)
Loss rate: it refers to the rate of loss in the process of segmentation of microbial image by an algorithm. The calculation equation of this index is as follows:(19)A=1−t−i×100%,wheret refers to the planned quantity of microbial image experiment, and i refers to the total amount of microbial images.


## 2.1. Multithreshold Microbial Image Segmentation

The process of multithreshold microbial image segmentation algorithm using improved deep reinforcement learning is shown in Figure1.Figure 1
A multithreshold microbial image segmentation process.Analysis of Figure1 shows that the microbial image is preprocessed to remove the noise contained in the image. The calculation of the number of image thresholds is completed by analyzing the number of wave peaks in the grey histogram to determine the number of thresholds. The foreground target of the microbial image is enhanced to achieve the initial segmentation of the background and foreground. The image segmentation model based on ResNet-Unet is constructed, and the improvement of the segmentation model is completed by introducing the null convolution and double Q network mechanisms, and the initial segmentation results are input to the improved model to obtain the segmentation results of the relevant multithreshold microbial images.


## 2.2. Microbial Image Denoising Based on Wavelet Transform

In this paper, the wavelet transform method is used to realize the noise reduction of microbial image. Its principle is as follows: setfi,j,i,j=1,2,⋯N as the original microbial image, in which the value of N is among the integer power of 2, i,j is the pixel point of the image, and the grey value of the pixel is expressed as fi,j. When a large amount of noise is covered in the original multithreshold microbial image, the noisy microbial image can be described by the following equation [11]:(1)vi,j=fi,j+bi,j,where bi,j is the noise in the microbial image, which meets the Gaussian distribution, the mean value is 0 is, σ2 is the variance which is independent just as fi,j. It is processed using the wavelet method, which is described by the following equation:(2)Zvi,j=Zfi,j+Zbi,j,where for noisy microbial images, Zvi,j is the wavelet coefficient. For the original microbial image, Zfi,j is wavelet coefficient. The noise in the microbial image, Zbi,j is wavelet coefficient.The multithreshold microbial image noise reduction process based on wavelet transform is as follows:Step 1: wavelet transform is used to process noisy microbial imagesvi,j. After s layers of orthogonal redundancy decomposition, the wavelet coefficient can be determined, expressed as Wvi,js,j, and j=1,2,⋯s.Step 2: the estimation of noise varianceσ2 is realized from different decomposition levels and directions [12]. The following calculation equation is as follows:(3)σ^bi,js,j=medianWvi,js,j0.6634,Step 3: the parameters of the noise threshold are determined. For the original microbial image,σfi,j is the variance of Zfi,j. A satisfies a Gaussian distribution whose variance Equation can be described by the following equation. Zvi,j satisfies Gaussian distribution, the calculation equation is as follows:(4)σ^vi,j2s,j=∑i,j=1NjZvi,j2s,jN2j.According toσvi,j2=σfi,j2+σbi,j2, it can be determined that:(5)σ^fi,js,j=maxσ^vi,j2s,j−σ^bi,j2s,j1/2.Step 4: determine the threshold coefficient, expressed asβ. Through β, it can update the thresholds of different high-frequency sub-bands in different decomposition layers. The calculation Equation of β is expressed as follows:(6)β=logLk/j1/24.08,where for wavelet coefficient afterj layers of decomposition, Lk is the length of layer k.Step 5: determine the threshold calculation Equation, described as follows:(7)Ts,j=βσ^bi,j2s,jσ^fi,js,j−1.The wavelet coefficient of the denoised image is determined by the wavelet soft threshold processing method, and its equation is described as follows:(8)Z˜vi,js,j=WSTZvi,js,j.whereWST⋅ is the soft threshold processing function. The method to obtain the original microbial image fi,j is to adopt the inverse wavelet transform based on determining Z˜vi,js,j.Based on the above-given process, the noise reduction processing of microbial images can be obtained and realized.


## 2.3. Determination of Multithreshold for Microbial Images

In the multithreshold segmentation of the microbial image, the selection of threshold number has a direct impact on the segmentation effect of the microbial image [13]. The accurate threshold number can effectively avoid the misclassification of false peaks in the histogram of the microbial image and improve the segmentation effect of the microbial image. For the microbial image, obtain its grey histogram, expressed as h. Suppose U is the number of grey levels contained, i is the grayscale, and i∈0,U−1. The mean value of frequency is expressed as have whose solution can be obtained by have=meanh. In this paper, the threshold amount of microbial image is determined by the following steps:Step 1: the denoise-colour or microbial image is obtained, perform the grey operation on it, and obtain the grey histogram.Step 2: according to the obtained grey histogram, the frequency comparison relative to the grey value is completed in corresponding order. For any grey value, when its frequency is greater than that of the adjacent grey value, the determination of the wave crest can be realized, that is, the frequency relative to the grey value.Step 3: the histogram peak can be obtained through step 2, but there are some pseudopeaks in it. To ensure that the false peak misclassification problem does not occur, setD as the peak spacing, and consider that there is only one peak in D area.Step 4: most of the pixels in the microbial image are covered by each target. To accurately determine the wave crest [14], it is necessary to further screen the wave crest, and the screening rule is set that its grey value should be higher than have.Step 5: setAi as the wave peak, which can be expressed by the frequency of the grey value of i, and An refers to the number of wave peaks. The solution Equation is described as follows:(9)An=countAik.whereAik−D<Aik<Aik+D and Aik>have. k refers to the frequency, and k∈1,2,…,n. Aik refers to the wave peak with the frequency as k.Step 6: thus, the threshold number of microbial images can be obtained, expressed asm. The following solution equation is described as follows:(10)m=An−1.


## 2.4. Foreground Target Enhancement of Microbial Image Using Mean Iterative Threshold Segmentation

The mean iterative threshold segmentation method is widely used in multithreshold image segmentation because of its low computational difficulty [15, 16]. Therefore, this paper uses this algorithm to enhance the foreground target of multithreshold microbial images. Inputting the enhanced microbial image into the image segmentation model of improved deep reinforcement learning can effectively eliminate the characteristics of worthless background and avoid the influence of image background on multithreshold microbial image target segmentation. The algorithm can reduce the complexity of image segmentation and reduce the parameters of the multithreshold microbial image segmentation model and the number of training samples.The principle of this method is to divide the microbial image into two categories, namely, foreground and background, according to the different grey attributes of microbial image foreground and background. The grey levels of each category show different characteristics. The classification of image pixels is realized by threshold T. This method can distinguish the foreground and background of microbial images. The specific steps are as follows:Step 1: set the original threshold value T. The average grey value of the microbial image is taken as its initial value.Step 2: the foreground and background of the microbial image are distinguished by threshold valueT, which are expressed as E1, E2. When the grey value of the pixel is higher than the threshold value T, it is classified into E1. Otherwise, it is classified as E2.Step 3: get the solution of the mean grey value of all the pixels ofE1, E2, which are expressed as μ1 and μ2.Step 4: according to the threshold numberm of microbial image m determined in Section 2.2, the threshold value T is adjusted by the following equation:(11)T=2m−1μ1+μ2.Step 5: repeat steps 2, 3, and 4, and the algorithm ends when the average change of the last two times is lower than the set value.Step 6: thus, the new threshold valueT′ can be obtained. Through T′, the distinction between the foreground and background of the microbial image is realized.


## 2.5. Multithreshold Microbial Image Segmentation Model Using ResNet-UNet

Resnet-34 is a residual convolution neural network. Its structure has 34 layers and includes three parts: independent convolution module, residual convolution module, mean pooling, and full connection module [17]. The independent convolution module consists of a size of 7 × 64 channel convolution kernel of 7 and 3 × 3. The former can realize the convolution operation of the input image, and the latter is used to perform the maximum pool processing of the image. The residual convolution part contains four residual convolution blocks with differences. For any residual convolution block, it is realized by twice convolution, batch regularization, and ReLu. Taking the residual convolution block as a processing unit, complete the residual processing of the residual convolution module with 3, 4, 6, and 3 different times, respectively, and then “smooth” the feature vector. Through mean pooling and full connection, the microbial image is segmented and the segmented image is obtained, to reduce the impact of network gradient disappearance on network performance [18].The UNet network is composed of the front and rear parts, in which the former is used to obtain microbial image features, and the latter can realize upsampling of image information [19]. According to the dimension difference of the feature map, it can be divided into 5 layers from top to bottom, and 3 layers × 3. The convolution and ReLu adjustable linear unit are used as the structure of each layer. For the adjacent two layers, the feature extraction structure reduces the size of the feature image and improves the feature dimension through feature cutting and downsampling. The upsampling structure enlarges the feature size of the microbial image through feature difference and upsampling and reduces its dimension. For the same layer, the front and back structure of the net network is completed by feature replication and splicing. Then, after the last processing of the upper sampling structure, execute 1 × 1. Convolution and ReLu operation can realize the segmentation of microbial image and obtain the target segmentation image. The ResNet-UNet designed in this paper takes ResNet-34 residual convolution neural network as the feature image extraction network of microbial image. Based on the basic principle of the UNet network, the resolution restoration of microbial images is realized through upsampling, feature replication, and splicing.The specific steps are as follows: take the microbial image processed in Section2.3 as the input of the ResNet-UNet network, and its size is 512 × 512. The characteristic map of the microbial image is obtained by ResNet -34 network, but its network structure needs to be adjusted, its independent convolution and residual convolution modules are retained, and the mean pooling and full connection modules are deleted. To ensure the tight connection between ResNet -34 network and the U-Net network, the number of convolution channels of the fourth residual convolution module of the ResNet-UNet network needs to be designed to have the same number of convolution channels as the convolution at the end layer of ResNet -34 network. According to the principle of the UNet network, the difference processing of the image is realized through upsampling to achieve the purpose of image resolution restoration. When there are features with the same size in upsampling and extracting the features of the image, they need to be spliced and fused, which can be realized through feature replication and splicing processing. After that, three tests were carried out twice 3 × 3 convolutions, batch regularization, and ReLu processing. The sigmoid function is applied to the last layer convolution of the upsampling structure to ensure that the ResNet-UNet network has a strong learning ability and improves the classification accuracy. Then the nonlocal operation is used to realize the integration of microbial image information. Through 1 × 1 convolution and upsampling processing, the initial segmentation of the multithreshold microbial image is realized.


## 2.6. Improvement of Multithreshold Microbial Image Segmentation Model

### 2.6.1. Void Convolution

To ensure that the ResNet-UNet network has a strong feature extraction ability and make the obtained microbial image feature map contain complete image details, this paper improves the ResNet-UNet network, introduces void convolution, and replaces the original convolution based on the original mean pooling and the deletion of the full connection module. While ensuring that the network parameters do not increase, the receptive field of the image is improved so that the details of the microbial image are not lost. For void convolution, it can be regarded as porous convolution, i.e., inserting pixels with a 0-pixel value between the pixels of the original convolution core, to increase the size of the convolution core and improve the image receptive field [20, 21].


### 2.6.2. DualQ Network Mechanism

The improved ResNet-UNet network realizes the panoramic perception of multithreshold microbial image features, obtains the microbial image feature map, takes it as the training data set of reinforcement learning, fully considers the dualQ network mechanism, and introduces the target Net as a network copy of the depth Q value. The target value function is determined through the ResNet-UNet network and provided to the depth Q-value network [22]. While continuously adjusting it, the target Net, give play to the decision-making performance of the deep Q-value network and complete the feature extraction of the microbial map and the collaborative operation of parameters. Through the positive feedback mechanism, the error of the collaborative operation process is adjusted, to establish the best segmentation mechanism for the multithreshold microbial image. The objective function gradient function equation is as follows:(12)∇θJπθ=∫AS∫ASρπs∇θπθs,aQπs,adads=Es⋅ρπ,a⋅πθ∇θlogπθa|sQπs,a,where state information and its collection are represented as s and S, the set of action a is A; the strategy is π, the network parameter is θ. Qπs,a is used to describe the expected reward of choosing an action a when the status is s, the probability distribution function is expressed as ρ. According to its value, the determination of the optimal action in the current state can be realized, E is the reward for the best action in the current status.According to the objective function gradient function of equation (12), the deterministic strategy expression is obtained, use function μ and according to the action information, actions can be obtained. μ is regarded as the optimal action strategy at=μst|θμ. The quantitative analysis of multi-threshold microbial image segmentation can be realized by the following equation:(13)Jμθ=∫ASρμs∫ASrs,μθsds=Es⋅ρμrs,μθs.Under competitive conditions, equation (13) has poor stationarity. Therefore, this paper performs the first-order derivation. At this time, the deterministic strategy gradient can be described by the following equation:(14)∇θJμθ=∫ASρμs∫ASQμs,a∇θμθs|a=μθds=Es⋅ρπQμs,a∇θμθs|a=μθ.The compatibility of equation (14) is outstanding. It can complete the segmentation of various multithreshold microbial images in continuous adaptive learning and obtain the best-segmented image. Due to the variety and diversity of microorganisms, there are many characteristics of microorganisms [23]. Therefore, in this paper, the strategy network μ is regarded as the actor. The function s,a is fitted through the value network and regarded as critical, to complete the recognition and segmentation of different multithreshold microbial images. The expression of the objective function is described by the following equation:(15)Jθμ=Eθμr1+Eθμγr2+Eθμγ2r3+…Eθμγ2rn.


## 2.6.1. Void Convolution

To ensure that the ResNet-UNet network has a strong feature extraction ability and make the obtained microbial image feature map contain complete image details, this paper improves the ResNet-UNet network, introduces void convolution, and replaces the original convolution based on the original mean pooling and the deletion of the full connection module. While ensuring that the network parameters do not increase, the receptive field of the image is improved so that the details of the microbial image are not lost. For void convolution, it can be regarded as porous convolution, i.e., inserting pixels with a 0-pixel value between the pixels of the original convolution core, to increase the size of the convolution core and improve the image receptive field [20, 21].


## 2.6.2. DualQ Network Mechanism

The improved ResNet-UNet network realizes the panoramic perception of multithreshold microbial image features, obtains the microbial image feature map, takes it as the training data set of reinforcement learning, fully considers the dualQ network mechanism, and introduces the target Net as a network copy of the depth Q value. The target value function is determined through the ResNet-UNet network and provided to the depth Q-value network [22]. While continuously adjusting it, the target Net, give play to the decision-making performance of the deep Q-value network and complete the feature extraction of the microbial map and the collaborative operation of parameters. Through the positive feedback mechanism, the error of the collaborative operation process is adjusted, to establish the best segmentation mechanism for the multithreshold microbial image. The objective function gradient function equation is as follows:(12)∇θJπθ=∫AS∫ASρπs∇θπθs,aQπs,adads=Es⋅ρπ,a⋅πθ∇θlogπθa|sQπs,a,where state information and its collection are represented as s and S, the set of action a is A; the strategy is π, the network parameter is θ. Qπs,a is used to describe the expected reward of choosing an action a when the status is s, the probability distribution function is expressed as ρ. According to its value, the determination of the optimal action in the current state can be realized, E is the reward for the best action in the current status.According to the objective function gradient function of equation (12), the deterministic strategy expression is obtained, use function μ and according to the action information, actions can be obtained. μ is regarded as the optimal action strategy at=μst|θμ. The quantitative analysis of multi-threshold microbial image segmentation can be realized by the following equation:(13)Jμθ=∫ASρμs∫ASrs,μθsds=Es⋅ρμrs,μθs.Under competitive conditions, equation (13) has poor stationarity. Therefore, this paper performs the first-order derivation. At this time, the deterministic strategy gradient can be described by the following equation:(14)∇θJμθ=∫ASρμs∫ASQμs,a∇θμθs|a=μθds=Es⋅ρπQμs,a∇θμθs|a=μθ.The compatibility of equation (14) is outstanding. It can complete the segmentation of various multithreshold microbial images in continuous adaptive learning and obtain the best-segmented image. Due to the variety and diversity of microorganisms, there are many characteristics of microorganisms [23]. Therefore, in this paper, the strategy network μ is regarded as the actor. The function s,a is fitted through the value network and regarded as critical, to complete the recognition and segmentation of different multithreshold microbial images. The expression of the objective function is described by the following equation:(15)Jθμ=Eθμr1+Eθμγr2+Eθμγ2r3+…Eθμγ2rn.


## 2.7. Data Sets and Evaluation Index

Experimental data sets were used Earth Microbiome Project, Human gut microbiomes, and Human Microbiome Project. Earth Microbiome Project: EMP, founded in 2010, is a large-scale crowdsourcing work aimed at analyzing the global microbial community. The general premise is to study from the perspective of the microbial community itself. The researchers used amplicon sequencing, metagenomics, and metabolomics to analyze 200000 samples from these communities, including 100000 microbial images, which is very suitable for multithreshold microbial image segmentation. Human gut microbiomes: human gut microbiomes refer to all microorganisms inhabiting the human gastrointestinal tract. The different roles of intestinal microbiota in human health and disease have been rerecognized. Metagenomic research has changed our understanding of the taxonomy and functional diversity of human microbiota, such as promoting the maturation of the immune system and providing a direct barrier against pathogen colonization. The data set includes a series of studies on human intestinal microbiota, including images, papers, and patents. In this paper, two thousand microbial images are used for experimental tests to verify the application effect of different algorithms. Human Microbiome Project: the overall mission of the HMP is to generate resources to facilitate the characterization of the human microbiota to further our understanding of how the microbiome impacts human health and disease. The initial phase of the project, HMP1, established in 2008, characterized the microbial communities of 300 healthy individuals, across several different sites on the human body: nasal passages, oral cavity, skin, gastrointestinal tract, and urogenital tract. 16S rRNA sequencing was performed to characterize the complexity of microbial communities at each body site and to begin to ask to investigate whether there is a core healthy microbiome. Metagenomic whole genome shotgun sequencing provided insights into the functions and pathways present in the human microbiome.A total of five thousand microbial images were selected in the earth microbiome projects data set, human gut microbiomes data set, and Human microbiome project data set, the discount coefficient is 0.95, and the learning rate is 10−3. The value of strategy reward is [−1, 1], and the reward of action is −1. The parameters of the multithreshold microbial image segmentation model are adaptively selected to achieve a better effect of model training. Among them, four thousand images are used to train the segmentation model, and the other images are used for image testing. Different algorithms are applied to the segmentation of microbial images in the data set to analyze the image segmentation effect of this algorithm.(1)
Image denoising ability: due to the interference of noise factors when collecting microbial images, the visual effect of the obtained microbial images is poor. Therefore, the clearer the image after denoising, the stronger the denoising ability.(2)
Peak signal-to-noise ratio (PSNR): it is an index for evaluating image quality. The calculation Equation of this index is as follows:(16)PSNR=10×log102n−12MSE,whereMSE refers to the mean square error between an original image and a denoised image, and n refers to the number of bits of the sample value.(3)
Structural similarity (SSIM) is an important index to judge the contrast of segmented multithreshold microbial images and whether the structure is suitable and complete.(17)SSIMx,y=lx,yαcx,yβsx,yγ,wherex and y refer to the two input microbial images, lx,y refers to the brightness comparison function, cx,y refers to contrast comparison function, sx,y refers to structure comparison function, and α,β,γ are different constants.(4)
After using the algorithm to segment the microbial image, the similarity of the segmented image features compared with the original microbial image can be reflected by feature similarity (FSIM). The value range of FSIM is [0, 1], and the closer its value is to 1, the closer their characteristics are. The calculation Equation of this index is as follows:(18)FSIM=∑x∈ΩSLxPCmx∑x∈ΩPCmx,wherePCmx refers to phase consistency judgment function, and SLx refers to coupling functions of different terms.(5)
Loss rate: it refers to the rate of loss in the process of segmentation of microbial image by an algorithm. The calculation equation of this index is as follows:(19)A=1−t−i×100%,wheret refers to the planned quantity of microbial image experiment, and i refers to the total amount of microbial images.


## 3. Results and Discussion

In this paper, three microbial images randomly selected from the Earth microbiome projects data set, human gut microbiomes data set, and human microbiome Project data set are denoised first. By comparing the microbial images before and after denoising, the image denoising ability of this algorithm is analyzed as shown in Figure2.Figure 2
Comparison of microbial images before and after denoising. (a) Microbial image A before denoising. (b) Microbial image B after denoising. (c) Microbial image A before denoising. (d) Microbial image B after denoising. (e) Microbial image A before denoising. (f) Microbial image B after denoising.
(a)(b)(c)(d)(e)(f)According to Figure2. The three selected microbial images have a lot of noise, which has the problems of too high brightness and too large contrast, resulting in the loss of some details of the microbial image, which seriously affects the visual effect of the microbial image. After denoising with this algorithm, the supersaturation of the microbial image is suppressed, the brightness and contrast of the image are restored, and the microbial image information is complete and rich in detail. Therefore, it can be determined that the algorithm in this paper has image noise reduction performance, and the noise reduction effect is outstanding.The threshold number of the microbial image has a direct impact on its image segmentation effect. This algorithm is used to determine the threshold number of Earth Microbiome Project data set, Human gut microbiomes data set, and Human Microbiome Project data set. The Influence of different threshold numbers on the microbial image segmentation effect is analyzed by the PSNR index as shown in Figure3.Figure 3
Relationship between threshold number and PSNR.According to the data in Figure3, PSNR index is used to reflect the visual effect of the segmented microbial image, and the image distortion decreases with the increase of the PSNR index. This algorithm is used to determine the threshold number of microbial image. With the increasing number of thresholds of the microbial image, the PSNR index of the microbial image after segmentation shows an increasing trend. When the threshold number of the Earth Microbiome Project data set increases to 6, the growth trend of the PSNR index tends to be stable. Thus, it can be determined that when the number of thresholds is 6, the image can achieve the best segmentation effect; similarly, it can be determined that the threshold numbers of the Human gut microbiomes data set and Human Microbiome Project data set are 8 and 4, respectively.According to the relationship between the threshold number of different microbial images and SSIM, the image segmentation effect of this algorithm is analyzed. It is shown in Figure4.Figure 4
Relationship analysis between threshold number and SSIM.According to Figure4, with the increasing number of threshold values of microbial images, SSIM indexes show an upward trend. When the number of threshold values is small, SSIM indexes increase rapidly, the number of threshold values increases from 2 to 4, and SSIM indexes increase significantly. When the number of Human Microbiome Project data set threshold values reaches 4, SSIM indexes increase slowly. When the number of Human gut microbiomes data set threshold reaches 8, the SSIM index remains unchanged. When the number of Human Microbiome Project data set threshold increases to 6, the change in the SSIM index is very small. It can be seen that the threshold number of different data sets is different. When the threshold number of Earth Microbiome Project data set, Human gut microbiomes data set and Human Microbiome Project data set is 6, 8, and 4, respectively, it can have the best contrast.By analyzing the relationship between FSIM index and threshold number, this paper realizes the analysis of the multithreshold microbial image segmentation effect. It is shown in Figure5.Figure 5
Relationship analysis between threshold number and FSIM.The FSIM index showed an upward trend with the increasing number of thresholds. When the number of thresholds increases from 1 to 2, the FSIM increase of the three data sets is small. When the number reaches 4, the three FSIM curves show a rapid growth trend and continue to increase the threshold number. The FSIM index of the Earth Microbiome Project data set changes slightly. After the threshold number of the Human gut microbiomes data set is 8, the FSIM does not change, and after the threshold number of the Human Microbiome Project data set is 6, the FSIM index remains stable. From the above-given analysis, when the threshold number of three data sets is 6, 8, and 4, respectively, the characteristics of the segmented image can be closer to the original microbial image.Based on the microbial images of the Earth microbiome projects data set, human gut microbiomes dataset, and human microbiome Project data set, the image segmentation accuracy of this algorithm is verified by the loss rate. Therefore, this paper verifies the image segmentation accuracy of this algorithm by analyzing the model loss rate index under the conditions of training and testing. It is shown in Figure6.Figure 6
Image segmentation accuracy analysis of proposed algorithm.The training image set is used to complete the training of the multithreshold microbial image segmentation model, and the test image set is input into the trained model. The loss rate indexes of training and test image set gradually decrease with the increasing number of iterations. After 60 iterations, the segmentation model begins to converge, and the loss rate curve under the test condition fluctuates at the beginning of the iteration, which is due to the wide variety of microorganisms and the small variety of microorganisms in the training image set. However, this model can better learn the feature information of the microbial image and reduce the loss rate of the model, which is less than 0.05%. The experimental results show that this algorithm can effectively reduce the loss rate of the model and improve the segmentation accuracy of the multithreshold microbial image.The quantum-based image segmentation algorithm in Li et al. [6], the parallel Lévy grey wolf optimized segmentation algorithm in Li et al. [7], the segmentation algorithm based on the Intelligent Algorithm in Cai et al. [8], the Lévy flight improved multithreshold segmentation algorithm in Jia et al. [9], and the target recognition algorithm of deep reinforcement learning in Sun et al. [10] are used as comparison algorithms. The above-given algorithm and proposed algorithm were used to segment the three data sets, and the advantages of this proposed algorithm for image segmentation were analyzed by normalized absolute error (NAE), mean square error (MSE), and running time indexes. The results are shown in Table 1.Table 1
Comparison of segmentation performance of different algorithms in different data sets.
AlgorithmsData setsThreshold numberNAEMSERunning time (s)Li et al. [6]Earth microbiome project1540.97678.43215.886Human gut microbiomes1370.92570.53717.1406Human microbiome project1070.82664.31115.34Li et al. [7]Earth microbiome project134, 1700.7777.53210.0427Human gut microbiomes52, 113, 170, 1810.64668.44310.7601Human microbiome project73, 110, 1360.56162.4319.7423Cai et al. [8]Earth microbiome project59, 148, 1520.7578.5314.7921Human gut microbiomes58, 125, 186, 1720.62175.8114.7178Human microbiome project75, 1280.56169.5834.6341Jia et al. [9]Earth microbiome project61, 141, 165, 173, 224, 1000.47660.8305.3121Human gut microbiomes53, 120, 181, 169, 212, 141, 137, 1010.50157.0415.6212Human microbiome project83, 112, 150, 1360.36140.5605.2353Sun et al. [10]Earth microbiome project48, 138, 155, 182, 235, 1000.50353.8804.6748Human gut microbiomes69, 111, 177, 163, 202, 125, 114, 980.45746.3365.2346Human microbiome project88, 108, 124, 1470.35738.4834.2728ProposedEarth microbiome project63, 134, 185, 197, 225, 1000.31634.0154.0863Human gut microbiomes66, 118, 181, 177, 215, 116, 128, 1030.2130.5864.1408Human microbiome project91, 105, 132, 1480.28829.0433.804According to the data in Table1, different algorithms are used to segment three data sets. The number of thresholds determined by the algorithms in Li et al. [6], Li et al. [7], and Cai et al. [8] is less than other algorithms, which seriously affects the image segmentation performance of the algorithm. This is the reason why its NAE and MSE indexes are higher than other algorithms, and the operation efficiency of the above-given algorithms is low. The algorithms in Jia et al. [9], Sun et al. [10], and the algorithm in this paper is used to segment three data sets, the same threshold number can be determined. However, the NAE and MSE indexes of this algorithm are the lowest and can realize the efficient operation of image segmentation. The reason is that this algorithm can quickly and accurately learn the characteristic information of microbial image through the improved image segmentation model of deep reinforcement learning, and the loss of microbial image information is low. The experimental results show that the microbial image segmentation ability of this algorithm is outstanding and has significant operational advantages.


## 4. Conclusions

To address the problem of poor performance of multithreshold microbial image segmentation existing in traditional methods, this paper proposes a multithreshold microbial image segmentation algorithm using improved deep reinforcement learning. The results show that the proposed algorithm can improve the accuracy of multithreshold microbial image segmentation and has application performance. It can effectively address the problems of loss of image detail information and poor image visualization caused by noise interference in microbial images. The improved deep reinforcement learning segmentation model can efficiently learn microbial features of different morphologies and shapes and has better generalization performance. However, this paper did not verify the image detail information and image visual effect during the experimental process, which led to a decrease in the persuasive power of the experimental results. Therefore, this problem will be investigated in depth in the future as a way to improve the comprehensive performance of the proposed algorithm.



---

*Source: 5096298-2022-08-23.xml*