# A Robust Image Segmentation Framework Based on Nonlocal Total Variation Spectral Transform

Authors: Jianwei Zhang; Yue Shen; Zhaohui Zheng; Le Sun
Journal: Wireless Communications and Mobile Computing
 (2022)

---

## Abstract

Image segmentation plays an important role in various computer vision tasks. Nevertheless, noise always inevitably appears in images and brings a big challenge to image segmentation. To handle the problem, we study the nonlocal total variation (NLTV) spectral theory and build up an image segmentation framework with NLTV spectral transform to segment images with noise. Firstly, we decompose an image into the NLTV flow in the NLTV spectral transform, with which the max response time of each pixel is calculated. Secondly, a separation surface is constructed with the max response time to distinguish the objects and preserve the structure details in segmentation. Thirdly, the image is filtered by the surface in the NLTV spectral domain, and a rough segmentation result is obtained by means of an inverse transform. Finally, we use a binary process and morphological operations to refine the segmentation result. Experiments illustrate that our method can preserve edge structures effectively and has the ability to achieve competitive segmentation performance compared with the state-of-the-art approaches.


---

## Body

## 1. Introduction

Image segmentation refers to partitioning images into multiple homogeneous parts or objects. It plays a significant role in a broad range of computer vision applications, including scene understanding [1], image compression [2], and image retrieval [3, 4]. To date, two categories of segmentation methods have been widely proposed: data-driven methods [5–7] and model-driven methods [8–28].Among data-driven methods, the common strategy is to extract the semantic features of images using deep convolutional neural networks, based on which each pixel can obtain a semantic label to realize segmentation. The popular deep neural networks for semantic segmentation consist of FCN [5], U-Net [6], SegNet [7], etc., which can obtain satisfying segmentation results without any postprocess techniques. However, deep neural networks often suffer from high computational resource consumption and need a great mass of labeled data. Moreover, the interpretability of neural networks is always an Achilles’ heel. Therefore, model-driven methods are our research centrality.According to different segmentation strategies, model-driven methods can be further categorized as boundary-based methods, region-based methods, hybrid methods, and transform-based methods. Boundary-based methods separate objects from the background by edge or shape. The representative methods include edge detection [8–10] and graph-cut methods [11, 12]. The former uses intensity discontinuity to segment an object. Common edge detection operators contain Prewitt [8], Sobel [9], Roberts [9], and Canny [10]. Compared with the edge detection approaches, graph-cut-based methods can achieve better segmentation accuracy. Nonetheless, the extraction of gradients is sensitive to noise, which makes the boundary-based models produce unsatisfying segmentation results for noisy images.Region-based approaches recognize similar regions and complete segmentation by means of statistical techniques. The Chan-Vese model [13] and FCM [14] are representative works. The Chan–Vese model makes the contour curve close to the object boundary by minimizing the energy on both sides of the evolution curve [15]. Nevertheless, the Chan–Vese model fails to obtain satisfying results because of the intensity inhomogeneity. FCM improves the tolerance to ambiguity and obtains more reasonable segmentation results by introducing a membership matrix. However, FCM is unrobust to noise because of the fact that it merely considers gray-level information. To solve the problem, many variants of FCM [16–19] have been developed, which bring good segmentation performance. Nonetheless, the improved methods are still sensitive to the complex background and intensity inhomogeneity.Hybrid methods employ boundary information to detect the region of objects and then use region information to preserve the boundary structures. Recently, transition region (TR)-based image thresholding [20–23] has been proposed as a type of hybrid method. The method, firstly, uses edge detectors or statistical techniques to extract a transition region, which is a structure similar to the image edge, and then, it segments the image by a threshold, which is a gray level mean value of the transition region. TR-based image thresholding additionally exploits the spatial information to acquire more satisfying segmentation results. However, it is a global thresholding method, which is unrobust to intensity inhomogeneity.The aforementioned model-driven methods segment the image using spatial features, which results in sensitivity to noise. Differently, transform-based approaches, firstly, transform the image to a specific domain according to mathematical theories, where noise and image details have different performances. Then, denoised images are obtained by filtering and inverse transformation, on which postprocessing is performed to segment the image. As one of the popular transform approaches, wavelet transform is widely used in diverse computer vision tasks because of its ease of use and multiresolution processing ability. The common operation of the wavelet transform in image processing is to decompose the image to obtain multiscale sub-bands in the wavelet domain with the help of Mallat’s pyramid algorithm [24]. Then, filter the image by low-pass, band-pass, or high-pass filter to obtain the required features. Finally, the processed image can be obtained by inverse transform. To get satisfying segmentation results, wavelet transform is often combined with other segmentation methods, such as watershed segmentation [25], clustering approaches [26], and image thresholding methods [27]. For instance, the method in [25], firstly, decomposes the original image into a multiscale pyramid representation in the wavelet transform domain. Secondly, the watershed algorithm is applied to segment every image of the multiscale pyramid into several regions, including objects and background. Thirdly, the reverse wavelet transform is conducted on the split regions to get the next higher resolution representation. Finally, the size of split regions gradually becomes the same as that of regions in the ground truth to achieve the segmentation result. Nonetheless, wavelet transform-based methods are sensitive to contrast, and the segmentation results are influenced by the selection of wavelet basis functions.Recently, the NLTV spectral theory has been introduced [28] and has attracted people’s attention. The NLTV spectral transform can transform the image from the spatial domain to the spectral domain, in which objects with different contrast, size, and detailed structures can be distinguished well. Additionally, the NLTV spectral transform can preserve image structures because of its nonlocal operators [28]. To this end, we further discuss the performance of NLTV spectral theory and attempt to further enhance the applicability of the NLTV spectral transform. Inspired by the work [29], we demonstrate the sensitivity of the NLTV spectral transform to size, contrast, and its detailed structures in images with or without noise. We also indicate that the spectral transform is invariance to rotation and translation. Besides, we are motivated to put forward a robust image segmentation framework with NLTV spectral transform. The main process is as follows: firstly, the NLTV flow is imposed on an image to acquire the NLTV spectral transform, by which spectral response and a salient time map of the image are calculated. The elements in the salient time map represent the max response time of each pixel of the image. Secondly, we filter the salient time map by a Gaussian filter to remove the isolated points and perform a least-squares regression using a polynomial on the filtered map to fit a separation surface. Thirdly, the image is filtered by the surface in the NLTV spectral domain, followed by the NLTV inverse transform to obtain a rough segmentation result. Finally, we use morphological operators and a binary process to refine the segmentation result.It should be noticed that the total variation (TV) spectral transform-based method [30] has a similar idea in segmenting images with noise. However, the TV spectral transform used in [30] calculates the horizontal and vertical gradient of every pixel, which means only local information is selected to describe object features. In reference [30], the TV flow is obtained by iteratively solving the ROF model, and then the TV spectral transform is yielded. Considering that the edge detail of objects is lost for solving the ROF model, the guided filter is adopted to refine the object edge in [30]. In contrast to the spectral transform strategy in [30], our method pays more attention to the difference between one pixel and all other pixels in the image, termed nonlocal gradients, to achieve NLTV spectral transform. With the nonlocal information, the edge details can be effectively preserved when segmenting the object in a variety of noises. In addition, our segmentation framework does not introduce the guided filter, which may bring the noise from the original image to the segmentation result. We perform the experiments on synthetic, natural, and medical cell images, which demonstrate that the proposed method can achieve competitive segmentation performance compared with the state-of-the-art methods.Overall, the contributions of this work are twofold, which are as follows:(i)
We illustrate the properties of NLTV spectral transform by theoretical proof and experiments. The analysis demonstrates that objects with varying size, contrast, and detailed structures can be distinguished in the NLTV spectral domain. Additionally, the transform is invariant to rotation and translation. These properties indicate the feasibility of segmentation based on NLTV spectral transform.(ii)
We propose an image segmentation framework using NLTV spectral transform, which fits a separation surface to filter sub-bands in the NLTV spectral domain, and it obtains segmentation results by means of postprocessing. Our method can achieve satisfying results for images with diverse noise or complex texture.The rest of the article is structured as follows: section2 gives an overview of the NLTV spectral theory. Section 3 discusses the properties of NLTV transform and introduces our segmentation framework based on NLTV spectral transform. Section 4 illustrates the experimental results of the proposed method. At last, the paper is concluded in section 5.


## 2. Preliminaries

This section introduces the NLTV spectral transform framework [28]. The framework is made of several parts: nonlocal operators, NLTV flow, NLTV spectral transform, and spectral response.


### 2.1. Nonlocal Operators

According to continuous definitions on the graphs of nonlocal gradient and divergence [31], three nonlocal operators, namely nonlocal derivatives, nonlocal gradients, and nonlocal divergences, are defined as follows:LetΩ⊂R2 be a bounded domain and wX,Y≥0 be non-negative weights between any two points, X,Y∈Ω. In the view of graphs, these weights correspond to a certain relationship between these points. For simplicity, we assume that these weights are symmetric, which means wX,Y=wY,X. Then, Gilboa and Osher [28] extended the local derivative to a nonlocal version by the following definition:(1)∂YuX=uY−uXwX,Y,X,Y∈Ω,where uX is a real function, u:Ω⟶R,0<wX,Y<∞, and ∂YuX represents the partial derivatives of uX in the direction of point X and Y.Similar to local gradients derived from local partial derivatives, nonlocal gradient∇wuX:Ω⟶Ω×Ω is defined as the vector composed of all partial derivatives.(2)∇wuX,Y=uY−uXwX,Y,X,Y∈Ω.Before introducing nonlocal divergence, the definition of inner product for vectors is shown as below. Denoting vectors asv1⟶=v1X,Y, v2⟶=v2X,Y∈Ω×Ω, an inner product is defined as follows:(3)v1⟶,v2⟶=∫Ωv1X,Yv2X,YdY.Then nonlocal divergencedivwv⟶X:Ω×Ω⟶Ω is defined as the adjoint of nonlocal gradient. divwv⟶X=∫ΩvX,Y−vY,XwX,YdY.


### 2.2. NLTV Flow

The weight matrixW depends on the patch similarity. For fixed point X and arbitrary point Y in the image, WX,Y represents the weight between the points X and Y, which is defined as follows:(4)WX,Y=EX,Y∑Y∈ΩEX,Y,EX,Y=exp−PX−PY22σ2,where PX and PY represent the patches centered at points X and Y in the image, respectively. σ is a parameter to control the decay of the exponential function. EX,Y describes the similarity between the points X and Y.NLTV is divided into two types, including isotropic NLTV and anisotropic NLTV. The former is defined as follows:(5)JISO−NLTVu=∫Ω∫ΩuX−uY2wX,YdY1/2dX.The latter is defined as follows:(6)JANISO−NLTVu=∫Ω×ΩuX−uYwX,YdYdX.In our work, the anisotropic nonlocal TV is applied to calculate NLTV flow.(7)−∂u∂t∈∂uJNLTVu,u0,X=uX.


### 2.3. NLTV Transform

The sine and cosine functions are the basic functions in Fourier transform. These basic functions’ amplitude forms impulses in the Fourier domain. The work [28] generalized this to NLTV domain. By examining the elementary structures disks for NLTV functional, the second derivative in the time of NLTV flow is considered the representation of the impulse of the elementary structure. Hence, the NLTV transform is defined by the following:(8)ϕt=uttt,where t∈0,∞ is a time parameter of the NLTV flow equation (7), and utt is the second derivative in the time of the NLTV flow.For NLTV transform, the inverse transform reconstructs a signal or image from allϕt elements.(9)IX=∫0∞ϕt,Xdt+u¯,where u¯=1/Ω∫ΩuXdX is the residual part of NLTV transform, and it is also the mean value of the initial condition.


### 2.4. NLTV Spectral Response

Corresponding to the amplitude of the response in Fourier domain, the NLTV spectral response is defined as follows:(10)St=∫Ωϕt,XdX,t∈0,∞.The NLTV spectral response can roughly measure the importance of image information at different time scales in the NLTV spectral domain [28]. The main features of the image emerge at the time scale corresponding to the high response. Otherwise, the NLTV spectral transform could be considered negligible.


## 2.1. Nonlocal Operators

According to continuous definitions on the graphs of nonlocal gradient and divergence [31], three nonlocal operators, namely nonlocal derivatives, nonlocal gradients, and nonlocal divergences, are defined as follows:LetΩ⊂R2 be a bounded domain and wX,Y≥0 be non-negative weights between any two points, X,Y∈Ω. In the view of graphs, these weights correspond to a certain relationship between these points. For simplicity, we assume that these weights are symmetric, which means wX,Y=wY,X. Then, Gilboa and Osher [28] extended the local derivative to a nonlocal version by the following definition:(1)∂YuX=uY−uXwX,Y,X,Y∈Ω,where uX is a real function, u:Ω⟶R,0<wX,Y<∞, and ∂YuX represents the partial derivatives of uX in the direction of point X and Y.Similar to local gradients derived from local partial derivatives, nonlocal gradient∇wuX:Ω⟶Ω×Ω is defined as the vector composed of all partial derivatives.(2)∇wuX,Y=uY−uXwX,Y,X,Y∈Ω.Before introducing nonlocal divergence, the definition of inner product for vectors is shown as below. Denoting vectors asv1⟶=v1X,Y, v2⟶=v2X,Y∈Ω×Ω, an inner product is defined as follows:(3)v1⟶,v2⟶=∫Ωv1X,Yv2X,YdY.Then nonlocal divergencedivwv⟶X:Ω×Ω⟶Ω is defined as the adjoint of nonlocal gradient. divwv⟶X=∫ΩvX,Y−vY,XwX,YdY.


## 2.2. NLTV Flow

The weight matrixW depends on the patch similarity. For fixed point X and arbitrary point Y in the image, WX,Y represents the weight between the points X and Y, which is defined as follows:(4)WX,Y=EX,Y∑Y∈ΩEX,Y,EX,Y=exp−PX−PY22σ2,where PX and PY represent the patches centered at points X and Y in the image, respectively. σ is a parameter to control the decay of the exponential function. EX,Y describes the similarity between the points X and Y.NLTV is divided into two types, including isotropic NLTV and anisotropic NLTV. The former is defined as follows:(5)JISO−NLTVu=∫Ω∫ΩuX−uY2wX,YdY1/2dX.The latter is defined as follows:(6)JANISO−NLTVu=∫Ω×ΩuX−uYwX,YdYdX.In our work, the anisotropic nonlocal TV is applied to calculate NLTV flow.(7)−∂u∂t∈∂uJNLTVu,u0,X=uX.


## 2.3. NLTV Transform

The sine and cosine functions are the basic functions in Fourier transform. These basic functions’ amplitude forms impulses in the Fourier domain. The work [28] generalized this to NLTV domain. By examining the elementary structures disks for NLTV functional, the second derivative in the time of NLTV flow is considered the representation of the impulse of the elementary structure. Hence, the NLTV transform is defined by the following:(8)ϕt=uttt,where t∈0,∞ is a time parameter of the NLTV flow equation (7), and utt is the second derivative in the time of the NLTV flow.For NLTV transform, the inverse transform reconstructs a signal or image from allϕt elements.(9)IX=∫0∞ϕt,Xdt+u¯,where u¯=1/Ω∫ΩuXdX is the residual part of NLTV transform, and it is also the mean value of the initial condition.


## 2.4. NLTV Spectral Response

Corresponding to the amplitude of the response in Fourier domain, the NLTV spectral response is defined as follows:(10)St=∫Ωϕt,XdX,t∈0,∞.The NLTV spectral response can roughly measure the importance of image information at different time scales in the NLTV spectral domain [28]. The main features of the image emerge at the time scale corresponding to the high response. Otherwise, the NLTV spectral transform could be considered negligible.


## 3. Proposed Method

This section discusses the properties of the NLTV spectral transform and displays a segmentation method for images with noise using the NLTV spectral transform. Firstly, the seminal works [29, 30], which demonstrate the properties of TV spectral transform in images with or without noise, are extended to the NLTV spectral transform in motivation. Secondly, a segmentation method using NLTV spectral transform for images with noise is introduced.


### 3.1. Motivation

The section tries to research the properties of NLTV spectrum transform in images with or without noise. Theories and experiments without noise are shown, firstly. Then, the properties are extended to the noise condition by experiments. As known to all, the typical noises in digital images are additive noise, multiplicative noise, and impulse noise. For this reason, we corrupt the images with Gaussian noise, Salt & Pepper noise, and Speckle noise.


#### 3.1.1. Property 1: Sensitivity to Size

A short proof about the property is provided. For the sake of simplicity, we consider scaling with a gray level imagefX, where X=x,y∈Ω. Then, the image after scaling can be denoted as faX. With the above notations, we explore why NLTV spectral transform values over the time scale of images before and after scaling satisfy the following relationship:(11)ϕ˜t,X=aϕat,aX,where ϕ˜t,X and ϕt,X are NLTV spectral transforms corresponding to images before and after scaling, respectively. Notice that for the original image fX, the NLTV flow can be derived from the following partial differential equation:(12)−∂u∂t∈∂uJNL−TVu,u0,X=fX.Inspired by the case of TV, we consider the elementary structures called nonlocal disks for the imagefX. A set A can be used as a nonlocal disk when two conditions are satisfied [28]: 1) A is a nonlocal calibrable set. 2) The curvature is constant on the internal boundary of the set A.The characteristic function ofA is χAX=1,X∈A0,X∉A. Then, the explicit solution of problem (12) with u0,X=χAX is expressed as follows:(13)ut,X=1−tλAχAX,X∈A,0,X∉A,where λA=PerA/A and PerA and A are, respectively, perimeter and normal of A. In the same way, the NLTV flow of nonlocal disk A′ for the image faX is as follows:(14)u˜t˜,X˜=1−t˜λA′χA′X˜,X˜∈A′,0,X˜∉A′.The energy of points in the imagefX and faX decreases with the average speed of λA and λA′, respectively. It is worth noting that λA is equal to λA′ because the object patterns before and after scaling are similar. Hence, we have u˜t˜,X˜=ut,X with t˜=t/a and X˜=X/a. For time scaling is t˜=t/a, u˜t˜t˜t˜,X˜=a2uttt,X is obvious. Therefore, ϕ˜t/a,X/a=tu˜t˜t˜t˜,X˜/a=t⋅a2uttt,X/a=aϕt,X.Figure1 is an example showing how the NLTV spectral transform separates different size objects. The multiscale NLTV spectral descriptions of the pixels are shown in Figure 1(b), which shows that there is a positive correlation between the size and the time to reach the max spectral response. In addition, we can find that the disappearance order of objects in Figure 1(c) is consistent with the order of reaching max spectral response time in Figure 1(b). Figure 1(d) shows the visualization of sub-bands in the NLTV spectral domain, and it is a more intuitive interpretation of figure 1(b). Moreover, Figure 2 shows the sensitivity of NLTV spectral transform to size and similar performance in different noises.Figure 1
Demonstration of property 1. Signatures are distinguished because of their sensitivity to size. (a) Imagef. (b) Multiscale NLTV spectral descriptions of different pixels. (c) Results of NLTV flow of f. (d) Multiscale NLTV spectral components.
(a)(b)(c)(d)Figure 2
NLTV spectral transform on different size objects corrupted with different noises. (a) is the image corrupted with Gaussian (10% variance), Salt & Pepper (10% density), and Speckle noise (10% variance), respectively. (b) is the multiscale NLTV spectral descriptions of different pixels corrupted with noises. (a) Gaussian noise. (b) Salt & Pepper noise. (c) Speckle noise.
(a)(b)(c)


#### 3.1.2. Property 2: Sensitivity to Local Contrast

Combing the work [29], we attempt to provide a short proof. The image after gray-scale transformation by factor a is denoted as afX. Then, we plan to prove that the NLTV spectral signatures of fX and afX satisfy the following relationship:(15)ϕ˜at,X=ϕt,X.It is noting thatϕt,X is still related with characteristic function χAX mentioned in property 1. Copying the analysis of property 1, the NLTV flows of fX and faX are as follows:(16)ut,X=1−tλAχAX,X∈A,0,X∉A,u˜t˜,X˜=a1−t˜λA′χA′X˜,X˜∈A′,0,X˜∉A′,where t˜=at, A=A′, and X=X˜. u˜t˜,X˜=aut,X and u˜t˜t˜at˜,X˜=uttt,X/a are given. Therefore, ϕ˜at,X=atu˜t˜t˜t˜,X˜=at⋅uttt,X/a=ϕt,X.An example is demonstrated on a synthetic image without noise, as shown in Figure3. The image exhibited in figure 3(a) contains four different contrast squares with a black background. The NLTV spectral transform is calculated, and multiscale NLTV spectral descriptions of different pixels are shown in Figure 3(b). Figures 3(c) and 3(d) show more intuitive performance, which indicates that the low contrast squares disappear first. In addition, the NLTV spectral transform is implemented on different noises to verify its performance. As shown in Figure 4, except for small time scales, the NLTV spectral description has a similar performance, which demonstrates the sensitivity of the NLTV spectral transform to contrast images with noise.Figure 3
Demonstration of property 2. Signatures are distinguished because of their sensitivity to contrast. (a) Imagef. (b) Multiscale NLTV spectral descriptions of different pixels. (c) Results of NLTV flow of f. (d) Multiscale NLTV spectral components.
(a)(b)(c)(d)Figure 4
NLTV spectral transform on different local contrasts corrupted with different noises. (a) is the image corrupted with Gaussian (10% variance), Salt & Pepper (10% density), and Speckle noise (10% variance) respectively. (b) is the multiscale NLTV spectral descriptions of different pixels corrupted with noises. (a) Gaussian noise. (b) Salt & Pepper noise. (c) Speckle noise.
(a)(b)(c)


#### 3.1.3. Property 3: Sensitivity to Detailed Structures

Figure5 shows objects with diverse structures. Figure 5(b) shows that different objects have different time scales when reaching the max spectral response. Figures 5(c) and 5(d) show an intuitive description. The center square with high contrast is decomposed, firstly. Then, the square ring to which the blue point belongs starts to be decomposed. The black square ring is decomposed finally. The experiment indicates the sensitivity of the NLTV spectral transform to detailed structures. The phenomena are caused by the nonlinear property of the NLTV spectral transform. Assuming that images f and g make up the image h, the response of these images satisfies the following:(17)ϕh≠ϕf+ϕg.Figure 5
Demonstration of property 3. Signatures are distinguished because of their sensitivity to detailed structures. (a) Imagef. (b) Multiscale NLTV spectral descriptions of different pixels. (c) Results of NLTV flow of f. (d) Multiscale NLTV spectral components.
(a)(b)(c)(d)To observe the decomposition process of NLTV spectral transform within noise, examples are carried out on different noises. Figure6 shows the decomposition results of different pixels in diverse noises. It can be seen that, except for small time scales, the NLTV spectral description is similar to the case shown in figure 5(b). The experiments demonstrate that the NLTV spectral transform has a sensitivity to detailed structures.Figure 6
NLTV spectral transform on different structures corrupted with different noises. (a) is the image corrupted with Gaussian (10% variance), Salt & Pepper (10% density), and Speckle noise (10% variance), respectively. (b) is the multiscale NLTV spectral descriptions of different pixels corrupted with noises. (a) Gaussian noise. (b) Salt & Pepper noise. (c) Speckle noise.
(a)(b)(c)


#### 3.1.4. Property 4: Invariance to Rotation and Translation

Suppose the original image is denoted asfX,X∈Ω. Then, the image after rotation by angle θ about the origin is fRX, where R is the rotation matrix.(18)R=cosθsinθ−sinθcosθ,fX=fRX.Moreover, the image after translation by spatial shift on the original image isfX−d and fX=fX−d. In essence, the rotation or translation of the image is equal to rotating or translating the coordinate system in the original image. On the other hand, the NLTV spectral transform is invariant to the coordinate system and sensitive to derivatives. Therefore, the NLTV spectral transform is invariant to rotation and translation, i.e.,(19)ϕ˜fRXt,X=ϕft,RX,ϕ˜fX−at,X=ϕft,X−a.There are three groups of objects with different shapes in figure7(a). The objects in the same group have the same shape and contrast. Different objects have been translated in different positions and rotated at different angles. As figure 7(b) shows, the objects in the same group have a similar NLTV spectral description. More intuitive illustrations are displayed in figures 7(c) and 7(d), which present that the objects within the same group disappear simultaneously. Figure 8 shows the NLTV spectral descriptions of different pixels corrupted with noises. The bottom row of Figure 8 shows that the objects with the same shape have similar descriptions in large time scales, even though they have distinct rotations and translations.Figure 7
Demonstration of property 4. Signatures are similar to different rotations and translations. (a) Imagef. (b) Multiscale NLTV spectral descriptions of different pixels. (c) Results of NLTV flow of f. (d) Multiscale NLTV spectral components.
(a)(b)(c)(d)Figure 8
NLTV spectral transform on different groups corrupted with different noises. (a) is the image corrupted with Gaussian (10% variance), Salt & Pepper (10% density), and Speckle noise (10% variance), respectively. (b) is the multiscale NLTV spectral descriptions of different pixels corrupted with noises. (a) Gaussian noise. (b) Salt & Pepper noise. (c) Speckle noise.
(a)(b)(c)


### 3.2. NLTV Spectral Transform for Robust Image Segmentation

#### 3.2.1. Overview of the Proposed Segmentation Flowchart

Figure9 shows the flowchart of the proposed method. The method starts with the decomposition of an original image in the NLTV spectral domain. Then, the available information dimension of every pixel in the image increases from one to the number of time scales. To better get appropriate components, a soft threshold band-pass filter is selected to replace the traditional hard threshold band-pass filter. After obtaining the separation surface result, an inverse transform is used to get an abstract structure. The segmentation result is obtained with the help of the binary process and morphological operations.Figure 9
Flowchart of the proposed segmentation method using NLTV spectral transform.


#### 3.2.2. NLTV Spectral Decomposition

In the subsection, the process of image decomposition using the NLTV spectral transform is illustrated in detail. Assuming that the number of decomposition components isN, the NLTV flow xxx can be calculated with the help of formulae (6) and (7). According to the definition of the NLTV spectral transform described in formula (8), the second derivative of the element ui with respect to time scale needs to be computed. To speed up the calculation, the first and second derivatives are combined, expressed by formula (20).(20)utti,X=ui+1,X+ui−1,X−2⋅ui,XΔt2,where Δt is the time interval. NLTV transform is obtained based on utt by equation (21).(21)ϕi,X=utti,X⋅i⋅Δt.The NLTV spectral response can also be calculated using equation (10). The residual can be computed by equation (9). If the forward time difference uti=ui+1−ui/Δt is used to calculate the first derivatives, the residual part f¯ can be transformed into formula (22).(22)f¯=N+1⋅uN−N⋅uN+1.


#### 3.2.3. Object and Background Separation

After the decomposition of the original image in the NLTV spectral domain, the available information dimension of every pixel in the image increases from one to the number of time scales, i.e., the information used before decomposition is just pixel value. Inspired by the work [29], a separation surface is selected to effectively reduce the interference of noise on segmentation.To better characterize the feature of objects in the image, time parameterst1 and t2 are chosen to construct a time range t1,t2. By the above analysis of the four properties of NLTV spectral transform, the max response time is computed to describe the image. The max response time here is different from the spectral response of equation (10). As equation (10) shows, the spectral response calculates the element ϕ(t) of the image in the NLTV spectral domain and can reflect the significant part of the image. The NLTV element ϕt on the time scale t corresponding to the low response contains unimportant features, which can be discarded. However, formula (10) demonstrates that it fails to reflect the spatial information of the objects. To better analyze the performance of pixels in the NLTV spectral domain, the max response time is calculated. Specifically, the NLTV spectral transform, firstly, decomposes the image into several spectral components on a time scale, as shown in Figure 9. Then, every pixel in the image corresponds to a set of spectral responses. The time scale of the maximum spectral response is selected to indicate the performance of the local spatial information in the NLTV spectral domain. The maximum response time of pixels inside the same target tends to be close. Therefore, different objects of the image can be extracted by analyzing the max response time corresponding to each pixel. In other words, a salient time map TX for each point X is calculated by equation (23).(23)TX=argmaxiϕi,X,i∈t1,t2.To extract more meaningful information about the segmentation target, we fit a separation surface whose role is a band-pass filter to separate the target from undesired information. Firstly, the filtered max response mapTfilterX is obtained by performing the Gaussian filtering on TX to ensure the smoothness of separation surface. Then, the time scale corresponding to the maximum spectral response is stored as scatters, on which the least square regression is performed to finish fitting the surface TsurX. The fitted surface can be regarded as a soft threshold in the range of t1,t2. For a certain point X, the surface divides it into two parts, t1,TsurX and TsurX,t2. The latter time range is usually chosen for image description to reduce the effect of noise. Therefore, the band-pass filter for each point X with time range t1,t2 can be denoted as follows:(24)HHBPF,t1,t2i=0,1≤i<TsurX,1,TsurX≤i≤t2,0,t2<i≤N.


#### 3.2.4. Desired Objects Segmentation

Image reconstruction, which is also called inverse transform, is implemented after surface fitting. The time scale band represents the integration times of each pixel for the object. The target in the original image is easily obtained by integrating over a specific time scale using reconstruction formula (25).(25)Ix=∑t=TsurXTϕt,X+f¯.Binary processing is performed after inverse transform to obtain the segmentation mask. Finally, morphological operations are used to refine the final mask. By the above operations, the desired segmentation maskfoutput is obtained. To exhibit more details of the proposed method, Algorithm 1 shows the specific process of the NLTV spectral transform-based method for robust image segmentation.Algorithm 1: NLTV spectral transform-based method for robust image segmentation.
Input: gray imagef.Output: segmentation maskfoutput.(1)
Initialize: maximal time scaleT, time step Δt.(2)
Calculate the number of decomposition componentsN=T/Δt.(3)
Compute NLTV flowuii=0N+1 using equations (6) and (7).(4)
Calculate NLTV residual partf¯ using equation (22).(5)
fori=1,2,…,Ndo(6)
Compute the second derivatives in time of flow for each pixelX by equation (20).(7)
Achieve NLTV transform by equation (21).(8)
Calculate NLTV spectral response using equation (10).(9)
end for(10)
Select time parameterst1 and t2 according to the NLTV spectral response.(11)
Compute the salient time mapTX by equation (23).(12)
ObtainTfilterX by performing Gaussian filtering on TX.(13)
Get the fitted surfaceTsurX by performing least square regression on TfilterX.(14)
Reconstruct the resultIX using equation (25).(15)
Get the segmentation maskfbwX by thresholding segmentation on IX.(16)
Get the final maskfoutputX by performing morphological operations on fbwX.


## 3.1. Motivation

The section tries to research the properties of NLTV spectrum transform in images with or without noise. Theories and experiments without noise are shown, firstly. Then, the properties are extended to the noise condition by experiments. As known to all, the typical noises in digital images are additive noise, multiplicative noise, and impulse noise. For this reason, we corrupt the images with Gaussian noise, Salt & Pepper noise, and Speckle noise.


### 3.1.1. Property 1: Sensitivity to Size

A short proof about the property is provided. For the sake of simplicity, we consider scaling with a gray level imagefX, where X=x,y∈Ω. Then, the image after scaling can be denoted as faX. With the above notations, we explore why NLTV spectral transform values over the time scale of images before and after scaling satisfy the following relationship:(11)ϕ˜t,X=aϕat,aX,where ϕ˜t,X and ϕt,X are NLTV spectral transforms corresponding to images before and after scaling, respectively. Notice that for the original image fX, the NLTV flow can be derived from the following partial differential equation:(12)−∂u∂t∈∂uJNL−TVu,u0,X=fX.Inspired by the case of TV, we consider the elementary structures called nonlocal disks for the imagefX. A set A can be used as a nonlocal disk when two conditions are satisfied [28]: 1) A is a nonlocal calibrable set. 2) The curvature is constant on the internal boundary of the set A.The characteristic function ofA is χAX=1,X∈A0,X∉A. Then, the explicit solution of problem (12) with u0,X=χAX is expressed as follows:(13)ut,X=1−tλAχAX,X∈A,0,X∉A,where λA=PerA/A and PerA and A are, respectively, perimeter and normal of A. In the same way, the NLTV flow of nonlocal disk A′ for the image faX is as follows:(14)u˜t˜,X˜=1−t˜λA′χA′X˜,X˜∈A′,0,X˜∉A′.The energy of points in the imagefX and faX decreases with the average speed of λA and λA′, respectively. It is worth noting that λA is equal to λA′ because the object patterns before and after scaling are similar. Hence, we have u˜t˜,X˜=ut,X with t˜=t/a and X˜=X/a. For time scaling is t˜=t/a, u˜t˜t˜t˜,X˜=a2uttt,X is obvious. Therefore, ϕ˜t/a,X/a=tu˜t˜t˜t˜,X˜/a=t⋅a2uttt,X/a=aϕt,X.Figure1 is an example showing how the NLTV spectral transform separates different size objects. The multiscale NLTV spectral descriptions of the pixels are shown in Figure 1(b), which shows that there is a positive correlation between the size and the time to reach the max spectral response. In addition, we can find that the disappearance order of objects in Figure 1(c) is consistent with the order of reaching max spectral response time in Figure 1(b). Figure 1(d) shows the visualization of sub-bands in the NLTV spectral domain, and it is a more intuitive interpretation of figure 1(b). Moreover, Figure 2 shows the sensitivity of NLTV spectral transform to size and similar performance in different noises.Figure 1
Demonstration of property 1. Signatures are distinguished because of their sensitivity to size. (a) Imagef. (b) Multiscale NLTV spectral descriptions of different pixels. (c) Results of NLTV flow of f. (d) Multiscale NLTV spectral components.
(a)(b)(c)(d)Figure 2
NLTV spectral transform on different size objects corrupted with different noises. (a) is the image corrupted with Gaussian (10% variance), Salt & Pepper (10% density), and Speckle noise (10% variance), respectively. (b) is the multiscale NLTV spectral descriptions of different pixels corrupted with noises. (a) Gaussian noise. (b) Salt & Pepper noise. (c) Speckle noise.
(a)(b)(c)


### 3.1.2. Property 2: Sensitivity to Local Contrast

Combing the work [29], we attempt to provide a short proof. The image after gray-scale transformation by factor a is denoted as afX. Then, we plan to prove that the NLTV spectral signatures of fX and afX satisfy the following relationship:(15)ϕ˜at,X=ϕt,X.It is noting thatϕt,X is still related with characteristic function χAX mentioned in property 1. Copying the analysis of property 1, the NLTV flows of fX and faX are as follows:(16)ut,X=1−tλAχAX,X∈A,0,X∉A,u˜t˜,X˜=a1−t˜λA′χA′X˜,X˜∈A′,0,X˜∉A′,where t˜=at, A=A′, and X=X˜. u˜t˜,X˜=aut,X and u˜t˜t˜at˜,X˜=uttt,X/a are given. Therefore, ϕ˜at,X=atu˜t˜t˜t˜,X˜=at⋅uttt,X/a=ϕt,X.An example is demonstrated on a synthetic image without noise, as shown in Figure3. The image exhibited in figure 3(a) contains four different contrast squares with a black background. The NLTV spectral transform is calculated, and multiscale NLTV spectral descriptions of different pixels are shown in Figure 3(b). Figures 3(c) and 3(d) show more intuitive performance, which indicates that the low contrast squares disappear first. In addition, the NLTV spectral transform is implemented on different noises to verify its performance. As shown in Figure 4, except for small time scales, the NLTV spectral description has a similar performance, which demonstrates the sensitivity of the NLTV spectral transform to contrast images with noise.Figure 3
Demonstration of property 2. Signatures are distinguished because of their sensitivity to contrast. (a) Imagef. (b) Multiscale NLTV spectral descriptions of different pixels. (c) Results of NLTV flow of f. (d) Multiscale NLTV spectral components.
(a)(b)(c)(d)Figure 4
NLTV spectral transform on different local contrasts corrupted with different noises. (a) is the image corrupted with Gaussian (10% variance), Salt & Pepper (10% density), and Speckle noise (10% variance) respectively. (b) is the multiscale NLTV spectral descriptions of different pixels corrupted with noises. (a) Gaussian noise. (b) Salt & Pepper noise. (c) Speckle noise.
(a)(b)(c)


### 3.1.3. Property 3: Sensitivity to Detailed Structures

Figure5 shows objects with diverse structures. Figure 5(b) shows that different objects have different time scales when reaching the max spectral response. Figures 5(c) and 5(d) show an intuitive description. The center square with high contrast is decomposed, firstly. Then, the square ring to which the blue point belongs starts to be decomposed. The black square ring is decomposed finally. The experiment indicates the sensitivity of the NLTV spectral transform to detailed structures. The phenomena are caused by the nonlinear property of the NLTV spectral transform. Assuming that images f and g make up the image h, the response of these images satisfies the following:(17)ϕh≠ϕf+ϕg.Figure 5
Demonstration of property 3. Signatures are distinguished because of their sensitivity to detailed structures. (a) Imagef. (b) Multiscale NLTV spectral descriptions of different pixels. (c) Results of NLTV flow of f. (d) Multiscale NLTV spectral components.
(a)(b)(c)(d)To observe the decomposition process of NLTV spectral transform within noise, examples are carried out on different noises. Figure6 shows the decomposition results of different pixels in diverse noises. It can be seen that, except for small time scales, the NLTV spectral description is similar to the case shown in figure 5(b). The experiments demonstrate that the NLTV spectral transform has a sensitivity to detailed structures.Figure 6
NLTV spectral transform on different structures corrupted with different noises. (a) is the image corrupted with Gaussian (10% variance), Salt & Pepper (10% density), and Speckle noise (10% variance), respectively. (b) is the multiscale NLTV spectral descriptions of different pixels corrupted with noises. (a) Gaussian noise. (b) Salt & Pepper noise. (c) Speckle noise.
(a)(b)(c)


### 3.1.4. Property 4: Invariance to Rotation and Translation

Suppose the original image is denoted asfX,X∈Ω. Then, the image after rotation by angle θ about the origin is fRX, where R is the rotation matrix.(18)R=cosθsinθ−sinθcosθ,fX=fRX.Moreover, the image after translation by spatial shift on the original image isfX−d and fX=fX−d. In essence, the rotation or translation of the image is equal to rotating or translating the coordinate system in the original image. On the other hand, the NLTV spectral transform is invariant to the coordinate system and sensitive to derivatives. Therefore, the NLTV spectral transform is invariant to rotation and translation, i.e.,(19)ϕ˜fRXt,X=ϕft,RX,ϕ˜fX−at,X=ϕft,X−a.There are three groups of objects with different shapes in figure7(a). The objects in the same group have the same shape and contrast. Different objects have been translated in different positions and rotated at different angles. As figure 7(b) shows, the objects in the same group have a similar NLTV spectral description. More intuitive illustrations are displayed in figures 7(c) and 7(d), which present that the objects within the same group disappear simultaneously. Figure 8 shows the NLTV spectral descriptions of different pixels corrupted with noises. The bottom row of Figure 8 shows that the objects with the same shape have similar descriptions in large time scales, even though they have distinct rotations and translations.Figure 7
Demonstration of property 4. Signatures are similar to different rotations and translations. (a) Imagef. (b) Multiscale NLTV spectral descriptions of different pixels. (c) Results of NLTV flow of f. (d) Multiscale NLTV spectral components.
(a)(b)(c)(d)Figure 8
NLTV spectral transform on different groups corrupted with different noises. (a) is the image corrupted with Gaussian (10% variance), Salt & Pepper (10% density), and Speckle noise (10% variance), respectively. (b) is the multiscale NLTV spectral descriptions of different pixels corrupted with noises. (a) Gaussian noise. (b) Salt & Pepper noise. (c) Speckle noise.
(a)(b)(c)


## 3.1.1. Property 1: Sensitivity to Size

A short proof about the property is provided. For the sake of simplicity, we consider scaling with a gray level imagefX, where X=x,y∈Ω. Then, the image after scaling can be denoted as faX. With the above notations, we explore why NLTV spectral transform values over the time scale of images before and after scaling satisfy the following relationship:(11)ϕ˜t,X=aϕat,aX,where ϕ˜t,X and ϕt,X are NLTV spectral transforms corresponding to images before and after scaling, respectively. Notice that for the original image fX, the NLTV flow can be derived from the following partial differential equation:(12)−∂u∂t∈∂uJNL−TVu,u0,X=fX.Inspired by the case of TV, we consider the elementary structures called nonlocal disks for the imagefX. A set A can be used as a nonlocal disk when two conditions are satisfied [28]: 1) A is a nonlocal calibrable set. 2) The curvature is constant on the internal boundary of the set A.The characteristic function ofA is χAX=1,X∈A0,X∉A. Then, the explicit solution of problem (12) with u0,X=χAX is expressed as follows:(13)ut,X=1−tλAχAX,X∈A,0,X∉A,where λA=PerA/A and PerA and A are, respectively, perimeter and normal of A. In the same way, the NLTV flow of nonlocal disk A′ for the image faX is as follows:(14)u˜t˜,X˜=1−t˜λA′χA′X˜,X˜∈A′,0,X˜∉A′.The energy of points in the imagefX and faX decreases with the average speed of λA and λA′, respectively. It is worth noting that λA is equal to λA′ because the object patterns before and after scaling are similar. Hence, we have u˜t˜,X˜=ut,X with t˜=t/a and X˜=X/a. For time scaling is t˜=t/a, u˜t˜t˜t˜,X˜=a2uttt,X is obvious. Therefore, ϕ˜t/a,X/a=tu˜t˜t˜t˜,X˜/a=t⋅a2uttt,X/a=aϕt,X.Figure1 is an example showing how the NLTV spectral transform separates different size objects. The multiscale NLTV spectral descriptions of the pixels are shown in Figure 1(b), which shows that there is a positive correlation between the size and the time to reach the max spectral response. In addition, we can find that the disappearance order of objects in Figure 1(c) is consistent with the order of reaching max spectral response time in Figure 1(b). Figure 1(d) shows the visualization of sub-bands in the NLTV spectral domain, and it is a more intuitive interpretation of figure 1(b). Moreover, Figure 2 shows the sensitivity of NLTV spectral transform to size and similar performance in different noises.Figure 1
Demonstration of property 1. Signatures are distinguished because of their sensitivity to size. (a) Imagef. (b) Multiscale NLTV spectral descriptions of different pixels. (c) Results of NLTV flow of f. (d) Multiscale NLTV spectral components.
(a)(b)(c)(d)Figure 2
NLTV spectral transform on different size objects corrupted with different noises. (a) is the image corrupted with Gaussian (10% variance), Salt & Pepper (10% density), and Speckle noise (10% variance), respectively. (b) is the multiscale NLTV spectral descriptions of different pixels corrupted with noises. (a) Gaussian noise. (b) Salt & Pepper noise. (c) Speckle noise.
(a)(b)(c)


## 3.1.2. Property 2: Sensitivity to Local Contrast

Combing the work [29], we attempt to provide a short proof. The image after gray-scale transformation by factor a is denoted as afX. Then, we plan to prove that the NLTV spectral signatures of fX and afX satisfy the following relationship:(15)ϕ˜at,X=ϕt,X.It is noting thatϕt,X is still related with characteristic function χAX mentioned in property 1. Copying the analysis of property 1, the NLTV flows of fX and faX are as follows:(16)ut,X=1−tλAχAX,X∈A,0,X∉A,u˜t˜,X˜=a1−t˜λA′χA′X˜,X˜∈A′,0,X˜∉A′,where t˜=at, A=A′, and X=X˜. u˜t˜,X˜=aut,X and u˜t˜t˜at˜,X˜=uttt,X/a are given. Therefore, ϕ˜at,X=atu˜t˜t˜t˜,X˜=at⋅uttt,X/a=ϕt,X.An example is demonstrated on a synthetic image without noise, as shown in Figure3. The image exhibited in figure 3(a) contains four different contrast squares with a black background. The NLTV spectral transform is calculated, and multiscale NLTV spectral descriptions of different pixels are shown in Figure 3(b). Figures 3(c) and 3(d) show more intuitive performance, which indicates that the low contrast squares disappear first. In addition, the NLTV spectral transform is implemented on different noises to verify its performance. As shown in Figure 4, except for small time scales, the NLTV spectral description has a similar performance, which demonstrates the sensitivity of the NLTV spectral transform to contrast images with noise.Figure 3
Demonstration of property 2. Signatures are distinguished because of their sensitivity to contrast. (a) Imagef. (b) Multiscale NLTV spectral descriptions of different pixels. (c) Results of NLTV flow of f. (d) Multiscale NLTV spectral components.
(a)(b)(c)(d)Figure 4
NLTV spectral transform on different local contrasts corrupted with different noises. (a) is the image corrupted with Gaussian (10% variance), Salt & Pepper (10% density), and Speckle noise (10% variance) respectively. (b) is the multiscale NLTV spectral descriptions of different pixels corrupted with noises. (a) Gaussian noise. (b) Salt & Pepper noise. (c) Speckle noise.
(a)(b)(c)


## 3.1.3. Property 3: Sensitivity to Detailed Structures

Figure5 shows objects with diverse structures. Figure 5(b) shows that different objects have different time scales when reaching the max spectral response. Figures 5(c) and 5(d) show an intuitive description. The center square with high contrast is decomposed, firstly. Then, the square ring to which the blue point belongs starts to be decomposed. The black square ring is decomposed finally. The experiment indicates the sensitivity of the NLTV spectral transform to detailed structures. The phenomena are caused by the nonlinear property of the NLTV spectral transform. Assuming that images f and g make up the image h, the response of these images satisfies the following:(17)ϕh≠ϕf+ϕg.Figure 5
Demonstration of property 3. Signatures are distinguished because of their sensitivity to detailed structures. (a) Imagef. (b) Multiscale NLTV spectral descriptions of different pixels. (c) Results of NLTV flow of f. (d) Multiscale NLTV spectral components.
(a)(b)(c)(d)To observe the decomposition process of NLTV spectral transform within noise, examples are carried out on different noises. Figure6 shows the decomposition results of different pixels in diverse noises. It can be seen that, except for small time scales, the NLTV spectral description is similar to the case shown in figure 5(b). The experiments demonstrate that the NLTV spectral transform has a sensitivity to detailed structures.Figure 6
NLTV spectral transform on different structures corrupted with different noises. (a) is the image corrupted with Gaussian (10% variance), Salt & Pepper (10% density), and Speckle noise (10% variance), respectively. (b) is the multiscale NLTV spectral descriptions of different pixels corrupted with noises. (a) Gaussian noise. (b) Salt & Pepper noise. (c) Speckle noise.
(a)(b)(c)


## 3.1.4. Property 4: Invariance to Rotation and Translation

Suppose the original image is denoted asfX,X∈Ω. Then, the image after rotation by angle θ about the origin is fRX, where R is the rotation matrix.(18)R=cosθsinθ−sinθcosθ,fX=fRX.Moreover, the image after translation by spatial shift on the original image isfX−d and fX=fX−d. In essence, the rotation or translation of the image is equal to rotating or translating the coordinate system in the original image. On the other hand, the NLTV spectral transform is invariant to the coordinate system and sensitive to derivatives. Therefore, the NLTV spectral transform is invariant to rotation and translation, i.e.,(19)ϕ˜fRXt,X=ϕft,RX,ϕ˜fX−at,X=ϕft,X−a.There are three groups of objects with different shapes in figure7(a). The objects in the same group have the same shape and contrast. Different objects have been translated in different positions and rotated at different angles. As figure 7(b) shows, the objects in the same group have a similar NLTV spectral description. More intuitive illustrations are displayed in figures 7(c) and 7(d), which present that the objects within the same group disappear simultaneously. Figure 8 shows the NLTV spectral descriptions of different pixels corrupted with noises. The bottom row of Figure 8 shows that the objects with the same shape have similar descriptions in large time scales, even though they have distinct rotations and translations.Figure 7
Demonstration of property 4. Signatures are similar to different rotations and translations. (a) Imagef. (b) Multiscale NLTV spectral descriptions of different pixels. (c) Results of NLTV flow of f. (d) Multiscale NLTV spectral components.
(a)(b)(c)(d)Figure 8
NLTV spectral transform on different groups corrupted with different noises. (a) is the image corrupted with Gaussian (10% variance), Salt & Pepper (10% density), and Speckle noise (10% variance), respectively. (b) is the multiscale NLTV spectral descriptions of different pixels corrupted with noises. (a) Gaussian noise. (b) Salt & Pepper noise. (c) Speckle noise.
(a)(b)(c)


## 3.2. NLTV Spectral Transform for Robust Image Segmentation

### 3.2.1. Overview of the Proposed Segmentation Flowchart

Figure9 shows the flowchart of the proposed method. The method starts with the decomposition of an original image in the NLTV spectral domain. Then, the available information dimension of every pixel in the image increases from one to the number of time scales. To better get appropriate components, a soft threshold band-pass filter is selected to replace the traditional hard threshold band-pass filter. After obtaining the separation surface result, an inverse transform is used to get an abstract structure. The segmentation result is obtained with the help of the binary process and morphological operations.Figure 9
Flowchart of the proposed segmentation method using NLTV spectral transform.


### 3.2.2. NLTV Spectral Decomposition

In the subsection, the process of image decomposition using the NLTV spectral transform is illustrated in detail. Assuming that the number of decomposition components isN, the NLTV flow xxx can be calculated with the help of formulae (6) and (7). According to the definition of the NLTV spectral transform described in formula (8), the second derivative of the element ui with respect to time scale needs to be computed. To speed up the calculation, the first and second derivatives are combined, expressed by formula (20).(20)utti,X=ui+1,X+ui−1,X−2⋅ui,XΔt2,where Δt is the time interval. NLTV transform is obtained based on utt by equation (21).(21)ϕi,X=utti,X⋅i⋅Δt.The NLTV spectral response can also be calculated using equation (10). The residual can be computed by equation (9). If the forward time difference uti=ui+1−ui/Δt is used to calculate the first derivatives, the residual part f¯ can be transformed into formula (22).(22)f¯=N+1⋅uN−N⋅uN+1.


### 3.2.3. Object and Background Separation

After the decomposition of the original image in the NLTV spectral domain, the available information dimension of every pixel in the image increases from one to the number of time scales, i.e., the information used before decomposition is just pixel value. Inspired by the work [29], a separation surface is selected to effectively reduce the interference of noise on segmentation.To better characterize the feature of objects in the image, time parameterst1 and t2 are chosen to construct a time range t1,t2. By the above analysis of the four properties of NLTV spectral transform, the max response time is computed to describe the image. The max response time here is different from the spectral response of equation (10). As equation (10) shows, the spectral response calculates the element ϕ(t) of the image in the NLTV spectral domain and can reflect the significant part of the image. The NLTV element ϕt on the time scale t corresponding to the low response contains unimportant features, which can be discarded. However, formula (10) demonstrates that it fails to reflect the spatial information of the objects. To better analyze the performance of pixels in the NLTV spectral domain, the max response time is calculated. Specifically, the NLTV spectral transform, firstly, decomposes the image into several spectral components on a time scale, as shown in Figure 9. Then, every pixel in the image corresponds to a set of spectral responses. The time scale of the maximum spectral response is selected to indicate the performance of the local spatial information in the NLTV spectral domain. The maximum response time of pixels inside the same target tends to be close. Therefore, different objects of the image can be extracted by analyzing the max response time corresponding to each pixel. In other words, a salient time map TX for each point X is calculated by equation (23).(23)TX=argmaxiϕi,X,i∈t1,t2.To extract more meaningful information about the segmentation target, we fit a separation surface whose role is a band-pass filter to separate the target from undesired information. Firstly, the filtered max response mapTfilterX is obtained by performing the Gaussian filtering on TX to ensure the smoothness of separation surface. Then, the time scale corresponding to the maximum spectral response is stored as scatters, on which the least square regression is performed to finish fitting the surface TsurX. The fitted surface can be regarded as a soft threshold in the range of t1,t2. For a certain point X, the surface divides it into two parts, t1,TsurX and TsurX,t2. The latter time range is usually chosen for image description to reduce the effect of noise. Therefore, the band-pass filter for each point X with time range t1,t2 can be denoted as follows:(24)HHBPF,t1,t2i=0,1≤i<TsurX,1,TsurX≤i≤t2,0,t2<i≤N.


### 3.2.4. Desired Objects Segmentation

Image reconstruction, which is also called inverse transform, is implemented after surface fitting. The time scale band represents the integration times of each pixel for the object. The target in the original image is easily obtained by integrating over a specific time scale using reconstruction formula (25).(25)Ix=∑t=TsurXTϕt,X+f¯.Binary processing is performed after inverse transform to obtain the segmentation mask. Finally, morphological operations are used to refine the final mask. By the above operations, the desired segmentation maskfoutput is obtained. To exhibit more details of the proposed method, Algorithm 1 shows the specific process of the NLTV spectral transform-based method for robust image segmentation.Algorithm 1: NLTV spectral transform-based method for robust image segmentation.
Input: gray imagef.Output: segmentation maskfoutput.(1)
Initialize: maximal time scaleT, time step Δt.(2)
Calculate the number of decomposition componentsN=T/Δt.(3)
Compute NLTV flowuii=0N+1 using equations (6) and (7).(4)
Calculate NLTV residual partf¯ using equation (22).(5)
fori=1,2,…,Ndo(6)
Compute the second derivatives in time of flow for each pixelX by equation (20).(7)
Achieve NLTV transform by equation (21).(8)
Calculate NLTV spectral response using equation (10).(9)
end for(10)
Select time parameterst1 and t2 according to the NLTV spectral response.(11)
Compute the salient time mapTX by equation (23).(12)
ObtainTfilterX by performing Gaussian filtering on TX.(13)
Get the fitted surfaceTsurX by performing least square regression on TfilterX.(14)
Reconstruct the resultIX using equation (25).(15)
Get the segmentation maskfbwX by thresholding segmentation on IX.(16)
Get the final maskfoutputX by performing morphological operations on fbwX.


## 3.2.1. Overview of the Proposed Segmentation Flowchart

Figure9 shows the flowchart of the proposed method. The method starts with the decomposition of an original image in the NLTV spectral domain. Then, the available information dimension of every pixel in the image increases from one to the number of time scales. To better get appropriate components, a soft threshold band-pass filter is selected to replace the traditional hard threshold band-pass filter. After obtaining the separation surface result, an inverse transform is used to get an abstract structure. The segmentation result is obtained with the help of the binary process and morphological operations.Figure 9
Flowchart of the proposed segmentation method using NLTV spectral transform.


## 3.2.2. NLTV Spectral Decomposition

In the subsection, the process of image decomposition using the NLTV spectral transform is illustrated in detail. Assuming that the number of decomposition components isN, the NLTV flow xxx can be calculated with the help of formulae (6) and (7). According to the definition of the NLTV spectral transform described in formula (8), the second derivative of the element ui with respect to time scale needs to be computed. To speed up the calculation, the first and second derivatives are combined, expressed by formula (20).(20)utti,X=ui+1,X+ui−1,X−2⋅ui,XΔt2,where Δt is the time interval. NLTV transform is obtained based on utt by equation (21).(21)ϕi,X=utti,X⋅i⋅Δt.The NLTV spectral response can also be calculated using equation (10). The residual can be computed by equation (9). If the forward time difference uti=ui+1−ui/Δt is used to calculate the first derivatives, the residual part f¯ can be transformed into formula (22).(22)f¯=N+1⋅uN−N⋅uN+1.


## 3.2.3. Object and Background Separation

After the decomposition of the original image in the NLTV spectral domain, the available information dimension of every pixel in the image increases from one to the number of time scales, i.e., the information used before decomposition is just pixel value. Inspired by the work [29], a separation surface is selected to effectively reduce the interference of noise on segmentation.To better characterize the feature of objects in the image, time parameterst1 and t2 are chosen to construct a time range t1,t2. By the above analysis of the four properties of NLTV spectral transform, the max response time is computed to describe the image. The max response time here is different from the spectral response of equation (10). As equation (10) shows, the spectral response calculates the element ϕ(t) of the image in the NLTV spectral domain and can reflect the significant part of the image. The NLTV element ϕt on the time scale t corresponding to the low response contains unimportant features, which can be discarded. However, formula (10) demonstrates that it fails to reflect the spatial information of the objects. To better analyze the performance of pixels in the NLTV spectral domain, the max response time is calculated. Specifically, the NLTV spectral transform, firstly, decomposes the image into several spectral components on a time scale, as shown in Figure 9. Then, every pixel in the image corresponds to a set of spectral responses. The time scale of the maximum spectral response is selected to indicate the performance of the local spatial information in the NLTV spectral domain. The maximum response time of pixels inside the same target tends to be close. Therefore, different objects of the image can be extracted by analyzing the max response time corresponding to each pixel. In other words, a salient time map TX for each point X is calculated by equation (23).(23)TX=argmaxiϕi,X,i∈t1,t2.To extract more meaningful information about the segmentation target, we fit a separation surface whose role is a band-pass filter to separate the target from undesired information. Firstly, the filtered max response mapTfilterX is obtained by performing the Gaussian filtering on TX to ensure the smoothness of separation surface. Then, the time scale corresponding to the maximum spectral response is stored as scatters, on which the least square regression is performed to finish fitting the surface TsurX. The fitted surface can be regarded as a soft threshold in the range of t1,t2. For a certain point X, the surface divides it into two parts, t1,TsurX and TsurX,t2. The latter time range is usually chosen for image description to reduce the effect of noise. Therefore, the band-pass filter for each point X with time range t1,t2 can be denoted as follows:(24)HHBPF,t1,t2i=0,1≤i<TsurX,1,TsurX≤i≤t2,0,t2<i≤N.


## 3.2.4. Desired Objects Segmentation

Image reconstruction, which is also called inverse transform, is implemented after surface fitting. The time scale band represents the integration times of each pixel for the object. The target in the original image is easily obtained by integrating over a specific time scale using reconstruction formula (25).(25)Ix=∑t=TsurXTϕt,X+f¯.Binary processing is performed after inverse transform to obtain the segmentation mask. Finally, morphological operations are used to refine the final mask. By the above operations, the desired segmentation maskfoutput is obtained. To exhibit more details of the proposed method, Algorithm 1 shows the specific process of the NLTV spectral transform-based method for robust image segmentation.Algorithm 1: NLTV spectral transform-based method for robust image segmentation.
Input: gray imagef.Output: segmentation maskfoutput.(1)
Initialize: maximal time scaleT, time step Δt.(2)
Calculate the number of decomposition componentsN=T/Δt.(3)
Compute NLTV flowuii=0N+1 using equations (6) and (7).(4)
Calculate NLTV residual partf¯ using equation (22).(5)
fori=1,2,…,Ndo(6)
Compute the second derivatives in time of flow for each pixelX by equation (20).(7)
Achieve NLTV transform by equation (21).(8)
Calculate NLTV spectral response using equation (10).(9)
end for(10)
Select time parameterst1 and t2 according to the NLTV spectral response.(11)
Compute the salient time mapTX by equation (23).(12)
ObtainTfilterX by performing Gaussian filtering on TX.(13)
Get the fitted surfaceTsurX by performing least square regression on TfilterX.(14)
Reconstruct the resultIX using equation (25).(15)
Get the segmentation maskfbwX by thresholding segmentation on IX.(16)
Get the final maskfoutputX by performing morphological operations on fbwX.


## 4. Experiment Results

### 4.1. Data and Settings

To evaluate the performance of the proposed method, synthetic, natural, and medical images are used for experiments. 1) The first experiment contains 3 groups of synthetic images whose textures are taken from the Brodatz Textures dataset [32]. Speckle, Salt & Pepper, and Gaussian noises are added to each group of synthetic images separately. 2) The second experiment contains 3 groups of natural images taken from the MSRA-1000 dataset [33]. 3) The third experiment contains 1 group of cell images, which is taken from the Fluo-N2DH-SIM + dataset [34]. Three different types of noises are also added to natural and medical images.We compare our segmentation method with four classical methods, i.e., the C-V model [13], FCM [14], FRFCM [19], and wavelet segmentation method (WSM) [27], which are used in the experiments. The experiments are implemented using the MATLAB R2020b platform and a PC with 16 GB RAM.The parameter settings for the proposed method are as follows: experiments show that when the image is transformed into the NLTV domain, detailed information is located in a low time scale. Large scale, which is close toT, contains less important information. Objects are mostly distributed in the middle scale. Hence, a middle-scale time range t1,t2 is selected. In the following experiments, t1 is set to T/5 and t2 is set to 3T/5. The parameters T and Δt are set to 9 and 0.03, respectively.


### 4.2. Quantitative Metrics

To quantitatively evaluate the performance of segmentation effect, four different metrics are chosen: FPR [21], FNR [21], dice similarity coefficient (DICE) [35], and segmentation accuracy (SA) [36].To measure the difference between segmentation results and ground truths, FPR and FNR are chosen in the subsequent experiments. The former calculates the number of background pixels classified as object pixels relative to the total background pixels. FNR measures the number of object pixels classified as background pixels relative to the total object pixels. FPR and FNR are defined as follows:(26)FPR=BR∩OGBG,FNR=OR∩BGOG,where BR and BG represent the number of background pixels in the segmentation results and ground truths, respectively. Additionally, OR and OG are the number of object pixels in the segmentation results and ground truths, respectively.DICE measures segmentation accuracy by calculating the degree of spatial overlap. Specifically, for the result regionA and target region B,(27)DICEA,B=2A∩BA+B,where ∩ means the intersection of two sets. The value range of DICE is [0, 1]. The higher DICE indicates that the segmentation result is more precise. DICEA,B=1 demonstrates that the segmentation result is the most complete, while DICEA,B=0 shows that the segmentation result is the worst.Another evaluation metric is SA, which can assess the number of well-classified pixels in the image. The definition of SA is given as follows:(28)SA=∑i=1NfitruthN,where fitruth means the correctly segmented pixel and N denotes the total number of pixels in an image.


### 4.3. Parameter Analysis

This section analyzes the effects ofΔt and T on the segmentation results of the proposed method through an experiment. The experiment was carried out on MSRA-1000, and the average SA was used as an indicator to show the influence of two parameters on the segmentation accuracy. The average SA was calculated by averaging the SA of all images on the dataset. The parameter Δt ranges from 0.01 to 0.1, and the step is 0.01. Additionally, the maximal time scale T ranges from 1 to 10, and the interval is 1. Figure 10 demonstrates the results for different Δt and T. The proposed method achieves the best performance when Δt=0.03 and T=9.Figure 10
Effects of differentΔt and T on the average SA.


### 4.4. Synthetic Images

The first experiment was implemented on three synthetic images, which are shown in Figure11. The first row shows a synthetic image containing multiple repeating structures and a dark grid-like background. A simple synthetic image, which has an irregular object, is arranged in the middle row. The object in the bottom row is complex and has a texture with inhomogeneous contrast. Moreover, three images are separately contaminated with Speckle (10% variance), Salt & Pepper (10% density), and Gaussian (10% variance) noise.Figure 11
Segmentation results on synthetic images corrupted by different noises. (a) Original images. (b) Images (from top to bottom) that are corrupted with Speckle (10% variance), Salt & Pepper (10% density), and Gaussian noises (10% variance), respectively. (c) C-V Model. (d) FCM. (e) FRFCM. (f) WSM (db). (g) WSM (haar). (h) Proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)Table1 lists the quantitative evaluations of different segmentation methods on various images. Combining with Figure 11 and Table 1, FCM got wrong segmentation results because of its sensitivity to noise. FRFCM achieved a good result on the first image and got a high DICE and SA value as shown in Table 1. However, it failed to distinguish the second and the third image because of the inhomogeneous contrast. WSM, which is based on spectral analysis, can remove the influence of noise. However, as Figure 11 shows, WSM oversmoothed the edge and damaged the edge details. Meanwhile, WSM was unable to segment objects accurately on the second and third images. The reason is that WSM is sensitive to inhomogeneous contrast. The C-V model obtained the segmentation results of all images more correctly. One of the reasons was that the C-V model relies on an initial contour, which provides prior information about the approximate position of the object. Nevertheless, the C-V model was sensitive to noise. On the second and third images, the C-V model was unable to accurately segment the targets. The noises slowed down the convergence speed of the algorithm and made the method fall into the local minimum problem. However, the proposed method achieved the best results in all methods. The NLTV spectral transform-based method can segment the objects exactly and can reduce the influence of inhomogeneous contrast at the same time. The reason is that our method can segment objects, combining object size, contrast, and structures. As shown in Table 1, the proposed method got a high FNR on the second synthetic image, which intended an under-segmentation. The problem was caused by the morphological operators in the output of the proposed method, which may cause edge corrodes.Table 1
Evaluation metrics of compared methods for synthetic images, which are corrupted with Speckle (10% variance), Salt & Pepper (10% density), and Gaussian noise (10% variance), respectively.
ImageMetricC-V modelFCMFRFCMWSM (db)WSM (haar)Proposed method1FPR0.00250.04120.00100.05800.04020.0009FNR0.05010.19770.06550.04280.04920.0290DICE0.95960.75640.96040.78890.83280.9821SA0.99070.93970.99100.94080.95580.99562FPR0.01670.21110.13550.02410.03260.0038FNR0.02310.07430.01870.03890.03680.0273DICE0.96660.75310.84270.95100.94150.9798SA0.98080.82730.89590.97180.96590.98843FPR0.03580.31510.23520.19630.19840.0235FNR0.02890.09010.02170.02530.02550.0229DICE0.90750.54460.64020.67750.67510.9333SA0.96410.72490.80150.83260.83080.9748The best two results are highlighted in bold and italics fonts.


### 4.5. Nature Images

To further discuss the proposed method’s segmentation ability for images with various noises, the second experiment was performed on three natural images, which are shown in Figures12, 13, and 14. The object that has a similar contrast to the surroundings is shown in Figure 12. Figure 13 displays a complex scene that has lots of tiny structures in the background. The object in Figure 14 is a piece of paper containing words, and the words will interfere with segmentation methods. Moreover, three images are separately contaminated with Speckle (10%, 20%, and 30% variance), Salt & Pepper (10%, 20%, and 30% density), and Gaussian (10%, 20%, and 30% variance) noise.Figure 12
Segmentation results on image “star” corrupted by Speckle noise. (a) Original images. (b) Images (from top to bottom) that are corrupted with 10%, 20%, and 30% variance of Speckle noise, respectively (c) C-V model. (d) FCM. (e) FRFCM; (f) WSM (db); (g) WSM (haar); (h) Proposed method.Figure 13
Segmentation results on image “snowman” corrupted by Salt & pepper noise. (a) Original images. (b) Images (from top to bottom) that are corrupted with 10%, 20%, and 30% variance of Speckle noise, respectively. (c) C-V Model. (d) FCM; (e) FRFCM. (f) WSM (db). (g) WSM (haar). (h) Proposed method.Figure 14
Segmentation results on image “paper scrap” corrupted by Gaussian noise. (a) Original images. (b) Images (from top to bottom) that are corrupted with 10%, 20%, and 30% variance of Speckle noise, respectively. (c) C-V Model. (d) FCM. (e) FRFCM. (f) WSM (db). (g) WSM (haar). (h) Proposed method.As Figure12 shows, the object has a similar contrast to the surrounding border. FCM separated the noise while segmenting the object because of its sensitivity to noise. FRFCM had better results than FCM, however, it still had wrong segmentation for noise. WSM can remove the influence of noise. However, WSM failed to remove the impact of inhomogeneous contrast. The C-V model achieved accurate segmentation results because of its initial contour. From Table 2, it can be seen that the C-V model had similar DICE and SA values with the proposed method. However, the C-V model was difficult to segment corner structure because of noises. The proposed method can better preserve structural information while segmenting.Table 2
Evaluation metrics of compared methods for “star,” which are corrupted with Speckle (10%, 20%, and 30% variance) noise.
Noise level (%)MetricC-V modelFCMFRFCMWSM (db)WSM (haar)Proposed method10FPR0.00140.32990.21720.29530.29340.0010FNR0.06360.11700.14560.02280.00440.0544DICE0.96120.26330.35530.30670.31170.9667SA0.99530.68230.79100.71890.72130.995720FPR0.00140.34310.26770.29790.29350.0011FNR0.05730.12960.03540.01210.01070.0578DICE0.96160.25340.32940.30710.31090.9649SA0.99500.66960.74610.71690.72120.995530FPR0.00100.34540.26680.30010.29130.0010FNR0.07180.11550.09470.01120.01700.0584DICE0.95990.25580.32070.30590.31130.9652SA0.99490.66830.74600.71500.72310.9950The best two results are highlighted in bold and italics fonts.Figure13 shows the algorithms’ performances on the natural images, which are corrupted by Salt & Pepper noise. Table 3 shows the corresponding quantitative metrics. In Figure 13, there are lots of small objects in the background, which have a similar contrast to the object. When these small targets are contaminated with Salt & Pepper noise, they cause serious interference with segmentation methods, which mainly rely on contrast. Figure 13 shows that WSM has a good result; however, it is unable to segment the areas surrounding the object correctly. Table 4 shows that the C-V model has better results than WSM; however, it still has an incorrect segmentation of the background. Because of the sensitivity of the NLTV spectral transform to contrast, size, and structures, the proposed method can still separate objects when the background has small size structures.Table 3
Evaluation metrics of compared methods for “snowman,” which are corrupted with Speckle (10%, 20%, and 30% variance) noise.
Noise level (%)MetricC-V modelFCMFRFCMWSM (db)WSM (haar)Proposed method10FPR0.01370.14060.07370.05320.04670.0049FNR0.05770.17560.04080.08480.09830.0890DICE0.94720.69880.85560.86420.86750.9524SA0.97830.85310.93320.94060.94310.981020FPR0.01400.18090.07630.05700.04970.0053FNR0.05530.17680.04020.07640.08830.0979DICE0.94830.65460.85200.86220.86820.9505SA0.97870.82070.93120.93910.94280.980430FPR0.02140.22630.07850.05420.05480.0175FNR0.05290.16510.03800.08800.07640.0997DICE0.93450.61780.85050.86030.86560.9479SA0.97250.78700.93010.93890.94080.9732The best two results are highlighted in bold and italics fonts.Table 4
Evaluation metrics of compared methods for “paper scrap,” which are corrupted with Speckle (10%, 20%, and 30% variance) noise.
Noise level (%)MetricC-V modelFCMFRFCMWSM (db)WSM (haar)Proposed method10FPR0.00130.41300.41380.50430.50630.0002FNR0.13020.09620.02650.01540.02300.0688DICE0.92650.38130.40230.35850.35570.9754SA0.98090.62710.63400.55570.55330.993920FPR0.00160.42360.41580.50590.50770.0005FNR0.13100.11440.02020.02250.01620.0743DICE0.92500.36860.40370.35620.35680.9725SA0.98050.61520.63330.55380.55270.993230FPR0.00160.42390.41640.51700.51370.0011FNR0.13310.11290.02300.01680.01880.0893DICE0.92390.36930.40340.35250.35340.9661SA0.98030.61540.6330.54460.54720.9917The best two results are highlighted in bold and italics fonts.Figure14 shows the segmentation results on the natural image when it is corrupted with different levels of Gaussian noise. Table 4 shows the corresponding quantitative metrics of algorithms. The natural image is difficult for segmentation methods because it has complex texture like words inside, which will affect the integrity of the segmentation results. The C-V model was capable of dealing with the background, however, it was unable to handle the interference of the internal texture of the object. FRFCM and WSM dealt with the effect of noise and internal texture but failed to remove the interference caused by contrast. Moreover, WSM cannot obtain accurate edge information of targets. As shown in Figure 14, WSM expanded the object and the edge details disappeared. However, our method can deal with the interference made by noise. The NLTV spectral transform was sensitive to local contrast and size. Hence, it can separate the low-contrast words on the paper scrap. Because of the contrast and structure difference between the paper scrap and the background, the proposed method can separate the object from the background and extract the object’s edge details correctly. Table 4 shows that the proposed method has high FNR values. From Figure 14, the bottom edge in the results of the proposed method is a little expanded, and the left edge is obviously corroded. The main reason is that the morphological operator makes the segmentation result corroded.


### 4.6. Medical Image

The proposed method was evaluated on a medical image in this part. Because the medical image has a black background and the inference of speckle noise on the image is not obvious, the experiment was implemented on an image with Gaussian noise and Salt & Pepper noise. As Figure15 shows, the top row is a cell image, which is contaminated with Gaussian noise, and the bottom row is the cell image corrupted with Salt & Pepper noise. On account of the noise, the initial contour of the C-V model generated a local minimum problem and was unable to be iteratively converged. As a result, the segmentation results of the C-V model can only be around the initial contour. FCM had wrong results because of its sensitivity to noise. FRFCM obtained the best result on the cell image corrupted with Salt & Pepper noise. However, Gaussian noise can cause FRFCM to generate an over-segmentation. WSM can better remove the influence of Gaussian noise. However, WSM achieved high FPR values in the cell image corrupted with Salt & Pepper noise, which intends the over-segmentation. Nevertheless, our method obtained good segmentation performances in both noises. Our method achieved the best results on the image corrupted with Gaussian noise and obtained the second-best performance in Salt & Pepper noise. Table 5 shows that the proposed method achieves a high FNR value, which implies under-segmentation. As shown in the bottom row in Figure 15, the proposed method is difficult to segment the cells that have both small size and low contrast.Figure 15
Segmentation results on image “cell” corrupted by Gaussian and Salt & Pepper noise. (a) Original images. (b) Images (from top to bottom) that are corrupted with Gaussian (5% variance) and Salt & Pepper (5% density) noise, respectively. (c) C-V Model. (d) FCM. (e) FRFCM. (f) WSM (db). (g) WSM (haar). (h) Proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)Table 5
Evaluation metrics of compared methods for “cell” which are corrupted with gaussian (5% variance) and salt & pepper (5% density) noise.
NoiseMetricC-V modelFCMFRFCMWSM (db)WSM (haar)Proposed methodGaussianFPR0.48350.27750.13600.11700.10200.0019FNR0.79580.15770.02130.02770.03250.3540DICE0.09260.46880.66380.68920.71120.7409SA0.47790.73320.86400.87960.89200.9436Salt & PepperFPR0.12560.14880.00090.26660.24910.0180FNR0.24150.11400.38950.03760.05020.2780DICE0.60000.62160.81190.51630.52990.8026SA0.86400.84840.95620.75230.76720.9459The best two results are highlighted in bold and italics fonts.


## 4.1. Data and Settings

To evaluate the performance of the proposed method, synthetic, natural, and medical images are used for experiments. 1) The first experiment contains 3 groups of synthetic images whose textures are taken from the Brodatz Textures dataset [32]. Speckle, Salt & Pepper, and Gaussian noises are added to each group of synthetic images separately. 2) The second experiment contains 3 groups of natural images taken from the MSRA-1000 dataset [33]. 3) The third experiment contains 1 group of cell images, which is taken from the Fluo-N2DH-SIM + dataset [34]. Three different types of noises are also added to natural and medical images.We compare our segmentation method with four classical methods, i.e., the C-V model [13], FCM [14], FRFCM [19], and wavelet segmentation method (WSM) [27], which are used in the experiments. The experiments are implemented using the MATLAB R2020b platform and a PC with 16 GB RAM.The parameter settings for the proposed method are as follows: experiments show that when the image is transformed into the NLTV domain, detailed information is located in a low time scale. Large scale, which is close toT, contains less important information. Objects are mostly distributed in the middle scale. Hence, a middle-scale time range t1,t2 is selected. In the following experiments, t1 is set to T/5 and t2 is set to 3T/5. The parameters T and Δt are set to 9 and 0.03, respectively.


## 4.2. Quantitative Metrics

To quantitatively evaluate the performance of segmentation effect, four different metrics are chosen: FPR [21], FNR [21], dice similarity coefficient (DICE) [35], and segmentation accuracy (SA) [36].To measure the difference between segmentation results and ground truths, FPR and FNR are chosen in the subsequent experiments. The former calculates the number of background pixels classified as object pixels relative to the total background pixels. FNR measures the number of object pixels classified as background pixels relative to the total object pixels. FPR and FNR are defined as follows:(26)FPR=BR∩OGBG,FNR=OR∩BGOG,where BR and BG represent the number of background pixels in the segmentation results and ground truths, respectively. Additionally, OR and OG are the number of object pixels in the segmentation results and ground truths, respectively.DICE measures segmentation accuracy by calculating the degree of spatial overlap. Specifically, for the result regionA and target region B,(27)DICEA,B=2A∩BA+B,where ∩ means the intersection of two sets. The value range of DICE is [0, 1]. The higher DICE indicates that the segmentation result is more precise. DICEA,B=1 demonstrates that the segmentation result is the most complete, while DICEA,B=0 shows that the segmentation result is the worst.Another evaluation metric is SA, which can assess the number of well-classified pixels in the image. The definition of SA is given as follows:(28)SA=∑i=1NfitruthN,where fitruth means the correctly segmented pixel and N denotes the total number of pixels in an image.


## 4.3. Parameter Analysis

This section analyzes the effects ofΔt and T on the segmentation results of the proposed method through an experiment. The experiment was carried out on MSRA-1000, and the average SA was used as an indicator to show the influence of two parameters on the segmentation accuracy. The average SA was calculated by averaging the SA of all images on the dataset. The parameter Δt ranges from 0.01 to 0.1, and the step is 0.01. Additionally, the maximal time scale T ranges from 1 to 10, and the interval is 1. Figure 10 demonstrates the results for different Δt and T. The proposed method achieves the best performance when Δt=0.03 and T=9.Figure 10
Effects of differentΔt and T on the average SA.


## 4.4. Synthetic Images

The first experiment was implemented on three synthetic images, which are shown in Figure11. The first row shows a synthetic image containing multiple repeating structures and a dark grid-like background. A simple synthetic image, which has an irregular object, is arranged in the middle row. The object in the bottom row is complex and has a texture with inhomogeneous contrast. Moreover, three images are separately contaminated with Speckle (10% variance), Salt & Pepper (10% density), and Gaussian (10% variance) noise.Figure 11
Segmentation results on synthetic images corrupted by different noises. (a) Original images. (b) Images (from top to bottom) that are corrupted with Speckle (10% variance), Salt & Pepper (10% density), and Gaussian noises (10% variance), respectively. (c) C-V Model. (d) FCM. (e) FRFCM. (f) WSM (db). (g) WSM (haar). (h) Proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)Table1 lists the quantitative evaluations of different segmentation methods on various images. Combining with Figure 11 and Table 1, FCM got wrong segmentation results because of its sensitivity to noise. FRFCM achieved a good result on the first image and got a high DICE and SA value as shown in Table 1. However, it failed to distinguish the second and the third image because of the inhomogeneous contrast. WSM, which is based on spectral analysis, can remove the influence of noise. However, as Figure 11 shows, WSM oversmoothed the edge and damaged the edge details. Meanwhile, WSM was unable to segment objects accurately on the second and third images. The reason is that WSM is sensitive to inhomogeneous contrast. The C-V model obtained the segmentation results of all images more correctly. One of the reasons was that the C-V model relies on an initial contour, which provides prior information about the approximate position of the object. Nevertheless, the C-V model was sensitive to noise. On the second and third images, the C-V model was unable to accurately segment the targets. The noises slowed down the convergence speed of the algorithm and made the method fall into the local minimum problem. However, the proposed method achieved the best results in all methods. The NLTV spectral transform-based method can segment the objects exactly and can reduce the influence of inhomogeneous contrast at the same time. The reason is that our method can segment objects, combining object size, contrast, and structures. As shown in Table 1, the proposed method got a high FNR on the second synthetic image, which intended an under-segmentation. The problem was caused by the morphological operators in the output of the proposed method, which may cause edge corrodes.Table 1
Evaluation metrics of compared methods for synthetic images, which are corrupted with Speckle (10% variance), Salt & Pepper (10% density), and Gaussian noise (10% variance), respectively.
ImageMetricC-V modelFCMFRFCMWSM (db)WSM (haar)Proposed method1FPR0.00250.04120.00100.05800.04020.0009FNR0.05010.19770.06550.04280.04920.0290DICE0.95960.75640.96040.78890.83280.9821SA0.99070.93970.99100.94080.95580.99562FPR0.01670.21110.13550.02410.03260.0038FNR0.02310.07430.01870.03890.03680.0273DICE0.96660.75310.84270.95100.94150.9798SA0.98080.82730.89590.97180.96590.98843FPR0.03580.31510.23520.19630.19840.0235FNR0.02890.09010.02170.02530.02550.0229DICE0.90750.54460.64020.67750.67510.9333SA0.96410.72490.80150.83260.83080.9748The best two results are highlighted in bold and italics fonts.


## 4.5. Nature Images

To further discuss the proposed method’s segmentation ability for images with various noises, the second experiment was performed on three natural images, which are shown in Figures12, 13, and 14. The object that has a similar contrast to the surroundings is shown in Figure 12. Figure 13 displays a complex scene that has lots of tiny structures in the background. The object in Figure 14 is a piece of paper containing words, and the words will interfere with segmentation methods. Moreover, three images are separately contaminated with Speckle (10%, 20%, and 30% variance), Salt & Pepper (10%, 20%, and 30% density), and Gaussian (10%, 20%, and 30% variance) noise.Figure 12
Segmentation results on image “star” corrupted by Speckle noise. (a) Original images. (b) Images (from top to bottom) that are corrupted with 10%, 20%, and 30% variance of Speckle noise, respectively (c) C-V model. (d) FCM. (e) FRFCM; (f) WSM (db); (g) WSM (haar); (h) Proposed method.Figure 13
Segmentation results on image “snowman” corrupted by Salt & pepper noise. (a) Original images. (b) Images (from top to bottom) that are corrupted with 10%, 20%, and 30% variance of Speckle noise, respectively. (c) C-V Model. (d) FCM; (e) FRFCM. (f) WSM (db). (g) WSM (haar). (h) Proposed method.Figure 14
Segmentation results on image “paper scrap” corrupted by Gaussian noise. (a) Original images. (b) Images (from top to bottom) that are corrupted with 10%, 20%, and 30% variance of Speckle noise, respectively. (c) C-V Model. (d) FCM. (e) FRFCM. (f) WSM (db). (g) WSM (haar). (h) Proposed method.As Figure12 shows, the object has a similar contrast to the surrounding border. FCM separated the noise while segmenting the object because of its sensitivity to noise. FRFCM had better results than FCM, however, it still had wrong segmentation for noise. WSM can remove the influence of noise. However, WSM failed to remove the impact of inhomogeneous contrast. The C-V model achieved accurate segmentation results because of its initial contour. From Table 2, it can be seen that the C-V model had similar DICE and SA values with the proposed method. However, the C-V model was difficult to segment corner structure because of noises. The proposed method can better preserve structural information while segmenting.Table 2
Evaluation metrics of compared methods for “star,” which are corrupted with Speckle (10%, 20%, and 30% variance) noise.
Noise level (%)MetricC-V modelFCMFRFCMWSM (db)WSM (haar)Proposed method10FPR0.00140.32990.21720.29530.29340.0010FNR0.06360.11700.14560.02280.00440.0544DICE0.96120.26330.35530.30670.31170.9667SA0.99530.68230.79100.71890.72130.995720FPR0.00140.34310.26770.29790.29350.0011FNR0.05730.12960.03540.01210.01070.0578DICE0.96160.25340.32940.30710.31090.9649SA0.99500.66960.74610.71690.72120.995530FPR0.00100.34540.26680.30010.29130.0010FNR0.07180.11550.09470.01120.01700.0584DICE0.95990.25580.32070.30590.31130.9652SA0.99490.66830.74600.71500.72310.9950The best two results are highlighted in bold and italics fonts.Figure13 shows the algorithms’ performances on the natural images, which are corrupted by Salt & Pepper noise. Table 3 shows the corresponding quantitative metrics. In Figure 13, there are lots of small objects in the background, which have a similar contrast to the object. When these small targets are contaminated with Salt & Pepper noise, they cause serious interference with segmentation methods, which mainly rely on contrast. Figure 13 shows that WSM has a good result; however, it is unable to segment the areas surrounding the object correctly. Table 4 shows that the C-V model has better results than WSM; however, it still has an incorrect segmentation of the background. Because of the sensitivity of the NLTV spectral transform to contrast, size, and structures, the proposed method can still separate objects when the background has small size structures.Table 3
Evaluation metrics of compared methods for “snowman,” which are corrupted with Speckle (10%, 20%, and 30% variance) noise.
Noise level (%)MetricC-V modelFCMFRFCMWSM (db)WSM (haar)Proposed method10FPR0.01370.14060.07370.05320.04670.0049FNR0.05770.17560.04080.08480.09830.0890DICE0.94720.69880.85560.86420.86750.9524SA0.97830.85310.93320.94060.94310.981020FPR0.01400.18090.07630.05700.04970.0053FNR0.05530.17680.04020.07640.08830.0979DICE0.94830.65460.85200.86220.86820.9505SA0.97870.82070.93120.93910.94280.980430FPR0.02140.22630.07850.05420.05480.0175FNR0.05290.16510.03800.08800.07640.0997DICE0.93450.61780.85050.86030.86560.9479SA0.97250.78700.93010.93890.94080.9732The best two results are highlighted in bold and italics fonts.Table 4
Evaluation metrics of compared methods for “paper scrap,” which are corrupted with Speckle (10%, 20%, and 30% variance) noise.
Noise level (%)MetricC-V modelFCMFRFCMWSM (db)WSM (haar)Proposed method10FPR0.00130.41300.41380.50430.50630.0002FNR0.13020.09620.02650.01540.02300.0688DICE0.92650.38130.40230.35850.35570.9754SA0.98090.62710.63400.55570.55330.993920FPR0.00160.42360.41580.50590.50770.0005FNR0.13100.11440.02020.02250.01620.0743DICE0.92500.36860.40370.35620.35680.9725SA0.98050.61520.63330.55380.55270.993230FPR0.00160.42390.41640.51700.51370.0011FNR0.13310.11290.02300.01680.01880.0893DICE0.92390.36930.40340.35250.35340.9661SA0.98030.61540.6330.54460.54720.9917The best two results are highlighted in bold and italics fonts.Figure14 shows the segmentation results on the natural image when it is corrupted with different levels of Gaussian noise. Table 4 shows the corresponding quantitative metrics of algorithms. The natural image is difficult for segmentation methods because it has complex texture like words inside, which will affect the integrity of the segmentation results. The C-V model was capable of dealing with the background, however, it was unable to handle the interference of the internal texture of the object. FRFCM and WSM dealt with the effect of noise and internal texture but failed to remove the interference caused by contrast. Moreover, WSM cannot obtain accurate edge information of targets. As shown in Figure 14, WSM expanded the object and the edge details disappeared. However, our method can deal with the interference made by noise. The NLTV spectral transform was sensitive to local contrast and size. Hence, it can separate the low-contrast words on the paper scrap. Because of the contrast and structure difference between the paper scrap and the background, the proposed method can separate the object from the background and extract the object’s edge details correctly. Table 4 shows that the proposed method has high FNR values. From Figure 14, the bottom edge in the results of the proposed method is a little expanded, and the left edge is obviously corroded. The main reason is that the morphological operator makes the segmentation result corroded.


## 4.6. Medical Image

The proposed method was evaluated on a medical image in this part. Because the medical image has a black background and the inference of speckle noise on the image is not obvious, the experiment was implemented on an image with Gaussian noise and Salt & Pepper noise. As Figure15 shows, the top row is a cell image, which is contaminated with Gaussian noise, and the bottom row is the cell image corrupted with Salt & Pepper noise. On account of the noise, the initial contour of the C-V model generated a local minimum problem and was unable to be iteratively converged. As a result, the segmentation results of the C-V model can only be around the initial contour. FCM had wrong results because of its sensitivity to noise. FRFCM obtained the best result on the cell image corrupted with Salt & Pepper noise. However, Gaussian noise can cause FRFCM to generate an over-segmentation. WSM can better remove the influence of Gaussian noise. However, WSM achieved high FPR values in the cell image corrupted with Salt & Pepper noise, which intends the over-segmentation. Nevertheless, our method obtained good segmentation performances in both noises. Our method achieved the best results on the image corrupted with Gaussian noise and obtained the second-best performance in Salt & Pepper noise. Table 5 shows that the proposed method achieves a high FNR value, which implies under-segmentation. As shown in the bottom row in Figure 15, the proposed method is difficult to segment the cells that have both small size and low contrast.Figure 15
Segmentation results on image “cell” corrupted by Gaussian and Salt & Pepper noise. (a) Original images. (b) Images (from top to bottom) that are corrupted with Gaussian (5% variance) and Salt & Pepper (5% density) noise, respectively. (c) C-V Model. (d) FCM. (e) FRFCM. (f) WSM (db). (g) WSM (haar). (h) Proposed method.
(a)(b)(c)(d)(e)(f)(g)(h)Table 5
Evaluation metrics of compared methods for “cell” which are corrupted with gaussian (5% variance) and salt & pepper (5% density) noise.
NoiseMetricC-V modelFCMFRFCMWSM (db)WSM (haar)Proposed methodGaussianFPR0.48350.27750.13600.11700.10200.0019FNR0.79580.15770.02130.02770.03250.3540DICE0.09260.46880.66380.68920.71120.7409SA0.47790.73320.86400.87960.89200.9436Salt & PepperFPR0.12560.14880.00090.26660.24910.0180FNR0.24150.11400.38950.03760.05020.2780DICE0.60000.62160.81190.51630.52990.8026SA0.86400.84840.95620.75230.76720.9459The best two results are highlighted in bold and italics fonts.


## 5. Conclusion

We have analyzed the properties of NLTV spectral transform with the help of theoretical proof and experiments. Our analyses demonstrate that the object in an image corrupted with various noises can be separated its size, contrast, and detailed structure. The analyses also illustrate that the objects with same structures have similar descriptions in the NLTV spectral domain.Furthermore, we have developed a novel transform-based method that segments images based on the NLTV spectral transform. The approach, firstly, decomposes an image into many sub-bands in the NLTV spectral domain and utilizes the max response time to represent the image features. Then, to better divide the object and background, the sub-bands in the NLTV spectral domain are filtered by fitting the separation surface, which is calculated based on maximum response time. Next, the filtered image is reconstructed by an inverse transform to obtain the rough segmentation result. Finally, the segmentation mask is calculated using postprocess methods. Subjective and objective evaluations show that the proposed method effectively protects the edge details while segmenting the object in a variety of noises.However, one limitation of the proposed method is the high computational cost since the computation of nonlocal operators needs a long time and large memory storage. The other limitation of the method is the difficulty in fitting multiple separation surfaces accurately. We attempt to solve the aforementioned problems and develop a fast multiobject segmentation method in future work.



---

*Source: 1442745-2022-02-24.xml*