# Edge Detection in Color Images Based on DSmT

Jean Dezert, Zhun-Ga Liu, Grégoire Mercier

► To cite this version:

Jean Dezert, Zhun-Ga Liu, Grégoire Mercier. Edge Detection in Color Images Based on DSmT. 14th International Conference on Information Fusion (FUSION), Jul 2011, Chicago, United States. hal-01219500

HAL Id: hal-01219500

https://hal.science/hal-01219500

Submitted on 14 Jun 2021

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Edge Detection in Color Images Based on DSMT

Jean Dezert<fnref n="1" />, Zhun-ga Liu²,³, Grégoire Mercier³

1. Onera - The French Aerospace Lab, F-91761 Palaiseau, France, Email: jean.dezert@onera.fr

2. School of Automation, Northwestern Polytechnical University, Xi'an, China, Email: liuzhunga@gmail.com

3. Télécom Bretagne, Technopôle Brest-Iroise, 29238, France, Email: Gregoire.Mercier@telecom-bretagne.eu

**Abstract**—In this paper, we present a non-supervised methodology for edge detection in color images based on belief functions and their combination. Our algorithm is based on the fusion of local edge detectors results expressed into basic belief assignments thanks to a flexible modeling, and the proportional conflict redistribution rule developed in DSMT framework. The application of this new belief-based edge detector is tested both on original (noise-free) Lena's picture and on a modified image including artificial pixel noises to show the ability of our algorithm to work on noisy images too.

**Keywords:** Edge detection, image processing, DSMT, DST, fusion, belief functions.

## I. INTRODUCTION

Edge detection is one of most important tasks in image processing and its application to color images is still subject to a very strong interest [8], [10]–[12], [14] for example in teledetection, in remote sensing, target recognition, medical diagnosis, computer vision and robotics, etc. Most of basic image processing algorithms developed in the past for gray-scale images have been extended to multichannel images. Edge detection algorithms for color images have been classified into three main families [15]: 1) fusion methods, 2) multidimensional gradient methods and 3) vector methods depending on the position of where the recombination step applies [7]. In this paper, the method we propose uses a fusion method with a multidimensional gradient method. Our new unsupervised edge detector combines the results obtained by gray-scale edge detectors for individual color channels [3] to define bba's from the gradient values which are combined using Dezert-Smarandache Theory [17] (DSMT) of plausible and paradoxical reasoning for information fusion. DSMT has been proved to be a serious alternative to well-known Dempster-Shafer Theory of mathematical evidence [16] specially for dealing with highly conflicting sources of evidences. Some supervised edge detectors based on belief functions computed from gaussian pdf assumptions and Dempster-Shafer Theory can be found in [1], [21]. In this work, we show through very simple examples how edge detection can be performed based on DSMT fusion techniques with belief functions without learning (supervision). The interest for using belief functions for edge detection comes from their ability to model more adequately uncertainties with respect to the classical probabilistic modeling approach, and to deal with conflicting information due to spatial changes in the image or noises. This paper is organized as follows: In section 2 we briefly recall the basics of DSMT and the fusion rule we use. In section 3, we present

in details our new edge detector based on belief functions and their fusion. Results of our new algorithm tested on the original Lena's picture and its noisy version are presented in section 4 with a comparison to the classical Canny's edge detector. Conclusions and perspectives are given in section 5.

## II. BASICS OF DSMT

The purpose of DSMT [17] is to overcome the limitations of DST [16] mainly by proposing new underlying models for the frames of discernment in order to fit better with the nature of real problems, and proposing new efficient combination and conditioning rules. In DSMT framework, the elements $\theta_i$, $i = 1, 2, \dots, n$ of a given frame $\Theta$ are not necessarily exclusive, and there is no restriction on $\theta_i$ but their exhaustivity. The hyper-power set $D^\Theta$ in DSMT, the hyper-power set is defined as the set of all composite propositions built from elements of $\Theta$ with operators $\cup$ and $\cap$. For instance, if $\Theta = \{\emptyset, \theta_1, \theta_2, \theta_1 \cap \theta_2, \theta_1 \cup \theta_2\}$. A (generalized) basic belief assignment (bba for short) is defined as the mapping $m : D^\Theta \to [0, 1]$. The generalized belief and plausibility functions are defined in almost the same manner as in DST. More precisely, from a general frame $\Theta$, we define a map $m(\cdot) : D^\Theta \to [0, 1]$ associated to a given body of evidence $B$ as

$$m(\emptyset) = 0 \quad \text{and} \quad \sum_{A \in D^\Theta} m(A) = 1 \quad (1)$$

The quantity $m(A)$ is called the generalized basic belief assignment/mass (or just "bba" for short) of $A$.

The generalized credibility and plausibility functions are defined in almost the same manner as within DST, i.e.

$$\text{Bel}(A) = \sum_{\substack{B \subseteq A \\ B \in D^{\Theta}}} m(B) \quad \text{and} \quad \text{Pl}(A) = \sum_{\substack{B \cap A \neq \emptyset \\ B \in D^{\Theta}}} m(B) \quad (2)$$

Two models<fnref n="1" /> (the free model and hybrid model) in DSMT can be used to define the bba's to combine. In the free DSMT model, the sources of evidence are combined without taking into account integrity constraints. When the free DSMT model does not hold because the true nature of the fusion problem under consideration, we can take into account some known integrity constraints and define bba's to combine using the proper hybrid DSMT model. All details of DSMT with

<footnote n="1">¹Actually, Shafer's model, considering all elements of the frame as truly exclusive, can be viewed as a special case of hybrid model.</footnote>
many examples can be easily found in [17] available freely on the web. In this paper, we will work only with Shafer's model of the frame where all elements $\theta_i$ of $\Theta$ are assumed truly exhaustive and exclusive (disjoint) and therefore $D^\Theta$ reduces the classical power set $2^\Theta$ and generalized belief functions reduces to classical ones as within DST framework. Aside offering the possibility to work with different underlying models (not only Shafer's model as within DST), DSmT offers also new efficient combination rules based on proportional conflict redistribution (PCR rules no 5 and no 6) for combining highly conflicting sources of evidence. In DSmT framework, the classical pignistic transformation $BetP(\cdot)$ is replaced by the one by the more effective DSmP(\cdot) transformation to estimate the subjective probabilities of hypotheses for decision-making support once the combination of bba's has been obtained. Before presenting our new edge detector, we just recall briefly what are the PCR5 fusion rule and the DSmP transformation. All details, justifications with examples on PCR5 and DSmP can be found freely from the web in [17], Vols. 2 & 3 and will not be reported here.

### A. PCR5 fusion rule

The Proportional Conflict Redistribution Rule no. 5 (PCR5) is used generally to combine bba's in DSmT framework. PCR5 transfers the conflicting mass only to the elements involved in the conflict and proportionally to their individual masses, so that the specificity of the information is entirely preserved in this fusion process. Let $m_1(\cdot)$ and $m_2(\cdot)$ be two independent<fnref n="2" /> bba's, then the PCR5 rule is defined as follows (see [17], Vol. 2 for full justification and examples): $m_{PCR5}(\emptyset) = 0$ and $\forall X \in 2^\Theta \setminus \{\emptyset\}$

$$ m_{PCR5}(X) = \sum_{\substack{X_1, X_2 \in 2^\Theta \\ X_1 \cap X_2 = X}} m_1(X_1)m_2(X_2) + \sum_{\substack{X_2 \in 2^\Theta \\ X_2 \cap X = \emptyset}} \left[ \frac{m_1(X)^2 m_2(X_2)}{m_1(X) + m_2(X_2)} + \frac{m_2(X)^2 m_1(X_2)}{m_2(X) + m_1(X_2)} \right] \quad (3) $$

where all denominators in (3) are different from zero. If a denominator is zero, that fraction is discarded. Additional properties of PCR5 can be found in [5]. Extension of PCR5 for combining qualitative bba's can be found in [17], Vol. 2 & 3. All propositions/sets are in a canonical form. A variant of PCR5, called PCR6 has been proposed by Martin and Osswald in [17], Vol. 2, for combining $s > 2$ sources. The general formulas for PCR5 and PCR6 rules are given in [17], Vol. 2 also. PCR6 coincides with PCR5 when one combines two sources. The difference between PCR5 and PCR6 lies in the way the proportional conflict redistribution is done as soon as three or more sources are involved in the fusion. For example, let's consider three sources with bba's $m_1(\cdot)$, $m_2(\cdot)$ and $m_3(\cdot)$, $A \cap B = \emptyset$ for the model of the frame $\Theta$, and $m_1(A) = 0.6$, $m_2(B) = 0.3$, $m_3(B) = 0.1$. With PCR5 the partial conflicting mass $m_1(A)m_2(B)m_3(B) = 0.6 \cdot 0.3 \cdot 0.1 = 0.018$

is redistributed back to A and B only with respect to the following proportions respectively: $x_A^{PCR5} = 0.01714$ and $x_B^{PCR5} = 0.00086$ because the proportionalization requires

$$ \frac{x_A^{PCR5}}{m_1(A)} = \frac{x_B^{PCR5}}{m_2(B)m_3(B)} = \frac{m_1(A)m_2(B)m_3(B)}{m_1(A) + m_2(B)m_3(B)} $$

$$ \text{that is} \quad \frac{x_A^{PCR5}}{0.6} = \frac{x_B^{PCR5}}{0.03} = \frac{0.018}{0.6 + 0.03} \approx 0.02857 $$

$$ \text{thus} \quad \begin{cases} x_A^{PCR5} = 0.60 \cdot 0.02857 \approx 0.01714 \\ x_B^{PCR5} = 0.03 \cdot 0.02857 \approx 0.00086 \end{cases} $$

With the PCR6 fusion rule, the partial conflicting mass $m_1(A)m_2(B)m_3(B) = 0.6 \cdot 0.3 \cdot 0.1 = 0.018$ is redistributed back to A and B only with respect to the following proportions respectively: $x_A^{PCR6} = 0.0108$ and $x_B^{PCR6} = 0.0072$ because the PCR6 proportionalization is done as follows:

$$ \frac{x_A^{PCR6}}{m_1(A)} = \frac{x_{B,2}^{PCR6}}{m_2(B)} = \frac{x_{B,3}^{PCR6}}{m_3(B)} = \frac{m_1(A)m_2(B)m_3(B)}{m_1(A) + m_2(B) + m_3(B)} $$

that is

$$ \frac{x_A^{PCR6}}{0.6} = \frac{x_{B,2}^{PCR6}}{0.3} = \frac{x_{B,3}^{PCR6}}{0.1} = \frac{0.018}{0.6 + 0.3 + 0.1} = 0.018 $$

thus

$$ \begin{cases} x_A^{PCR6} = 0.6 \cdot 0.018 = 0.0108 \\ x_{B,2}^{PCR6} = 0.3 \cdot 0.018 = 0.0054 \\ x_{B,3}^{PCR6} = 0.1 \cdot 0.018 = 0.0018 \end{cases} $$

and therefore with PCR6, one gets finally the following redistributions to A and B:

$$ \begin{cases} x_A^{PCR6} = 0.0108 \\ x_B^{PCR6} = x_{B,2}^{PCR6} + x_{B,3}^{PCR6} = 0.0054 + 0.0018 = 0.0072 \end{cases} $$

From the implementation point of view, PCR6 is simpler to implement than PCR5. Very basic Matlab codes for PCR5 and PCR6 fusion rules can be found in [17], [18].

### B. DSmP transformation

DSmP probabilistic transformation is a serious alternative to the classical pignistic transformation which allows to increase the probabilistic information content (PIC), i.e. to reduce Shannon entropy, of the approximated subjective probability measure drawn from any bba. Justification and comparisons of DSmP(\cdot) w.r.t. $BetP(\cdot)$ and to other transformations can be found in details in [6], [17], Vol. 3, Chap. 3. DSmP transformation is defined<fnref n="3" /> by $DSmP_{\epsilon}(\emptyset) = 0$ and $\forall X \in 2^\Theta \setminus \{\emptyset\}$

$$ DSmP_{\epsilon}(X) = \sum_{Y \in 2^{\Theta}} \frac{\sum_{Z \subseteq X \cap Y} m(Z) + \epsilon \cdot |X \cap Y|}{\sum_{\substack{Z \subseteq Y \\ |Z|=1}} m(Z) + \epsilon \cdot |Y|} m(Y) \quad (4) $$

<footnote n="2">²I.e. each source provides its bba independently of the other sources.</footnote>

<footnote n="3">³Here we work on classical power-set, but DSmP can be defined also for working with other fusion spaces, hyper-power sets or super-power sets if necessary.</footnote>
where $|X \cap Y|$ and $|Y|$ denote the cardinals of the sets $X \cap Y$ and $Y$ respectively; $\epsilon \ge 0$ is a small number which allows to increase the PIC value of the approximation of $m(.)$ into a subjective probability measure. Usually $\epsilon = 0$, but in some particular degenerate cases, when the $DSmP_{\epsilon=0}(.)$ values cannot be derived, the $DSmP_{\epsilon>0}$ values can however always be derived by choosing $\epsilon$ as a very small positive number, say $\epsilon = 1/1000$ for example in order to be as close as we want to the highest value of the PIC. The smaller $\epsilon$, the better/bigger PIC value one gets. When $\epsilon = 1$ and when the masses of all elements $Z$ having $|Z| = 1$ are zero, $DSmP_{\epsilon=1}(.) = BetP(.)$, where the pignistic transformation $BetP(.)$ is defined by [19]:

$$BetP\{X\} = \sum_{Y \in 2^{\Theta}} \frac{|Y \cap X|}{|Y|} m(Y) \quad (5)$$

with convention $|\emptyset|/|\emptyset| = 1$.

### C. DS combination rule

Dempster-Shafer (DS) rule of combination is the main historical (and still widely used) rule proposed by Glenn Shafer in his milestone book [16]. Very passionate debates have emerged in the literature about the justification and the behavior of this rule from the famous Zadeh's criticism in [22]. We don't plan to reopen this endless debate and just want to recall briefly here how it is mathematically defined. Let's consider a given discrete and finite frame of discernment $\Theta = \{\theta_1, \theta_2, \dots, \theta_n\}$ of exclusive and exhaustive hypotheses (a.k.a satisfying Shafer's model) and two independent bba's $m_1(.)$ and $m_2(.)$ defined on $2^\Theta$, then DS rule of combination is defined by $m_{DS}(\emptyset) = 0$ and $\forall X \neq \emptyset$ and $X \in 2^\Theta$:

$$m_{DS}(X) = \frac{1}{1 - K_{12}} \sum_{\substack{X_1, X_2 \in 2^\Theta \\ X_1 \cap X_2 = X}} m_1(X_1) m_2(X_2) \quad (6)$$

where $K_{12} \triangleq \sum_{X_1, X_2 \in 2^\Theta} m_1(X_1) m_2(X_2)$ represents the total conflict between sources. If $K_{12} = 1$, the sources of evidence are in full conflict and DS rule cannot be applied. DS rule is commutative and associative and can be extended for the fusion of $s > 2$ sources as well. The main criticism about such concerns its unexpected/counter-intuitive behavior as soon as the degree of conflict between sources becomes high (see [17], Vol.1, Chapter 5 and references therein for details and examples).

### D. Decision-making support

Decisions are achieved by computing the expected utilities of the acts using either the subjective/pignistic $BetP\{\cdot\}$ (usually adopted in DST framework) or $DSmP(.)$ (as suggested in DSmT framework) as the probability function needed to compute expectations. Usually, one uses the maximum of the pignistic probability as decision criterion. The maximum of $BetP\{\cdot\}$ is often considered as a prudent betting decision criterion between the two other decision strategies (max of

plausibility or max. of credibility which appears to be respectively too optimistic or too pessimistic). It is easy to show that $BetP\{\cdot\}$ is indeed a probability function (see [19], [20]) as well as $DSmP(.)$ (see [17], Vol.2). The max of $DSmP(.)$ is considered as more efficient for practical applications since $DSmP(.)$ is more informative (it has a higher PIC value) than $BetP(.)$ transformation.

### III. EDGE DETECTION BASED ON DSmT AND FUSION

In this work, we use the most common RGB (Red-Green-Blue) representation of the digital color image where each layer (channel) R, G and B consists in a matrix of $n_i \times n_j$ pixels. The discrete value of each pixel in a given color channel is assumed in a given absolute interval of color intensity $[c_{min}, c_{max}]$. The principle of our new Edge detector based on DSmT is very simple and consists in the following steps:

#### A. Step 1: Construction of bba's

Let's consider a given channel (color layer) and denote it as $L$ which can represent either the Red (R) color layer, the Green (G) color layer or the Blue (B) color layer, or any other channel in a more general case for multispectral images. For simplicity, we focus our work and presentation here on color images only.

Apply an edge detector algorithm for each color channel $L$ to get for each pixel $x_{ij}^L$, $i = 1, 2, \dots, n_i$, $j = 1, 2, \dots, n_j$ an associated bba $m_{ij}^L(.)$ expressing the local belief that this pixel belongs or not to an edge. The frame of discernment $\Theta$ used to define the bba's is very simple and is defined as

$$\Theta = \{\theta_1 \triangleq \text{Pixel} \in \text{Edge}, \theta_2 \triangleq \text{Pixel} \notin \text{Edge}\} \quad (7)$$

$\Theta$ is assumed to satisfy Shafer's model (i.e. $\theta_1 \cap \theta_2 = \emptyset$). It is clear that many (binary) edge detection algorithms are available in the image processing literature but here we want a "smooth" algorithm able to provide both the belief of each pixel to belong or not to an edge and also the uncertainty one has on the classification of this pixel. In this subsection, we present a very simple algorithm for accomplishing this task at the color channel level. Obviously the quality of the algorithm used in this first step will have a strong impact of the final result and therefore it is important to focus research efforts on the development of efficient algorithms for realizing this step as best as possible.

As in Sobel method [9], two $3 \times 3$ kernels are convolved with the original image $A^L$ for each layer $L$ to calculate approximations of the derivatives - one for horizontal changes, and one for vertical. We then obtain two gradient images $G_x^L$ and $G_y^L$ for each layer $L$ represent the horizontal and vertical derivative approximations for each pixel $x_{ij}^L$. The x-coordinate is defined as increasing in the right-direction, and the y-coordinate is as increasing in the down-direction. At each pixel $x_{ij}^L$ of the color layer $L$, the gradient magnitude $g_{ij}^L$ can be estimated by the combination of the two gradient approximations as:

$$g_{ij}^L = \frac{1}{\sqrt{2}} (G_x^L(i, j)^2 + G_y^L(i, j)^2)^{1/2} \quad (8)$$
where

$$G_x^L = \frac{1}{8} \begin{pmatrix} -1 & 0 & 1 \\ -2 & 0 & 2 \\ -1 & 0 & 1 \end{pmatrix} * A^L;$$

$$G_y^L = \frac{1}{8} \begin{pmatrix} -1 & -2 & -1 \\ 0 & 0 & 0 \\ 1 & 2 & 1 \end{pmatrix} * A^L;$$

and where * denotes the 2-dimensional convolution operation.

In Sobel's detection method, the edge detection for a pixel $x_{ij}$ of a gray image is declared based on a hard thresholding of $g_{ij}$ value. Such Sobel detector is sensitive to noise and it can generate false alarms. In this work, $g_{ij}^L$ values are used only to define the mass function (bba) of each pixel in each layer over the power-set of $\Theta$ defined in (7). If the value $g_{ij}^L$ value of a pixel is big, it implies that this pixel is more likely to belong to an edge. If $g_{ij}^L$ value of the pixel $x_{ij}^L$ is low then our belief that it belongs to an edge must be low too. Such very simple and intuitive modeling can be obtained directly from the sigmoid functions commonly used as activation function in neural networks, or as fuzzy membership in the fuzzy subsets theory as explained below.

Let's consider the sigmoid function defined as

$$f_{\lambda,t}(g) \triangleq \frac{1}{1 + e^{-\lambda(g-t)}} \quad (9)$$

$g$ is the gradient magnitude of the pixel under consideration. $t$ is the abscissa of the inflection point of the sigmoid which can be selected by $t = p \cdot \max(g)$ where $p$ is a proportion parameter and $\cdot$ is the scalar product operator. When working with noisy images, $p$ always increases with the level of noise. $\lambda$ is the slope of the tangent at the inflection point.

It can be easily verified that the bba $m_{ij}^L(\cdot|g_{ij}^L)$ satisfying the expected behavior can be obtained by the fusion<fnref n="4" /> of the two following simple bba's defined by:

<table><thead><tr><th>focal element</th><th>$m_1(.)$</th><th>$m_2(.)$</th></tr></thead><tbody><tr><td>$\theta_1$</td><td>$f_{\lambda,t_e}(g)$</td><td>0</td></tr><tr><td>$\theta_2$</td><td>0</td><td>$f_{-\lambda,t_n}(g)$</td></tr><tr><td>$\theta_1 \cup \theta_2$</td><td>$1 - f_{\lambda,t_e}(g)$</td><td>$1 - f_{-\lambda,t_n}(g)$</td></tr></tbody></table>

with $0 < t_n < t_e < 255$, $\lambda > 0$.

$t_e$ is the lower threshold for the edge detection, and $t_n$ is the upper threshold for the non edge detection. Thus, $[t_n, t_e]$ corresponds to our uncertainty decision zone and the $g_{ij}^L$ values lying in this interval correspond to the unknown decision state. The bounds (thresholds) $t_n$ and $t_e$ can be tuned based on the average gradients values of the image, and the length $t_e - t_n$ depends on the level of the noise. If the image is very noisy, it means the information is very uncertain, and the length of the interval $[t_n, t_e]$ can become large. Otherwise, it is small. Because of structure of these two simple bba's,

the fusion obtained with PCR5, DS of even with DSm hybrid (DSmH) rules of combination provide globally similar results and therefore the choice of the fusion rule here does not really matter to build $m_{ij}^L(\cdot|g_{ij}^L)$ as shown on the figures 1-3. PCR5, which is the most specific fusion rule (it reduces the level of belief committed to the uncertainty), is used in this work to generate $m_{ij}^L(\cdot|g_{ij}^L)$.

<figure><img src="image_2.png" /><figcaption>Figure 1. Computation of $m_{ij}^L(\cdot|g_{ij}^L)$ from $m_1(.)$ and $m_2(.)$ with $[t_n, t_e] = [60, 100]$ and $\lambda = 0.09$.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Figure 2. Computation of $m_{ij}^L(\cdot|g_{ij}^L)$ from $m_1(.)$ and $m_2(.)$ with $[t_n, t_e] = [50, 80]$ and $\lambda = 0.06$.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Figure 3. Computation of $m_{ij}^L(\cdot|g_{ij}^L)$ from $m_1(.)$ and $m_2(.)$ with $[t_n, t_e] = [30, 40]$ and $\lambda = 0.04$.</figcaption></figure>

<footnote n="4">⁴with DS, PCR5 or even with DSmH rule [17].</footnote>
In summary, $m_{ij}^L(\cdot|g_{ij}^L)$ can be easily constructed from the choice of thresholding parameters $t_e, t_n$ defining the uncertainty zone of the gradient values, the slope parameter $\lambda$ of sigmoids, and of course from the gradient magnitude $g_{ij}^L$. This approach is very easy to implement and very flexible since it depends on the parameters which are totally under the control of the user.

## B. Step 2: Fusion of bba's $m_{ij}^L(\cdot)$

Many combination rules like DS rule, Dubois & Prade rule Yager's rule, and so on can be used with our approach. In this work, we just make investigations based on the two most well-known rules (DS and PCR5 rule proposed in DST and DSmT respectively). So we use either DS or PCR5 rule to combine the three bba's $m_{ij}^R(\cdot), m_{ij}^G(\cdot)$ and $m_{ij}^B(\cdot)$ for each pixel $x_{ij}$ in order to get the global bba $m_{ij}(\cdot)$ to estimate the degree of belief of the belonging of $x_{ij}$ to an edge in the given image. Since PCR5 is not associative, we must apply the general PCR5 formula for combining the 3 sources (channels) altogether<fnref n="5" /> as explained in details in [17], Vol.2, Chap. 1 & 2. A suboptimal approach requiring less computations would consist in applying a PCR5 sequential fusion of these bba's in such a way that the two least conflicting bba's are combined at first by PCR5 and then combine again with PCR5 the resulting bba's with the third one according to (3). The more simple PCR6 rule could also be used instead of PCR5 as well - see [17], Vol. 2.

## C. Step 3: Decision-making

The output of step 2 is the set of $N_i \times N_j$ bba's $m_{ij}(\cdot)$ associated to each pixel $x_{ij}$ of the image in the whole color space (R,G,B). $m_{ij}(\cdot)$ commits some degree of belief to $\theta_1 \triangleq \text{Pixel} \in \text{Edge}$, to $\theta_2 \triangleq \text{Pixel} \notin \text{Edge}$ and also to the uncertainty $\theta_1 \cup \theta_2$. The binary decision-making process consists in declaring if the pixel $x_{ij}$ under consideration belongs or not to an edge from the bba $m_{ij}(\cdot)$, or in a more complicated manner from $m_{ij}(\cdot)$ and the bba's of its neighbours. In this paper, we just recall the principal methods based on the use of $m_{ij}(\cdot)$.

Based on $m_{ij}(\cdot)$ only, how to decide $\theta_1$ or $\theta_2$? Many approaches have been proposed in the literature for answering this question when working with a n-D frame $\Theta$. The pessimistic approach consists in declaring the hypothesis $\theta_i \in \Theta$ which has the maximum of credibility, whereas the optimistic approach consists in declaring the hypothesis which has the maximum of plausibility. When the cardinality of the frame $\Theta$ is greater than two, these two approaches can yield to a different final decision. In our particular application and since our frame $\Theta$ has only two elements, the final decision will be the same if we use the max of credibility or the max of plausibility criterion. Other decision-making methods suggest, as a good balance between aforementioned pessimistic and optimistic approaches, to approximate the bba at first into a

subjective probability measure from a suitable probabilistic transformation, and then to choose the element of $\Theta$ which has the highest probability. In practice, one suggests to take as final decision the argument of the max of $\text{BetP}(\cdot)$ or of the max of $\text{DSmP}(\cdot)$. In our binary frame case however these two approaches also provide the same final decision as with the max of credibility approach. This can be easily proved from $\text{BetP}(\cdot)$ or $\text{DSmP}(\cdot)$ formulas. Indeed, let's consider $m(\theta_1) > m(\theta_2) > 0$ with $m(\theta_1) + m(\theta_2) + m(\theta_1 \cup \theta_2) = 1$ (which means that $\theta_1$ is taken as final decision because it has a higher credibility than $\theta_2$), then one gets as approximate subjective probabilities:

$$ \text{BetP}(\theta_1) = m(\theta_1) + m(\theta_1 \cup \theta_2)/2 \equiv m(\theta_1) + K $$

$$ \text{BetP}(\theta_2) = m(\theta_2) + m(\theta_1 \cup \theta_2)/2 \equiv m(\theta_2) + K $$

$$ \text{DSmP}(\theta_1) = m(\theta_1)[1 + \frac{m(\theta_1 \cup \theta_2)}{m(\theta_1) + m(\theta_2)}] \equiv m(\theta_1)[1 + K'] $$

$$ \text{DSmP}(\theta_2) = m(\theta_2)[1 + \frac{m(\theta_1 \cup \theta_2)}{m(\theta_1) + m(\theta_2)}] \equiv m(\theta_2)[1 + K'] $$

where $K$ and $K'$ are two positive constants. From these expressions, one sees that if $m(\theta_1) > m(\theta_2) > 0$, then also $\text{BetP}(\theta_1) > \text{BetP}(\theta_2)$ and $\text{DSmP}(\theta_1) > \text{DSmP}(\theta_2)$ and thus the final decision based on max of $\text{BetP}(\cdot)$ or max of $\text{DSmP}(\cdot)$ is finally the same. Note that when $m(\theta_1) = m(\theta_2)$, no rational decision can be drawn from $m(\cdot)$ and only a random decision procedure or ad-hoc method can be used in such particular case.

In summary, one sees that when working with a binary frame $\Theta$, all common decision-making strategies provide the same final decision and therefore there is no interest to use a complex decision-making procedure in that case and that's why we can adopt here the max of belief as final decision-making criterion in our simulations. Note that aside the final decision and because we have $m(\theta_1 \cup \theta_2)$, we are able (if we want) also to plot the level of uncertainty related with such decision (not presented in this paper).

# IV. SIMULATIONS RESULTS

In this section we present the results of our new edge detection algorithm tested on two color images for different parameter settings.

## A. Test on original Lena's picture

Lena Soderberg picture is one of the most used image for testing image processing algorithms in the literature [4] and therefore we propose to test our algorithm on this reference image. This image can be found as part of the USC SIPI Image Database in their "miscellaneous" collection available at http://sipi.usc.edu/database/index.php. The original Lena's picture scan is shown on Fig. 4-(a). The figure 5-(a)-(c) shows the edge detection on each channel (layer) based on the bba's $m_{ij}^L(\cdot|g_{ij}^L)$ in section III-A. One sees that the edges in different channels are different, and the task of our proposed algorithm

<footnote n="5">⁵i.e. a generalization of the PCR5 formula described in section II-A.</footnote>
<figure><img src="image_5.png" /><figcaption>(a) Original Lena image</figcaption></figure>

<figure><img src="image_6.png" /><figcaption>(b) Lena with noise</figcaption></figure>

<figure><img src="image_7.png" /><figcaption>Figure 5. Edge detections in each channel.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Figure 6. Canny's edge detector on Lena's gray image.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 7. Sobel's edge detector on Lena's gray image.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 8. DS-based edge detector on Lena's color image.</figcaption></figure>

<figure><img src="image_11.png" /><figcaption>Figure 9. PCR5 edge detector on Lena's color image.</figcaption></figure>

is to combine efficiently the underlying bba's $m_{ij}^L(\cdot|g_{ij}^L)$ generating the subfigures 5-(a)–(c).

Sobel [9] and Canny [2] edge detectors are commonly used in image processing community and that's why we make comparison of our new edge detector w.r.t. Canny's and Sobel's approaches. Canny and Sobel edge detectors are applied directly to the gray image converted from the original Lena color image Fig. 4-(a). The figures 6–9 show the results of the different edge detectors on Lena's picture. In our simulations, we took $\lambda = 0.06$, and $t_g$ defined as $t = p \cdot \max(g)$ in each layer, was taken with $p_n = 0.17$ and $p_e = 0.19$, corresponding to gradient thresholds $[t_n^R, t_n^L] = [15, 17]$, $[t_e^G, t_e^L] = [13, 14]$ and $[t_n^B, t_e^B] = [11, 13]$. The max of credibility, plausibility, DSmP or BetP for decision-making to generate final result provide the same decision as explained in the section III-C which is normal in this binary frame case.

One sees that finally on the clean (noise-free) Lena's picture, our edge detector provides close performances to Sobel's detector applied on Lena's grey image. Canny's detector seems to provide a better ability to detect some edges in Lena's picture than our method, but it also generates much more false
alarms too. It is worth noting that the results provided by DS-based or PCR5-based edge detectors show a coarse location of the edges. So it is quite difficult to draw a clear and fair conclusion between these edge detectors since it highly depends on what we want, i.e. the reduction of false alarms or the reduction of miss-detections.

## B. Test on Lena's picture with noise

In this simulation, we show how our edge detector works on a noisy image. Sampling of independent Gaussian noise $N(0, \sigma^2)$ is added to each pixels of each layer of the original Lena's picture as seen on Fig. 4-(b). In the presented simulation, $\sigma^2 = 1100$ which correspond approximatively to the value of the variance of the blue channel and half the variance of the others. Local edge detection for each layer based on $m_{ij}^L(\cdot|g_{ij}^L)$ is shown on Fig. 10-(a)–(c), where the red points represent the ignorant pixel which commits the most belief to the ignorance $\theta_1 \cup \theta_2$. As shown in Fig 10, the edge detection in each channel is very noisy. Our method allows to commit automatically highest belief value to uncertainty for most of pixels associated to an edge which actually correspond to noises.<fnref n="6" /> The edge detection based on fusion result are interesting as shown by Fig.11 and Fig. 12 because it shows the ability of our edge detector to suppress the noise effects. For comparison, we give on Fig. 13 and Fig.14, the performance of Canny and Sobel edge detectors applied classically on the noisy gray-level Lena's picture. In this simulation, we took $\lambda = 0.06$, and $t$ using $p_n = 0.22 \cdot \max(g)$ and $p_e = 0.39 \cdot \max(g)$ in each layer with $[t_n^R, t_n^R] = [36, 20]$, $[t_n^G, t_n^G] = [35, 19]$ and $[t_n^B, t_n^B] = [31, 18]$. The decision-making is still based on max of credibility.

The visual comparison and analysis of results shown of figures 11–12 clearly indicates that our edge detector based on the fusion of belief constructed on each layer works much better than the edge detection applied separately on each layer. There is no ignorant pixel corresponding to red color according to the fusion results, since the fusion process of DS or PCR5 rule effectively decrease the uncertainty. Our results show also clearly that Canny and Sobel edge detectors applied to noisy gray-level Lena's picture are very sensitive to the noise perturbations. Our proposed method (based on DS rule or on PCR5 rule) is more robust to the noise perturbations and provides better results than Sobel or Canny edge detector for such noisy image. For this tested image, it appears that the results using DS and PCR5 rules are very close, because there is not too much conflict actually between bba's of layers and one know that in such case PCR5 rule behavior is close to DS rule behavior. DS rule is usually good enough in the low conflict case, whereas PCR5 rule is preferred for the combination of high conflicting sources of evidence. So the preference of PCR5 with respect to DS rule for edge detection must be guided by the level of conflict which appears in the layers of the color image that we need to process.

<figure><img src="image_12.png" /><figcaption>Figure 10. Edge detections in each channel on noisy image.</figcaption></figure>

<figure><img src="image_13.png" /><figcaption>Figure 11. DS edge detector on noisy Lena's color image.</figcaption></figure>

## V. CONCLUSIONS AND PERSPECTIVES

A new unsupervised edge detector for color image based on belief functions has been proposed in this work. The basic belief assignment (bba) associated with the edge of a pixel in each channel of the image is defined according to its gradient magnitude, and one can easily model the uncertainty about our belief it belong or not to an edge. PCR5 and DS rules have been applied in this work to combine these bba's to get the global bba for final decision-making. Other rules of combination of bba's could also have been used instead but they are known to be less efficient than PCR5 or DS rules in high and low conflict cases respectively. The

<figure><img src="image_14.png" /><figcaption>Figure 12. PCR5 edge detector on noisy Lena's color image.</figcaption></figure>

<footnote n="6">⁶So we are also able at layer level to filter these pixels (false alarms) before applying the fusion. This has not yet be done in this work.</footnote>
<figure><img src="image_15.png" /><figcaption>Figure 13. Sobel's edge detector on noisy Lena's gray image.</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>Figure 14. Canny's edge detector on noisy Lena's gray image.</figcaption></figure>

fusion process is able to reduce noise perturbations because the noises are assumed to be independent between channels. The final decision making on the edge can be made either on the maximum of credibility, plausibility, *DSmP* or *BetP* values as well. The first simulation done on original Lena's picture shows that our edge detector works as well as the classical Sobel's edge detector and it provides less false alarms than with Canny's detector, but seems to generate more miss-detections. In our second simulation based on noisy Lena image, the results show that our new edge detector is more robust to the noise perturbations than Sobel or Canny classical edge detectors. As possible improvement of this algorithm and for further research, we would like to include some morphological or connectivity constraints at a higher level of processing and develop automatic technique for threshold selection. The application of this new approach of edge detection to satellite multispectral images is under investigations.

## REFERENCES

[1] S. Ben Chaabane, F. Fnaiech, M. Sayadi, E. Brassart, *Relevance of the Dempster-Shafer evidence theory for image segmentation*, 3rd International conference on Signals, Circuits and Systems, Nov. 6-8th, Medenine, Tunisia, 2009.

[2] J.F. Canny, *A Computational Approach to Edge Detection*, IEEE Transactions on Pattern Analysis and Machine Intelligence, 8(6):679-698, 1986.

[3] C.J. Delcroix, M. A. Abidi, *Fusion of edge maps in color images*, in Proc. SPIE Int. Soc. Opt. Eng. 1001, pp. 454-554, 1988.

[4] N. Devillard, *Image Processing: the Lena story*, 2006. http://ndevilla.free.fr/lena/, see also http://www.lenna.org.

[5] J. Dezert, F. Smarandache, *Non Bayesian conditioning and deconditioning*, Int. Workshop on Belief Functions, Brest, France, April 2010.

[6] J. Dezert, F. Smarandache, *A New Probabilistic Transformation of Belief Mass Assignment (DSmP)*, in Proc. of Fusion 2008 Int. Conf., Cologne, Germany, June 30-July 3, 2008.

[7] A.N. Evans, *Nonlinear Edge Detection in color images*, Chap. 12, pp. 329-356, in [13].

[8] A.K. Jain, *Fundamentals of Digital Image Processing*, Prentice-Hall, Englewood Cliffs, NJ, 1991.

[9] E.P. Lyvers and OR. Mitchell, *Precision Edge Contrast and Orientation Estimation*, IEEE Transactions on Pattern Analysis and Machine Intelligence, 10(6):927-937,1988.

[10] A. Koschan, M. Abidi, *Detection and classification of edges in color images*, IEEE Signal Processing Magazine, vol. 22, no. 1, pp. 64-73, 2005.

[11] A. Koschan, M. Abidi, *Digital Color Image Processing*, John Wiley Press, NJ, USA, 376 pages, May 2008.

[12] D. Marr, E. Hildreth, *Theory of Edge Detection*, in Proceedings of the Royal Society London 207, pp. 187-217, 1980.

[13] S. Marshall, G. L. Sicuranza (Editors), *Advances in nonlinear signal and image processing*, EURASIP Book series on Signal Processing and Communications, Hindawi Publishing Corporation, 2006.

[14] K. N. Plataniotis, A. N. Venetsanopoulos, *Color Image Processing and Applications*, Springer, New York, NY, USA, 2000.

[15] M. A. Ruzon, C. Tomasi, *Edge, junction, and corner detection using color distributions*, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 23, no. 11, pp. 1281-1295, 2001.

[16] G. Shafer, *A Mathematical Theory of Evidence*, Princeton Univ. Press, 1976.

[17] F. Smarandache, J. Dezert (Editors), *Advances and Applications of DSmT for Information Fusion*, American Research Press, Rehoboth, Vol.1-3, 2004-2009.

[18] F. Smarandache, J. Dezert, J.-M. Tacnet, *Fusion of sources of evidence with different importances and reliabilities*, in Proceedings of Fusion 2010 conference, Edinburgh, UK, July 2010.

[19] P. Smets, *Constructing the pignistic probability function in a context of uncertainty*, Uncertainty in Artificial Intelligence, Vol. 5, pp. 29-39, 1990.

[20] P. Smets, R. Kennes, *The transferable belief model*, Artif. Intel., 66(2), pp. 191-234, 1994.

[21] P. Vannoorenberghe, O. Colot, D. De Brucq, *Color Image Segmentation Using Dempster-Shafer's Theory*, ICIP 99. Proc. of 1999 International Conference on Image Processing, Kobe, Japan, Vol. 4, pp. 300-304, 1999.

[22] L. Zadeh, *On the validity of Dempster's rule of combination*, Memo M79/24, Univ. of California, Berkeley, USA, 1979.