# Transmission Estimation by Complex Assumption with Occlusion Handling

Authors: Jiantao Liu; Xiaoxiang Yang; Mingzhu Zhu; Bingwei He
Journal: Mathematical Problems in Engineering
 (2019)

---

## Abstract

Transmission estimation is a critical step in single-image dehazing. The estimate of each pixel describes the portion of the scene radiance that is degraded by hazing and finally reaches the image sensor. Transmission estimation is an underconstrained problem, and, thus, various assumptions, priors, and models are employed to make it solvable. However, most of the previous methods did not consider the different assumptions simultaneously, which, therefore, did not correctly reflect the previous assumptions in the final result. This paper focuses on this problem and proposes a method using an energy function that clearly defines the optimal transmission map and combines the assumptions from three aspects: fidelity, smoothness, and occlusion handling, simultaneously. Fidelity is measured by a novel principle derived from the dark channel prior, smoothness is described by the assumption of piecewise smoothening, and occlusion handling is achieved based on a new proposed feature. The transmissions are estimated by searching for the optimal solution of the function that can retain all the employed assumptions simultaneously. The proposed method is evaluated on the synthetic images of two datasets and various natural images. The results show that there is remarkable fidelity and smoothness in the transmission and that a good performance is exhibited for haze removal.


---

## Body

## 1. Introduction

Images captured in hazy weather have a limited visibility and low contrast [1] owing to the optical attenuation resulting from the scattering of the hydrometeors. There is a high demand for restoring haze-free images from a single hazy input. In photography, this requires correction of the color shift of the photos and enhancement of the overall quality of the photos. In computer vision domain, it provides an estimate of the scene radiance, which is an important input in most analysis or recognition algorithms.As illustrated in Figure1, a hazy image is widely modeled as a combination of scene radiance J and global airlight A, as expressed in(1)Ix=Jxtx+A1-tx,where t is the transmission map, which controls the scene attenuation and amount of haze in every pixel. When the haze is evenly dispersed, the transmission is inversely proportional to the scene depth as(2)tx=e-βdx,where d is the depth map and β is the scattering coefficient, commonly assumed to be a constant [1–3].Figure 1
Example of hazy image formation. A hazy image is modeled as a combination of the scene radiance and global airlight, with the combining ratio described by the transmission map. Red/blue colors represent large/small transmissions, corresponding to near/far-scene points. (The airlight is selected manually. The scene radiance and transmission map are offered by the proposed method.)Given a hazy image, the airlight can be estimated by selecting the brightest pixels [2] or concatenating the maximum values of each channel [4]. More robustly, to avoid the disturbance caused by bright objects, Heet al. [3] suggested selecting the brightest pixel among the pixels having the most high brightness values of the dark channel. These methods rely on the presence of the sky or dense haze regions. Alternatively, the method of Bermanet al. [5] exhibits a comparatively better performance. With a known airlight, recovering the scene radiance, which is the objective of haze removal, is still an under-constrained problem. Additional assumptions, priors, or models (these terms are used alternatively below to express additional constraints) are employed explicitly or implicitly to previous methods to make the problem solvable.An important category of dehazing methods is enhancement based, which focuses on enhancing the visibility of a hazy image. The employed assumptions are commonly decided based on the scene radiance, and, for visibility, they include high contrast [2, 6, 7], saliency and exposedness [6, 7], luminance, and saturation [7], or multiple quality assessments [8]. The haze removal results are robust, having a high visibility even when dealing with underwater, night time, or poor exposure images. However, these methods commonly do not follow the model of (1) and offer airlight and transmission estimates. An unnatural appearance might be yielded in some cases owing to the lack of physical interpretation. In comparison, a physical-based method is more valid and attracts more attention.The assumptions of physical-based methods are selected by studying the laws of nature or intuition. A simple intuition is that a pixel with a dense haze appears white. Based on this, Zhuet al. [9] and Liet al. [10] made assumptions (Zhuet al. [9] using machine learning) that there was a simple relationship between the transmission and hazy color at each pixel. However, this assumption failed to provide a viable initial transmission map. The above two methods relied on a guided filter [11] during regularization, which further undermined the assumption (discussed in Section 2.1). Fattal [12] assumed that the transmission and surface shading were locally uncorrelated. The assumption was quite physically reasonable and made impressive contributions. However, it was shown by Heet al. [3] that this method could not well handle strongly hazy images. Nishinoet al. [13] modeled the scene radiance and gradient of the scene depth using heavy-tail distributions and assumed them to be independent. It was shown by Fattal [14] that the above method tended to underestimate the transmission and produce oversaturated results. Considering independency to be reasonable but not sufficient, assumptions regarding the quality of the scene radiance or transmission are still required.The dark channel prior proposed by Heet al. [3] is a clearly defined prior based on the scene radiance, which attracts significant attention and has tremendous potential [15–24] despite receiving criticism for being invalid for uniformly bright objects [19, 25]. However, previously reported methods considered the prior but undermined its merits owing to their employment with the local constant assumption, a Laplacian matting [26], or a guided filter [11] (discussed in Section 2.1). Since the proposal of the dark channel prior, increasing publications have been paying more attention to integrating their assumptions in a prior or model, such as the color-lines model [14], haze-lines model [25], patch recurrence model [27], and model for remote sensing images [28]. The color-lines model proposed by Fattal [14] describes haze-free images containing numerous small patches, which are distributed by the internal colors on a line across the origin in the RGB space. After mixing with the haze, these lines are shifted toward the point of the airlight color. the airlight-intercept indicates the transmission of the pixels on this line. Although the above method exhibits a good performance in some cases, there are also numerous small patches in the resultant images distributed by their internal colors on a line that crosses the airlight vector and not the origin. The method does not consider this condition. Bermanet al. [25] proposed a non-local method with the haze-lines model. A haze-line is a gathering of colors in a hazy image that lie on a ray starting from the airlight color in the RGB-space. There are two assumptions for each haze-line: a) pixels on the same haze-line are assumed to have the same scene radiance, and b) the pixel farthest one from the airlight point is assumed to be haze-free. The first assumption has ambiguities as discussed by Bermanet al. [25], and the second one is not close to the reality. Therefore, the initial transmission estimates are of low fidelity in some regions and cause oversmoothness after regularization. The patch recurrence model describes the fact that small patches tend to repeat abundantly inside a natural haze-free image, regardless of the scale. Bahatet al. [27] employed this model and solved the transmission map that introduced the maximal internal patch recurrence in the haze removal method. Assuming a haze-free image has maximal internal patch recurrence is questionable. Indeed, for haze removal, the method produces numerous structures and sometimes more than natural.There are also methods that build priors by learning [29, 30], but it is difficult to summarize them explicitly. It is recommended to refer to the reviews and comparisons provided by Ancutiet al. [31], Liet al. [32], or Singhet al. [33].The priors and models introduced above require to work with the assumptions of transmission consistency (introduced in Section2.1), which is similar to the fidelity and smoothness measurements in energy minimization. However, few methods attempt to apply them simultaneously; instead they estimate an initial map without considering the smoothness and then regularize it with an ambiguous trade-off for the fidelity. For example, Bermanet al. [25] and Fattal [14] made a trade-off between the initial estimate and smoothness. The fidelity was simply defined as the weighted distance between the initial estimate and solution, which did not truly reflectthe degree of compliance of the models. As a comparison, Bahatet al. [27] measured fidelity by counting the number of internal patch recurrences and found it to be steadily maintain with its major assumption.In this study, we focus on this problem and propose an energy function that combines different assumptions to define a good transmission map. All of the employed assumptions are considered simultaneously while searching the optimal solution, which is the estimate of the transmission map. Another drawback in the previous methods is that they barely consider the occlusion problem. This usually occurs when a distant scene is partially occluded and divided into parts by a closer one, such as a wall behind a furcated trunk and the sky behind a net. These divided scenes are commonly assigned transmissions similar to the surroundings and appear veiled in the haze removal result. In this study, we also handle this problem by searching and linking the pixels of a divided scene, settling the constraints to extend the energy function. Along with the fidelity and smoothness assumptions, the effect of occlusion handling is also truly reflected in the optimal solutions.The contributions of this study are listed below: (a) To solve the problem that traditional methods usually generate an initial transmission map based on partial assumptions and then refine it based on other parts, undermining the former parts, we propose a method that simultaneously considers different assumptions in an optimization framework. (c) To solve the problem that the transmissions of an occluded distant scene are usually over-estimated, we propose a novel feature to establish relationships between the fragmented part and major part and then develop them into a term in our optimization framework.This paper is organized as follows. Section2 introduces several assumptions of fidelity, smoothness, and occlusion handling. Related works based on these assumptions and their issues are also discussed. Section 3 introduces the basic algorithm and proposed energy function along with the solving process. The experiments are performed on synthetic images of two datasets and various natural images, which are illustrated in Section 4. The conclusion and future work described are in Section 5.


## 2. Background

### 2.1. Assumption of Smoothness

The assumptions for smoothness widely used in transmission maps include: local constant assumption, local linear assumption, and piecewise smoothness assumption.According to the local constant, assumption transmissions in a small patch are the same, i.e.,(3)ty=tx,y∈Ωx,where Ω(x) is a small patch centered at x. Such an assumption is far from the reality, but easy to use. Therefore, it is used to assist in the initial transmission estimation [2, 3, 14]. For example, Tanet al. [2] aimed to solve a transmission map that improved the contrast of the haze removal result maximally. However, transmissions cannot be solved pixel by pixel because contrast has no definition on a single pixel. Therefore, the local constant assumption was employed, and thus, the pixels in Ω(x) were enhanced based on the same transmission, t(x), with the optimal t(x) being that introduced the maximal contrast.The local constant assumption causes theblock effect, as shown in Figure 2(b). This effect was previously removed by employing regularization with the piecewise smoothness assumption or local linear assumption. If a block (related to the shape of Ω(x) in (3)) is small, it can be removed without significantly changing the initial estimate, as in the method of Fattal [14]. Otherwise, a Laplacian matting [26] or guided filter [11], which is better at block effect removal, is employed [3].Figure 2
Shortcomings in the transmission estimation. Columns from left to right: hazy image and our estimate; initial and final estimate of Heet al. [3] and Bermanet al. [25]. The subtitles of the last two columns describe the defect of each transmission map, discussed in Section 2.1.Laplacian matting and guided filters are widely employed to regularize the initial transmission map [3, 15, 17]. These algorithms adopt the local linear assumption, i.e.,(4)ty=∑cacIy+b,y∈Ωx,where (a,b) are linear coefficients assumed to be constant in Ω(x) and c is the color channel. It is assumed that in a small patch, Ω(x), a transmission is linear to the hazy colors.The local linear assumption is also much different from the reality because by definition a transmission is only related to the amount of haze (or related to the depth when the haze is evenly dispersed). Moreover, inappropriately, different from employing the local constant assumption, most methods employing a Laplacian matting or guided filter retain such an assumption in their final result, causing the phenomenon ofredundant details. As shown in Figure 2(e), the clock, building textures, and vehicles, which are unrelated to the transmissions, can be clearly identified from the map.The piecewise smoothness assumption describes the intuition that similar pixels adjacent to each other have similar transmissions. Commonly, the similarity is measured in a hazy image, and the assumption is employed in the smoothness term of the energy functions [14, 25] as(5)Et=Dt+λtx-ty2Ix-Ix2, where D(t) is a data term and λ balances the fidelity and smoothness. The piecewise smoothness assumption is more in accordance with the reality than the previous two assumptions. However, it causes over0-smoothness if the data term has an inappropriately low weight. Examples are shown in Figures 2(c) and 2(f), taken from Bermanet al. [25]. Here, the method is based on the haze-lines model, which is not robustly applicable everywhere. The regions that do not fit the model well have low weights in the regularization. The smoothness term plays a strong role in these regions, and, thus, there is over-smoothness; e.g., edges are noticeable between the buildings and sky.In the proposed method, the piecewise smoothness assumption is chosen because it is the closest one to the reality. The over-smoothness is suppressed because the data term of the proposed energy function is strong everywhere. An example of our estimate is shown in Figure2(d).


### 2.2. Assumption of Fidelity

Although various assumptions for fidelity have been proposed (introduced in Section1), few have an explicit definition and are widely tested. In this respect, the dark channel prior proposed by Heet al. [3] is superior among the known assumptions. It has a brief and clear definition based on the scene radiance, as expressed in (6)mincminy∈ΩxJcy=0. Although the prior is defined to be only valid in non-sky regions, it is actually also compatible with the sky because the sky can be explained as a purely dark region with zero transmissions. (this might cause an unnatural appearance on the sky, as discussed by Wanget al. and Cuiet al., which is ignored in this study).The prior has exhibited its tremendous effectiveness in single-image dehazing and attracted numerous research studies [15, 17, 19]. However, most research works have not exploited its advantages fully. For examples, Heet al. [3] estimated an initial transmission map based on the dark channel prior and local constant assumption, which caused the block effect. Laplacian matting was then employed to remove this effect, and the local linear assumption was introduced, which introduced redundant details. Such details reduce the contrast of a haze removal result, as discussed by Fattal [14] (Note that a higher contrast does not typically imply a better result). Wanget al. [17] assumed that the dark channel prior and local constant assumption were satisfied in each superpixel. Although their method avoided the block effect, the assumption was questionable because the colors in each superpixel tend to be similar, and the dark channel prior easily fails with similar colors. Furthermore, the method employed a guided filter to regularize its initial estimate, also generating redundant details. Similarly, Wanget al. [19] employed a Laplacian matting patch by patch, and Luanet al. [15] used a guided filter for regularization. Both of them did not avoid the redundant details.Although the method of Heet al. [3] has been surpassed by new methods in many comparative studies [32], the prior itself is outstanding. In this paper, we show that the prior can be used to construct a data term for an energy function that clearly reflects the degree of compliance of the solution. With this, the prior again exhibits its superiority.


### 2.3. Assumption for Occlusion Handling

Figure3 illustrates an example of the occlusion problem, where the plants near the road are partially occluded and divided by the plants close to the camera (framed in yellow). Without occlusion handling, these isolated parts might be assigned transmissions similar to their surroundings, which overestimate the due values, as shown in Figure 3(d).Figure 3
Example of occlusion. (a) hazy image with occlusion problems; estimate of (b) Heet al. [3] and (c) Bermanet al. [25]; and the proposed method (d) without occlusion handling and (e) with occlusion handling.
(a)
Hazy image (b)
Heet al. [3] (c)
Bermanet al. [25] (d)
The proposed method (e)
With occlusion handlingAmong the introduced methods (in Section1), that by Fattal [14] is the only one that generates specific solutions for such problems (However, the code is not available to us). The method adopts the similar assumption: pixels with similar colors have similar transmissions. Obviously, this is rather a false positive assumption. The method of Heet al. [3] can slightly handle this problem attributed to the local linear assumption concealed in the Laplacian matting; but it is not effective particularly when the isolated part is extremely small, as shown in Figure 3(b). The method of Bermanet al. [25] did not introduce any patch-based assumptions; although its initial estimate was fully free from this problem, it was undermined by the regularization with the shortcoming of over-smoothness, as shown in Figure 3(c).In this study, the pixels of a common but divided scene are searched based on a novel feature, i.e., with the assumption that pixels with similar such features have similar transmissions. This method is similar to the method of Fattal [14], but gives more attention to suppress the false positive cases. The result of the proposed method with occlusion handling is shown in Figure 3(e).


## 2.1. Assumption of Smoothness

The assumptions for smoothness widely used in transmission maps include: local constant assumption, local linear assumption, and piecewise smoothness assumption.According to the local constant, assumption transmissions in a small patch are the same, i.e.,(3)ty=tx,y∈Ωx,where Ω(x) is a small patch centered at x. Such an assumption is far from the reality, but easy to use. Therefore, it is used to assist in the initial transmission estimation [2, 3, 14]. For example, Tanet al. [2] aimed to solve a transmission map that improved the contrast of the haze removal result maximally. However, transmissions cannot be solved pixel by pixel because contrast has no definition on a single pixel. Therefore, the local constant assumption was employed, and thus, the pixels in Ω(x) were enhanced based on the same transmission, t(x), with the optimal t(x) being that introduced the maximal contrast.The local constant assumption causes theblock effect, as shown in Figure 2(b). This effect was previously removed by employing regularization with the piecewise smoothness assumption or local linear assumption. If a block (related to the shape of Ω(x) in (3)) is small, it can be removed without significantly changing the initial estimate, as in the method of Fattal [14]. Otherwise, a Laplacian matting [26] or guided filter [11], which is better at block effect removal, is employed [3].Figure 2
Shortcomings in the transmission estimation. Columns from left to right: hazy image and our estimate; initial and final estimate of Heet al. [3] and Bermanet al. [25]. The subtitles of the last two columns describe the defect of each transmission map, discussed in Section 2.1.Laplacian matting and guided filters are widely employed to regularize the initial transmission map [3, 15, 17]. These algorithms adopt the local linear assumption, i.e.,(4)ty=∑cacIy+b,y∈Ωx,where (a,b) are linear coefficients assumed to be constant in Ω(x) and c is the color channel. It is assumed that in a small patch, Ω(x), a transmission is linear to the hazy colors.The local linear assumption is also much different from the reality because by definition a transmission is only related to the amount of haze (or related to the depth when the haze is evenly dispersed). Moreover, inappropriately, different from employing the local constant assumption, most methods employing a Laplacian matting or guided filter retain such an assumption in their final result, causing the phenomenon ofredundant details. As shown in Figure 2(e), the clock, building textures, and vehicles, which are unrelated to the transmissions, can be clearly identified from the map.The piecewise smoothness assumption describes the intuition that similar pixels adjacent to each other have similar transmissions. Commonly, the similarity is measured in a hazy image, and the assumption is employed in the smoothness term of the energy functions [14, 25] as(5)Et=Dt+λtx-ty2Ix-Ix2, where D(t) is a data term and λ balances the fidelity and smoothness. The piecewise smoothness assumption is more in accordance with the reality than the previous two assumptions. However, it causes over0-smoothness if the data term has an inappropriately low weight. Examples are shown in Figures 2(c) and 2(f), taken from Bermanet al. [25]. Here, the method is based on the haze-lines model, which is not robustly applicable everywhere. The regions that do not fit the model well have low weights in the regularization. The smoothness term plays a strong role in these regions, and, thus, there is over-smoothness; e.g., edges are noticeable between the buildings and sky.In the proposed method, the piecewise smoothness assumption is chosen because it is the closest one to the reality. The over-smoothness is suppressed because the data term of the proposed energy function is strong everywhere. An example of our estimate is shown in Figure2(d).


## 2.2. Assumption of Fidelity

Although various assumptions for fidelity have been proposed (introduced in Section1), few have an explicit definition and are widely tested. In this respect, the dark channel prior proposed by Heet al. [3] is superior among the known assumptions. It has a brief and clear definition based on the scene radiance, as expressed in (6)mincminy∈ΩxJcy=0. Although the prior is defined to be only valid in non-sky regions, it is actually also compatible with the sky because the sky can be explained as a purely dark region with zero transmissions. (this might cause an unnatural appearance on the sky, as discussed by Wanget al. and Cuiet al., which is ignored in this study).The prior has exhibited its tremendous effectiveness in single-image dehazing and attracted numerous research studies [15, 17, 19]. However, most research works have not exploited its advantages fully. For examples, Heet al. [3] estimated an initial transmission map based on the dark channel prior and local constant assumption, which caused the block effect. Laplacian matting was then employed to remove this effect, and the local linear assumption was introduced, which introduced redundant details. Such details reduce the contrast of a haze removal result, as discussed by Fattal [14] (Note that a higher contrast does not typically imply a better result). Wanget al. [17] assumed that the dark channel prior and local constant assumption were satisfied in each superpixel. Although their method avoided the block effect, the assumption was questionable because the colors in each superpixel tend to be similar, and the dark channel prior easily fails with similar colors. Furthermore, the method employed a guided filter to regularize its initial estimate, also generating redundant details. Similarly, Wanget al. [19] employed a Laplacian matting patch by patch, and Luanet al. [15] used a guided filter for regularization. Both of them did not avoid the redundant details.Although the method of Heet al. [3] has been surpassed by new methods in many comparative studies [32], the prior itself is outstanding. In this paper, we show that the prior can be used to construct a data term for an energy function that clearly reflects the degree of compliance of the solution. With this, the prior again exhibits its superiority.


## 2.3. Assumption for Occlusion Handling

Figure3 illustrates an example of the occlusion problem, where the plants near the road are partially occluded and divided by the plants close to the camera (framed in yellow). Without occlusion handling, these isolated parts might be assigned transmissions similar to their surroundings, which overestimate the due values, as shown in Figure 3(d).Figure 3
Example of occlusion. (a) hazy image with occlusion problems; estimate of (b) Heet al. [3] and (c) Bermanet al. [25]; and the proposed method (d) without occlusion handling and (e) with occlusion handling.
(a)
Hazy image (b)
Heet al. [3] (c)
Bermanet al. [25] (d)
The proposed method (e)
With occlusion handlingAmong the introduced methods (in Section1), that by Fattal [14] is the only one that generates specific solutions for such problems (However, the code is not available to us). The method adopts the similar assumption: pixels with similar colors have similar transmissions. Obviously, this is rather a false positive assumption. The method of Heet al. [3] can slightly handle this problem attributed to the local linear assumption concealed in the Laplacian matting; but it is not effective particularly when the isolated part is extremely small, as shown in Figure 3(b). The method of Bermanet al. [25] did not introduce any patch-based assumptions; although its initial estimate was fully free from this problem, it was undermined by the regularization with the shortcoming of over-smoothness, as shown in Figure 3(c).In this study, the pixels of a common but divided scene are searched based on a novel feature, i.e., with the assumption that pixels with similar such features have similar transmissions. This method is similar to the method of Fattal [14], but gives more attention to suppress the false positive cases. The result of the proposed method with occlusion handling is shown in Figure 3(e).


## 3. Proposed Method

### 3.1. The Algorithm

The haze model of (1) is followed in this paper. In our practice, few hazy images do not contain a small part of the sky or dense haze region; the airlight estimation approach of Heet al. [3] maintains its efficient most of the time. Therefore, we simply employ it to estimate the airlight. The transmissions are estimated by searching an approximately optimal configuration of the proposed energy function. With a known airlight and transmissions, the scene radiance is recovered through(7)Jx=Ix-Atx+A.Considering the residual error, aerial perspective, and noise of the sky region [3], the transmissions are slightly increased by t=(t+0.1)/1.1 before dehazing, and the removal results are brightened by I=I0.9.


### 3.2. The Energy Function

The proposed energy function is(8)Et=∑xDx+∑x,y∈C1Kx,y+∑x,y∈C2Ox,y,where D is the data term, measuring the fidelity of the transmissions at each pixel, K is the smoothness term for the piecewise smoothness, and O is the constraint term for the occlusion handling. (x,y)∈C1 or C2 imply that the two pixels are neighbors in the image space or feature space.


### 3.3. The Data Term

The data term is derived from the internal constraints of the haze model and dark channel prior. Because the scene radiance is non-negative, we have(9)Jxtx=Ix-A+Atx≥0, and, thus,(10)tx≥1-IxA≥tLBx=1-mincIxA,where tLB(x) is the lower bound of the transmission of pixel x.Briefly, pixels satisfyingmincJc(x)=0 are called dark pixels. The transmission of each dark pixel has only one possibility: it is equal to its lower bound; this is because(11)tx=1-mincAc1-txAc=1-mincIcxAc=tLBx.A pixel in a hazy image can be confirmed to be a dark pixel only if it satisfies mincIc(x)=0, and there is still no additional evidence of the possible transmissions of most pixels. In this situation, the local constant assumption can be employed, but it may cause the block effect, as we have discussed above. In this paper, we propose another assumption that is much looser than the local constant assumption. For each pixel x, at least one dark pixel in Ω(x) has the same transmission as x (the dark pixel must exist in Ω(x) as illustrated in the dark channel prior).Because every pixel inΩ(x) is a potential dark pixel with transmission tLB(y), which might equal t(x), we collect all such potentials in a label set as(12)tx∈Lx=tLBy∣y∈Ωx,tLBy>tLBx,where Ω is a circular mask with radius r=0.1min⁡(w,h) (w,h are the image width and height). For any two values in L(x), prior or assumption cannot be used to determine that which is better. Thus, the data term is designed only to avoid the values not in L(x), as(13)Dx=0,fortx∈Lx∞,fortx∉Lx.Note that the data term plays a strong role in every pixel of the hazy image, and, thus, the defect of over-smoothness is suppressed.


### 3.4. Smoothness Term

Based on the piecewise smoothness assumption, the proposed smoothness term is(14)Kx,y=tx-ty2tLBx-tLBy2,x,y∈C1,where the set of adjacent pixel pairs, C1, which is defined in four-connected. The term is a traditional one, except that the similarity between the pixels is measured on tLB not on I, as in the studies by Fattal [14] and Bermanet al. [25]. This method starts from the intuition that two pixels may still have similar transmissions even if they are distinct in color. Instead, tLB is more in accordance with the intuition on the haze amount, and sometimes it is used as an initial estimate of the transmissions [9, 27]. An example is shown in Figure 4. In the measurement of the similarity in tLB(x), the result is slightly more consistent and obvious on the structures circled in yellow. Most of time, measuring on I or tLB has little difference, but on tLB is faster because it only contains one channel.Figure 4
Example of measuring the similarity of the pixels on different maps. From left to right: hazy image, the lower bound map of the transmissions, and the estimate of the transmissions by measuring the similarity in the hazy image and lower bound map.


### 3.5. Constraint Term

The constraint term is proposed to handle the occlusion problem. The key point of occlusion handling is to recognize the pixels from a common but divided scene. It is a challenging task and nearly impossible for a low-level method. However, we do not have to recognize all the pixels, but only need to establish several links between them, such as the link illustrated in Figure5(a). The sky region in the figure is divided by the wires of the suspension. With only the piecewise smoothness assumption, the divided sky regions have no relationship with the major one. They are considered as some types of white objects in the front and are assigned transmissions similar to their surroundings, as shown in Figure 5(c). By settling such links, these regions are collected and assigned reasonable values, as shown in Figure 5(d). Because several links are required, the method to search the pixels of a common but divided scene is allowed to be quite a true negative, making the problem easier to solve than before.Figure 5
Example of occlusion handling. (a) Hazy image and a link we intend to build; (b) hazy image with features, pixels marked in the same color are supposed to have similar transmissions; (c) estimate of the transmissions without occlusion handling; (d) estimate of the transmissions with occlusion handling.
(a)
Hazy image (b)
Linked pixels (c)
The estimate (d)
With occlusion handlingA novel feature is proposed as(15)Fx=Ix,I∗x,L∗x,where(16)I∗x=argmaxIy,y∈Ω∗xIx-Iy2,with Ω∗ being a small mask with side length l∗=0.02min⁡(w,h), and L∗ includes the texture content.(17)L∗x=tLBy∣y∈Ω∗x.The shorter the distance between F(x) and F(y), the higher the possibility that pixels x and y belong to a common scene. Figure 5(b) shows two sets of pixels with very similar features, marked as blue and yellow. By linking the pixels of yellow, several isolated sky regions are successfully related to the major one (there are two pixels of yellow in the major sky region). By linking the pixels of blue, several isolated regions are collected together. However, these regions are not linked to the major one directly; they are linked with the pixels of yellow color in the image space, as circled in green in the figure. Note that there are hundreds of such sets of pixels, which are scattered in the suspension. Numerous links are settled between them, and, finally, they seem to be dispersed sky regions, as an integral one.The first two elements of the feature in (15) are based on the intuitions that (a) there must be some notable difference between the divided scene and its covering; (b) the same colors frequently recur in the divided scene; (c) the same colors frequently recur in the covering. Figure 6 illustrates such intuitions. It shows several pixel-pairs with colors turning from white to brown, which can be found in abundance between the divided background and its covering: the tree. The relationships we intend to build can be easily settled by searching such pixel-pairs and linking them. However, this is rather a false positive method, and, as mentioned above, the method can only be true negative.Figure 6
Example of occlusion handling. From left to right: hazy image and several pixel-pairs with similar colors; and the estimate of the transmissions without and with occlusion handling.The last element of the feature is proposed to tighten the search criteria; it is based on the intuition that small patches centered at the edges between the divided scene and its covering contain similar colors. Then, the similarity in the color is replaced by the similarity inL∗ based on the following considerations: (a) pixels similar to each other in I are similar to those in tLB; (b) L∗ is easier to use than a set of colors; (c) if a set of pixels (x1,x2,…) with different label sets is found to have very strong relationships (suppose their transmissions must be the same), there might be fewer choices for their solutions (L(x1)∪L(x2)∪⋯→0). Therefore, the similarity in L∗ reflects the risk of the solution space reduction more intuitively.Considering the features of all the pixels in a hazy image yields a point cloud in the feature space. Pixels close to each other in the feature space should be linked and assigned a large weight and vice versa. Such links are built as a graph construction problem, which can be solved well based on thek-nearest neighbors method or the b-matching method [34]. However, the built graph might be extremely large to solve. In practice, we simplify the graph construction by partitioning the feature space into numerous super-blocks and only linking the pixels of the features in the same super-block. The partition is based on the first six dimensions of the feature space (RGB for I and I∗), and the super-blocks are not of the same size, as shown in Figure 7. The unequal partitioning is because all the intuitions described above occur in a haze-free image, and the conditions are stricter while the haze is denser, and, approximately, the color is whiter.Figure 7
Partition of the feature space. Pixels of the features located in the same block are considered to have the same transmission. Moreover, the whiter the colors, the smaller the block. Note that both I(x) and I∗(x) are three-dimensional, and the blocks displayed are actually super-blocks in a six-dimensional space.Given pixel-pair(x,y) with features in the common super-block, it is included in set C2 if it also satisfies(18)distL∗x,L∗yL∗x∪L∗y<0.2,where dist is the Hamming distance and L∗(x) provides the texture content. Each pair of pixels in C2 is assumed to have exactly the same transmission; thus, the constraint term is(19)Ox,y=0,fortx=ty∞,fortx≠ty,x,y∈C2.Despite the approach of occlusion handling being to avoid a false positive, it still occurs in some cases; e.g., several white containers on a cargo ship, as shown in Figure5(d) are recognized as a part of the sky and assigned transmissions lower than the reality. As mentioned above, perfect occlusion handling is nearly impossible for a low-level method, and there must be a tradeoff. However, as illustrated in Section 4, the benefit of such an approach is much more obvious than its defect in the haze removal results; thus, it is typically employed in the energy function.


### 3.6. The Solving Process

Based on the definition of each term, minimization of the energy function in (8) is a discrete optimization problem that can be solved by the graph cut-based algorithm. More specifically, the problem is a multi-label problem; for an eight-bit image, there are 256 labels for the transmission of each pixel, and the node number is exactly the pixel number. Because the distance measurement between t(x),t(y) in the smoothness term is not a metric, αβ-swap [35–37] is employed to solve the optimal solution, as shown in Figure 8(b).Figure 8
Example of regularization. (a) hazy image; (b) optimal solution based on the αβ-swap; (c) approximate solution based on α-expansion with several changes in the energy function; (d) final solution regularized from the approximate solution.
(a)
Hazy image (b)
Optimal solution (c)
Approximate solution (d)
After regularizationHowever, in practice, employing anαβ-swap for such a large graph with numerous labels is intolerably time-consuming. Therefore, two modifications are made: (a) while solving the transmissions, the input images are compressed into five-bits; thus, there are only 32 labels. (b) The distance measurement between t(x),t(y) is replaced by the L1-norm (i.e., tx-ty); thus, α-expansion [35–37] can be employed instead, which is more robust and efficienct. Figure 8(c) shows the solution after such modifications. As can be seen, it remains close to the optimal solution except at the several abrupt edges, and it is most obvious in the sky region. The method of Farbmanet al. [38] is employed to remove this defect by preserving the edges of the transmissions only on the positions that have notable edges of color. Figure 8(d) shows the final solution.Note that different from most methods we introduce in Section1 the solved initial estimate based on a group of assumptions. It is then regularized based on another group, we have a clear definition on what is the best case, and such approximation is no more than a fast approach. The concept of considering the assumptions simultaneously is still to be implemented. The difference can be determined by examining how clearly the employed assumptions are reflected in the solution and how close the three solutions in Figure 8 are compared to the initial and final results obtained from the other methods displayed in Figure 2,To better illustrate the proposed method, a flowchart is presented in Figure9. As discussed above, the energy function consists of the data term, smoothness term, and constraint term. The assumptions are considered simultaneously in the optimization process.Figure 9
Flowchart of the proposed method.


## 3.1. The Algorithm

The haze model of (1) is followed in this paper. In our practice, few hazy images do not contain a small part of the sky or dense haze region; the airlight estimation approach of Heet al. [3] maintains its efficient most of the time. Therefore, we simply employ it to estimate the airlight. The transmissions are estimated by searching an approximately optimal configuration of the proposed energy function. With a known airlight and transmissions, the scene radiance is recovered through(7)Jx=Ix-Atx+A.Considering the residual error, aerial perspective, and noise of the sky region [3], the transmissions are slightly increased by t=(t+0.1)/1.1 before dehazing, and the removal results are brightened by I=I0.9.


## 3.2. The Energy Function

The proposed energy function is(8)Et=∑xDx+∑x,y∈C1Kx,y+∑x,y∈C2Ox,y,where D is the data term, measuring the fidelity of the transmissions at each pixel, K is the smoothness term for the piecewise smoothness, and O is the constraint term for the occlusion handling. (x,y)∈C1 or C2 imply that the two pixels are neighbors in the image space or feature space.


## 3.3. The Data Term

The data term is derived from the internal constraints of the haze model and dark channel prior. Because the scene radiance is non-negative, we have(9)Jxtx=Ix-A+Atx≥0, and, thus,(10)tx≥1-IxA≥tLBx=1-mincIxA,where tLB(x) is the lower bound of the transmission of pixel x.Briefly, pixels satisfyingmincJc(x)=0 are called dark pixels. The transmission of each dark pixel has only one possibility: it is equal to its lower bound; this is because(11)tx=1-mincAc1-txAc=1-mincIcxAc=tLBx.A pixel in a hazy image can be confirmed to be a dark pixel only if it satisfies mincIc(x)=0, and there is still no additional evidence of the possible transmissions of most pixels. In this situation, the local constant assumption can be employed, but it may cause the block effect, as we have discussed above. In this paper, we propose another assumption that is much looser than the local constant assumption. For each pixel x, at least one dark pixel in Ω(x) has the same transmission as x (the dark pixel must exist in Ω(x) as illustrated in the dark channel prior).Because every pixel inΩ(x) is a potential dark pixel with transmission tLB(y), which might equal t(x), we collect all such potentials in a label set as(12)tx∈Lx=tLBy∣y∈Ωx,tLBy>tLBx,where Ω is a circular mask with radius r=0.1min⁡(w,h) (w,h are the image width and height). For any two values in L(x), prior or assumption cannot be used to determine that which is better. Thus, the data term is designed only to avoid the values not in L(x), as(13)Dx=0,fortx∈Lx∞,fortx∉Lx.Note that the data term plays a strong role in every pixel of the hazy image, and, thus, the defect of over-smoothness is suppressed.


## 3.4. Smoothness Term

Based on the piecewise smoothness assumption, the proposed smoothness term is(14)Kx,y=tx-ty2tLBx-tLBy2,x,y∈C1,where the set of adjacent pixel pairs, C1, which is defined in four-connected. The term is a traditional one, except that the similarity between the pixels is measured on tLB not on I, as in the studies by Fattal [14] and Bermanet al. [25]. This method starts from the intuition that two pixels may still have similar transmissions even if they are distinct in color. Instead, tLB is more in accordance with the intuition on the haze amount, and sometimes it is used as an initial estimate of the transmissions [9, 27]. An example is shown in Figure 4. In the measurement of the similarity in tLB(x), the result is slightly more consistent and obvious on the structures circled in yellow. Most of time, measuring on I or tLB has little difference, but on tLB is faster because it only contains one channel.Figure 4
Example of measuring the similarity of the pixels on different maps. From left to right: hazy image, the lower bound map of the transmissions, and the estimate of the transmissions by measuring the similarity in the hazy image and lower bound map.


## 3.5. Constraint Term

The constraint term is proposed to handle the occlusion problem. The key point of occlusion handling is to recognize the pixels from a common but divided scene. It is a challenging task and nearly impossible for a low-level method. However, we do not have to recognize all the pixels, but only need to establish several links between them, such as the link illustrated in Figure5(a). The sky region in the figure is divided by the wires of the suspension. With only the piecewise smoothness assumption, the divided sky regions have no relationship with the major one. They are considered as some types of white objects in the front and are assigned transmissions similar to their surroundings, as shown in Figure 5(c). By settling such links, these regions are collected and assigned reasonable values, as shown in Figure 5(d). Because several links are required, the method to search the pixels of a common but divided scene is allowed to be quite a true negative, making the problem easier to solve than before.Figure 5
Example of occlusion handling. (a) Hazy image and a link we intend to build; (b) hazy image with features, pixels marked in the same color are supposed to have similar transmissions; (c) estimate of the transmissions without occlusion handling; (d) estimate of the transmissions with occlusion handling.
(a)
Hazy image (b)
Linked pixels (c)
The estimate (d)
With occlusion handlingA novel feature is proposed as(15)Fx=Ix,I∗x,L∗x,where(16)I∗x=argmaxIy,y∈Ω∗xIx-Iy2,with Ω∗ being a small mask with side length l∗=0.02min⁡(w,h), and L∗ includes the texture content.(17)L∗x=tLBy∣y∈Ω∗x.The shorter the distance between F(x) and F(y), the higher the possibility that pixels x and y belong to a common scene. Figure 5(b) shows two sets of pixels with very similar features, marked as blue and yellow. By linking the pixels of yellow, several isolated sky regions are successfully related to the major one (there are two pixels of yellow in the major sky region). By linking the pixels of blue, several isolated regions are collected together. However, these regions are not linked to the major one directly; they are linked with the pixels of yellow color in the image space, as circled in green in the figure. Note that there are hundreds of such sets of pixels, which are scattered in the suspension. Numerous links are settled between them, and, finally, they seem to be dispersed sky regions, as an integral one.The first two elements of the feature in (15) are based on the intuitions that (a) there must be some notable difference between the divided scene and its covering; (b) the same colors frequently recur in the divided scene; (c) the same colors frequently recur in the covering. Figure 6 illustrates such intuitions. It shows several pixel-pairs with colors turning from white to brown, which can be found in abundance between the divided background and its covering: the tree. The relationships we intend to build can be easily settled by searching such pixel-pairs and linking them. However, this is rather a false positive method, and, as mentioned above, the method can only be true negative.Figure 6
Example of occlusion handling. From left to right: hazy image and several pixel-pairs with similar colors; and the estimate of the transmissions without and with occlusion handling.The last element of the feature is proposed to tighten the search criteria; it is based on the intuition that small patches centered at the edges between the divided scene and its covering contain similar colors. Then, the similarity in the color is replaced by the similarity inL∗ based on the following considerations: (a) pixels similar to each other in I are similar to those in tLB; (b) L∗ is easier to use than a set of colors; (c) if a set of pixels (x1,x2,…) with different label sets is found to have very strong relationships (suppose their transmissions must be the same), there might be fewer choices for their solutions (L(x1)∪L(x2)∪⋯→0). Therefore, the similarity in L∗ reflects the risk of the solution space reduction more intuitively.Considering the features of all the pixels in a hazy image yields a point cloud in the feature space. Pixels close to each other in the feature space should be linked and assigned a large weight and vice versa. Such links are built as a graph construction problem, which can be solved well based on thek-nearest neighbors method or the b-matching method [34]. However, the built graph might be extremely large to solve. In practice, we simplify the graph construction by partitioning the feature space into numerous super-blocks and only linking the pixels of the features in the same super-block. The partition is based on the first six dimensions of the feature space (RGB for I and I∗), and the super-blocks are not of the same size, as shown in Figure 7. The unequal partitioning is because all the intuitions described above occur in a haze-free image, and the conditions are stricter while the haze is denser, and, approximately, the color is whiter.Figure 7
Partition of the feature space. Pixels of the features located in the same block are considered to have the same transmission. Moreover, the whiter the colors, the smaller the block. Note that both I(x) and I∗(x) are three-dimensional, and the blocks displayed are actually super-blocks in a six-dimensional space.Given pixel-pair(x,y) with features in the common super-block, it is included in set C2 if it also satisfies(18)distL∗x,L∗yL∗x∪L∗y<0.2,where dist is the Hamming distance and L∗(x) provides the texture content. Each pair of pixels in C2 is assumed to have exactly the same transmission; thus, the constraint term is(19)Ox,y=0,fortx=ty∞,fortx≠ty,x,y∈C2.Despite the approach of occlusion handling being to avoid a false positive, it still occurs in some cases; e.g., several white containers on a cargo ship, as shown in Figure5(d) are recognized as a part of the sky and assigned transmissions lower than the reality. As mentioned above, perfect occlusion handling is nearly impossible for a low-level method, and there must be a tradeoff. However, as illustrated in Section 4, the benefit of such an approach is much more obvious than its defect in the haze removal results; thus, it is typically employed in the energy function.


## 3.6. The Solving Process

Based on the definition of each term, minimization of the energy function in (8) is a discrete optimization problem that can be solved by the graph cut-based algorithm. More specifically, the problem is a multi-label problem; for an eight-bit image, there are 256 labels for the transmission of each pixel, and the node number is exactly the pixel number. Because the distance measurement between t(x),t(y) in the smoothness term is not a metric, αβ-swap [35–37] is employed to solve the optimal solution, as shown in Figure 8(b).Figure 8
Example of regularization. (a) hazy image; (b) optimal solution based on the αβ-swap; (c) approximate solution based on α-expansion with several changes in the energy function; (d) final solution regularized from the approximate solution.
(a)
Hazy image (b)
Optimal solution (c)
Approximate solution (d)
After regularizationHowever, in practice, employing anαβ-swap for such a large graph with numerous labels is intolerably time-consuming. Therefore, two modifications are made: (a) while solving the transmissions, the input images are compressed into five-bits; thus, there are only 32 labels. (b) The distance measurement between t(x),t(y) is replaced by the L1-norm (i.e., tx-ty); thus, α-expansion [35–37] can be employed instead, which is more robust and efficienct. Figure 8(c) shows the solution after such modifications. As can be seen, it remains close to the optimal solution except at the several abrupt edges, and it is most obvious in the sky region. The method of Farbmanet al. [38] is employed to remove this defect by preserving the edges of the transmissions only on the positions that have notable edges of color. Figure 8(d) shows the final solution.Note that different from most methods we introduce in Section1 the solved initial estimate based on a group of assumptions. It is then regularized based on another group, we have a clear definition on what is the best case, and such approximation is no more than a fast approach. The concept of considering the assumptions simultaneously is still to be implemented. The difference can be determined by examining how clearly the employed assumptions are reflected in the solution and how close the three solutions in Figure 8 are compared to the initial and final results obtained from the other methods displayed in Figure 2,To better illustrate the proposed method, a flowchart is presented in Figure9. As discussed above, the energy function consists of the data term, smoothness term, and constraint term. The assumptions are considered simultaneously in the optimization process.Figure 9
Flowchart of the proposed method.


## 4. Experiment

The methods of Choiet al. [8], Menget al. [4], Heet al. [3], and Bermanet al. [25] are compared with the proposed method. We validate these methods by (a) quantitatively comparing the accuracy of the haze removal results on synthetic images; (b) comparing the quality of the transmission maps and haze removal results qualitatively on natural images. The codes for these methods, including the proposal, are available (Choi et al. [8], http://live.ece.utexas.edu/research/fog/index.html. Meng et al. [4], http://www.escience.cn/people/menggaofeng/research.html. He et al. [3], https://github.com/sjtrny/Dark-Channel-Haze-Removal. Berman et al. [25], https://github.com/danaberman/non-local-dehazing.).Most methods have their own strategy for estimating the airlight, but they generally differ slightly. We exclude this factor in our experiments by giving the estimate directly (for synthetic images) or uniformly as by Heet al. [3] (for natural images). The haze removal results tend to be dark [3], and different methods employ different enhancement approaches to improve the visual appeal. We also exclude this factor by employing a gamma correction (the index is 0.9) uniformly, except for enhancement-based methods, which should consider this problem internally.


### 4.1. Quantitative Results

For real hazy images, the ground truth of the scene radiance and transmission map is not available. Thus the quantitative comparison is made on synthetic images. Two datasets of synthetic images are employed: D-HAZY [31] and the one of Fattal [14]. D-HAZY is built based on the Middelbury and NYU Depth datasets. Only a part of the Middlebury dataset is used because of the quality of the depth map.Fattal [14] produced hazy images by combining a haze-free image and an arbitrary airlight, which might be counterintuitive, such as, in case a white balanced image is covered with a mazarine haze. In fact, the airlight is related to the haze-free image as J=ρA [39], where ρ is the reflectance. This implies that the airlight is a ground truth and a casual assumption might be unfair. Therefore, we reproduce the dataset by employing a white airlight, which might not be exactly the truth but is much more natural. Furthermore, the sky regions in the haze-free images are also painted white according to the airlight. Another problem of the dataset is that the ground truths of the transmission map are of low quality; thus, we improve them manually. A comparison of the original and improved samples is shown in Figure 10 (pay attention to the sky region in the hazy and haze-free images; the hollows and border in the right in the transmission map).Figure 10
Example of the modification to on the dataset of Fattal [14]. Columns from left to right: the original hazy image in the dataset of Fattal [14] and the modified one, ground truths of the haze-free image; and ground truths of the transmission map.Because the ground truths of haze-free images are known, SSIM [40] (the structural similarity index) is employed to evaluate the results. Table 1 lists the results using D-HAZY and illustrates the outstanding of the proposed method. Figure 11 shows the sample ofMask in Table 1. The result of Choiet al. [8] is gloomy. The result of Menget al. [4] has an obvious halo-effect in the vicinity of the straw hat and left-hand basket. The result of Heet al. [3] is veiled but is obvious on the wall, chairs, and left-hand basket. The results of Bermanet al. [25] are over brightened and saturated; furthermore, the table appears inconsistent with the brightness on the right. Comparatively, our results have the tone and brightness close to the ground truth, and the above problems do not appear.Table 1
SSIM comparison on D-HAZY [31].
Choi [8] Meng [4] He [3] Berman [25] Ours Adiron. 0.733 0.912 0.871 0.849 0.896 Backpack 0.720 0.901 0.915 0.790 0.888 Bicycle1 0.835 0.728 0.868 0.823 0.784 Cable 0.610 0.653 0.561 0.545 0.652 Class.1 0.668 0.859 0.877 0.862 0.890 Couch 0.775 0.779 0.763 0.664 0.826 Flowers 0.660 0.803 0.814 0.841 0.837 Jadeplant 0.566 0.675 0.648 0.670 0.721 Mask 0.644 0.798 0.809 0.703 0.818 Motor. 0.689 0.823 0.802 0.672 0.794 Piano 0.748 0.802 0.845 0.836 0.857 Pipes 0.633 0.689 0.651 0.590 0.707 Playroom 0.767 0.828 0.827 0.831 0.854 Playtable 0.729 0.875 0.872 0.772 0.876 Recycle 0.619 0.874 0.890 0.841 0.895 Shelves 0.776 0.951 0.951 0.901 0.903 Shopvac 0.412 0.738 0.620 0.532 0.660 Sticks 0.807 0.956 0.933 0.872 0.954 Storage 0.653 0.856 0.859 0.840 0.860 Sword1 0.644 0.898 0.862 0.675 0.901 Sword2 0.550 0.895 0.604 0.550 0.878 Umbrella 0.674 0.751 0.747 0.677 0.760 Vintage 0.934 0.792 0.932 0.793 0.873  mean 0.689 0.819 0.805 0.745 0.829 median 0.674 0.823 0.845 0.790 0.857Figure 11
Example of the comparison based on D-HAZY [31]. The hazy image is namedMask in the datasets.Table2 lists the results based on the samples modified from the datasets of Fattal [14] (the mean and median are not displayed owing to the small sample size). The same with the configuration of Fattal [14]; for each sample of the haze-free image, three extra tests are performed by adding Gaussian noises of different standard deviations σ to the hazy image. As shown, our method also has a competitive performance, and the rank is not much affected by the noise in the samples,Mansion andReindeer.Table 2
SSIM comparison using synthetic images based on dataset of Fattal [14].
Choi [8] Meng [4] He [3] Berman [25] Ours Church 0.826 0.921 0.931 0.901 0.936 σ=0.010 0.764 0.680 0.824 0.867 0.718 σ=0.025 0.602 0.564 0.636 0.709 0.560 σ=0.050 0.436 0.429 0.453 0.474 0.406  Lawn 0.668 0.943 0.910 0.915 0.919 σ=0.010 0.634 0.864 0.835 0.867 0.844 σ=0.025 0.569 0.734 0.709 0.708 0.714 σ=0.050 0.462 0.555 0.543 0.498 0.537  Mansion 0.815 0.920 0.920 0.893 0.930 σ=0.010 0.790 0.897 0.895 0.865 0.910 σ=0.025 0.729 0.820 0.818 0.777 0.847 σ=0.050 0.632 0.688 0.674 0.643 0.698  Raindeer 0.783 0.897 0.924 0.883 0.919 σ=0.010 0.775 0.885 0.907 0.869 0.910 σ=0.025 0.743 0.841 0.848 0.803 0.870 σ=0.050 0.672 0.745 0.736 0.680 0.766  Road 0.699 0.896 0.881 0.910 0.911 σ=0.010 0.658 0.779 0.810 0.878 0.801 σ=0.025 0.547 0.672 0.677 0.730 0.672 σ=0.050 0.424 0.517 0.513 0.515 0.510


### 4.2. Qualitative Results

Figures12–15 show the comparisons performed on natural images (Choiet al. [8] did not estimate the transmissions). These methods are evaluated by examining the performances based on: global, local, and detailed appearance, and occlusion handling. While comparing, the performance of each method is recorded in Table 3, where a check implies the method performs robustly well compared to the other methods, a circle represents that sometimes failure occurs, and cross implies that the method is comparatively weak.Table 3
Comparison of the problems avoiding natural images.
Choi [8] Meng [4] He [3] Berman [25] Ours Global appr. × √ √ ○ √ Local appr. √ × ○ × ○ Detail appr. × × × ○ √Figure 12
Comparison on a natural image ofTian’anmen.
(a)
Hazy image (b)
Meng et al. [4] (c)
He et al. [3] (d)
Choi et al. [8] (e)
Berman et al. [25] (f)
OursFigure 13
Comparison on a natural image of aForest.
(a)
Hazy image (b)
Meng et al. [4] (c)
He et al. [3] (d)
Choi et al. [8] (e)
Berman et al. [25] (f)
OursFigure 14
Comparison on natural imageRiver.
(a)
Hazy image (b)
Meng et al. [4] (c)
He et al. [3] (d)
Choi et al. [8] (e)
Berman et al. [25] (f)
OursFigure 15
Comparison on a natural image ofPumpkins.
(a)
Hazy image (b)
Meng et al. [4] (c)
He et al. [3] (d)
Choi et al. [8] (e)
Berman et al. [25] (f)
OursGlobal appearance implies how much the haze removal result is in accordance with the intuition on the brightness, saturation, or tone in general. For physical-based method with a well estimated airlight, counter-intuitional brightness or saturation is found and the transmission map is overestimated or underestimated globally. It can be derived from (7) that(20)dJxdtx=A-Ix1t2x,which implies that the intensity of each channel of the haze removal results (affects the brightness and saturation) has a positive correlation with the transmissions. In the displayed samples, the transmission maps of Bermanet al. [25] are underestimated; thus, the results are gloomy or oversaturated, as is obvious in the samples ofTian’anmen,Forest, andRiver. In comparison, the transmission maps of Heet al. [3] are robust in an appropriate range, and the results are of reasonable brightness and saturation. The proposed method remains close to that of Heet al. [3] because both are based on the dark channel prior. The transmission maps of Menget al. [4] are slightly higher than the ones of Heet al. [3], but do not affect the haze removal results noticeably. The results of Choiet al. [8] are the most gloomy and appear to have an unnatural tone.Local appearance is evaluated by examining the colors of specific scenes or objects. In the result of Menget al. [4], as shown in Figure 12, there is a serious halo-effect in the vicinity of the building and the trees on its right. The transmissions appear to gradually vary on the boundaries. In the result of Menget al. [4], as shown in Figure 13, the transmission edges are not according to the color edges, and in the result, the method produces several blocks of haze. It seems that the estimated transmission of Menget al. [4] is not well constrained in the image space. The local linear assumption employed by Heet al. [3] and the piecewise smoothness assumption employed by Bermanet al. [25] and the proposal counter this phenomenon. Therefore, the problem is avoided in the results of these methods.In the result of Bermanet al. [25], Figure 13, the path suddenly becomes gloomy and the leaves turn burned dark in the middle of the scope. This failure is also reflected by the transmissions, which do not reflect these structures consistently. Along with the path, the transmissions fast fall to to reach a low value. In comparison, the transmissions of the other methods fall more gently, resulting in a less gloomy result. Along with the branches and leaves in the front, the transmissions of Bermanet al. [25] also fall, whereas the transmissions of the other methods steadily maintain high values. A similar phenomenon also occurs in the result of Bermanet al. [25], as shown in Figure 14, where the lawn suddenly turns gloomy in the left.In the result of the proposed method, as depicted in Figure14, the goose on the left is colored in purple. The transmissions of the goose are underestimated owing to a false positive case of the occlusion handling, which links the pixels of the goose with the background.In the result of Heet al. [3], as depicted in Figure 13, the sky region is blurry in the transmission map, mixed with the region of the distant path. Thus, the branches distant from this mixed region are veiled compared to the other methods.The local appearances of the results of Choiet al. [8] are consistent with their global appearances, and do not show abrupt changes.Detailed appearance is examined in small regions, textures, and edges. Despite the transmission maps of Heet al. [3] and the proposal being similar to the global appearance, the haze removal results of Heet al. [3] typically have a less contrast than results. This is because redundant details are present in its transmission maps, as proved by Fattal [14]. This problem exists obviously in all the transmission maps of Heet al. [3], which reflect all the color variance in the hazy image without any differences. The root of the problem is the employment of the local linear assumption while concealed in the Laplacian matting (or guided filter), which is also widely employed by the latest dehazing methods [15, 17, 19]. In the result of Choiet al. [8], the details are fully lost in the front of Figure 12 and tree of Figure 14.The occlusion problem also undermines the haze removal result in detail. It easily occurs in the regions of: the gaps between the flags and pillars, as in Figure12; the gaps among the branches on the right scope, as in Figure 13; and the background behind the gaps of the trunk, as in Figure 14. Without occlusion handling, these regions appear to be veiled and their transmissions tend to be overestimated. The method of Menget al. [4] barely handles this problem. The method of Bermanet al. [25] is not patch-based and thus, is affected slghtly in its transmission maps. However, this advantage is unstable owing to the defect of over-smoothness, such as shown in the result in Figure 12, which fails at occlusion handling. The method of Heet al. [3] also slightly handles this problem because of the local linear assumption. Although its transmission maps reflect the variances, they do not do this completely in these regions. In Figure 13, the gaps among the branches are still veiled. In comparison, the proposed method performs well in these regions and is most outstanding for the gaps in Figure 13.A summary of the comparisons is presented in Table3. The proposed method performs well in most respects. The employed assumptions are truly reflected in the result. The dark channel prior fully shows its power and leads to a good performance in terms of the global appearance. The piecewise smoothness assumption and occlusion handling lead to a good performance for the local and detailed appearances. A shortcoming of the proposal is that the occlusion handling might lead to inappropriate links, causing underestimation of the transmissions, as shown in Figure 14. However, under a general condition, it simply introduces some details in the transmissions, as shown in Figure 15, where the effects on the haze removal results are unnoticeable. However, it provides a significant improvement once the occlusion problem occurs. Therefore, the function is typically valid if manual selection is undesired.


## 4.1. Quantitative Results

For real hazy images, the ground truth of the scene radiance and transmission map is not available. Thus the quantitative comparison is made on synthetic images. Two datasets of synthetic images are employed: D-HAZY [31] and the one of Fattal [14]. D-HAZY is built based on the Middelbury and NYU Depth datasets. Only a part of the Middlebury dataset is used because of the quality of the depth map.Fattal [14] produced hazy images by combining a haze-free image and an arbitrary airlight, which might be counterintuitive, such as, in case a white balanced image is covered with a mazarine haze. In fact, the airlight is related to the haze-free image as J=ρA [39], where ρ is the reflectance. This implies that the airlight is a ground truth and a casual assumption might be unfair. Therefore, we reproduce the dataset by employing a white airlight, which might not be exactly the truth but is much more natural. Furthermore, the sky regions in the haze-free images are also painted white according to the airlight. Another problem of the dataset is that the ground truths of the transmission map are of low quality; thus, we improve them manually. A comparison of the original and improved samples is shown in Figure 10 (pay attention to the sky region in the hazy and haze-free images; the hollows and border in the right in the transmission map).Figure 10
Example of the modification to on the dataset of Fattal [14]. Columns from left to right: the original hazy image in the dataset of Fattal [14] and the modified one, ground truths of the haze-free image; and ground truths of the transmission map.Because the ground truths of haze-free images are known, SSIM [40] (the structural similarity index) is employed to evaluate the results. Table 1 lists the results using D-HAZY and illustrates the outstanding of the proposed method. Figure 11 shows the sample ofMask in Table 1. The result of Choiet al. [8] is gloomy. The result of Menget al. [4] has an obvious halo-effect in the vicinity of the straw hat and left-hand basket. The result of Heet al. [3] is veiled but is obvious on the wall, chairs, and left-hand basket. The results of Bermanet al. [25] are over brightened and saturated; furthermore, the table appears inconsistent with the brightness on the right. Comparatively, our results have the tone and brightness close to the ground truth, and the above problems do not appear.Table 1
SSIM comparison on D-HAZY [31].
Choi [8] Meng [4] He [3] Berman [25] Ours Adiron. 0.733 0.912 0.871 0.849 0.896 Backpack 0.720 0.901 0.915 0.790 0.888 Bicycle1 0.835 0.728 0.868 0.823 0.784 Cable 0.610 0.653 0.561 0.545 0.652 Class.1 0.668 0.859 0.877 0.862 0.890 Couch 0.775 0.779 0.763 0.664 0.826 Flowers 0.660 0.803 0.814 0.841 0.837 Jadeplant 0.566 0.675 0.648 0.670 0.721 Mask 0.644 0.798 0.809 0.703 0.818 Motor. 0.689 0.823 0.802 0.672 0.794 Piano 0.748 0.802 0.845 0.836 0.857 Pipes 0.633 0.689 0.651 0.590 0.707 Playroom 0.767 0.828 0.827 0.831 0.854 Playtable 0.729 0.875 0.872 0.772 0.876 Recycle 0.619 0.874 0.890 0.841 0.895 Shelves 0.776 0.951 0.951 0.901 0.903 Shopvac 0.412 0.738 0.620 0.532 0.660 Sticks 0.807 0.956 0.933 0.872 0.954 Storage 0.653 0.856 0.859 0.840 0.860 Sword1 0.644 0.898 0.862 0.675 0.901 Sword2 0.550 0.895 0.604 0.550 0.878 Umbrella 0.674 0.751 0.747 0.677 0.760 Vintage 0.934 0.792 0.932 0.793 0.873  mean 0.689 0.819 0.805 0.745 0.829 median 0.674 0.823 0.845 0.790 0.857Figure 11
Example of the comparison based on D-HAZY [31]. The hazy image is namedMask in the datasets.Table2 lists the results based on the samples modified from the datasets of Fattal [14] (the mean and median are not displayed owing to the small sample size). The same with the configuration of Fattal [14]; for each sample of the haze-free image, three extra tests are performed by adding Gaussian noises of different standard deviations σ to the hazy image. As shown, our method also has a competitive performance, and the rank is not much affected by the noise in the samples,Mansion andReindeer.Table 2
SSIM comparison using synthetic images based on dataset of Fattal [14].
Choi [8] Meng [4] He [3] Berman [25] Ours Church 0.826 0.921 0.931 0.901 0.936 σ=0.010 0.764 0.680 0.824 0.867 0.718 σ=0.025 0.602 0.564 0.636 0.709 0.560 σ=0.050 0.436 0.429 0.453 0.474 0.406  Lawn 0.668 0.943 0.910 0.915 0.919 σ=0.010 0.634 0.864 0.835 0.867 0.844 σ=0.025 0.569 0.734 0.709 0.708 0.714 σ=0.050 0.462 0.555 0.543 0.498 0.537  Mansion 0.815 0.920 0.920 0.893 0.930 σ=0.010 0.790 0.897 0.895 0.865 0.910 σ=0.025 0.729 0.820 0.818 0.777 0.847 σ=0.050 0.632 0.688 0.674 0.643 0.698  Raindeer 0.783 0.897 0.924 0.883 0.919 σ=0.010 0.775 0.885 0.907 0.869 0.910 σ=0.025 0.743 0.841 0.848 0.803 0.870 σ=0.050 0.672 0.745 0.736 0.680 0.766  Road 0.699 0.896 0.881 0.910 0.911 σ=0.010 0.658 0.779 0.810 0.878 0.801 σ=0.025 0.547 0.672 0.677 0.730 0.672 σ=0.050 0.424 0.517 0.513 0.515 0.510


## 4.2. Qualitative Results

Figures12–15 show the comparisons performed on natural images (Choiet al. [8] did not estimate the transmissions). These methods are evaluated by examining the performances based on: global, local, and detailed appearance, and occlusion handling. While comparing, the performance of each method is recorded in Table 3, where a check implies the method performs robustly well compared to the other methods, a circle represents that sometimes failure occurs, and cross implies that the method is comparatively weak.Table 3
Comparison of the problems avoiding natural images.
Choi [8] Meng [4] He [3] Berman [25] Ours Global appr. × √ √ ○ √ Local appr. √ × ○ × ○ Detail appr. × × × ○ √Figure 12
Comparison on a natural image ofTian’anmen.
(a)
Hazy image (b)
Meng et al. [4] (c)
He et al. [3] (d)
Choi et al. [8] (e)
Berman et al. [25] (f)
OursFigure 13
Comparison on a natural image of aForest.
(a)
Hazy image (b)
Meng et al. [4] (c)
He et al. [3] (d)
Choi et al. [8] (e)
Berman et al. [25] (f)
OursFigure 14
Comparison on natural imageRiver.
(a)
Hazy image (b)
Meng et al. [4] (c)
He et al. [3] (d)
Choi et al. [8] (e)
Berman et al. [25] (f)
OursFigure 15
Comparison on a natural image ofPumpkins.
(a)
Hazy image (b)
Meng et al. [4] (c)
He et al. [3] (d)
Choi et al. [8] (e)
Berman et al. [25] (f)
OursGlobal appearance implies how much the haze removal result is in accordance with the intuition on the brightness, saturation, or tone in general. For physical-based method with a well estimated airlight, counter-intuitional brightness or saturation is found and the transmission map is overestimated or underestimated globally. It can be derived from (7) that(20)dJxdtx=A-Ix1t2x,which implies that the intensity of each channel of the haze removal results (affects the brightness and saturation) has a positive correlation with the transmissions. In the displayed samples, the transmission maps of Bermanet al. [25] are underestimated; thus, the results are gloomy or oversaturated, as is obvious in the samples ofTian’anmen,Forest, andRiver. In comparison, the transmission maps of Heet al. [3] are robust in an appropriate range, and the results are of reasonable brightness and saturation. The proposed method remains close to that of Heet al. [3] because both are based on the dark channel prior. The transmission maps of Menget al. [4] are slightly higher than the ones of Heet al. [3], but do not affect the haze removal results noticeably. The results of Choiet al. [8] are the most gloomy and appear to have an unnatural tone.Local appearance is evaluated by examining the colors of specific scenes or objects. In the result of Menget al. [4], as shown in Figure 12, there is a serious halo-effect in the vicinity of the building and the trees on its right. The transmissions appear to gradually vary on the boundaries. In the result of Menget al. [4], as shown in Figure 13, the transmission edges are not according to the color edges, and in the result, the method produces several blocks of haze. It seems that the estimated transmission of Menget al. [4] is not well constrained in the image space. The local linear assumption employed by Heet al. [3] and the piecewise smoothness assumption employed by Bermanet al. [25] and the proposal counter this phenomenon. Therefore, the problem is avoided in the results of these methods.In the result of Bermanet al. [25], Figure 13, the path suddenly becomes gloomy and the leaves turn burned dark in the middle of the scope. This failure is also reflected by the transmissions, which do not reflect these structures consistently. Along with the path, the transmissions fast fall to to reach a low value. In comparison, the transmissions of the other methods fall more gently, resulting in a less gloomy result. Along with the branches and leaves in the front, the transmissions of Bermanet al. [25] also fall, whereas the transmissions of the other methods steadily maintain high values. A similar phenomenon also occurs in the result of Bermanet al. [25], as shown in Figure 14, where the lawn suddenly turns gloomy in the left.In the result of the proposed method, as depicted in Figure14, the goose on the left is colored in purple. The transmissions of the goose are underestimated owing to a false positive case of the occlusion handling, which links the pixels of the goose with the background.In the result of Heet al. [3], as depicted in Figure 13, the sky region is blurry in the transmission map, mixed with the region of the distant path. Thus, the branches distant from this mixed region are veiled compared to the other methods.The local appearances of the results of Choiet al. [8] are consistent with their global appearances, and do not show abrupt changes.Detailed appearance is examined in small regions, textures, and edges. Despite the transmission maps of Heet al. [3] and the proposal being similar to the global appearance, the haze removal results of Heet al. [3] typically have a less contrast than results. This is because redundant details are present in its transmission maps, as proved by Fattal [14]. This problem exists obviously in all the transmission maps of Heet al. [3], which reflect all the color variance in the hazy image without any differences. The root of the problem is the employment of the local linear assumption while concealed in the Laplacian matting (or guided filter), which is also widely employed by the latest dehazing methods [15, 17, 19]. In the result of Choiet al. [8], the details are fully lost in the front of Figure 12 and tree of Figure 14.The occlusion problem also undermines the haze removal result in detail. It easily occurs in the regions of: the gaps between the flags and pillars, as in Figure12; the gaps among the branches on the right scope, as in Figure 13; and the background behind the gaps of the trunk, as in Figure 14. Without occlusion handling, these regions appear to be veiled and their transmissions tend to be overestimated. The method of Menget al. [4] barely handles this problem. The method of Bermanet al. [25] is not patch-based and thus, is affected slghtly in its transmission maps. However, this advantage is unstable owing to the defect of over-smoothness, such as shown in the result in Figure 12, which fails at occlusion handling. The method of Heet al. [3] also slightly handles this problem because of the local linear assumption. Although its transmission maps reflect the variances, they do not do this completely in these regions. In Figure 13, the gaps among the branches are still veiled. In comparison, the proposed method performs well in these regions and is most outstanding for the gaps in Figure 13.A summary of the comparisons is presented in Table3. The proposed method performs well in most respects. The employed assumptions are truly reflected in the result. The dark channel prior fully shows its power and leads to a good performance in terms of the global appearance. The piecewise smoothness assumption and occlusion handling lead to a good performance for the local and detailed appearances. A shortcoming of the proposal is that the occlusion handling might lead to inappropriate links, causing underestimation of the transmissions, as shown in Figure 14. However, under a general condition, it simply introduces some details in the transmissions, as shown in Figure 15, where the effects on the haze removal results are unnoticeable. However, it provides a significant improvement once the occlusion problem occurs. Therefore, the function is typically valid if manual selection is undesired.


## 5. Conclusion

In this paper, we emphasize on the problem that in single-image dehazing most previous methods do not consider their priors or assumptions simultaneously. Commonly, these methods provide an initial estimate of the transmission map based on one group of assumptions and then regularize it based on another group. The fidelity measurement in the regularization in the previous methods is the distance between the initial estimate and solution and did not correctly reflect the degree of compliance with the previous assumptions, thus undermining them.Several well-known priors and assumptions are introduced, and the relative methods employing them are also discussed. An energy function that collects selective priors and assumptions is proposed. It provides a clear definition of the desired transmission map. Furthermore, a delicate approach to use the dark channel prior with a novel feature for occlusion handling is proposed. The transmission map is estimated by searching the optimal configuration of the energy function, and all the employed priors and assumptions are considered simultaneously in the solving process. Comparison with the-state-of-the-art methods shows that the proposed method delivers an outstanding performance.Because the major prior employed in the proposal is the dark channel prior, our method also inherits the limitation of the prior: when the scene objects are inherently similar to the atmospheric light and no shadow is cast on them, the dark channel prior is invalid [3]. Our method underestimates the transmission of these objects. Another problem is that the proposed feature for occlusion handling is a false positive in some cases, finally resulting in an underestimation of the transmissions. However, the framework of the energy minimization can be easily extended. For any newly generated prior, an assumption or a feature can be tested to construct new terms and produce better results.



---

*Source: 1651958-2019-03-24.xml*