# HarmoDehazeDiffusion: Multi-Scale Spatial Fusion Network for Dense Haze Removal
## 3 The Method
### The Depth Fusion Diffusion Model  
When the camera's position is fixed, the concentration and depth of haze in specific regions of an image are closely related. Therefore, depth information is crucial for haze removal. Based on this observation, unlike previous image dehazing works [38] that have used depth estimation to augment the dataset, we propose a Depth Fusion Diffusion Model (DFDM) to incorporate a depth map as a conditional guide for the step-by-step denoising process of the diffusion model to enhance the network's perception of image depth.  
The conditional diffusion model [34; 35; 36] learns a conditional reverse process \(p_{\theta}(x_{0:T}\mid y)\) without modifying the diffusion process \(\mathrm{q}(x_{0:T}\mid x_{0})\) for \(fx\), such that the sampled image has high fidelity to the data distribution conditioned on \(y\). During training, we sample \((x_{0},\,y,D)\sim\mathrm{q}(x,\,y)\) from a data distribution (e.g., the haze-free image \(x\), the hazy image \(y\), and the depth map \(D\)). Our training approach is outlined in Algorithm 1. We learn the depth-aware reverse process  
Figure 1: Overview of the proposed HarmoDehazeDiffusion. Our model first uses a pre-trained depth estimation model to predict the depth map of the input image. Then, we concatenate the depth map with the original image to guide the denoising process of the Depth Fusion Diffusion Model. In the diffusion model, we enhance the network’s ability to perceive long-range features using a multi-scale feature extraction layer and use a spatial fusion module to fuse the shallow texture information and deep lighting information. Finally, an RGB haze-free image is obtained.  
\[p_{\theta}(x_{0:T}|y,d)=p(x_{T})\prod_{t=1}^{T}p_{\theta}(x_{t-1}|x_{t},y,D), \tag{1}\]  
We can marginalize the Gaussian diffusion process and directly sample the intermediate term \(x_{t}\) from the haze-free image \(x_{0}\) via \(x_{t}=\sqrt{\delta_{t}}x_{0}+\sqrt{1-\delta_{t}}e\), \(\alpha_{t}=1-\beta_{t}\), \(\delta_{t}=\prod_{t=1}^{T}\alpha_{i}\), and \(\epsilon\sim\mathcal{N}(0,I)\), where \(\beta_{t}\) is the noise schedule. It has the same dimensionality as \(x_{0}\). The denoising network \(Den\) takes the hazy image \(y\), the intermediate variable \(x_{t}\), the depth map \(D\), and the time step t as its input, predicting the noise map \(e_{t}\) as follows:  
\[e_{t}=Den(\sqrt{\delta_{t}}x_{0}+\sqrt{1-\delta_{t}}\epsilon,y,D,t). \tag{2}\]  
Following [31], the diffusion objective function is  
\[\mathcal{L}_{diff}=\mathbb{E}_{x_{0},t,\epsilon}\|e_{t}-\epsilon_{t}\|_{F}^{2}. \tag{3}\]  
It is noteworthy that we avoided directly applying a depth estimation method [37] to hazy images for depth prediction. This strategic decision stems from the limitation of atmospheric scattering in hazy conditions causing detail degradation and texture loss, which would lead to some deviations in the estimations when applying depth estimation algorithms not specifically trained on such degraded scenarios. To address this critical challenge, we implemented a data refinement strategy: First, we utilized the method [37] to estimate depth maps from the corresponding haze-free images in the dataset. Subsequently, we constructed training pairs by combining these estimated depth maps with their original hazy counterparts. This synthesized dataset was then employed to fine-tune the depth estimation model through transfer learning, enabling it to adaptively learn the latent relationship between haze-affected visual patterns and their underlying depth information.  
```
Input: The hazy image \(y\), the non-hazy image \(x\), and the depth map \(D\) whilenot convergeddo
1\(t\sim\text{Uniform}\{1,\dots,T\}\);
2\(\epsilon\sim\mathcal{N}(0,I)\);
3\(e_{t}=Den(x_{t},y,D,t)\);
4 Perform gradient descent steps on \(\
abla_{\theta}\mathcal{L}_{\text{total}}\left(\theta\right)\);
5
6 end while return\(\theta\)
```  
**Algorithm 1**Depth Fusion Diffusion Model.