# Retinex-Based Relighting for Night Photography
## 3 Proposed Method
### Illuminating Saliency Map  
For illuminating images, lighting source points should be in human-visual saliency areas. The well-known saliency map [48] indicates that the human eyes focus on distinctive areas in the natural scene. For relighting night photography, we assume that the light source points should be located in high-reflectance areas. Therefore, the saliency map for this relighting is based on the reflectance map. A simple solution uses reflectance images directly. However, there are some half pixels due to blurring or object boundaries. When the new lighting point is in a mixed area, lights spread across both foreground and background objects. Therefore, we erode the reflectance image and diffuse it using edge-preserving filtering to avoid this problem. First, we convert the image to grayscale and apply SSR to extract the salient light areas in the input image. Examples of input color and grayscale images, \(\mathbf{L}\), and \(\mathbf{R}\) from SSR are shown in Figure 2, Figure 3a, and Figure 3b, respectively. To compute SSR, we use a sliding-DCT-based convolution [25; 49; 50; 51], which has a constant-time property per pixel for convolution. The FFT computational order is \(O(\log N)\) per pixel, where \(N\) is the number of image pixels, while that of sliding-DCT is \(O(1)\). SSR is known to produce white shadows near the boundaries of objects, i.e., halo artifacts. The halo causes problems in the next step of dithering. Therefore, we erode the reflectance area to extract the more pronounced light component without the halo effect by min-convolution and iterative joint bilateral filtering (JBF) [22; 23]. Next, we erode the reflectance image \(\mathbf{R}\), which is defined as follows:  
\[\mathbf{R}_{\mathbf{p}}^{\text{min}}=\inf_{q\in\Omega_{\mathbf{p}}^{\text{min}}}\mathbf{R}_{q}, \tag{4}\]  
where \(\Omega_{\mathbf{p}}^{\text{min}}\) is a set for the convolution area, also called min-convolution. We use OpenCV's implementation, which is \(O(1)\) order per pixel and is based on recursive filtering [24]. After the min-convolution, we diffuse the processed reflectance map using an iterative version of JBF. JBF is a variant of bilateral filtering (BF) [52], and the difference between BF and JBF is only the guidance signal for measuring color distance. BF measures the color distance for kernel weight from the filtering target of the input signal, while JBF measures  
Figure 3: Luminance \(\mathbf{L}\) and reflectance \(\mathbf{R}\) for each method: \(\mathbf{L}\) is shared for each method, and the output is normalized for visibility. The proposed reflectance is SSR+min-filter+JBF. Figure 2: Input color and its grayscale images. it from an additional guide signal. Let the guide signal be the input \(\mathbf{f}\) and the \(t\)-th iterative input and output signal be \(\mathbf{S}_{\mathbf{p}}^{t}\) and \(\mathbf{S}_{\mathbf{p}}^{t+1}\), respectively. Iterative JBF is defined by  
\[\mathbf{S}_{\mathbf{p}}^{t+1}=\frac{\sum_{\mathbf{q}\in\Gamma_{\mathbf{p}}^{\text{opt}}}w_{s}( \mathbf{p},\mathbf{q})w_{r}(\mathbf{f}_{\mathbf{p}},\mathbf{f}_{\mathbf{q}})\mathbf{S}_{\mathbf{q}}^{t}}{\sum_{ \mathbf{q}\in\Gamma_{\mathbf{p}}^{\text{opt}}}w_{s}(\mathbf{p},\mathbf{q})w_{r}(\mathbf{f}_{\mathbf{ p}},\mathbf{f}_{\mathbf{q}})}, \tag{5}\]  
where \(\mathbf{S}^{1}=\mathbf{R}^{\text{min}}\), \(w_{s}:\mathcal{S}\times\mathcal{S}\mapsto\mathbb{R}\) is a spatial kernel, and \(w_{r}:\mathcal{R}\times\mathcal{R}\mapsto\mathbb{R}\) is a range kernel. The kernels are based on the Gaussian distribution:  
\[w_{s}(\mathbf{p},\mathbf{q})=\exp\left(-\frac{\|\mathbf{q}-\mathbf{p}\|_{2}^{2}}{2\sigma_{s}^ {2}}\right),\quad w_{r}(\mathbf{f}_{\mathbf{p}},\mathbf{f}_{\mathbf{q}})=\exp\left(-\frac{\| \mathbf{f}_{\mathbf{q}}-\mathbf{f}_{\mathbf{p}}\|_{2}^{2}}{2\sigma_{r}^{2}}\right), \tag{6}\]  
where \(\sigma_{s}\in\mathbb{R}_{+}\) is a spatial scale and \(\sigma_{r}\in\mathbb{R}_{+}\) is a range scale.