# CNN-Based Illumination Estimation with Semantic Information
## 2 Technical Approach
### Color Constancy Approach  
In general, given an RGB image, F, the color constancy approach is designed to estimate the global illuminant color \(I_{g}=(\text{r},\text{ g},\text{ b})\) (or color cast), using a canonical light source color, usually perfect white \(\left(\frac{1}{\sqrt{3}},\ \frac{1}{\sqrt{3}},\ \frac{1}{\sqrt{3}}\right)^{T}\), and normalize the estimated global illuminant color \(\hat{I}_{g}=\frac{\hat{I}_{g}}{\|\hat{I}_{g}\|}\). The approach then replaces the estimated global illuminant color with the normalized global illuminant color. However, in real-life scenes, there exist multiple illuminants, which possibly impact on the perceived color of an object. To address this problem, conventional methods attempt to estimate a single global illuminant color. Similarly, the proposed approach is designed to estimate \(f_{\theta}\) and get the replacement, \(f_{\theta}(F)=\hat{I}_{g}\), and notably uses the convolutional neural network (CNN) to estimate \(f_{\theta}\), which gets the replacement closer to the ground truth illuminant color. \(\theta\) refers to parameters. Let \(\hat{I}_{g}^{*}\) defined as the ground truth illuminant color. During dataset learning, the CNN minimizes a loss function. The loss function represents an angular error (in degrees) between the estimated color \(\hat{I}_{g}\) and the ground truth illuminant color \(\hat{I}_{g}^{*}\), described as follows:  
\[L(\hat{I}_{g})=\frac{180}{\pi}cos^{-1}(\hat{I}_{g}\times\hat{I}_{g}^{*}) \tag{1}\]  
In the CNN, \(f_{\theta}\) is the estimation of all the semantically informative regions, ideally avoiding any repercussion of ambiguous light. Equation (2) explains how to calculate the final global illumination estimation. Let \(R=\{R_{1},R_{2},R_{3},\cdots,\ R_{n}\}\) be the local regions in \(F\), and \(\text{g}(R_{i});\ i=1,\ 2,\ 3,\ \cdots,\ n\) be the output of the regional illuminant color estimation, \(R_{i}\). The \(f_{\theta}(F)\) is the normalization of the sum of the product of semantic information, \(\text{c}(R_{i})\), and regional illumination estimation, \(\text{g}(R_{i})\), and as a result delivers the final global illuminant estimation color as follows [22]:  
\[f_{\theta}(F)=\hat{I}_{g}=\text{norm}\left(\sum_{i\in R}\text{c}(R_{i})\text{ g}(R_{i})\right) \tag{2}\]  
Intuitively, supposing that \(R_{i};\ i=1,\ 2,\ 3,\ \cdots,\ n\) are local regions that contain useful semantic information for illuminant estimation, \(\text{c}(R_{i})\) should be large values. In detail, the semantic pooling in Equation (2) is described as follows:  
\[\hat{I}_{g}=\sum_{i\in R}c_{i}\hat{I}_{i};\ i=1,\ 2,\ 3,\ \cdots,\ n \tag{3}\]  
where  
\[c_{i}=\sqrt{\sum_{x\in N}{(\hat{I}_{i})^{2}}};\ if\ \begin{cases}c_{ mean,\ i}<\text{color\_threshold}&c_{i}\\ \text{other\ wise}&0\end{cases}  \tag{4}\]where \(x\) is the coordinate of local region in the image and \(N\) is the total number of pixels in the local region. \(c_{mean,\ i}\) refers to mean of local semantic information. \[\hat{I}_{g}=\frac{I_{g}}{\left\|{I_{g}}\right\|_{2}}=\frac{1}{\left\|{I_{g}} \right\|}\sum_{i\in R}c_{i}\hat{I}_{i};\ i=1,\ 2,\ 3,\ \cdots,\ n \tag{5}\]  
where \(c_{i}\) and \(\hat{I}_{i}\) refer to semantic information and local illuminant estimation function, respectively. Using the chain rule, Equation (5) is transformed into Equation (6) below:  
\[\frac{\partial L(\hat{I}_{g})}{\partial\hat{I}_{i}}=\frac{c_{i}}{\left\|{I_{g }}\right\|_{2}}\times\frac{\partial L(\hat{I}_{g})}{\partial\hat{I}_{g}} \tag{6}\]  
In Equation (6), the estimation \(\hat{I}_{i}\) has different magnitudes with different semantic information \(c_{i}\). In estimating local illuminants, semantic information serves as a mask within the salience region, which helps prevent the proposed network from learning noisy data.