# Guided Facial Skin Color Correction
## 3 Proposed Method
### Guide Image Filtering via Optimization  
GIF is a sort of joint filtering (also called cross filtering) that uses a guide image to correct the input image. The energy function of GIF is based on the image matting method [3]. The data fidelity term is also known as the _local linear model_, and we focus on the data fidelity term, but the design objective is different. As a similar method, Ref. [19] has been proposed in high dynamic range imaging. Our guide image filtering reconstructs an image \(\mathbf{x}\in\mathbb{R}^{3N}\) where the facial skin color region has a corrected color by using the input image \(\mathbf{y}\in\mathbb{R}^{3N}\) and the color grading result, where \(N\) is a number of pixels. In the RGB color space (\(\mathbb{R}^{3}\)), let the pixel value at pixel \(j\) of the color corrected image to be solved be \(\mathbf{x}_{j}\), the input image be \(\mathbf{y}_{j}\), and the guide image be \(\mathbf{g}_{j}\) which are given by the color grading in Section 3.3. Figure 5 shows each image. Using them, we formulate our guide image filtering as the following convex optimization problem:  
\[\min_{\mathbf{x},\mathbf{A},\mathbf{b}}\ \underbrace{\sum_{i}\Biggl{(}\sum_{j \in w_{i}}\|\mathbf{x}_{j}-\mathbf{A}_{i}\mathbf{y}_{j}-\mathbf{b}_{i}\|_{2}^ {2}+\varepsilon\|\mathbf{A}_{i}\|_{F}^{2}\Biggr{)}}_{\text{data fidelity}} +\ \underbrace{\iota_{\mathcal{S}_{\text{s}}}(\mathbf{x})+\iota_{\mathcal{S}_{ \text{s}}}(\mathbf{x})}_{\text{constraints}}. \tag{5}\]  
First, the data fidelity terms, which correspond to [3], reflect textures and local contrasts of \(\mathbf{y}\) onto \(\mathbf{x}\). As for the symbols, \(\mathbf{A}_{i}\in\mathbb{R}^{3\times 3}\) and \(\mathbf{b}_{i}\in\mathbb{R}^{3}\) are a scaling matrix and an offsetting vector to approximate \(\mathbf{y}_{j}\) to \(\mathbf{x}_{j}\) for each pixel \(i\). \(w_{i}\) is a square window around  
Figure 4: Each extracted facial skin color region and their distributions. The top row shows (**a**) target color image; (**b**) input image; and (**c**) color graded image. The bottom row shows their color distributions in the RGB color space. Figure 5: Guide image filtering. From left: Input image, Guide image, Target color in \(\Omega_{B}\) and \(\Omega_{S}\), and Color transformed image. pixel \(i\). \(\|\cdot\|_{2}\) and \(\|\cdot\|_{F}\) denote the \(\ell_{2}\) norm and Frobenius norm, respectively. Next, the constraint terms are expressed by the following indicator function and conditions:  
\[\iota_{C}(\mathbf{x}):=\left\{ \begin{array}{ll}0&\text{if }\mathbf{x}\in \mathcal{C},\\ +\infty&\text{otherwise},\end{array} \right. \tag{6}\]  
\(\mathcal{C}_{S}\) and \(\mathcal{C}_{B}\) are conditions and given by  
\[ \begin{split}\mathcal{C}_{S}&:=\left\{\mathbf{x}\mid\sqrt{ \sum_{p\in\Omega_{S}}\|\mathbf{x}_{p}-\mathbf{g}_{p}\|_{2}^{2}}\leq\eta_{S} \right\},\\ \mathcal{C}_{B}&:=\left\{\mathbf{x}\mid\sqrt{\sum_{q\in\Omega_{B}}\| \mathbf{x}_{q}-\mathbf{y}_{q}\|_{2}^{2}}\leq\eta_{B}\right\}.\end{split}  \tag{7}\]  
The term \(\iota_{C_{S}}(\mathbf{x})\) works on the facial skin color region \(\Omega_{S}\), and brings the facial skin color close to the guide image \(\mathbf{g}\). The term \(\iota_{C_{B}}(\mathbf{x})\) works on the background region \(\Omega_{B}\), and keeps the color of the background the same as the original image \(\mathbf{y}\). For reducing undesirable artifacts, we design the constraint that the summation of the color differences obtained at each pixel does not exceed \(\eta\). Note that we purposefully represent the second and third terms as constraint formulation over unconstrained one, e.g., the second term can be replaced with a regularizer \(\lambda\sqrt{\sum_{p\in\Omega_{S}}\|\mathbf{x}_{p}-\mathbf{g}_{p}\|_{2}^{2}}\), where \(\lambda\) is a balancing parameter. This is because \(\eta\) can be controlled more intuitively than \(\lambda\), and it allows us to adaptively change \(\eta\) depending on the area of \(\Omega_{s}\) (see the next Section 4).