# Conditional GAN-Based Two-Stage ISP Tuning Method: A Reconstruction-Enhancement Proxy Framework
## 3 Methods
### Overall Proxy Tuning Process
The proxy objective is as follows:  
\[\mathcal{W}^{*}=\underset{\{\mathcal{W}\}}{\arg\min}\sum_{i=1}^{N}f_{diff}(f_ {proxy}(I_{i},\mathcal{P};\mathcal{W}),f_{ISP}(I_{i},\mathcal{P})) \tag{4}\]  
where \(f_{proxy}\) is a differentiable proxy, \(\mathcal{W}\) is the training parameter of \(f_{proxy}\), and \(f_{diff}\) is a function used to evaluate the magnitude of the difference between the image output by \(f_{proxy}\) and the image output by \(f_{ISP}\). By adjusting and training \(\mathcal{W}\), the sum of \(f_{diff}\) for \(I\) is minimized, and by getting \(\mathcal{W}^{*}\), the differentiable proxy of ISP is obtained. Then, by adjusting the input \(\mathcal{P}\) of \(f_{proxy}\), the optimal effect of the proxy under \(\mathcal{T}_{HV|CV}\) is achieved, and \(\mathcal{P}\) is given to target \(f_{ISP}\) to achieve the optimal effect under \(\mathcal{T}_{HV|CV}\). The optimization function is as follows:  
\[\mathcal{P}^{*}=\underset{\{\mathcal{P}\}}{\arg\max}\sum_{i=1}^{N}\mathcal{T} _{HV|CV}(f_{proxy}(I_{i},\mathcal{P};\mathcal{W}^{*})),I_{i}\in\mathcal{V} \tag{5}\]  
By fixing \(\mathcal{W}^{*}\), \(f_{proxy}\) can adjust the overall image effect on \(\mathcal{V}\) only by adjusting \(\mathcal{P}\), and maximize its score \(\mathcal{S}\) on HV and CV. Let the \(\mathcal{P}\) with the largest \(\mathcal{S}\) be the tuned \(\mathcal{P}^{*}\). Then, the best ISP effect on \(\mathcal{T}_{HV|CV}\) is obtained by reusing \(\mathcal{P}^{*}\) in \(f_{ISP}\). As described above, the overall ISP proxy tuning process involves one dataset generation session and two training sessions as shown in Figure 1. The first session, as depicted in Equation (4), involves training the proxy model by minimizing the error between the proxy model output and the actual ISP output. The second session, as depicted in Equation (5), aims to obtain parameters suitable for the actual ISP by maximizing the evaluation score of the proxy model in HV and CV. Nevertheless, in the proxy training process of [13; 20], the parameters within the ISP are treated as equivalent, and the proxy is trained via the same parameter network. This approach results in the end-to-end proxy having to simultaneously learn both low-level (denoising) and high-level (color style) features during the subsequent parameter tuning phase, potentially leading to conflicts in parameter objectives and a decline in proxy performance. Additionally, the employed end-to-end network solely relies on pixel-level loss, thus failing to effectively capture high-frequency details and maintain global consistency in images. As discussed in [14], considering Formula (1), the target ISP pipeline can be divided into two categories: reconstruction and enhancement. For a general ISP pipeline, the processing effect of the two types of modules on images is completely different. Thus, we can split the ISP pipeline into two stages through weak correlation: reconstruction stage and enhancement stage with different optimization goals. Thus, the ReEn-GAN is proposed, which uses a two-stage generative adversarial network as a proxy. By splitting the ISP proxy with all parameters into two different proxies with their own parameters, the above problems can be effectively reduced. The new optimization functions are shown as follows:  
\[ \begin{cases}\mathcal{W}_{Re}^{*}=\operatorname*{arg\,min}_{\{\mathcal{W}_{Re} \}}\sum_{i=1}^{N}f_{diff}[f_{Re}(I_{i},\mathcal{P}_{Re};\mathcal{W}_{Re}),f_{ ISP-Re}(I_{i},\mathcal{P}_{Re})],I_{i}\in\mathcal{V}\\ \mathcal{W}_{En}^{*}=\operatorname*{arg\,min}_{\{\mathcal{W}_{En}\}}\sum_{i=1 }^{N}f_{diff}\{f_{En}[f_{ISP-Re}(I_{i},\mathcal{P}_{Re}),\mathcal{P}_{En}; \mathcal{W}_{En}],f_{ISP}(I_{i},\mathcal{P})\},I_{i}\in\mathcal{V}\\ \mathcal{P}^{*}=\operatorname*{arg\,max}_{\{\mathcal{P}\}}\sum_{i=1}^{N} \mathcal{T}_{HV|CV}\{f_{En}[f_{Re}(I_{i},\mathcal{P}_{Re};\mathcal{W}_{Re}^{* }),\mathcal{P}_{En};\mathcal{W}_{En}^{*}]\},(\mathcal{P}_{En},\mathcal{P}_{Re })\in\mathcal{P}\end{cases}