# Conditional GAN-Based Two-Stage ISP Tuning Method: A Reconstruction-Enhancement Proxy Framework
## 3 Methods
### Overall Proxy Tuning Process  
As shown in Figure 1, the tuning method proposed consisted of three steps: step 1, dataset generation; step 2, proxy training; step 3, parameter tuning. In the first step, by injecting randomly selected parameters that cover the parameter space extensively and RAW images into the target ISP, the corresponding ReRGB images and EnRGB images are obtained. Thus, paired RAW-RGB-parameter data are collected. In the second step, by taking paired data as the dataset, the proposed ReEn-GAN is trained separately and jointly to get a two-stage proxy network fitting the target ISP. In the third step, with the ground truth image in the original dataset as the optimization target, parameters with a two-stage proxy are tuned to meet the optimal situation of downstream HV and CV tasks, and the optimal parameters for the target ISP are obtained. The ultimate goal of ISP tuning is to find the optimal parameters for downstream tasks. For the target ISP pipelines to be tuned, the basic functions are as follows:  
\[f_{ISP}(I,P)=f_{n}(f_{n-1}(\cdots f_{1}(I,P_{1})\cdots,P_{n-1}),P_{n}) \tag{1}\]  
where \(f_{1}\) to \(f_{n}\) are modules in the ISP pipeline that sequentially process sensor signals, \(P_{1}\) to \(P_{n}\) are parameters for each module, \(I\) is the sensor information input to \(f_{ISP}\), and  
Figure 1: Illustration of the proposed two-stage proxy auto-tuning method. and \(I\) jointly affect the overall output image of \(f_{ISP}\). When I is fixed or follows a certain distribution, adjusting \(P\) appropriately can significantly affect the image output by \(f_{ISP}\). In order to evaluate whether ISP output images can meet certain performance indicators and downstream tasks, an evaluation system is needed as follows:  
\[\mathcal{S}=\sum_{i=1}^{N}\mathcal{T}_{HV|CV}(f_{ISP}(I_{i},\mathcal{P})),I_{i} \in\mathcal{V} \tag{2}\]  
where \(\mathcal{T}_{HV|CV}\) is the evaluation criterion for \(f_{ISP}\) in a certain HV or CV scenario, \(I\) belongs to a certain image set or scene \(\mathcal{V}\), and \(\mathcal{S}\) is the score of \(f_{ISP}\) in the scene. \(\mathcal{T}_{HV|CV}\) and \(\mathcal{V}\) are strictly related; different \(\mathcal{V}\) requires different \(\mathcal{T}_{HV|CV}\) as the evaluation function; when \(\mathcal{V}\) and \(\mathcal{T}\) are fixed, the only way to improve \(\mathcal{S}\) is to change \(\mathcal{P}\) and find the optimal \(\mathcal{S}\) for \(\mathcal{T}_{HV|CV}\) under the condition of adjusting \(\mathcal{P}\). The optimization function is as follows:  
\[\mathcal{P}^{*}=\underset{\{\mathcal{P}\}}{\arg\min}\sum_{i=1}^{N}\mathcal{T} _{HV|CV}(f_{ISP}(I_{i},\mathcal{P})),I_{i}\in\mathcal{V} \tag{3}\]  
Because the ISP module is in the form of a pipeline, the modules will affect each other. The effect of the latter module \(f_{i}\) depends on both its own \(\mathcal{P}_{i}\) and the processing results of all the previous modules \(\{f_{j}|j<i\}\). Therefore, it is difficult to optimize the overall \(\mathcal{T}_{HV|CV}\) by adjusting the parameters of a single separate module. At the same time, because \(\mathcal{P}\) in ISP is basically nonlinear and complex, and most of the existing hardware ISP algorithms do not disclose all details to the public, the traditional iterative search method is complex and time-consuming [18]; the complexity of \(\mathcal{P}\) also determines that it is difficult to use gradient propagation and reverse search methods to search the optimal parameters. Those above problems make the ISP \(\mathcal{T}_{HV|CV}\)-oriented parameter tuning problem become a nonlinear optimization problem in complex space. When there is a clear evaluation function \(\mathcal{T}_{HV|CV}\), some methods refer to the use of a proxy method [13] for HV and CV.