# Conditional GAN-Based Two-Stage ISP Tuning Method: A Reconstruction-Enhancement Proxy Framework
## 1 Introduction  
In modern image processing systems, the image signal processor (ISP) transforms the raw signals captured by image sensors into image signals comprehensible to both human eyes and machines. Modern ISPs consist of numerous independent yet interconnected modules. These modules sequentially convert the raw signals into image signals, while also processing and optimizing the image information to better align with human visual requirements or machine vision needs. During their design, these modules adhere to the principle of divide and conquer, breaking down the image processing tasks into distinct steps. Each critical step handles specific image processing tasks, including demosaicing, noise reduction, white balance, and color correction. Through the pipeline of all modules within the ISP, the image undergoes processing and optimization [1]. On the one hand, the image processing effect of each module in the ISP is not constant [2]. Modern ISPs need to adjust the algorithm for different scenes. The adjustment here is more about the optimization of the algorithm itself than the optimization of the ISP pipeline. In order to make the ISP adapt to more scenarios and obtain better adaptability, modern ISPs usually expose many super parameters and configuration interfaces of the algorithm to the outside world, so that the algorithm can adapt to more scenarios. Inthe actual development scenario, professional image tuning experts will coordinate the ISP and the entire image acquisition system (including cameras, image sensors, ISP and back-end processing), and set the optimal parameter set for the ISP in different scenarios. This process is time-consuming, and it is difficult to obtain the optimal parameter set for the scene, and the process of tuning is subject to the experience and subjective evaluation of tuning experts. On the other hand, ISP output images are not only used for Human Visions (HV). In many scenarios, ISP output images are often used for Computer Vision (CV) tasks such as image recognition, image segmentation [3], etc. Under the traditional ISP tuning method, the goal of tuning is mainly based on the subjective evaluation of image tuning experts [4] and some commonly used image quality standards [5; 6; 7; 8]. These image quality standards cannot strictly correspond to downstream tasks. The excellence of the image quality standard does not mean that the accuracy of the CV task is high, and the traditional image tuning cannot carry out specific tuning for the CV task [9], which leads to a certain impact on the accuracy of the CV task when ISP is used for some downstream tasks [10; 11]. Therefore, the method of tuning for CV tasks not based on tuning experts is gradually emerging [12]. To facilitate the feedback of CV task losses to the ISP tuning system, an agent-based ISP tuning approach is employed. This method fits the ISP pipeline using a differentiable network, allowing the loss from downstream tasks to be backpropagated to the input parameters of the proxy ISP. This enables parameter adjustments tailored to specific tasks. Subsequently, the tuned parameters are passed to the hardware ISP, achieving optimal parameter configuration for the ISP. The existing proxy-based ISP tuning methods came from end-to-end proxy training methods [13]. The advantage of the end-to-end proxy method lies in the fact that the proxy network can capture the sequential effects of all levels of modules in the ISP pipeline on image processing. However, due to the complexity of the ISP pipeline, a single network often fails to distinguish the optimization objectives of different type modules, resulting in conflicts in optimization directions. This paper utilizes the different optimization objectives of different type modules in the ISP pipeline [14] and proposes a two-stage ISP proxy tuning method based on reconstruction and enhancement, thus avoiding the tuning interference and parameter search difficulties caused by end-to-end approaches. In this paper, our contributions can be summarized as follows:  
1.