# Conditional GAN-Based Two-Stage ISP Tuning Method: A Reconstruction-Enhancement Proxy Framework
## 3 Methods
### Reconstruction Stage  
In the ISP pipeline, the primary objective of the reconstruction phase is to transform sensor RAW data into high-quality linear RGB images while preserving the original details within the images. For the scenario of ISP reconstruction, in order to optimize the proxy network more effectively and make it suitable for Re-stage scenarios, a specific network structure needs to be used. There are many network structures to choose from, and for image-to-image tasks, we first consider the U-net model [33] as the network structureused for image segmentation tasks. Due to its encoder-decoder structure, the information at all levels of the image can be transmitted across layers through skip connections, making it suitable for image transfer tasks that preserve image details. Meanwhile, in image reconstruction task scenarios, the corresponding ISP module is mainly used for image restoration tasks such as noise reduction, sharpening, and demosaicing, with higher requirements for image details. Therefore, U-NET is often used as the network structure in many AI-ISP applications [23]. At the same time, as a RAW-to-RGB transfer task, image reconstruction requires the network to be able to specifically adapt to the input of RAW images and the output of RGB images, as well as to preserve the structure and information of the input image well. Therefore, based on the four-layer U-Net, we changed the input structure of the network to accept RAW input and ISP parameter input to preserve the original information of the image while the network has sufficient scalability to obtain information on the influence of different ISP parameters on the image. For the discriminator, in order to obtain detailed information of the image, we use patch discriminator to obtain the \(\mathcal{L}_{GAN}\), which allows D1 to better control the image generated by G1 through patch level of loss. Generally speaking, the generator employs an optimized encoder-decoder architecture akin to U-Net, while the discriminator utilizes patch discriminators and loss functions described in Equation (7), as depicted in Figure 3.  
During the reconstruction phase, generator \(G1\) is required to capture the spatial transformation features of the image from RAW to RGB in the ISP pipeline, while also implementing image detail processing such as denoising and sharpening. The basic structure of \(G1\) is an encoder-decoder U-Net with four layers of skip connections. In order to reduce the problems of image artifacts and edge blur caused by the direct input of Bayer images into the network [32], pixel shuffle [34] is applied before the input layer of the network. By rearranging the pixels of the Bayer array, the raw image in the shape of \(H\times W\times 1\) is replaced with \(H/2\times W/2\times 4\)\(\{R,G,B,G\}\) images, which reduces the size of the RAW image and decreases the computational complexity of the network, while avoiding the edge blur and mosaic problems caused by the direct processing of Bayer images [35]. The input image in the shape of \(H/2\times W/2\times 4\) undergoes one flat convolution and four down-flat convolutions to encode the image. In order to enable the network to better learn the impact of \(P_{Re}\) on the image effect, the image will be concatenated with the corresponding size parameter matrix before each convolution layer, so that all information of \(P_{Re}\) can be captured in each convolution. The output data size of the encoder is \(H/32\times W/32\times 256\), and the data enters the decoding stage to restore the details of the image at all levels and achieve the overall reconstruction of the image. At the same time, the concatenated data is linked to the decoder through a skip connection [33], which directly transfers low-level features (such as edges, textures, etc.) to high-level feature maps, helping the network preserve detailed information during the reconstruction process and reducing the problem of vanishing gradients, thereby accelerating the training process and improving reconstruction quality.