# Conditional GAN-Based Two-Stage ISP Tuning Method: A Reconstruction-Enhancement Proxy Framework
## 3 Methods
### Enhancement Stage  
In the enhancement stage, the training target of the proxy is the module used for image enhancement in ISP. The proxy takes plain \(\{R,G,B\}\) as the image input, learns the enhancement module to perform color correction, white balance correction, and other effects on the image to obtain the proxy, and outputs the enhanced \(\{R,G,B\}\) image. Similar to the reconstruction stage, in the proxy training of the reconstruction stage, the main goal of the network is also the conversion task of image-to-image. Therefore, U-Net is also used as the basic network structure, and the encoder-decoder structure and skip connection are used to achieve the synthesis and restoration of information at various levels of the image while preserving a certain amount of network learning space for learning the impact  
Figure 3: The structure of reconstruction stage.  
of ISP parameter changes on the image. Unlike Re-stage, En-stage serves as a fitting for ISP image reconstruction step, mainly fitting modules such as color correction and white balance in ISP. Therefore, it has lower modeling requirements for image details and higher fitting requirements for overall chromaticity and contrast. Therefore, a network of En-stage can be designed to more efficiently learn the impact of ISP parameters on image effects based on the above characteristics. We use a three-layer U-Net as the network backbone. Compared to the four-layer U-Net used in restoration, we reduce the number of layers to reduce the overall feature extraction scale of the network while retaining a certain feature extraction ability. At the same time, we use a PPM [37] module to enable the network to capture global information at various levels of the image, which corresponds to the color and texture information of the image. Meanwhile, due to the En-stage processing image pairs of RGB-to-RGB, a normal image input layer is used. For the discriminator, in order to obtain the overall information of the image, we use global discriminator to obtain the \(\mathcal{L}_{GAN}\), which allows D2 to better control the image generated by G2 through global level of loss. Generally speaking, the enhancement stage uses an encoder-decoder structure similar to U-Net with pyramid pooling module(PPM) [37] as Generator, and evaluates the output images at global levels using global discriminators and loss functions described in Equation (8). The structure of the enhancement stage is shown in Figure 4.  
The enhancement stage mainly requires G2 to capture color restoration or tone information in ISP, so the restoration of details at various levels of the image does not need to be like G1, but only needs to focus on the color information of the image and the style information of each level of the image. Therefore, the structure of G2 is similar to G1, using a skip connection encoder-decoder structure similar to U-Net, reducing one layer of encoding and decoding hierarchy and adding a pyramid pooling module(PPM) [37] to enhance the decoder's multi-scale context awareness ability, achieving better color restoration effect. The input of G2 is a reconstructed three-channel RGB image in size of \(H\times W\times 2\). After one flat convolution and three down-flat convolutions, the encoded information in size of \(H/8\times W/8\times 64\) is obtained. After being encoded by the encoder, the data enters the PPM module and is up-sampled through four different-sized max pooling layers of \(([2,2],[4,4],[16,16],[32,32])\) to capture the image information of each level. It is  
Figure 4: The structure of enhancement stage.  
then concatenated with the original encoded information and sent to the decoder. In the decoding section, the output image EnRGB in the shape of \(H\times W\times 3\) is obtained through three layers of up convolutions and one layer of flat convolution. In \(D2\), in order to enable the discriminator to better obtain global perceptual information of the image for color reproduction, global-GAN is used as the discriminators to capture global color information.