RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Problem Definition
Throughout this paper, we aim to tackle the photo finishing tuning problem. Given an input image I_{0} , an image processing pipeline f_{\text{PIPE}} maps the input to a finished image I_{\text{FINISHED}}=f_{\text{PIPE}}(I_{0},P) , controlled by a set of parameters P . Our task is to solve the inverse problem: given an input image I_{0} and a tuning target I_{g} (the goal condition in the RL framework), how to find the parameters that reach this target:

\arg\min_{P}\mathcal{L}(I_{g},f_{PIPE}(I_{0},P))

The goal image I_{g} can vary between different tasks. For photo finishing tuning, the goal image shares the same content as the input, and the tuning target is to minimize the distance between the pipeline output and the goal image I_{g} . For photo stylization tuning, the goal image is a style target with different content than the input, and the tuning target is to generate an output that matches the style of the goal image. Note that unlike artistic style transfer gatys2016image , we focus on photorealistic style transfer xia2020joint , where the processing pipeline does not change the content of an image.
Goal Conditioned Reinforcement Learning
We are the first to introduce a reinforcement learning (RL) approach to the photo finishing tuning task by formulating it as an end-to-end policy learning problem. Inspired by human experts who use iterative trial and error in photo finishing, our method models the task as a decision process with multi-step feedback, akin to a Markov Decision Process. As shown in the bottom row of Fig. 2 , our policy iteratively tunes the parameters of a given image processing pipeline to match a goal image. At each step, it takes the currently retouched image and the goal image as inputs, and then outputs the next action. This RL-based framework efficiently predicts pipeline parameters, requiring only minimal iterations (e.g., 10 queries to the image processing pipeline). Additionally, since RL optimization bypasses the need for gradient flow from the image processing pipeline, our system can handle any black-box pipeline, regardless of complexity.

The RL process is formally defined as follows. Let \mathcal{S} be the state space, \mathcal{O} be the observation space, \mathcal{A} be the action space, \mathcal{T} be the transition function, \mathcal{R} be the reward function, \mathcal{G} be the goal distribution, \rho_{0} be the initial state distribution, and \gamma the discount factor. All these forms a Goal-conditioned Partially-Observed Markov Decision Process \left(\mathcal{S},\mathcal{O},\mathcal{A},\mathcal{T},\mathcal{R},\mathcal{G},\rho_{0},\gamma\right) liu2022goalconditioned .

In each tuning episode t , the agent is given a goal image I_{g}\in\mathcal{G} , as well as an observation o_{t}\in\mathcal{O} consists of the retouched image I_{t} at step t along with all historical actions and observations.
The action a_{t} is the parameter set P used by image processing pipeline to generate the output image at the next step. And the transition function \mathcal{T}:\mathcal{S}\times\mathcal{A}\rightarrow\mathcal{S} is the image processing pipeline f_{\text{PIPE}} defined in Sec. 3.1 .
The reward function is \mathcal{R}(s,I_{g}) for s\in\mathcal{S} and I_{g}\in\mathcal{G} .
We aim to learn a goal-conditioned policy \pi(a|o,I_{g}):\mathcal{S}\times\mathcal{G}\rightarrow\mathcal{A} that maps from observation o and goal I_{g} to the next action a , maximizing the sum of discounted rewards \mathbb{E}_{s_{0}\sim\rho_{0},I_{g}\sim\mathcal{G}}\sum_{t}\gamma^{t}\mathcal{R}(s_{t},I_{g}) .
Our policy \pi(a|o,I_{g}) is a deterministic policy \mu_{\theta} parameterized by \theta , outputting continuous actions a_{t}=\mu_{\theta}(o_{t},I_{g}) .
Reward Function and Training Objectives
We provide a general RL-based framework for photo tuning tasks. One can train our goal-conditioned end-to-end policy with different reward functions to resolve different photo tuning tasks including photo finishing tuning and photo stylization tuning. The policy is optimized with twin-delayed DDPG (TD3) algorithm fujimoto2018addressing-td3 .

Reward function for photo finishing tuning. When goal image I_{g} is the photo-finished input image, we measure the distance between current image I_{t} and I_{g} with PSNR metric. The reward function is calculated as the difference between PSNR values of consecutive steps:

 r_{t}=PSNR(I_{t+1},I_{g})-PSNR(I_{t},I_{g})

Instead of using \ell_{2} -distance to measure image distance, we use PSNR, the negative logarithm of \ell_{2} -distance. This design ensures the policy receives appropriate rewards even when the current image is close to the goal, encouraging fine-grained tuning of pipeline parameters.

Reward function for photo stylization tuning. When goal image I_{g} is a style image with arbitrary content and style, we measure the distance between the input and goal images with a style score:

StyleScore_{t}=\sum_{i=1}^{N_{S}}\left\|G_{i}[I_{g}]-G_{i}[I_{t}]\right\|_{2}+\lambda_{0}\|H(I_{t}^{Y}),H(I_{g}^{Y})\|_{2}+\lambda_{1}\|H(I_{t}^{UV}),H(I_{g}^{UV})\|_{2}, (3)

where N_{S} denotes the number of layers from a pre-trained VGG-19 simonyan2014very-vgg model used to extract features.
Following gatys2016image , the style is captured using Gram matrices G_{i}[\cdot]=F_{i}[\cdot]F_{i}[\cdot]^{T} , with F_{i} representing feature maps. Additionally, the \ell_{2} -distances of histograms for the Y and UV channels are included to align luminance and color palettes. The overall reward is calculated by the change in style score across consecutive steps, penalized by the difference in content features F_{i} :

 r_{t}= ~StyleScore_{t}-StyleScore_{t+1}-\lambda_{2}\sum_{i=1}^{N_{C}}\left\|F_{i}[I_{g}]-F_{i}[I_{t}]\right\|_{2}

Policy optimization. We optimize our goal-conditioned policy with off-policy TD3 fujimoto2018addressing-td3 algorithm.
Specifically, TD3 learns two Q value function Q_{\phi_{1}} and Q_{\phi_{2}} , optimized by mean square Bellman error minimization:

 y(r_{t},s_{t+1},d)= ~r_{t}+\gamma(1-d)\min_{i=1,2}Q_{\phi_{i,targ}}(s_{t+1},a^{\prime}(s_{t+1})), (5) L(\phi_{i})= ~\underset{s_{t}\sim{\mathcal{D}}}{\mathbb{E}}\left[\left(Q_{\phi_{i}}(s_{t},a_{t})-y(r_{t},s_{t+1},d)\right)^{2}\right], (6)

where Q_{\phi_{i,\text{targ}}} is the exponential moving average of Q_{\phi_{i}} , a^{\prime}(s_{t+1}) is given by target policy with clipped gaussian noise, and d is the termination signal. The state transition pair (s_{t},a_{t},r_{t},s_{t+1},d) is sampled from a replay buffer \mathcal{D} . With Q functions, the policy \mu_{\theta}(\cdot) is learned by maximizing Q_{\phi_{1}} :

 L(\theta)=\underset{s_{t}\sim{\mathcal{D}}}{\mathbb{E}}\left[Q_{\phi_{1}}(s_{t},\mu_{\theta}(s_{t}))\right]

More details about our reward functions and policy optimization are in the appendix.
Experiments
In the first subsection, we provide the details of datasets and task settings for photo finishing tuning and photo stylization tuning, along with a description of the evaluation metrics. In the second subsection, we demonstrate our experimental results and compare them to zeroth-order optimization mosleh2020hardware and first-order optimization tseng2019hyperparameter ; tseng2022neural baseline methods.
Ablation studies are conducted in the last subsection, which investigates the impact of each component of our state representation.
We also provide supplementary qualitative results in the Appendix.
Tasks Settings and Datasets
Datasets. We use the MIT-Adobe FiveK Dataset fivek , a renowned resource in the field of photo retouching, which comprises 5,000 photographs captured using DSLR cameras by various photographers. This dataset is notable for providing images in raw format alongside the retouching outcomes of five experts. For our study, we selected 4,500 images to serve as the training dataset, with the remaining 500 images designated as the validation dataset.
In our method, random parameters are employed to generate the target images, which are used as training data pairs. In the task of photo finishing tuning, the datasets including both the expert C retouched targets and randomly generated targets are utilized for evaluation. The expert C retouched targets are optimized using CMA-ES to ensure they are reachable by our image processing pipeline. For the photo stylization tuning task, we have curated a collection of 200 diverse style images from the Lightroom Discover website 1 1 1 https://lightroom.adobe.com/learn/discover , following shi2022spaceedit .

To further evaluate our method and demonstrate its generalizability, we test our RL-based framework directly on the HDR+ dataset hdrplus . We used the official subset of the HDR+ dataset, which consists of 153 scenes, each containing up to 10 raw photos. The aligned and merged frames are used as the input, expertly tuned images serve as the photo-finishing targets.

Implementation details. We conduct all experiments on an image processing pipeline consisting of standard image processing operations, including exposure, color balance, saturation, contrast, tone mapping (highlight and shadow), and texture (sharpness and smoothing), with nine adjustable parameters in total, similar to tseng2022neural . Thus, the agent’s action space is comprised of fine continuous actions corresponding to these nine pipeline parameters.
During the policy inference, the input and goal images are resized to the resolution of 64 × 64. Additionally, a 3-level Laplacian pyramid of both input and goal images is constructed and fed into the policy network to capture high-frequency details from the original resolution. Then the output parameters from the policy network are fed to the image processing pipeline along with full-resolution images to produce high-resolution results.
We train our policy using the standard TD3 algorithm fujimoto2018addressing-td3 and set the termination of our RL policy to trigger when the episode length reaches the maximum threshold (10 steps), ensuring efficiency. Further details can be found in the appendix.

Evaluate metrics. Similar to hu2018exposure ; tseng2019hyperparameter ; tseng2022neural , we employ the Peak Signal-to-Noise Ratio (PSNR), the Structural Similarity Index Measure (SSIM) wang2004image , and the Learned Perceptual Image Patch Similarity (LPIPS) zhang2018unreasonable as our evaluation metrics. To assess the quality of stylization, we conduct user studies to evaluate our methods, offering a subjective measure of image quality based on viewer assessments.
Ablation Study on State Representation
As has been shown in our main experiment, our RL-based approach significantly outperforms the previous method in terms of photo finishing quality and efficiency, demonstrating that RL is more suitable for the photo finishing tuning task. In this subsection, we focus on the photo finishing state representation we propose to better fit RL in our task. Specifically, we conduct experiments on photo finishing tuning task using FiveK Random-Target dataset, in order to study the contribution of each specific representation in our photo finishing state representation.

As shown in Tab. 4.2 , we set our baseline RL as RL policy trained with a naive CNN-based encoder taking the concatenated input and goal images as input.
This baseline achieves only 32.17 dB of PSNR. In Ex_{2} , our dual-path feature representation F^{D} improves the photo finishing quality, as it better encodes local features and global image characteristics critical to our photo tuning task.
Since traditional photo statistics contain invariant features about global statistics that are difficult to learn solely using a network, the photo statistics representation F^{S} also helps to boost photo finishing quality as shown in Ex_{3} .
Moreover, as parameters and results from all previous RL steps help in the decision process, the historical action representation is also useful as shown in Ex_{4} .
With the proposed three representations combined, our RL policy can be guided to effectively tune image processing pipeline parameters given input and goal images of any photo finishing style, as evidenced by the superior performance in Ex_{5} .
Conclusion
In this work, we propose an RL-based photo finishing tuning algorithm that efficiently tunes the parameters of a black-box image processing pipeline to match any tuning target. Our approach encompasses several key innovations. Firstly, we integrate goal-conditioned RL into the realm of photo-finishing tuning. Secondly, we propose a photo finishing state representation comprising three principal components essential for training an effective RL policy network: a CNN-based feature representation that encodes both global and local image features, a photo statistics representation designed to align the photographic statistics between the input and the target, and an embedding of historical actions. We assess the effectiveness of our framework through comprehensive experiments on both photo finishing tuning and photo stylization tuning. The experimental results affirm that our goal-conditioned policy is an adept photo-finishing tuner, capable of exerting efficient and fine-grained control over image processing pipeline parameters to fulfill a variety of objectives.
Currently, our method exclusively supports conditional inputs in the form of images and lacks the capability to process non-image types such as textual inputs. Moving forward, we intend to broaden our research to include multi-modal conditional inputs.
Acknowledgements
This work is supported by Shanghai Artificial Intelligence Laboratory and RGC Early Career Scheme (ECS) No. 24209224. We also extend our gratitude to Quanyi Li for his insightful discussions and valuable comments.
A.1 Photo Finishing Tuning Results on FiveK Dataset
We show more photo finishing tuning visualization results on the FiveK-Target evaluation dataset in Fig. 7 , showcasing its superior performance in terms of color and brightness compared to all other methods.
A.2 Photo Stylization Tuning Results
We show more photo stylization tuning visualization results in Fig. 8 , showcasing its superior performance in tuning input images to any style goal that is outside the training goal distribution.
C.1 Training Details of TD3 Algorithm
We state the details of TD3 fujimoto2018addressing-td3 algorithm training in this subsection. TD3 is an off-policy RL algorithm. When the policy sample transition pair (s_{t},a_{t},r_{t},s_{t+1},d) to form the replay buffer \mathcal{D} , a gaussian noise is added to encourage exploration: a=\text{clip}\left(\mu_{\theta}(s)+\epsilon,a_{Low},a_{High}\right) , where \epsilon\sim\mathcal{N}(0,\sigma) . We set \sigma=0.1 for a trade-off between exploration and exploitation.

To compute the target action in Q update target:

 L(\phi_{i})= ~\underset{s_{t}\sim{\mathcal{D}}}{\mathbb{E}}\left[\left(Q_{\phi_{i}}(s_{t},a_{t})-\left(r_{t}+\gamma(1-d)\min_{i=1,2}Q_{\phi_{i,targ}}(s_{t+1},a^{\prime}(s_{t+1}))\right)\right)^{2}\right], (8)

The next action of s_{t+1} comes from the target policy with a clipped noise, so that incorrect action peak produced by sub-optimal Q value estimation is smoothed out, as in the following equation:

 a^{\prime}(s_{t+1})= ~clip\left(\mu_{\theta_{targ}}(s_{t+1})+clip(\epsilon_{1},-c,c),a_{Low},a_{High}\right),\;\;\;\;\;\epsilon_{1}\sim\mathcal{N}(0,\sigma) (9)

In this implementation, we set \epsilon_{1}=0.2 , which is twice the value of \epsilon .
TD3 updates the policy less frequently than the Q-function. We set the policy to update half as frequently as the Q network update. The optimization objectives of the policy are already given in the main paper. To ensure a robust update of the policy and Q network, TD3 adopts an Exponential Moving Average (EMA) strategy to update the target network Q and policy network which is used to compute the Q value target above.

\phi_{targ,1} \leftarrow\rho\phi_{targ,1}+(1-\rho)\phi, (10) \phi_{targ,2} \leftarrow\rho\phi_{targ,2}+(1-\rho)\phi, (11) \theta_{targ} \leftarrow\rho\theta_{targ}+(1-\rho)\theta (12)

We set the EMA update rate \rho=0.99 in all our experiments.
The optimizer is performed using Adam optimizer kingma2014adam with (\beta_{1},\beta_{2})=(0.9,0.999) . We set the learning rate to specific values for policy and value network, that is, 1e-4 for action and 2e-4 for Q network. We set batch size as 64 for both photo finishing tuning and photo stylization tuning experiments. We set the discount factor \gamma=0.9 .
All experiments are conducted on NVIDIA RTX 4090 GPUs.
Furthermore, we implement the CMAES method tseng2019hyperparameter based on open-source framework pymoo .
C.2 Other Training Details
Reward design. For our detailed reward design, as described in Sec. 3.4 , we utilize the PSNR metric of consecutive steps for photo finishing tuning. In photo stylization tuning, we adopt the style score as the main reward and add a content negative reward to prevent the policy from taking drastic actions that hinder photo stylization quality. Specifically, we set \lambda_{0} and \lambda_{1} in \text{StyleScore}_{t} to 100 and 50, respectively, and set \lambda_{2}=0.5 . For the VGG features selected to compute the style score, we follow huang2017arbitrary-adain to set N_{S}=N_{C}=4 , selecting relu_{1…4}_1 features to form \{F_{i}\}_{i=1...4} , which is then used to compute then gram matrix and the content regularization term.

RL termination. Since our RL policy freely explores the image processing pipeline parameter space, we need to terminate the current episode if the state collapses, meaning the policy outputs parameters that render an abnormal image. Specifically, we calculate the average pixel intensity of the image I_{t} as \mathcal{I}_{t} and set this value to be within the range of \mathcal{I}_{min} and \mathcal{I}_{max} . For each rollout, if \mathcal{I}_{t} is out of range, we terminate the state and do not save it to our replay buffer \mathcal{D} .

Network architecture. Our approach learns an end-to-end policy that maps from input images and goal specifications to the next action to take (the next parameters of the image processing pipeline.) The state representation is s_{t} as described in Sec. 3.3 , which is fed into a 4-layer multi-layer perceptron network (MLP), with a continuous action space of 9 parameters. Each hidden layer of the MLP network is of width 512.
C.3 Baselines Implementation Details
Implementation details for CMAES hansen2006cma ; mosleh2020hardware . CMAES is a gradient-free search (zeroth order optimization) method using an evolution strategy. As mosleh2020hardware does not provide code, we implemented CMAES using the pymoo library pymoo , enabling parallel execution on multi-core CPUs and achieving reasonable performance. This baseline does not require training.

Implementation details for Cascaded Proxy tseng2022neural . For proxy network training, we followed the architecture in tseng2022neural , using 3 consecutive 1 \times 1 convolutions for pointwise ISP operations and 5 consecutive 3 \times 3 convolutions for areawise operations. We trained with the Adam optimizer, a learning rate of 1e-4, a batch size of 512, and 100 epochs, as recommended. Camera metadata was extracted from DNG files, as described in tseng2022neural .
Since tseng2022neural does not release its dataset, we used the MIT-Adobe FiveK dataset, as in our method. Following Section 5 of tseng2022neural , we used 1,000 raw images from FiveK and sampled 100 points for each ISP hyperparameter.

Implementation details for Monolithic Proxy tseng2019hyperparameter . The architecture uses a single UNet to approximate the ISP pipeline, with hyperparameters conditioned by concatenating extra planes to the features. We trained the proxy with the Adam optimizer, a learning rate of 1e-4, a batch size of 512, and 100 epochs.
The training set generation and first-order optimization method are the same as for tseng2022neural .
Figure 1: In this work, we propose an RL-based photo finishing tuning algorithm that efficiently tunes the parameters of a black-box image processing pipeline to match any tuning target. The RL-based solution (top row) takes only about 10 iterations to achieve a similar PSNR as the 500-iteration output of a zeroth-order algorithm (bottom row). Our method demonstrates fast convergence, high quality, and no need for a proxy.; In this work, we propose an RL-based photo finishing tuning algorithm that efficiently tunes the parameters of a black-box image processing pipeline to match any tuning target. The RL-based solution (top row) takes only about 10 iterations to achieve a similar PSNR as the 500-iteration output of a zeroth-order algorithm (bottom row). Our method demonstrates fast convergence, high quality, and no need for a proxy.
Figure 2: The overall framework. Top row: at each step, our policy maps the current image and the goal image to action (new parameters), with the help of our state representation consisting of dual-path features, photo statistics, and historical actions. Bottom row: visualization of iterative tuning trajectory of our RL-based photo finishing framework.; The overall framework. Top row: at each step, our policy maps the current image and the goal image to action (new parameters), with the help of our state representation consisting of dual-path features, photo statistics, and historical actions. Bottom row: visualization of iterative tuning trajectory of our RL-based photo finishing framework.
Figure 3: Photo finishing tuning results on FiveK dataset with expert C target. The visual results of our method are closest to the target image, especially in terms of color and brightness.; Photo finishing tuning results on FiveK dataset with expert C target. The visual results of our method are closest to the target image, especially in terms of color and brightness.
Figure 4: Photo stylization tuning results. Compared with CMAES hansen2006cma ; mosleh2020hardware , monolithic proxy tseng2019hyperparameter , and cascaded proxy tseng2022neural , our output matches the best with the style goal.; Photo stylization tuning results. Compared with CMAES hansen2006cma ; mosleh2020hardware , monolithic proxy tseng2019hyperparameter , and cascaded proxy tseng2022neural , our output matches the best with the style goal.
Figure 5: Qualitative comparison on the HDR + photo finishing tuning task. These comparisons illustrate that our method remains closer to the target even when dealing with input and target images outside the training distribution.; Qualitative comparison on the HDR + photo finishing tuning task. These comparisons illustrate that our method remains closer to the target even when dealing with input and target images outside the training distribution.
Figure 6: Results of the user study on the photo stylization tuning task. Each bar represents the number of votes each method received, where participants select the images they believed most closely resembled target images.; Results of the user study on the photo stylization tuning task. Each bar represents the number of votes each method received, where participants select the images they believed most closely resembled target images.
Figure 7: Additional results on photo finishing tuning task.; Additional results on photo finishing tuning task.
Figure 8: Additional results on photo stylization tuning task.; Additional results on photo stylization tuning task.
Figure 9: Options in question 1 of our user study.; Options in question 1 of our user study.
Figure 10: Options in question 2 of our user study.; Options in question 2 of our user study.
Figure 11: Options in question 3 of our user study.; Options in question 3 of our user study.
Figure 12: Options in question 4 of our user study.; Options in question 4 of our user study.
Figure 13: Options in question 5 of our user study.; Options in question 5 of our user study.
Figure 14: Options in question 6 of our user study.; Options in question 6 of our user study.
Figure 15: Options in question 7 of our user study.; Options in question 7 of our user study.
Figure 16: Options in question 8 of our user study.; Options in question 8 of our user study.
Figure 17: Options in question 9 of our user study.; Options in question 9 of our user study.
Figure 18: Options in question 10 of our user study.; Options in question 10 of our user study.