Goal Conditioned Reinforcement Learning for Photo Finishing Tuning
Authors: Jiarui Wu
Yujin Wang
Lingen Li
Zhang Fan
Tianfan Xue
Venue: NeurIPS 2024 poster (2024)
Keywords: Photo Finishing
Image Processing
Image Signal Processor Tuning
Reinforcement Learning
Primary area: machine_vision

Abstract: Photo finishing tuning aims to automate the manual tuning process of the photo finishing pipeline, like Adobe Lightroom or Darktable. Previous works either use zeroth-order optimization, which is slow when the set of parameters increases, or rely on a differentiable proxy of the target finishing pipeline, which is hard to train.
To overcome these challenges, we propose a novel goal-conditioned reinforcement learning framework for efficiently tuning parameters using a goal image as a condition. Unlike previous approaches, our tuning framework does not rely on any proxy and treats the photo finishing pipeline as a black box. Utilizing a trained reinforcement learning policy, it can efficiently find the desired set of parameters within just 10 queries, while optimization based approaches normally take 200 queries. Furthermore, our architecture utilizes a goal image to guide the iterative tuning of pipeline parameters, allowing for flexible conditioning on pixel-aligned target images, style images, or any other visually representable goals. We conduct detailed experiments on photo finishing tuning and photo stylization tuning tasks, demonstrating the advantages of our method.

Goal Conditioned Reinforcement Learning for Photo Finishing Tuning
Jiarui Wul Yujin Wang Lingen Li Fan Zhang', Tianfan Xue²
Shanghai Al Laboratory "The Chinese University of Hong Kong (wj024, tfxue]@ie. cuhk, edu. hk, Agli0link cuhk. edu, hk, {wangyuj in, zhangf an) @ jpa.. org. cn
Abstract
Photo finishing tuning aims to automate the manual tuning process of the photo finishing pipeline, like Adobe Lightroom or Darktable. Previous works either use zeroth-order optimization, which is slow when the set of parameters increases, rely a differentiable proxy of the target finishing pipeline, which is hard to train. To overcome these challenges, we propose a novel goal-conditioned reinforcement learning framework for efficiently tuning parameters using goal image as a condition. Unlike previous approaches, our tuning framework does not rely on any proxy and treats the photo finishing pipeline black box. Utilizing a trained reinforcement learning policy, it can efficiently find the desired set of parameters within just 10 queries, while optimization based approaches normally take 200 queries. Furthermore, our architecture utilizes goal image to guide the iterative tuning of pipeline parameters, allowing for flexible conditioning on pixel-aligned target images, style images, or any other visually representable goals. We conduct detailed experiments on photo finishing tuning and photo stylization tuning tasks, demonstrating the advantages of our method. Project website: http:///wwnnmggiiggggggiiiibb/// Pixpo//RRRiPPPPeeeeii Tuner/
Introduction
Image processing pipelines (ISPs) are widely used by photographers and artists retouch images to match their desired appearance. Existing pipelines like Adobe Lightroom and Darktable allow users to interactively tweak meaningful sliders such as exposure, white balance, and contrast, which control the pipeline to perform a series of non-destructive edits to the input image. Though users can manually tune the slider parameters, it is laborious and time-consuming even for experienced experts. To this end, automatic photo finishing algorithms have been introduced to automate the process and have drawn growing attention in the community
In this work, we aim to design an automatic tuning algorithm for black-box non-differentiable image processing pipelines. Although some preliminary research tries to propose fully differentiable image processing pipelines [1] Z or approximate the existing pipelines using neural networks (2] they only support limited set of image processing operations. On the other side, most commercial image processing pipelines, like Adobe Lightroom, are still black-box and non-differentiable, with a set of tunable parameters exposed to users. Under this setup, the goal of pipeline tuning is to automatically find the set of optimal parameters to achieve a desired image appearance. named the tuning target. More specifically, in this work, we study two different tuning targets. One is a target image with the same content as the input, but with a different rendering style, and we call it photo finishing tuning. The other is target style image with different content, and the algorithm similar way as the style target, and we call that photo stylization tuning.
render the target in
"This work was done while Jiarui Wu interned at Shanghai AI Laboratory. *Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Method
C 0.27 seconds
Ours
RL-baed
No Proky Needed High Quality
Tuning
Convergence
38.92 dB
Input Image
70.45 secords
Cascaded
Proxy
X Requires Proxy. Poor Slow Convergance
Rast-Ordor
Tuning
15.82 dB
18.69 dB
Tuning Target
90.4
CMAES
Zeroth-Crdes Orddr
v/ No Prozy Needed
Tuning
v Reasonable Quality
29.85 dB
X Slower Convergence
Input)
500
Tuning Step
Figure I: In this work, we propose an RL-based photo finishing tuning algorithm that efficiently tunes the parameters black-box image processing pipeline match any tuning target. The RL-based solution (top row) takes only about 10 iterations to achieve similar PSNR as the 500-iteration output of a zeroth-order algorithm (bottom row). Our method demonstrates fast convergence, high quality, and need for a proxy.
One previous solution for photo finishing tuning is to use zeroth-order or first-order optimization. However, both of them are either consuming or limited small set of parameters. Zeroth-order optimizations [9 [1 20 4 are gradient-free searching methods and thus are normally very slow when
the search space increases. First-order optimizations [28] 32 29 accelerate the searching process
using gradient descent, but they either require the processing pipeline itself to be differentiable, or a neural proxy pre-trained to find a differentiable proxy of the original pipeline. For complex commercial imaging pipeline, like cellphone camera pipelines, this proxy may not fully reproduce original pipelines
[28]. Neural plex pipeline into small modules, but this does not apply black-box dynamically reconfigurable pipelines (32]. Considering all these limitations, this brings challenging question: there an efficient parameter-searchheeee
(29) improves the proxy accuracy by breaking a com-
algorithm that is applicable to non-differentiable finishing pipelines?
To solve this challenge, we propose novel goal-conditioned reinforcement learning (RL) approach dedicated to photo finishing tuning. At each RL iteration, the policy network takes the tuning target and the currently tuned image as input, and finds a better set parameters that makes the finishing results closer to the target. This RL-based searching algorithm has several advantages over traditional optimization methods. Compared with the zeroth-order solution, the RL policy can more accurately predict the potential searching direction, while zeroth-order searching tries. As a result, RL searching is much more efficient. As shown in Fig. [ the RL-based solution (top row) only takes about 10 iterations to reach similar PSNR the 500-iteration output of a zeroth-order algorithm (bottom row). Also, compared with first-order optimization, RL-based tuning directly optimizes the non-differentiable image processing pipeline without a differential proxy. Therefore, it is not limited by the variety and complexity of image processing operations and pipelines, achieving much better tuning results. As shown in Fig. I RL-based tuning reaches 38.92dB (top row), while the first-order solution only obtains 18. 18.69dB (middle row).
only rely less effective
To train an efficient RL policy for tuning, we also propose novel state representation dedicated to this task. The state representation should model the relationship between the photo editing space and policy. To achieve that, state representation consists of three key components: CNN-based feature representation to encode global and local features, a photo statistics representation to match the photographic statistics between the input and the goal, and embedding of historical actions. These representations better fit the RL policy into our task and guide the policy to generate the next set of parameters effectively. Lastly, we also design reward functions for both photo finishing tuning and photo stylization tuning, enabling our framework to tune photos to different targets.
We validate the effectiveness of our framework with extensive experiments on both photo finishing tuning and photo stylization tuning. Our RL-bbased framework significantly outperforms previous methods in both tasks in terms of both efficiency and image quality. Experimental results demonstrate

---

that our condiiioid policy is an efficient photo finishing tuner capable of performing fine- grained control on image processing pipeline parameters achieve various goals.
2 Related Work
ISP tuning. Recent developments in end-to-end AI-based ISP pipelines show potential as alternatives to traditional mobile ISPs [3] [4 25]. Yet, traditional parametric ISPs remain preferred in consumer cameras for their controllability. efficiency, and interpretability. These systems require expert-driven, labor-intensive tuning to enhance image quality and achieve desired aesthetic effects. Efforts to streamline this process are ongoing, aiming to reduce the need for manual adjustments. Gradient- based optimization is notable strategy in this area. Tseng et al. [28] have applied differentiable
black-box proxies to simplify ISP tuning. Further research by Tseng et al. [2]] has opened up the ISP pipeline into white-box manageable modules, learning differentiable proxies for each module and subsequently integrating them, which improves the tuning performance. Additionally, Qin et [23] introduced an attention-based CNN approach for scene-aware ISP tuning. However, this method lacks integration of sequence-specifte prior knowledge. They later developed a framework for predicting ISP hyper-parameters sequentially (24). optimizing parameters based on their relationships and similarities. In different approach, Mosleh et al. (20) utilized a genetic evolutionary algorithm with zero-order stochastic solver[9) [ to directly optimize hardware-specific image processing pipelines, methods. Moreover, Nishimura et al. [21 explored
circumventing the constraints of gradient-base derivative- free optimization, employing nonlinear techniques and automatic reference generation for effective automation of image quality adjustments. Despite these advancements, the complexity of the image processing pipeline and the vast parameter space continue to challenge the efficiency of tuning methods.
Photo stylization. Automating image stylization, which is straightforward for humans, poses challenges for machines. Significant research has been conducted to bridge this gap. Karras et al. 115 pioneered StyleGAN, network manipulating the latent space control image styles at various scales. Building on this, Brooks et al. [2] developed InstructPix2Pix which allows users to guide the stylization process through textual instructions, enhancing user-machine interaction. However, these methods often lack explainability and may alter the original image content. Further exploring transparency, Hu al. [II] proposed reinforcement learning-based white-box photo-finisher, though its need for explicit gradients limits compatibility with traditional systems. Kosugi et al. [8] addressed style diversity using unpaired data based on reinforcement learning. Despite these advances, these methods are restricted to a single style during training, limiting the user's ability to control the pipeline produce images of any desired style during inference. Moreover, Tseng et al. [29] optimized proxy networks for image processing modules via style loss function, achieving promising results but facing limitations in handling complex modules and style variability during inference.
3 Method
3.1 Problem Definition
Throughout this paper, we aim tackle the photo finishing tuning problem. Given input image Ju, an image processing pipeline frirE maps the input to a finished image IINISEED = fewe (To, P). controlled by set of parameters P. Our task is to solve the inverse problem: given input image In and a tuning target I (the goal condition in the RL. framework), how to find the parameters that reach this target:
arg min L(Ig, SPIPE(Jo. P)).
P
The goal image I, can vary between different tasks. For photo finishing tuning, the goal image shares the same content the input, and the tuning target is to minimize the distance between the pipeline output and the goal image 4g. For photo stylization tuming, the goal image is style target with different content than the input, and the tuning target is generate an output that matches the style of the goal image. Note that unlike artistic style transfer [6], we focus on photorealistic style transfer 31] where the processing pipeline does not change the content of an image.
3.2
Goal Conditioned Reinforcement Learning
We are the first to introduce reinforcement learning (RL) approach to the photo finishing tuning task by formulating it as an end-to-end policy learning problem. Inspired by human experts who use
3

---

Goal (Target) Image
Dual-path Feature Representation
State Rep.
00
Policy Network
D00 CNN Encoder
000
Goal-conditioned Policy
Local path
Observations
Global path
de
Input (Step Image
Photo Statisties Representation F
Historical Actions pf
Image Stathetier
paramm(()
Low Ao
Pipeline Parameters
Gool Image Statictics
Histograms
param(()
RL Tuning Step
RL Tuning Step 2
RL Tuning Step
Tuning
Tuning Trajectory
Image Processing Pipeline
Step 1 Image
Step Image
Step 3 Image
Output Image
Figure 2: The overall framework. Top row: at each step. our policy maps the current image and the goal image to action (new parameters), with the help of our state representation consisting of dual-path features, photo statistics, and historical actions. Bottom row: visualization of iterative tuning trajectory of our RL-based photo finishing framework.
iterative trial and error in photo finishing, method models the task decision process with multi-step feedback, akin to a Markov Decision Process. As shown in the bottom row of Fig. 2 policy iteratively tunes the parameters of given image processing pipeline to match a goal image. At each step, it takes the currently retouched image and the goal image as inputs, and then outputs the next action. This RL-based framework efficiently predicts pipeline parameters, requiring only minimal iterations (e.g., 10 queries to the image processing pipeline). Additionally, since RL optimization bypasses the need for gradient flow from the image processing pipeline, system can handle any black-box pipeline, regardless of complexity.
The RL process is formally defined follows. Let S be the state space, O be the observation space, A be the action space, T be the transition function, R be the reward function, g be the goal distribution, Po be the initial state distribution, and 7 the discount factor. All these forms Goal-conditioned Partially-Obsere
Markov Decision Process (S,0,A, T, R. Pu, [9]]
In each tuning episode I, the agent is given goal image I, E G, as well as observation O consists of the retouched image I, at step / along with all historical actions and observations. The action is the parameter set P used by image processing pipeline to generate the output image at the next step. And the transition function T S A - S is the image processing pipeline friPE defined in Sec. 3.1 The reward function is R (. Ig) for S and E G. We aim to learn a goal-conditioned policy (a]o, Ig) : x g + A that maps from observation o and goal Ig to the next action maximizing the sum of discounted rewards Eno-po. 1n E. R'R(s, Lg). Our policy T(alo, In) is deterministic policy μe parameterized by 0, outputting continuous actions a: (o:; Ig).
3.3 Photo Finishing State Representation
State presentation is also critical for the success of the proposed RL policy. This is particularly important in challenging scenarios where the policy must tune parameters for unseen goals of any style. Our experiment in Sec. 4.3 also shows that simple concatenation of the input and goal images

---

yields sub-optimal result. Therefore, we design comprehensive photo finishing state representation to extract features from observations that are critical for photo finishing tuning.
Specifically, our representation consists of three components: a CNN-based dual-path feature rep- resentation to encode both global and local features, photo statisties representation match the traditional photographic statistics between input and goal images, and an embedding of historical actions. Details of each component are described below.
Dual-path feature representation. In our dual-path feature representation, we seek to extract both global and local features from both input and goal images, inspired by [8]. This is because the tuning task requires not only global image characteristics such as overall color, tone, average intensity, and scene category, but also local features associated with texture, highlight, and shadow. As shown in Fig. 2 the architecture begins with a stride-2 convolutional encoder to reduce spatial resolution and extract initial low-level features. It then splits into two paths: a local path and global path. The local path { includes two stride-1 convolutional layers, preserving spatial resolution to extract local features. The global path {G* Ji=1...NG has one stride-2 convolutional layer and three fully-connected layers, providing a global scene summary vector. This global feature encapsulates essential global image characteristics such overall color and tone, as well as global notion of scene category (light condition or indoor/outdoor). We fuse the global and local features by adding the global feature at each x, y spatial location of the local feature: F = (GNNe + L where is the ReLU activation. This results in a dual-path feature representation F.D.
Photo statistics representation. Simply relying on a CNN-based policy may lead to unsatisfactory results with input and goal images outside of training distribution. To better represent invariant features across diverse styles and content of both input and goal images, we introduce a photo statistics representation, which matches traditional image statistics such as a histogram between input and goal. Global photo statistics, such as histograms, are critical in global image processing operations, such as exposure control [22). highlight, and shadow. Since they cannot be well represented by conventional convolutional neural networks due to limited receptive fields [29], propose pre- compute these statistics and concatenate them into our state representation. Specifically, we compute histograms Hl on the RGB channels of input and goal images and map these to fixed dimension using a linear layer H' = Linear( Lineer((Hrgg luminance, median, contrast, and saturation of both input and goal images to form the photo statistics representation F
H Hr (Tg)). )) This feature is then concatenated with the
Policy network. At last, we combine the dual-path feature representation and photo statistics representation with a historical action embedding FM Linear(d1:; (1:),, where & the fz- distance of image I, and goal image I As shown in Fig. 2 the input of policy network can be formulated as = Concat( (PP;; P.; ). The policy network is multi-layer perceptron network (MLP) that maps from the current state and the goal representation to the next action to take. We choose deterministic policy to directly output continuous action a, = ua(o,, 9) = po(s)). We use the same architecture to estimate the value function for RL updates.
3.4 Reward Function and Training Objectives
We provide a general RL-based framework for photo tuning tasks. One can train our goal-conditioned end-to-end policy with different reward functions to resolve different photo tuning tasks including photo finishing tuning and photo stylization tuning. The policy is optimized with twin-delayed DDPG (TD3) algorithm (5).
Reward function for photo finishing tuning. When goal image I, is the photo-finished input image, measure the distance between current image I, and Iy with PSNR metric. The reward function is calculated as the difference between PSNR values of consecutive steps:
(2) Instead of using fı-distance to measure image distance, we use PSNR, the negative logarithm of fy-distance, This design ensures the policy receives appropriate rewards even when the current image is close to the goal, encouraging fine- fine-grained tuning of pipeline parameters.
PSNR(I++1: 1g) - PSNR 1g).
Reward function for photo stylization tuning. When goal image Ig is a style image with arbitrary content and style, we measure the distance between the input and goal images with a style score:
N₂
StyleScore, Σ
-G[[[])GG[[A] - G. + (011((1) ), HOT ll + AIIHH H ll2, (3)
5

---

where Ns denotes the number of layers from a pre-trained VGG-19 27 model used to extract features. Following [6]. the style is captured using Gram matrices G [:I"", with F, representing feature maps. Additionally, the fg-distances of histograms for the Y and UV channels are included to align luminance and color palettes. The overall reward is calculated by the change in style score across consecutive steps, penalized by the difference in content features F.:
rt StyleScore, StyleScore, A2
[F F(41112
Policy optimization. We optimize our goal-conditioned policy with off off-policy TD3 [5] algorithm. Specifically, TD3 learns two Q value function Q and Q optimized by mean square Bellman error minimization:
y(r S141, d) + 7(1 d) min Qui (s1+1, a' (s1+1)),
(5)
1=122
L( = E arDD
[Q az) az) - u(re s141, d))
(6)
where Qo is the exponential moving average of QΦ a' 1 is given by target policy with clipped gaussian noise, and d is the termination signal. The state transition pair (st, az Tt, 81451 d) is sampled from replay buffer D. With Q functions, the policy (=(-) is learned by maximizing Q
L
E (Q (st, ue (s:)))
More details about our reward functions and policy optimization are in the appendix.
4 Experiments
In the first subsection, we provide the details of datasets and task settings for photo finishing tuning and photo stylization tuning, along with a description of the evaluation metrics. the second subsection, we demonstrate experimental results and compare them to zeroth-order optimization (00 and first-order optimization [28 29] baseline methods. Ablation studies are conducted the last subsection, which investigates the impact of each component of our state representation. We also provide supplementary qualitative results in the Appendix.
4.1 Tasks Settings and Datasets
Datasets. We use the MIT-Adob FiveK Dataset (3), a renowned resource in the field of photo retouching, which comprises 5,000 photographs captured using DSLR cameras by various photogra- phers. This dataset is notable for providing images in raw format alongside the retouching outcomes of five experts. For our study, we selected 4,500 images to serve the training dataset, with the remaining 500 images designated as the validation dataset. In our method, random parameters are employed generate the target images, which are used as training data pairs. In the task of photo finishing tuning, the datasets including both the expert C retouched targets and randomly generated targets are utilized for evaluation. The expert C retouched targets are optimized using CMA-ES to ensure they are reachable by our image processing pipeline. For the photo stylization tuning task, we have curated collection of 200 diverse style images from the Lightroom Discover website following [26).
To further evaluate our method and demonstrate its generalizability, we directly on the HDR+ dataset [1]. We used the official subset of the HDR+ dataset, which consists of 153 scenes, each containing up to 10 raw photos. The aligned and merged frames used as the input, expertly tuned images serve as the mmoto-finishing targets.
RL-based framework
image processing pipeline consisting of standard image processing operations, including exposure, color balance, saturation, contrast, tone mapping (highlight and shadow). and texture (sharpness and smoothing), with nine adjustable parameters in total, similar to [29]. Thus, the agent's action space is comprised of fine continuous actions corresponding to these nine pipeline parameters. During the policy inference, the input and goal images are resized to the resolution of 64 x 64. Additionally, a 3-level Laplacian pyramid of
Implementation details. We conduct all experiments

---

Table 1: photo finishing tuning experimental results on the FiveK validation datasets with FiveK targets (expert- and random targets. Queries represent the times of query image processing pipeline.
Eval Dataset
FiveK Target
Random- Target
Method
PSNR
SSIM
LPIPS
Queries,
PSNR
SSIM
LPIPS
Queries.
CMAES 20
28.53
0.9586
0.0968
200
32.29
0.9754
0.0827
200
Monolithic Proxy 28
21.71
0.9104
0.2144
21.08
0.9251
0.2785
Cascaded Proxy
29
22.31
0.9115
0.1939
21.40
0.9213
0.2613
Ours
35,89
0.9764
0.0305
10
38.46
0.9814
0.0128
10
Input
Monelthic Procy
Cvscaded Prony
CMARS
Ours
Yarget
Figure 3: Photo finishing tuning results on FiveK dataset with expert C target. The visual results of method are closest the target image, especially in terms of color and brightness.
both input and goal images is constructed and fed into the policy network to capture high- frequecyy details from the original resolution. Then the output parameters from the policy network are fed to image processing pipeline along with full-resolution images to produce high-resolution results. We train policy using the standard TD3 algorithm (5) and set the termination of our RL policy to trigger when the episode length reaches the maximum threshold (10 steps), ensuring efficiency. Further details can be found in the appendix.
we employ the Peak Signal-to-Noise Ratio (PSNR), the Structural Similarity Index Measure (SSIM) (30, and the Learned Perceptual Image Patch Similarity (LPIPS) 33 as our evaluation metrics. assess the quality of stylization, we conduct user studies to evaluate our methods, offering subjective measure of image quality based on viewer assessments.
Evaluate metries. Similar to 1I
28
4.2 Results
Results of photo finishing tuning. To evaluate the efficacy of our framework on the photo finishing tuning Task, we conduct two experiments on the FiveK validation datasets, using both expert-C targets and random targets. Our method is compared against the monolithic proxy- based approach [28), the cascaded proxy-basee method [29], and the search-based method proposed [20). Consistent with [29], utilize 100 iterations during inference for the proxy-based methods. Additionally, record the number of times that the photo-finiste finishiin pipeline was queried in both search-basee methods and our approach.
As illustrated in Tab. [ our method outperforms the others across all metries on both Fivek-Target and Rargem-Target.. The monolithic proxy-based method struggles with accurately representing the complex image processing pipeline, leading to suboptimal performance. While the cascaded proxy method can incrementally enhance tuning performance over its monolithic counterpart, it suffers from accumulated errors and difficulties in approximating certain operations, such as texture, resulting in poorer performance. Notably, our method only requires querying 10 times image processing pipeline,
7

---

Table Experimental results demonstrating efficiency across varying input resolutions. Our method significantly outperforms other methods, achieving a speed enhancement of 260 times relative to the cascaded proxy method (2 at 720P resolution, and 117 times faster than the CMAES
approach 99 20 at 4K resolution.
Methods
Monolithic Proxy [X]
Cascaded Proxy 29
CMAFS 9
Ours
Resolution
720P
2K
720P
1K
2K
4K
720P
1K
720P
1K
4K
Time(s)
7.67
17.89
33.07
0OM
70.45
0OM
OOM
OOM
36.16
51.82
91.86
144.02
0.27
0.33
0.47
1.23
Inpot
Momalithie
Cascaded Proxy
CMARS
Ours
Target
Figure 4: Photo stylization tuning results. Compared with CMAES 20 monolithic proxy and cascaded proxy 29 our output matches the best with the style goal.
28
whereas the performance of the search-based method, ours. Further, the visualization results depicted in Fig. demonstrate that our method produces visual outcomes that most closely match the target images, particularly in terms of color and brightness, when compared to all other methods. More visualization can be found in Fig. 7 of Appendix A
with 200 queries, remained inferior
Efficiency. To evaluate the efficiency of approach, conducted speed testing experiments system equipped with an AMD EPYC 7402 (48C) @ 2.8 GHz CPU, 8 NVIDIA RTX 4090 GPUs with 24GB of RAM each, 512 GB of memory, and running CentOS 7.9. In line with 29), applied 200 iterations during inference for both the proxy-based methods [28 29 and the search- method 20 We measured the execution time for each method across four different input resolutions.
As indicated Tab. 2 our method demonstrated superior efficiency, requiring only 1.23 seconds per exccution with 4K input. While the monolithic proxy-based method outperformed the search-bare based method in terms of speed, it faced limitations as input resolutions increased, leading to out-of -et-of-mmmory (OOM) errors once GPU memory was exceeded. The cascaded proxy-based method, which includes MLP networks, was the slowest and most prone to memory overflows due to its intensive memory demands. The search-based method primarily depends CPU performance. Despite being executed on a high- gesh-performncce server, the CMAES method 20 requires 144 seconds to process 4K input image, which is considerably slow.
Results of photo stylization tuning. We conduct qualitative comparisons and user studies on photo stylization tuning task to demonstrate the effectiveness of our goal-conditioned RI. framework. Our policy was trained using the FiveK training dataset, and all methods were evaluated on input and style image pairs collected from the Adobe Lightroom Discover website. For all baseline methods, we adopt the same style score described Sec 3.4 as optimization objectives, ensuring fair comparison.
As shown in Fig. 4 our method produces results closer to the style goal image compared to the baseline. Notably, our method achieves better results with only 10 queries to the image processing pipeline, whereas the baseline methods require 200 queries, making them orders of magnitude slower. important to note that the style images from the Lightroom Discover website have different distribution than our training dataset. Despite this, our method adapts directly to the target style image distribution during testing, whereas the baseline methods require time-consuming optimization of testing data. These experimental results demonstrate that our RL-based framework can efficiently tune
8

---

Ingut
Monolthic Prosy
CMAES
Casaded Proxy
Target
Figure 5: Qualitative comparison on the HDR+ photo finishing tuning task. These comparisons illustrate that our method remains closer to the target even when dealing with input and target images outside the training distribution.
Table 3: Photo finishing tuning experimental results on addition HDR+ datasets 10 with HDR expert-tuned targets.
Eval Dataset
HDR+ Target
Method
PSNR
SSIM1
LPIPS]
Queries.
CMAES g 20
28.08
0.9539
0.1307
200
Greedy Search [6
25.79
0.9212
0.1542
200
Monolithic Proxy
17.80
0.8940
0.3044
Cascaded Proxy
18.90
0.8982
0.2797
Ours
31.54
0.9652
0.0563
10
images different unseen styles, showcasing that our photo finishing state representation (Sec. 3.3 has the capability to generalize to versatile goals outside of training distributions. More visualization with versatile goal images can found Fig. g of Appendix A.2
To rigorously evaluate the effectiveness of our method in photo stylization tuning, we implemented a subjective user study comprising 20 questions. In each question, participants were presented with images generated by four different boethods-monolinn own approach. Participants were asked identify up to two images that most closely resembled given target image. The study was conducted online, garnering 65 responses from a diverse group of individuals selected randomly from the internet.
proxy, cascaded proxy, CMAES, and our
The aggregated preferences are visually summarized in Fig. 6 The data clearly show that our method is perceived by the majority of participants producing results that most closely match the target images, highlighting its superiority stylization tuning tasks.
Cross dataset generalization. We conducted additional evaluations using the HDR+ dataset [10 to demonstrate our RL-based framework's 's ability to generalize effectively to unseen datasets. We compare to basclines including CMAES 1 20). Cascaded Proxy (29). Monolithic Proxy (28), and Greedy Search [66] Tab. 3 we report PSNR, SSIM, LPIPS, and queries to the ISP pipeline. The results demonstrate that our RL policy generalizes effectively to unseen data, achieving higher photo-finishin finishing quality than methods directly tuned on the test dataset. Qualitative comparisons in Fig. slsow that results are closer targets, even with input and target images outside the training distribution.
In Tab. our RL-based method achieves PSNR of 31.54 on the HDR+ photo-finishing task, and it outperforms all baselines. This shows that our RI. policy, trained on the FiveK dataset, effectively generalizes to the HDR+ dataset. Such out-of-distribution capability is facilitated by proposed

---

871
Table 4: Ablation study on each component of our finishing state representation. RI. denotes a baseline naively using CNN trained with RI., FD, F$ F# denotes each of our state representa- tion respectively.
CMAES
550
108
M.P. 102
400
200
1000
Ours
CMAES
Cascaded Proxy (C. PJ) Monolithic Proxy (M.
RL
FD
8
PSNRT
SSIMt
Figure 6: Results of the user study on the photo stylization tuning task. Each bar represents the number of votes each method received, where participants select the images they believed most closely resembled target images.
Ea1
32.17
0.968
E22
35.15
0.973
Exg
37.61
0,980
Exa
35,96
0.976
v
38.46
0.983
photo- featto-fiishigggeo state representation, which extracts invariant features for photo finishing, allowing adaptation diverse inputs and goals beyond the training distributions. The CMAES ( 20 baseline shows consistent results on HDR+ compared to FiveK, as it is directly optimized on the test dataset without prior training. However, proxy-based methods (28 29 perform worse because the proxy network trained on FiveK does not generalize well to HDR+, leading to incorrect gradients and poorer photo-finishing quality.
4.3 Ablation Study on State Representation
As has been shown in our main experiment, our RL-based approach significantly outperforms the previous method in terms of photo finishing quality and efficiency, demonstrating that RL is more suitable for the photo finishing tuning task. In this subsection, we focus on the photo finishing state representation we propose to better fit RL in our task. Specifically, we conduct experiments on photo finishing tuning task using FiveK Random- Target dataset, in order to study the contribution of each specific representation in our photo finishing state representation.
As shown in Tab. 4 we set our baseline RL as RL policy trained with a naive CNN-based encoder taking the concatenated input and goal images as input. This baseline achieves only 32.17 dB of PSNR. In Exg, our dual-path feature representation FD improves the photo finishing quality, as better encodes local features and global image characteristies critical to our photo tuning task. Since traditional photo statistics contain invariant features about global statistics that are difficult to learn solely using a network, the photo statistics representation SS also helps to boost photo finishing quality as shown in Exx, Moreover, parameters and results from all previous RL steps help in the decision process, the historical action representation is also useful as shown in Ezg, With the proposed three representations combined, our RL policy can be guided to effectively tune image processing pipeline parameters given input and goal images of any photo finishing style, as evidenced by the superior performance in Far..
5 Conclusion
In this work, we propose an RL-based photo finishing tuning algorithm that efficiently tunes the parameters of black-box image processing pipeline to match any tuning target. Our approach encompasses several key innovations. Firstly, we integrate goal-conditioned RL into the realm of photo-finishing tuning. Secondly, we propose a photo finishing state representation comprising three principal components essential for training an effective RL policy network: CNN-based feature representation that encodes both global and local image features, photo statistics representation designed to align the photographic statistics between the input and the target, and an embedding of historical actions. We assess the effectiveness of our framework through comprehensive experiments on both photo finishing tuning and photo stylization tuning. The experimental results affirm that coal-conditioned policy is adept choto-finishinn tuner, capable of exerting efficient and fine- grained control over image processing pipeline parameters to fulfill a variety of objectives. Currently, method exclusively supports conditional inputs in the form of images and lacks the capability to process non-image types such as textual inputs. Moving forward, we intend to broaden our research to include multi-modal conditional inputs.
10

---

Acknowledgements
This work is supported by Shanghai Artificial Intelligence Laboratory and RGC Early Career Scheme (ECS) No. 24209224. We also extend our gratitude to Quanyi Li for his insightful discussions and valuable comments.
References
[1] Blank and K. Deb. pymoo: Multi-objective optimization in python. IEEE Access, 8:89497- 89509, 2020.
[2] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18392-18402, 2023.
[3] Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand. Learning photographic global tonal adjustment with a database of input/ output image pairs. In CVPR, 2011.
[4] Pin- Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pages 15-26, 2017.
[5] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning, pages 1587-1596. PMLR, 2018
[6] Leon A Gatys, Alexander S Beker, and Matthias Bethge. Image style transfer using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and patlern recognition, pages 2414-2423, 2016.
[7] Michaël Gharbi, Gaurav Chaurasia, Sylvain Paris, and Frédo Durand. Deep joint demosaicking and denoising. ACM Transactions on Graphics (ToG), 35(6): 2016.
[8] Michaël Gharbi, Jiawen Chen, Jonathan Barron, Samuel W Hasinoff, and Frédo Durand. Deep bilateral learning for real-time image enhancement. TOG.
[9] Nikolaus Hansen. The ema evolution strategy: comparing review. Towards a new evolutionary computation: Advances in the estimation of distribution algorithms, pages 75-102, 2006.
[10] Sam Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T. Barron, Florian Kainz, Jiawen Chen, and Mare Levoy. Burst photography for high dynamic range and low-light imaging on mobile cameras. SIGGRAPH Asia, 2016.
[11] Yuanming Hu, Hao He, Chenxi Xu, Baoyuan Wang, and Stephen Lin. Exposure: A white-box photo post-proeessing framework. TOG.
[12] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normalization. In ICCV, 2017.
[13] Andrey Ignatov, Lue Van Gool, and Radu Timofte. Replacing mobile camera isp with a single deep learning model. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pages 536-537, 2020.
[14] Wooseok Jeong and Seung-Woi Won Jung. Rawtobit: A fully end-to-end camera isp network. In European Conference Computer Vision, pages 497-513. Springer, 2022.
[15] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4401-4410, 2019.
[16] Heewon Kim and Kyoung Mu Lee. Learning controllable isp for image enhancement. IFEE Transactions Image Processing, 33:867-880, 2023.

---

[17] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[18] Satoshi Kosugi and Toshihiko Yamasaki. Unpaired image enhancement featuring reinforcement- learning-controlle image editing software. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11296-11303 2020.
[19] Minghuan Liu, Menghui Zhu, and Weinan Zhang. Goal-conditioned reinforcement learning: Problems and solutions, 2022.
[20] Ali Mosich, Avinash Sharma, Emmanuel Onzon, Fahim Mannan, Nicolas Robidoux, and Felix Heide. Hardware-in-ther -oophe-ooopppp end-to-end optimization of camera image processing pipelines. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7529-7552 2020.
[21] Jun Nishimura, Timo Gerasimow, Rao Sushma, Aleksandar Sutie, Chyuan-Tyyng Wu, and Gilad Michael. Automatic isp image quality tuning using nonlinear optimization. In 2018 25th IEEE International Conference Image Processing (ICIP), pages 2471-2475. IEEE, 2018.
[22] Emmanuel Onzon, Fahim Mannan, and Felix Heide. Neural auto-exposure for high-dynamic range object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2021.
[23] Haina Qin, Longfei Han, Juan Wang. Congxuan Zhang, Yanwei Li, Bing Li, and Weiming Hu. Attention-aware learning for hyperparameter prediction in image processing pipelines. In European Conference on Computer Vision, pages 271-287 Springer, 2022.
[24] Haina Qin, Longfei Han, Weihua Xiong, Juan Wang. Wentao Ma. Bing Li, and Weiming Hu. Learning to exploit the sequence-specific prior knowledge for image processing pipelines optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22314-22323, 2023.
[25] Ardhendu Shekhar Tripathi, Martin Danelljan, Samarth Shukla, Radu Timofte, and Luc Van Gool. Transform your smartphone into a dslr camera: Learning the isp in the wild. In European Conference on Computer Vision, pages 625-641. Springer, 2022.
[26] Jing Shi, Ning Xu, Haitian Zheng, Alex Smith. Jiebo Luo, and Chenliang Xu. Spaceedit: Learning a unified editing space for open-domain image color editing. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 19730-19739, 2022.
[27] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognitio. In International Conference Learning Representations (ICLR), 2015.
[28) Ethan Tseng, Felix Yu, Yuting Yang, Fahim Mannan, Karl ST Arnaud, Derek Nowrouzezahrai, Jean-François Lalonde, and Felix Heide. Hyperparameter optimization in black-box image processing using differentiable proxies. ACM Trans. Graph., 38(4):77-1, 2019.
[29] Ethan Tseng, Yuxuan Zhang, Lars Jebe, Xuaner Zhang, Zhihao Xia, Yifei Fan, Felix Heide, and Jiawen Chen. Neural choto-finishing ACM Transactions on Graphics, 41(6)33555526, 2022.
[30] Zhou Wang, Alan C Bovik, Hamid Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
[31] Xide Xia, Meng Zhang, Tianfan Xue, Zheng Sun, Hui Fang, Brian Kulis, and Jiawen Chen. Joint bilateral learning for real-time universal photorealistic style transfer. In Computer ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020. Proceedings, Part VIII 16, pages 327-342. Springer, 2020.
[32] Ke Yu, Zexian Li, Yue Peng, Chen Change Loy, and Jinwei Gu. Reconfigisp: Reconfigurable camera image processing pipeline. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4248 4257, 2021.
[33] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 586-595, 2018.
12

---

A Additional Results
A.1 Photo Finishing Tuning Results on FiveK Dataset
We show more photo finishing tuning visualization results on the TiveK-Targes evaluation dataset in Fig. 7 showcasing its superior performance in terms of color and brightness compared all other methods.
Iaput
Monolithi
Cascaded
CMAES
Targel
Figure 7: Additional results photo finishing tuning task.
Skylization Goal
Input Image
Figure 8: Additional results photo stylization tuning task.
13

---

A.2 Photo Stylization Tuning Results
We show more photo stylization tuning visualization results in Fig. 8 showeasing its superior performance in tuning ipuut images to any style goal that is outside the training goal distribution.
B Image Processing Pipeline
The detailed image processing pipeline in this paper encompasses various standard operations, including:
(1) Exposure: Adjusts the overall brightness of the image, primarily affecting the amount of light captured. Increasing exposure makes the image brighter while decreasing it makes the image darker. This is useful for correcting overexposed or underexposed photos. This operation includes one slider parameter.
(2) Color Balance:
Used adjust the color temperature and tint of an image, altering its overall color feel. Adjusting the color temperature (blue to yellow) and tint (green to magenta) can make the image appear warmer cooler. This operation includes three slider parameters.
(3) Saturation:
Controls the intensity of colors in the image. Increasing saturation makes the colors more vivid and lively, whereas decreasing saturation reduces color intensity, making the image appear softer or closer to black and white. This operation includes one slider parameter.
(4) Contrast: Adjusts the difference between the brightest and darkest parts of the image. Increasing contrast makes dark areas darker and bright areas brighter, enhancing the depth and dimension of the image. Decreasing contrast brings these areas closer to mid-grey, reducing the image's s depth. This operation includes one slider parameter.
(5) Tone Mapping: This operation compresses high dynamic range input to a smaller dynamic range, which is affected by the Highlights and Shadows sliders, respectively. Highlights: Adjusts the brightness of the brightest areas the image without affecting overall exposure. This helps recover details in overexposed areas. Shadows: Adjusts the brightness of the darkest areas, helping reveal details shadowed regions without changing the overall exposure. This operation includes two slider parameters.
(6) Texture:
Enhances reduces the detail and texture in the image without affecting colors. Enhancing texture can make details in the image clearer, while reducing texture can smooth out the image, often used in portrait photos for skin treatment. This operation includes one slider parameter.
C Additional Implementation Details and Training Strategy
C.1 Training Details of TD3 Algorithm
We state the details of TD3 (5] algorithm training in this subsection. TD3 is an off-policy RL algorithm. When the policy sample transition pair (st, de, Ft, St+1, d) to form the replay buffer Tt, D. gaussian noise is added to encourage exploration: a = clip (po(s) + Low, Higgh), where o), We set 0.1 for trade-off between exploration and exploitation.
To compute the target action in Q update target:
((a.(10 - (12 rr+ (r
27
I(bi)
E Se/DD
min Qeran ($141, d' (0t41)) 1=12
The next action of 8 comes from the target policy with clipped noise, so that incorrect action peak produced by sub-optimal Q value estimation smoothed out, as in the following equation:
a' d((s411) clip (mmim (5t+1) 1) + clip(e1, -c, c), aoo (High)
El (0,
In this implementation, set é 0.2, which is twice the value of e. TD3 updates the policy less frequently than the Q-function. We set the policy to update half as frequently as the Q network update. The optimization objectives of the policy already given in the main paper. To ensure robust
14

---

update of the policy and Q network, TD3 adopts an Exponential Moving Average (EMA) strategy to update the target network Q and policy network which used to compute the Q value target above.
Puarg11 - Pirurg,1 1 + (1 p)g,
(10)
Puurg22 +- porurg22 + (1 p)
(11)
Orag f- plragg + +1 - pp)
(12)
We set the EMA update rate p 0.99 in all our experiments. The optimizer performed using Adam optimizer [17 with (31, (22) (0.9, 0.999). We set the learning rate to specific values for policy and value network, that is, le-4 action and 2e-4 for Q network. We set hatch size as 64 for both photo finishing tuning and photo stylization tuning experiments. We set the discount factor y = 0.9. All experiments are conducted on NVIDIA RTX 4090 GPUs. Furthermore, we implement the CMAES based on open-source framework [
method 288
C.2 Other Training Details
Reward design. For our detailed reward design, as described in Sec. 3.4 we utilize the PSNR metric of consecutive steps for photo finishing tuning. In photo stylization tuning, we adopt the style score as the main reward and add content negative reward prevent the policy from taking drastic actions that hinder photo stylization quality. Specifically, we set Àg and X1 in StyleScore, to 100 and 50, respectively, and set X2 0.5. For the VGG features selected to compute the style score, we follow [12 to set Ng- = Ne = 4, selecting relu_(11..4 (1...4] features to form [F)).=1...4 which is then used to compute then gram matrix and the content regularization term.
RL termination. Since our RL policy freely explores the image processing pipeline parameter space, we need to terminate the current episode if the state collapses, meaning the policy outputs parameters that render an abnormal image. Specifically, we calculate the average pixel intensity of the image 4 as Ie and set this value be within the range of Imai and Imaax- For each rollout, if Ie is out of range, we terminate the state and do not save it to our replay buffer D.
Network architecture. Our approach learns end-to-end policy that maps from input images and goal specifications to the next action to take (the next parameters the image processing pipeline.) The state representation is s, described in Sec. B.3] which is fed into 4-layer multi-layer perceptron network (MLP), with a continuous action space of parameters. Each hidden layer of the MLP network is of width 512.
C.3 Baselines Implementation Details
Implementation details for CMAES 9 20). CMAES is a gradient-free search (zeroth order optimization) method using evolution strategy. As 20] does not provide code, we implemented CMAES using the pymoo library [. enabling parallel execution on multi-core CPUs and achieving reasonable performance. This baseline does not require training.
Implementation details for Cascaded Proxy [29]. For proxy network training. we followed the architecture in (29]. using 3 consecutive 1 x convolutions for pointwise ISP operations and 5 consecutive x convolutions for areawise operations. We trained with the Adam optimizer, a learning rate of 1e-4, a batch size of 512, and 100 epochs, recommended. Camera metadata was extracted from DNG files, as described in [29] Since [29] does not release its dataset, we used the MIT-Adobe FiveK dataset, in our method. Following Section 5 of [29], we used 1,000 raw images from FiveK and sampled 100 points for each ISP hyperparameter.
Implementation details for Monolithic Proxy (28). The architecture uses single UNet to approxi- mate the ISP pipeline, with hyperparameters conditioned by concatenating extra planes the features. We trained the proxy with the Adam optimizer, a learning rate of le-4, a batch size of 512, and 100 epochs. The training set generation and first-order optimization method are the same as for (29
D Detailed Content of the User Study
We conduct a human subject study on the user preference among results of methods for photo stylization tuning given target style images. In this appendix section, we provide the original questions that we used to ask for users' feedback.
15

---

For each of the 20 questions in our user study, the order of the methods was randomized and labeled as B, C, and to ensure the anonymity of the techniques used. Participants were not aware of which method corresponded to each label, eliminating any potential bias in their selections. The prompt for all questions remained consistent: "Among the four images on the right (labeled A, B, C, and D), which one(s) most closely resembles the target image on the left? You may select to two images you believe they are equally similar to the target image. Otherwise, please choose only one."
Detailed below are the options provided for each of the 20 questions.
Tangat
Figure 9: Options in question 1 of our user study.
Target
Figure 10: Options in question 2 of our user study,
Target
Figure 11: Options in question 3 of our user study,
16

---

Figure 12: Options question 4 of our user study.
Target
Figure 13: Options in question 5 of our user study.
Target
Figure 14: Options in question 6 of our user study.
Target
Figure 15: Options in question 7 of our user study.
Figure 16: Options in question 8 of our user study.
Target
Figure 17: Options in question 9 of our user study.
Target
Figure 18: Options in question 10 of our user study.
17

---

Figure 19: Options question 11 of our user study.
Target
Figure 20: Options in question 12 of our user study.
Target
Figure 21: Options in question 13 of our user study.
Target
Figure 22: Options in question 14 of our user study.
Figure 23: Options question 15 of our user study.
Target
Figure 24: Options in question 16 of our user study.
Target
Figure 25: Options in question 17 of our study.
18

---

Figure 26: Options in question 18 of our user study.
Target
Figure 27: Options in question 19 of our user study.
Target
Figure 28: Options in question 20 of our user study.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental details are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

tuning research, the same as Answer: [No]
Justification: Codes and datasets will be made publicly available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/dde/eeee Cooeeeoodeesssseeeeeee 7/nips.com htte://alllllgiiiee///oeeeuaaass ssiniiiicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
Details are provided Section 4.1 and Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: This follows the convention in previous works.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The information on the computer resources shown in Section 4.1 and
Appendix C
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We strictly adhere to the NeurlPS Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no direct negative societal impact for photo-finishiin tuning.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all original papers and make sure that usage is legal.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We construct experiments based public datasets and models.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification: Details of our user study, including the text of instructions and question options, are presented in Appendix D
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "1ulTh89oSv", "reviewer": "Reviewer_ASBs", "summary": "This paper applies goal-conditioned reinforcement learning (RL) to photo finishing tuning. With only 10 queries, it demonstrates that goal-conditioned RL can achieve better performance than zeroth-order approaches that require 500 queries. Additionally, this method is non-differentiable, which might make it more suitable for commercial scenarios. Experiments show that this work can achieve better performance than previous efforts.", "questions": "Please refer to the weakness section.", "limitations": "Please refer to the weakness section.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+ The first great application to adapt goal-conditioned RL to the photo finishing tuning task.\n+ A novel design for the photo finishing state representation.\n+ Achieved good performance on the evaluation benchmark.", "weaknesses": "- Some details are missing: The methods used in this work were trained on the training set, but it is unknown whether the compared methods also underwent the training stage.\n- More evaluation on additional datasets is preferred.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer ASBs\n\nWe sincerely thank you for reviewing our work. After reading your comments carefully, we summarize the following questions.\n\n## Q1: Details on Baselines\n\n***\n\nIn our main paper, we compare three baselines: (1) CMAES: a zeroth order optimization method, that does not need training, (2) Monolithic Proxy: first-order optimization through a single monolithic neural network proxy, that requires training a proxy network, and (3) Cascade proxy: first-order optimization through multiple cascaded neural network proxies, also requires training proxy networks. \n\nEach of the baseline's implementation details is available in Q3 of **To All** section. Here, We briefly  summarize the principal of each of these baselines:\n\n1. **CMAES** [18]: CMAES is a gradient-free search (zeroth order optimization) method using an evolution strategy. For our photo finishing tuning task, it directly optimizes the parameters of the image processing pipeline to match the target rendering style. Therefore, it does not undergo a training stage and optimizes directly on the validation set for multiple iterations instead. Such method faces no generalization challenges but converges very slowly as the searching processing is blind and brute-force. It also lacks exploration capability and is prone to stuck in local minimal with complex objectives.\n2. **Monolithic Proxy** [26]: is a gradient-based optimization method (e.g. gradient descent). Since our task involves tuning an image processing pipeline (ISP) that is a non-differentiable black box, the gradient cannot flow through the ISP pipeline directly to drive the optimization of ISP parameters. Therefore, it requires training a neural network proxy to approximate the behavior of the ISP pipeline. The differential nature of the neural network proxy allows gradient flow to directly optimize ISP parameters during inference. However, for a complex image processing pipeline, this proxy is hard to train and may not fully reproduce original pipelines.\n3. **Cascaded Proxy** [27]: is also a gradient-based optimization like [26], but the proxy neural network has a different architecture. The difference is that: the Monolithic Proxy uses a single UNet for the proxy, but the Cascaded Proxy improves the UNet with multiple small neural networks that are cascaded to address the vanishing gradient problem. Yet it still faces challenges with error accumulation, limited generalization capability of neural proxy, and cannot be applied to black-box pipeline.\n\nPlease refer to Q3 in the **To All** section for further implementation details for these baselines.\n\n## Q2: More Evaluation\n\n***\n\nPlease refer to Q1 in the **To All** section for the answer to this question. To summarize, we conducted further evaluations using the HDR+ dataset, demonstrating our RL-based framework's ability to generalize effectively to unseen datasets. Our method achieved a PSNR of 31.54 on the HDR+ photo-finishing tuning task, comparable to the 32.47 achieved on the FiveK dataset, and outperformed all baselines. This highlights our method's generalization capability across different datasets and its superior performance compared to other baselines, including CMAES, Greedy Search, Cascaded Proxy, and Monolithic Proxy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V3RaZnjzZG", "reviewer": "Reviewer_syDC", "summary": "Proposed Goal-Conditioned Reinforcement Learning for Photo Finishing Tuning. Specifically, the authors introduce a novel goal-conditioned reinforcement learning framework for parameter tuning in photo processing pipelines. Unlike existing methods, the proposed approach operates without relying on proxies and treats the pipeline as a black box. By leveraging a trained RL policy, it efficiently identifies optimal parameters in just 10 queries, contrasting with 500 queries typically required by zeroth-order methods. The proposed framework utilizes a goal image to guide iterative parameter tuning, allowing adaptation to diverse target images and styles. Experiments on photo finishing and stylization tasks validate the effectiveness and versatility of the proposed approach.", "questions": "(1) Why TD3? why not any other off-policy RL algorithm? Are there any specific reasons behind this choice?\n\n(2) Enhancing the paper with more ablation studies would elevate its quality. For instance, exploring the impact of RL versus a greedy algorithm on performance would provide valuable insights.", "limitations": "Yes, the authors adequately addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written. It is easy to follow. The proposed method is well-motivated. The proposed method seems intuitive and effective for photo-finishing tuning tasks.\n\nEmpirical evaluations on image-based datasets show the efficacy of the proposed method.", "weaknesses": "The motivations behind some algorithmic choices are not clear. \n\nAdditional ablation studies would enhance the persuasiveness of the findings.\n\nIt would be interesting to see the performance of the proposed method on other datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To reviewer syDC\nWe sincerely thank you for reviewing our work. After reading your comments carefully, we summarize the following questions.\n\n## Q1: Choice of RL Algorithm\n***\nIn our task, the choice of RL algorithm is not the primary factor driving performance; instead, our proposed state representation plays the most crucial role. We selected the TD3 algorithm due to its common usage and robustness to hyperparameters, but alternative algorithms like SAC or PPO would yield similar results.\n\nTo support this, we performed an additional ablation study comparing different RL algorithms. We trained our photo-finishing policy using SAC and PPO algorithms, keeping the state representation, policy network, reward design, and termination conditions identical to our original TD3 implementation. All implementations were based on the stable-baselines3 library in PyTorch. We evaluated each algorithm's performance on the photo-finishing tuning task using the FiveK-Random dataset.\n\n**Table 1: RL Algorithm Comparison**\n\n|      | TD3   | SAC   | PPO   | TD3 w/o state representation |\n| ---- | ----- | ----- | ----- | ---------------------------- |\n| PSNR | 38.26 | 37.10 | 36.73 | 32.17                        |\n\nAs shown above, different RL algorithms achieve comparable results on this task. However, removing the proposed state representation from the TD3 algorithm results in a significant performance drop, highlighting its importance. The results demonstrate that the state representation, not the specific choice of the RL algorithm, is the key factor behind our method's superior performance.\n\n## Q2: Additional Ablation over Greedy Algorithm\n***\nIn our paper, we have shown that our RL-based method achieves higher photo-finishing quality and efficiency than optimization-based methods. We compared our method with baselines using zeroth-order search and first-order optimization with differential proxies but did not previously compare it with a simple greedy algorithm. Here, we provide an additional ablation study using a greedy algorithm.\n\nFor the greedy algorithm, we followed Algorithm 1 from the [CRISP] paper, implementing a greedy search algorithm to find the optimal ISP parameters. This greedy algorithm mimics human behavior by incrementally adjusting each element towards the desired photo-finishing result until a stopping condition is met.\n\n> [CRISP] Learning Controllable ISP for Image Enhancement. TIP. 2023.\n\nAs shown in Algorithm 1 in the **Pseudo Code for Greedy Algorithm** section below, the greedy algorithm mimics user behavior by incrementally improving image quality to achieve the desired results.  The algorithm starts with the ISP parameters at $s_{init}$ and iteratively adjusts them with a step size $t$ until the stopping condition $K$ is reached. We set each element in $s_{init}$ to be 0, step size $t=0.1$, and iterations $K=200$ to be consistent with other optimization-based methods. We evaluated the greedy algorithm on the FiveK-Target and HDR+ datasets, with the results shown in Table 1 of the rebuttal PDF. We also present the results in the table below.\n\n| Greedy Tuning Algorithm Evaluation | PSNR  | SSIM   | LPIPS  | Queries |\n| ---------------------------------- | ----- | ------ | ------ | ------- |\n| FiveK Target Photo Finishing Task  | 26.14 | 0.9250 | 0.1380 | 200     |\n| HDR+ Target Photo Finishing Task   | 25.79 | 0.9212 | 0.1542 | 200     |\n\nFrom the results, it can be observed that while the intuitive greedy algorithm achieves reasonable performance, it performs worse than the CMAES baseline under the same iterations. This indicates that the greedy strategy is not as effective as the evolutionary strategy in the CMAES algorithm. However, the greedy algorithm performs better than proxy-based methods [26, 27] because it requires no proxy training and searches directly in the parameter space. Note that like CMAES, the greedy tuning algorithm requires at least hundreds of queries to the ISP pipeline to converge, which are very time-consuming.\n\nBoth greedy and CMAES are outperformed by our RL-based approach. This is because their searching processes are blind and brute-force, not conditioned on the input image, target image, or any prior on image processing operations. Our RL-based approach considers all these factors, requires no proxy training, and thus achieves better photo-finishing quality and efficiency.\n\n\n\n## Q3: More Evaluation Dataset\n***\nPlease refer to Q1 in the **To All** section for the answer to this question. To summarize, we conducted further evaluations using the HDR+ dataset, demonstrating our RL-based framework's ability to generalize effectively to unseen datasets. Our method achieved a PSNR of 31.54 on the HDR+ photo-finishing tuning task, comparable to the 32.47 achieved on the FiveK dataset, and outperformed all baselines. This highlights our method's generalization capability across different datasets and its superior performance compared to other baselines, including CMAES, Greedy Search, Cascaded Proxy, and Monolithic Proxy.\n\n\n\n## Pseudo Code for Greedy Algorithm\n\n***\n\n$\\textbf{Algorithm 1: Greedy Tuning Algorithm}$  \n\n***\n\n$\\textbf{Inputs: } s_{\\text{init}} \\in \\mathbb{R}^{D}, \\text{ Step size } t, \\text{ Stop condition } K, \\text{ Input image } x, \\text{ Tuning target image } \\hat{x}$  \n$\\textbf{Initialization: } e \\gets \\infty, s \\gets s_{\\text{init}}, d \\gets 1, i \\gets 0, k \\gets 0$  \n$\\textbf{While } k \\leq K \\textbf{ do}$  \n$\\quad \\textbf{if } e < \\text{MSE}(f_{\\text{PIPE}}(x, s), \\hat{x}) \\textbf{ then}$  \n$\\quad \\quad s_d \\gets s_d - t, d \\gets (d \\bmod D) + 1, i \\gets i + 1, k \\gets k + 1$  \n$\\quad \\quad \\textbf{if } i = D \\textbf{ then}$  \n$\\quad \\quad \\quad s \\gets s + t, d \\gets 1, i \\gets 0$  \n$\\quad \\quad \\textbf{end if}$  \n$\\quad \\textbf{else}$  \n$\\quad \\quad e \\gets \\text{MSE}(f_{\\text{PIPE}}(x, s), \\hat{x}), i \\gets 0, k \\gets 0, s_d \\gets s_d + t$  \n$\\quad \\textbf{end if}$  \n$\\textbf{end while}$  \n\n***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UsGXoiAWdL", "reviewer": "Reviewer_kK87", "summary": "This paper presents a method by which RL is used to drive the optimization of ISP hyperparameters for two tasks: (1) recovering photo finishing parameters and (2) mimicking reference style photo characteristics.", "questions": "Can the authors provide full details on how the baselines were implemented?  For me, acceptance is hinges on whether I can be convinced that the baselines were implemented correctly and are indeed that bad.", "limitations": "seems fine", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Clearly written.  Effective application of RL to photo finishing and stylization.  Quantitative evaluation showing substantially improved results on photo finishing in terms of quality, number of queries, and run time.  User study on photo stylization task.  Ablation study on state representations.", "weaknesses": "I'm a little skeptical of whether the baselines for [26] and [27] were implemented correctly.  What I don't really understand is that [26] does not provide code, [27] does provide code, but there's no mention as to whether they used the code from [27] and why the errors in the two methods look similar (almost all examples have a hue shift in a seemingly weird direction).  I also don't see the terms \"monolithic proxy\" or \"cascaded proxy\" in either of these two references.  I think these details at least need clarification for reproducibility.\n\nI don't really understand how CMAES was implemented.  The method is cited as being from [9][18], but then the appendix states that it was implemented from [26] using pymoo.  Some examples like the green image in Figure 4 make me skeptical that this baseline was implemented correctly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer kK87\nQ1 addresses the implementation details of three baselines. Q2 Q3 explain certain behavior (hue shift, green image) of baselines. Q4 extends extra details on baseline implementation and proves our baselines were implemented correctly. \n## Q1: Full Details on Baseline Implementations, Terms \"Cascaded & Monolithic\"\n***\nPlease refer to **Q3** in the **To All** section for how we implement each baseline, including code used and full details. Please also refer to **Q2** in the **To All** section for clarification of the terms \"cascaded & monolithic proxy\". \n\nReviewer pointed out an **incorrect citation of CMAES** in the appendix. On line 487, we make a typo on the citation of CMAES; the correct citation should be [18], not [26]. We apologize for this misunderstanding and will correct it in our paper. Since [18] does not provide code, we reproduce it using `pymoo`.\n## Q2: Similar Hue Shift for Proxy-Based Methods [26,27]\n***\nHue shift is a common artifact. As shown in Figure 2 of the PDF file, snapshots from the result figures in the [27] paper also exhibit severe hue shift. In our paper, [26] and [27] have similar hue shifts in some cases, and we believe it was a result of inherent challenges in proxy-based methods, and is due to similar errors in proxy networks. \n\nThe similar inaccuracies of proxy networks are due to the similar data bias introduced during proxy training data generation. [27] utilize a uniform strategy to generate proxy training data:\n\n>\"We uniformly sample 100 points for each slider when generating the data for each intermediate.\" --Section 5,[27]\n\nThis uniform sampling strategy introduces bias because sub-ranges of ISP parameters may not distribute evenly and often have non-linear effects. And ISP operations are black-box so the parameter range cannot be considered during data sampling. For example, the white balance parameter (RGB color channel coefficients) ranges from [0.6, 1.65], where 1 represents no gain. Uniform sampling results in more data points between [1, 1.65] than [0.6, 1], potentially leading to biased training data. Since the proxy network is purely data-driven, such data bias can lead to similar inaccuracies. As a result, both proxies tend to behave similarly, especially in complex ISP operations like color adjustments.\n\nThen, both methods use the same regression optimization approach with the Adam optimizer and identical hyperparameters, leading to similar optimization dynamics. It results in similar parameter shifts, particularly in white balance adjustments, where even slight inaccuracies can cause visible hue shifts. Importantly, these shifts do not occur in all cases, as shown in Figure 3 of our paper and Figure 1 of the rebuttal PDF. In conclusion, similar hue shifts are expected due to shared conditions and challenges in proxy-based ISP tuning.\n## Q3: Green Image in CMAES Figure 4\n***\nThe color shift in Figure 4 is an occasional failure case specific to the stylization task, not the photo finishing tuning task. It occurs because style loss is too complex for the CMAES to explore the search space, causing the optimization stuck in local minimal. In contrast, our RL-based method has superior exploration capabilities under such cases.\n## Q4: Extra Details on Each Baseline Implementation\n***\nWe extend extra details on the performance and implementation of each baseline, to prove our baselines' correctness. \n### Cascaded Proxy [27]\nBaseline [27] underperforms due to (1) our pipeline having more image processing operations, (2) the neural proxy cannot generalize well to unseen images, and (3) the unavailability of the dataset used in [27].\n\n**Accuracy Drop with More ISP Hyperparams**: As shown in the table below, proxy approximation accuracy and finishing quality of [27] decreases with more ISP operations. This is due to error accumulation in the proxy pipeline with more ISP operations, which leads to inaccurate gradient and suboptimal parameter recovery, causing the finishing quality to decrease.\n\n|Number of ISP Params|1|3|5|7|9|\n|-|-|-|-|-|-|\n|Proxy Approximation Accuracy of [27] (PSNR)|51.96|43.56|39.07|35.66|28.10|\n|Photo Finishing Tuning Quality of [27] (PSNR)|50.80|36.54|29.0|26.33|22.31|\n\nSince [27] recovers only 4 parameters while our experiment recovers 9, the finishing quality of is naturally lower in our paper. Additionally, ISP operations like sharpening are difficult for neural networks to approximate, as mentioned in [27], which further impacts performance.\n\n**Neural Proxy's Generalization**: The neural proxy learned in [27] struggles to generalize to unseen images, as shown in proxy accuracy drops on validation data:\n\n| |Train|Val|\n|-|-|-|\n|Proxy Approximation Accuracy (PSNR)|34.78|28.10|\n\nThis explains the poorer performance in our additional HDR+ experiments.\n\n**Dataset used in [27] not released**. Therefore, the numbers reported by [27] are not reproducible. We trained this baseline on the FiveK dataset, following the details in [27]. \n### Monolithic Proxy [26]\nOur implementation reported a PSNR result above 20 on the FiveK dataset, comparable to the result reported by [27] when they reproduced this baseline. Like [27], [26] struggles with complex ISP operations and generalization. The vanishing gradient issue, analyzed in [27], further affects its performance.\n### CMAES [18]\n**Reported PSNR Comparable to [27] Paper**: Our paper shows a PSNR above 28 for CMAES [18] on ISP tuning, comparable to the result reported by [27].\n\n**CMAES Performance with More Iterations**: In the table below, the performance of CMAES improves with more iterations, eventually reaching a PSNR of 32.12 on the FiveK dataset, comparable to RL's quality. While this indicates that our reproduction of the CMAES baseline is correct, the increased iterations lead to slower performance due to more queries to the ISP.\n\n|CMAES Iterations|10|100|200|500|\n|-|-|-|-|-|\n|PSNR|18.1|22.6|28.3|32.1|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ON8swgCEgl", "reviewer": "Reviewer_WirW", "summary": "This paper proposes a goal-conditioned reinforcement learning framework for photo finishing tuning. They introduce a novel state representation and treat the image processing pipeline as a black box, avoiding the need for differentiable proxies. The method can efficiently tune parameters to match various goals, including pixel-aligned target images and style images.", "questions": "Is the search-based method (CMAES) run on a single CPU core to obtain the inference time?", "limitations": "The paper's proposed method may require training a new model for each new incoming dataset. This limitation could potentially impact the framework's adaptability and efficiency when applied to diverse or frequently changing data sources.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow, clearly explaining the goal-conditioned reinforcement learning approach for photo finishing tuning.\n2. The visualization in Figure 1 effectively demonstrates the superiority of the proposed method compared to existing approaches, showing rapid convergence and high-quality results.", "weaknesses": "1. Compared to search-based methods, this proposed RL-based framework may not generalize well to unseen datasets.\n2. The efficiency comparison in Table 2 may not be entirely fair. While the CMAES method runs on CPUs as required, the paper doesn't explore potential speed-ups through multi-processing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer WirW\n\nWe sincerely thank you for reviewing our work. After reading your comments carefully, we summarize the following questions. \n\n## Q1: Cross-Dataset Generalization\n***\n\nPlease refer to Q1 in the **To All** section for a detailed answer to this question. To summarize, we conducted additional evaluations using the HDR+ dataset to demonstrate our RL-based framework's ability to generalize effectively to unseen datasets. Our method achieved a PSNR of 31.54 on the HDR+ photo-finishing tuning task, which is comparable to the 32.47 achieved on the FiveK dataset, and it outperforms all baselines.\n\nThis highlights our method's strong generalization capabilities across different datasets, outperforming other methods such as CMAES, Greedy Search, Cascaded Proxy, and Monolithic Proxy. Our approach leverages a robust state representation that captures invariant features for photo finishing, enabling it to adapt to diverse inputs and target outputs beyond the training distributions. These results affirm that our RL-based framework is not only effective in the scenarios it was trained on but also exhibits superior performance on datasets it has not previously encountered.\n\n\n## Q2: Efficiency Comparison of CMAEAS\n***\nOur implementation of CMAES takes full advantage of parallel computing capabilities on a high-performance 48-core CPU, ensuring a fair efficiency comparison for efficiency.\n\n**Parallelization Enabled by `pymoo` Library**: For the CMAES method, we implemented the baseline using the widely-used Python library `pymoo`, which supports parallelization. In our experiment, we enable the parallelization options provided by `pymoo`. This is done by setting ```classpymoo.algorithms.soo.nonconvex.cmaes.CMAES(... , parallelize=True, ...)``` and enabling `elementwise_renner` when calling the `pymoo` library. \n\n**Hardware Configuration**: Our speed testing experiments were conducted on a high-performance server-level system equipped with AMD EPYC 7402 (**48 Cores**) @ 2.8 GHz CPU, 8 NVIDIA RTX 4090 GPUs, 512 GB of RAM, and running CentOS 7.9. With this configuration, we utilized the full capabilities of the 48-core CPU to explore potential speed-ups for CMAES through multi-processing. This setup ensures that the CMAES method was not constrained by computational resources and accurately reflects the method's performance potential.\n\n**Multi-Core Execution**: To clarify, the reported inference times for the CMAES method were obtained by running the algorithm on all available 48 CPU cores, rather than a single core. By doing so, we provided a realistic comparison with other methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# To ALL

We sincerely thank all reviewers for your comments. We summarize the following questions and add more results to the rebuttal PDF file.

## **Q1. Evaluation on more datasets (Reviewer ASBs syDC WirW)**
***

We test our RL-based framework directly on an extra dataset (HDR+ dataset). The results show that our method generalizes well to unseen datasets and outperforms all baselines. On the photo finishing tuning task, our RL-based framework achieves a PSNR of 31.54, comparable to 32.47 on the FiveK dataset, and outperforms first-order and zeroth-order optimization methods by a large margin.

**Dataset:** To further evaluate our method and demonstrate its generalizability, we used the HDR+ dataset:

> [HDR+] Burst photography for high dynamic range and low-light imaging on mobile cameras. TOG. 2016.

We used the official subset of the HDR+ dataset, which consists of 153 scenes, each containing up to 10 raw photos. The aligned and merged frames are used as the input, expertly tuned images serve as the photo-finishing targets.

**Evaluation result**: To test the generalization capability of our RL-based framework, we directly tested our pre-trained model on the new HDR+ dataset. We compare to baselines including CMAES, Cascaded Proxy, Monolithic Proxy, and Greedy Search (requested by reviewer syDC). In Table 1 of the rebuttal PDF file, we report PSNR, SSIM, LPIPS, and queries to the ISP pipeline. We have also included test results from the FiveK-Target dataset in the table's left column for easy comparison.

The results demonstrate that our RL policy generalizes effectively to unseen data, achieving higher photo-finishing quality than methods directly tuned on the test dataset. Qualitative comparisons in Figure 1 of the PDF file show our results are closer to targets, even with input and target images outside the training distribution.

**Cross dataset generalization**: In Table 1 of the PDF file, our RL-based method achieves a PSNR of 31.54 on the HDR+ photo-finishing task, comparable to 32.47 on MIT-Adobe FiveK. This shows that our RL policy, trained on the FiveK dataset, effectively generalizes to the HDR+ dataset. Such out-of-distribution capability is facilitated by our proposed photo-finishing state representation, which extracts invariant features for photo finishing, allowing adaptation to diverse inputs and goals beyond the training distributions.

The CMAES baseline shows consistent results on HDR+ compared to FiveK, as it is directly optimized on the test dataset without prior training. However, proxy-based methods [26, 27] perform worse because the proxy network trained on FiveK does not generalize well to HDR+, leading to incorrect gradients and poorer photo-finishing quality.

## **Q2. Clarification for terms cascaded and monolithic proxy (Reviewer kK87)**
***

Both [26] and [27] use neural network proxies to approximate ISP pipelines. The core difference is in the architecture of the neural network proxy: [26] uses a single UNet for the proxy, while [27] uses multiple small neural networks cascaded to address the vanishing gradient problem of UNet. The term "monolithic proxy" is borrowed from [27], which refers to [26] as a monolithic proxy. We refer to [27] as a "cascaded proxy" to differentiate their approaches.


## **Q3. More details on the implementation of baselines (Reviewer ASBs kK87)**
***

We provide details on the code used and implementation for each baseline. To summarize, we used as much of the provided code as possible and followed every publicly available detail.

### 3.1 Implementation details for cascaded proxy [27]

**Code used:** Although [27] provides code, it is incomplete and cannot be run directly. For instance, only the UNet architecture from [26] is specified in the code, not the cascaded network proposed in [27]. There are also known bugs in the GitHub repository (in GitHub issue #2) that prevent the pipeline from running. We used as much of the provided code as possible and reproduced other parts by following detailed descriptions in [27], as shown below.

**Proxy network training:** With no complete code available, we strictly followed the architecture in Tables 1 and 2 of [27] appendix, using 3 1x1 convolutions for pointwise ISP operations and 5 3x3 convolutions for areawise operations. We trained with the Adam optimizer, a learning rate of 1e-4, a batch size of 512, and 100 epochs, as recommended. Camera metadata was extracted from DNG files, as described in [27].

**Training dataset:** Since [27] does not release its dataset, we used the MIT-Adobe FiveK dataset, as in our method. Following Section 5 of [27], we used 1,000 raw images from FiveK and sampled 100 points for each ISP hyperparameter.

**ISP hyperparameter optimization:** After training the proxy neural network, we used it as a differential proxy to approximate the ISP pipeline. As in [27], we use the gradient flow through the proxy to directly optimize ISP hyperparameters, following [27]'s optimization details with the same Adam optimizer, same loss, same learning rate, and same iterations.

### 3.2 Implementation details for monolithic proxy [26]

**Code used:** [26] does not provide code, but [27] includes code for the UNet proxy used in [26]. We used this code for reproduction.

**Proxy network training:** The architecture uses a single UNet to approximate the ISP pipeline, with hyperparameters conditioned by concatenating extra planes to the features. We trained the proxy with the Adam optimizer, a learning rate of 1e-4, a batch size of 512, and 100 epochs.

The training set generation and first-order optimization method are the same as for [27].

### 3.3 Implementation details for CMAES [18]

CMAES [9,18] is a zeroth order optimization method without a neural proxy. As [18] does not provide code, we implemented CMAES using the `pymoo` library, enabling parallel execution on multi-core CPUs and achieving reasonable performance. This baseline does not require training.

Meta-review: After a robust rebuttal period, the paper has been assigned the following ratings: weak accept, borderline accept (2x), and borderline reject. Two reviewers increased their ratings during the rebuttal process. The main lingering concerns from the borderline reject review relate to the motivation/justification for the selected RL algorithm and the value of including additional evaluation with more random seeds. The ACs agree that addressing these issues would strengthen the paper but do not believe they are sufficient to warrant rejection given the other strengths, including the magnitude of improvement over other approaches. The authors are encouraged to take the reviews and the post-rebuttal discussion into consideration as they prepare the camera-ready copy.
Decision: Accept (poster)