Goal Conditioned Reinforcement Learning for Photo Finishing Tuning

Authors: jiarui  wu
yujin  wang
lingen  li
fan  zhang
tianfan  xue
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 16:39:06

Abstract: Photo finishing tuning aims to automate the manual tuning process of the photo finishing pipeline, like Adobe Lightroom or Darktable. Previous works either use zeroth-order optimization, which is slow when the set of parameters increases, or rely on a differentiable proxy of the target finishing pipeline, which is hard to train.
To overcome these challenges, we propose a novel goal-conditioned reinforcement learning framework for efficiently tuning parameters using a goal image as a condition. Unlike previous approaches, our tuning framework does not rely on any proxy and treats the photo finishing pipeline as a black box. Utilizing a trained reinforcement learning policy, it can efficiently find the desired set of parameters within just 10 queries, while optimization based approaches normally take 200 queries. Furthermore, our architecture utilizes a goal image to guide the iterative tuning of pipeline parameters, allowing for flexible conditioning on pixel-aligned target images, style images, or any other visually representable goals. We conduct detailed experiments on photo finishing tuning and photo stylization tuning tasks, demonstrating the advantages of our method.

Main paper: 
 introduction: 
 Image processing pipelines (ISPs) are widely used by photographers and artists to retouch images to match their desired appearance. Existing pipelines like Adobe Lightroom and Darktable allow users to interactively tweak meaningful sliders such as exposure, white balance, and contrast, which control the pipeline to perform a series of non-destructive edits to the input image. Though users can manually tune the slider parameters, it is laborious and time-consuming even for experienced experts. To this end, automatic photo finishing algorithms have been introduced to automate the process and have drawn growing attention in the community [11] .In this work, we aim to design an automatic tuning algorithm for black-box non-differentiable image processing pipelines. Although some preliminary research tries to propose fully differentiable image processing pipelines [11, 7] or approximate the existing pipelines using neural networks [29] , they only support a limited set of image processing operations. On the other side, most commercial image processing pipelines, like Adobe Lightroom, are still black-box and non-differentiable, with a set of tunable parameters exposed to users. Under this setup, the goal of pipeline tuning is to automatically find the set of optimal parameters to achieve a desired image appearance, named the tuning target. More specifically, in this work, we study two different tuning targets. One is a target image with the same content as the input, but with a different rendering style, and we call it photo finishing tuning. The other is a target style image with different content, and the algorithm is to render the target in a similar way as the style target, and we call that photo stylization tuning. In this work, we propose an RL-based photo finishing tuning algorithm that efficiently tunes the parameters of a black-box image processing pipeline to match any tuning target. The RL-based solution (top row) takes only about 10 iterations to achieve a similar PSNR as the 500-iteration output of a zeroth-order algorithm (bottom row). Our method demonstrates fast convergence, high quality, and no need for a proxy.One previous solution for photo finishing tuning is to use zeroth-order or first-order optimization. However, both of them are either time-consuming or limited to a small set of parameters. Zeroth-order optimizations [9, 21, 20, 4] are gradient-free searching methods and thus are normally very slow when the search space increases. First-order optimizations [28, 32, 29] accelerate the searching process using gradient descent, but they either require the processing pipeline itself to be differentiable, or a neural proxy is pre-trained to find a differentiable proxy of the original pipeline. For a complex commercial imaging pipeline, like cellphone camera pipelines, this proxy may not fully reproduce original pipelines [28] . Neural photo-finishing [29] improves the proxy accuracy by breaking a complex pipeline into small modules, but this does not apply to black-box or dynamically reconfigurable pipelines [32] . Considering all these limitations, this brings a challenging question: Is there an efficient parameter-searching algorithm that is applicable to non-differentiable finishing pipelines?To solve this challenge, we propose a novel goal-conditioned reinforcement learning (RL) approach dedicated to photo finishing tuning. At each RL iteration, the policy network takes the tuning target and the currently tuned image as input, and finds a better set of parameters that makes the finishing results closer to the target. This RL-based searching algorithm has several advantages over traditional optimization methods. Compared with the zeroth-order solution, the RL policy can more accurately predict the potential searching direction, while zeroth-order searching can only rely on less effective tries. As a result, RL searching is much more efficient. As shown in Fig. 1 , the RL-based solution (top row) only takes about 10 iterations to reach a similar PSNR as the 500-iteration output of a zeroth-order algorithm (bottom row). Also, compared with first-order optimization, RL-based tuning directly optimizes the non-differentiable image processing pipeline without a differential proxy. Therefore, it is not limited by the variety and complexity of image processing operations and pipelines, achieving much better tuning results. As shown in Fig. 1 , RL-based tuning reaches 38.92dB (top row), while the first-order solution only obtains 18.69dB (middle row).To train an efficient RL policy for tuning, we also propose a novel state representation dedicated to this task. The state representation should model the relationship between the photo editing space and our policy. To achieve that, our state representation consists of three key components: a CNN-based feature representation to encode global and local features, a photo statistics representation to match the photographic statistics between the input and the goal, and an embedding of historical actions. These representations better fit the RL policy into our task and guide the policy to generate the next set of parameters effectively. Lastly, we also design reward functions for both photo finishing tuning and photo stylization tuning, enabling our framework to tune photos to different targets.We validate the effectiveness of our framework with extensive experiments on both photo finishing tuning and photo stylization tuning. Our RL-based framework significantly outperforms previous methods in both tasks in terms of both efficiency and image quality. Experimental results demonstrate that our goal-conditioned policy is an efficient photo finishing tuner capable of performing finegrained control on image processing pipeline parameters to achieve various goals. 
related work: 
 ISP tuning. Recent developments in end-to-end AI-based ISP pipelines show potential as alternatives to traditional mobile ISPs [13, 14, 25 ]. Yet, traditional parametric ISPs remain preferred in consumer cameras for their controllability, efficiency, and interpretability. These systems require expert-driven, labor-intensive tuning to enhance image quality and achieve desired aesthetic effects. Efforts to streamline this process are ongoing, aiming to reduce the need for manual adjustments. Gradientbased optimization is a notable strategy in this area. Tseng et al. [28] have applied differentiable black-box proxies to simplify ISP tuning. Further research by Tseng et al. [29] has opened up the ISP pipeline into white-box manageable modules, learning differentiable proxies for each module and subsequently integrating them, which improves the tuning performance. Additionally, Qin et al. [23] introduced an attention-based CNN approach for scene-aware ISP tuning. However, this method lacks integration of sequence-specific prior knowledge. They later developed a framework for predicting ISP hyper-parameters sequentially [24] , optimizing parameters based on their relationships and similarities. In a different approach, Mosleh et al. [20] utilized a genetic evolutionary algorithm with a zero-order stochastic solver [9] to directly optimize hardware-specific image processing pipelines, circumventing the constraints of gradient-based methods. Moreover, Nishimura et al. [21] explored derivative-free optimization, employing nonlinear techniques and automatic reference generation for effective automation of image quality adjustments. Despite these advancements, the complexity of the image processing pipeline and the vast parameter space continue to challenge the efficiency of tuning methods.Photo stylization. Automating image stylization, which is straightforward for humans, poses challenges for machines. Significant research has been conducted to bridge this gap. Karras et al. [15] pioneered StyleGAN, a network manipulating the latent space to control image styles at various scales. Building on this, Brooks et al. [2] developed InstructPix2Pix, which allows users to guide the stylization process through textual instructions, enhancing user-machine interaction. However, these methods often lack explainability and may alter the original image content. Further exploring transparency, Hu et al. [11] proposed a reinforcement learning-based white-box photo-finisher, though its need for explicit gradients limits compatibility with traditional systems. Kosugi et al. [18] addressed style diversity using unpaired data based on reinforcement learning. Despite these advances, these methods are restricted to a single style during training, limiting the user's ability to control the pipeline to produce images of any desired style during inference. Moreover, Tseng et al. [29] optimized proxy networks for image processing modules via a style loss function, achieving promising results but facing limitations in handling complex modules and style variability during inference.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Limitations are discussed in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem definition: 
 Throughout this paper, we aim to tackle the photo finishing tuning problem. Given an input image I 0 , an image processing pipeline f PIPE maps the input to a finished image I FINISHED = f PIPE (I 0 , P ), controlled by a set of parameters P . Our task is to solve the inverse problem: given an input image I 0 and a tuning target I g (the goal condition in the RL framework), how to find the parameters that reach this target:EQUATIONThe goal image I g can vary between different tasks. For photo finishing tuning, the goal image shares the same content as the input, and the tuning target is to minimize the distance between the pipeline output and the goal image I g . For photo stylization tuning, the goal image is a style target with different content than the input, and the tuning target is to generate an output that matches the style of the goal image. Note that unlike artistic style transfer [6] , we focus on photorealistic style transfer [31] , where the processing pipeline does not change the content of an image.We are the first to introduce a reinforcement learning (RL) approach to the photo finishing tuning task by formulating it as an end-to-end policy learning problem. Inspired by human experts who use iterative trial and error in photo finishing, our method models the task as a decision process with multi-step feedback, akin to a Markov Decision Process. As shown in the bottom row of Fig. 2 , our policy iteratively tunes the parameters of a given image processing pipeline to match a goal image. At each step, it takes the currently retouched image and the goal image as inputs, and then outputs the next action. This RL-based framework efficiently predicts pipeline parameters, requiring only minimal iterations (e.g., 10 queries to the image processing pipeline). Additionally, since RL optimization bypasses the need for gradient flow from the image processing pipeline, our system can handle any black-box pipeline, regardless of complexity.The RL process is formally defined as follows. Let S be the state space, O be the observation space, A be the action space, T be the transition function, R be the reward function, G be the goal distribution, ρ 0 be the initial state distribution, and γ the discount factor. All these forms a Goal-conditioned Partially-Observed Markov Decision Process (S, O, A, T , R, G, ρ 0 , γ) [19] .In each tuning episode t, the agent is given a goal image I g ∈ G, as well as an observation o t ∈ O consists of the retouched image I t at step t along with all historical actions and observations. The action a t is the parameter set P used by image processing pipeline to generate the output image at the next step. And the transition function T : S × A → S is the image processing pipeline f PIPE defined in Sec. 3.1. The reward function is R(s, I g ) for s ∈ S and I g ∈ G. We aim to learn a goal-conditioned policy π(a|o, I g ) : S × G → A that maps from observation o and goal I g to the next action a, maximizing the sum of discounted rewards E s0∼ρ0,Ig∼G t γ t R(s t , I g ). Our policy π(a|o, I g ) is a deterministic policy µ θ parameterized by θ, outputting continuous actions a t = µ θ (o t , I g ).State presentation is also critical for the success of the proposed RL policy. This is particularly important in challenging scenarios where the policy must tune parameters for unseen goals of any style. Our experiment in Sec. 4.3 also shows that a simple concatenation of the input and goal images yields a sub-optimal result. Therefore, we design a comprehensive photo finishing state representation to extract features from observations that are critical for photo finishing tuning. Specifically, our representation consists of three components: a CNN-based dual-path feature representation to encode both global and local features, a photo statistics representation to match the traditional photographic statistics between input and goal images, and an embedding of historical actions. Details of each component are described below.Dual-path feature representation. In our dual-path feature representation, we seek to extract both global and local features from both input and goal images, inspired by [8] . This is because the tuning task requires not only global image characteristics such as overall color, tone, average intensity, and scene category, but also local features associated with texture, highlight, and shadow. As shown in Fig. 2 , the architecture begins with a stride-2 convolutional encoder to reduce spatial resolution and extract initial low-level features. It then splits into two paths: a local path and a global path. The local path {L i } i=1...N L includes two stride-1 convolutional layers, preserving spatial resolution to extract local features. The global path {G i } i=1...N G has one stride-2 convolutional layer and three fully-connected layers, providing a global scene summary vector. This global feature encapsulates essential global image characteristics such as overall color and tone, as well as a global notion of scene category (light condition or indoor/outdoor). We fuse the global and local features by adding the global feature at each x, y spatial location of the local feature:F D x,y = σ(G N G + L N L x,y), where σ is the ReLU activation. This results in a dual-path feature representation F D . Photo statistics representation. Simply relying on a CNN-based policy may lead to unsatisfactory results with input and goal images outside of training distribution. To better represent invariant features across diverse styles and content of both input and goal images, we introduce a photo statistics representation, which matches traditional image statistics such as a histogram between input and goal. Global photo statistics, such as histograms, are critical in global image processing operations, such as exposure control [22] , highlight, and shadow. Since they cannot be well represented by conventional convolutional neural networks due to limited receptive fields [29] , we propose to precompute these statistics and concatenate them into our state representation. Specifically, we compute histograms H rgb on the RGB channels of input and goal images and map these to a fixed dimension using a linear layer H ′ = Linear((H rgb (I t ); H rgb (I g )). This feature is then concatenated with the luminance, median, contrast, and saturation of both input and goal images to form the photo statistics representation F S .Policy network. At last, we combine the dual-path feature representation and photo statistics representation with a historical action embedding F H t = Linear(a 1:t ; ℓ 1:t ), where ℓ t is the ℓ 2distance of image I t and goal image I g . As shown in Fig. 2 , the input of policy network can be formulated ass t = Concat(F D t ; F S t ; F H t ).The policy network is a multi-layer perceptron network (MLP) that maps from the current state and the goal representation to the next action to take. We choose deterministic policy to directly output continuous action a t = µ θ (o t , g) = µ θ (s t ). We use the same architecture to estimate the value function for RL updates.We provide a general RL-based framework for photo tuning tasks. One can train our goal-conditioned end-to-end policy with different reward functions to resolve different photo tuning tasks including photo finishing tuning and photo stylization tuning. The policy is optimized with twin-delayed DDPG (TD3) algorithm [5] .Reward function for photo finishing tuning. When goal image I g is the photo-finished input image, we measure the distance between current image I t and I g with PSNR metric. The reward function is calculated as the difference between PSNR values of consecutive steps:r t = PSNR(I t+1 , I g ) -PSNR(I t , I g ).(2) Instead of using ℓ 2 -distance to measure image distance, we use PSNR, the negative logarithm of ℓ 2 -distance. This design ensures the policy receives appropriate rewards even when the current image is close to the goal, encouraging fine-grained tuning of pipeline parameters.Reward function for photo stylization tuning. When goal image I g is a style image with arbitrary content and style, we measure the distance between the input and goal images with a style score:EQUATIONwhere N S denotes the number of layers from a pre-trained VGG-19 [27] model used to extract features. Following [6] , the style is captured using Gram matricesG i [•] = F i [•]F i [•]T , with F i representing feature maps. Additionally, the ℓ 2 -distances of histograms for the Y and UV channels are included to align luminance and color palettes. The overall reward is calculated by the change in style score across consecutive steps, penalized by the difference in content features F i :EQUATION)Policy optimization. We optimize our goal-conditioned policy with off-policy TD3 [5] algorithm. Specifically, TD3 learns two Q value function Q ϕ1 and Q ϕ2 , optimized by mean square Bellman error minimization:EQUATIONEQUATIONwhere Q ϕi,targ is the exponential moving average of Q ϕi , a ′ (s t+1 ) is given by target policy with clipped gaussian noise, and d is the termination signal. The state transition pair (s t , a t , r t , s t+1 , d) is sampled from a replay buffer D. With Q functions, the policy µ θ (•) is learned by maximizing Q ϕ1 :EQUATIONMore details about our reward functions and policy optimization are in the appendix.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [No] Justification: Codes and datasets will be made publicly available upon acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiments: 
 In the first subsection, we provide the details of datasets and task settings for photo finishing tuning and photo stylization tuning, along with a description of the evaluation metrics. In the second subsection, we demonstrate our experimental results and compare them to zeroth-order optimization [20] and first-order optimization [28, 29] baseline methods. Ablation studies are conducted in the last subsection, which investigates the impact of each component of our state representation. We also provide supplementary qualitative results in the Appendix.Datasets. We use the MIT-Adobe FiveK Dataset [3] , a renowned resource in the field of photo retouching, which comprises 5,000 photographs captured using DSLR cameras by various photographers. This dataset is notable for providing images in raw format alongside the retouching outcomes of five experts. For our study, we selected 4,500 images to serve as the training dataset, with the remaining 500 images designated as the validation dataset. In our method, random parameters are employed to generate the target images, which are used as training data pairs. In the task of photo finishing tuning, the datasets including both the expert C retouched targets and randomly generated targets are utilized for evaluation. For the photo stylization tuning task, we have curated a collection of 200 diverse style images from the Lightroom Discover website 3 , following [26] .To further evaluate our method and demonstrate its generalizability, we test our RL-based framework directly on the HDR+ dataset [10] . We used the official subset of the HDR+ dataset, which consists of 153 scenes, each containing up to 10 raw photos. The aligned and merged frames are used as the input, expertly tuned images serve as the photo-finishing targets.Implementation details. We conduct all experiments on an image processing pipeline consisting of standard image processing operations, including exposure, color balance, saturation, contrast, tone mapping (highlight and shadow), and texture (sharpness and smoothing), with nine adjustable parameters in total, similar to [29] . Thus, the agent's action space is comprised of fine continuous actions corresponding to these nine pipeline parameters. During the policy inference, the input and goal images are resized to the resolution of 128 × 128. Additionally, a 3-level Laplacian pyramid of both input and goal images is constructed and fed into the policy network to capture high-frequency details from the original resolution. Then the output parameters from the policy network are fed to the image processing pipeline along with full-resolution images to produce high-resolution results. We train our policy using the standard TD3 algorithm [5] and set the termination of our RL policy to trigger when the episode length reaches the maximum threshold (10 steps) or the PSNR metric reaches a threshold (60 dB), ensuring efficiency. Further details can be found in the appendix.Evaluate metrics. Similar to [11, 28, 29] , we employ the Peak Signal-to-Noise Ratio (PSNR), the Structural Similarity Index Measure (SSIM) [30] , and the Learned Perceptual Image Patch Similarity (LPIPS) [33] as our evaluation metrics. To assess the quality of stylization, we conduct user studies to evaluate our methods, offering a subjective measure of image quality based on viewer assessments.Results of photo finishing tuning. To evaluate the efficacy of our framework on the photo finishing tuning Task, we conduct two experiments on the FiveK validation datasets, using both expert-C targets and random targets. Our method is compared against the monolithic proxy-based approach [28] , the cascaded proxy-based method [29] , and the search-based method proposed [20] . Consistent with [29] , we utilize 100 iterations during inference for the proxy-based methods. Additionally, we record the number of times that the photo-finishing pipeline was queried in both search-based methods and our approach.As illustrated in Tab. 1, our method outperforms the others across all metrics on both FiveK-Target and Random-Target. The monolithic proxy-based method struggles with accurately representing the complex image processing pipeline, leading to suboptimal performance. While the cascaded proxy method can incrementally enhance tuning performance over its monolithic counterpart, it suffers from accumulated errors and difficulties in approximating certain operations, such as texture, resulting in poorer performance. Notably, our method only requires querying 10 times image processing pipeline, whereas the performance of the search-based method, even with 200 queries, remained inferior to ours. Further, the visualization results depicted in Fig. 4 demonstrate that our method produces visual Efficiency. To evaluate the efficiency of our approach, we conducted speed testing experiments on a system equipped with an AMD EPYC 7402 (48C) @ 2.8 GHz CPU, 8 NVIDIA RTX 4090 GPUs with 24GB of RAM each, 512 GB of memory, and running CentOS 7.9. In line with [29] , we applied 200 iterations during inference for both the proxy-based methods [28, 29] and the search-based method [20] . We measured the execution time for each method across four different input resolutions.As indicated in Tab. 2, our method demonstrated superior efficiency, requiring only 1.23 seconds per execution with 4K input. While the monolithic proxy-based method outperformed the search-based method in terms of speed, it faced limitations as input resolutions increased, leading to out-of-memory (OOM) errors once GPU memory was exceeded. The cascaded proxy-based method, which includes MLP networks, was the slowest and most prone to memory overflows due to its intensive memory demands. The search-based method primarily depends on CPU performance. Despite being executed on a high-performance server, the CMAES method [20] requires 144 seconds to process a 4K input image, which is considerably slow.Results of photo stylization tuning. We conduct qualitative comparisons and user studies on the photo stylization tuning task to demonstrate the effectiveness of our goal-conditioned RL framework.Our policy was trained using the FiveK training dataset, and all methods were evaluated on input and style image pairs collected from the Adobe Lightroom Discover website. For all baseline methods, we adopt the same style score described Sec.3.4 as optimization objectives, ensuring fair comparison.As shown in Fig. 4 , our method produces results closer to the style goal image compared to the baseline. Notably, our method achieves better results with only 10 queries to the image processing pipeline, whereas the baseline methods require 200 queries, making them orders of magnitude slower.It is important to note that the style images from the Lightroom Discover website have a different distribution than our training dataset. Despite this, our method adapts directly to the target style image distribution during testing, whereas the baseline methods require time-consuming optimization of testing data. These experimental results demonstrate that our RL-based framework can efficiently tune images to different unseen styles, showcasing that our photo finishing state representation (Sec. 3.3) To rigorously evaluate the effectiveness of our method in photo stylization tuning, we implemented a subjective user study comprising 20 questions. In each question, participants were presented with images generated by four different methods-monolithic proxy, cascaded proxy, CMAES, and our own approach. Participants were asked to identify up to two images that most closely resembled a given target image. The study was conducted online, garnering 65 responses from a diverse group of individuals selected randomly from the internet.The aggregated preferences are visually summarized in Fig. 6 . The data clearly show that our method is perceived by the majority of participants as producing results that most closely match the target images, highlighting its superiority in stylization tuning tasks.Cross dataset generalization. We conducted additional evaluations using the HDR+ dataset [10] to demonstrate our RL-based framework's ability to generalize effectively to unseen datasets. We compare to baselines including CMAES [9, 20] , Cascaded Proxy [29] , Monolithic Proxy [28] , and Greedy Search [16] . In Tab. 3, we report PSNR, SSIM, LPIPS, and queries to the ISP pipeline.The results demonstrate that our RL policy generalizes effectively to unseen data, achieving higher photo-finishing quality than methods directly tuned on the test dataset. Qualitative comparisons in Fig. 5 show that our results are closer to targets, even with input and target images outside the training distribution.In Tab. 3, our RL-based method achieves a PSNR of 31.54 on the HDR+ photo-finishing task, and it outperforms all baselines. This shows that our RL policy, trained on the FiveK dataset, effectively generalizes to the HDR+ dataset. Such out-of-distribution capability is facilitated by our proposed photo-finishing state representation, which extracts invariant features for photo finishing, allowing [28, 29] perform worse because the proxy network trained on FiveK does not generalize well to HDR+, leading to incorrect gradients and poorer photo-finishing quality.As has been shown in our main experiment, our RL-based approach significantly outperforms the previous method in terms of photo finishing quality and efficiency, demonstrating that RL is more suitable for the photo finishing tuning task. In this subsection, we focus on the photo finishing state representation we propose to better fit RL in our task. Specifically, we conduct experiments on photo finishing tuning task using FiveK Random-Target dataset, in order to study the contribution of each specific representation in our photo finishing state representation.As shown in Tab. 4, we set our baseline RL as RL policy trained with a naive CNN-based encoder taking the concatenated input and goal images as input. This baseline achieves only 32.17 dB of PSNR. In Ex 2 , our dual-path feature representation F D improves the photo finishing quality, as it better encodes local features and global image characteristics critical to our photo tuning task. Since traditional photo statistics contain invariant features about global statistics that are difficult to learn solely using a network, the photo statistics representation F S also helps to boost photo finishing quality as shown in Ex 3 . Moreover, as parameters and results from all previous RL steps help in the decision process, the historical action representation is also useful as shown in Ex 4 . With the proposed three representations combined, our RL policy can be guided to effectively tune image processing pipeline parameters given input and goal images of any photo finishing style, as evidenced by the superior performance in Ex 5 . 
conclusion: 
 In this work, we propose an RL-based photo finishing tuning algorithm that efficiently tunes the parameters of a black-box image processing pipeline to match any tuning target. Our approach encompasses several key innovations. Firstly, we integrate goal-conditioned RL into the realm of photo-finishing tuning. Secondly, we propose a photo finishing state representation comprising three principal components essential for training an effective RL policy network: a CNN-based feature representation that encodes both global and local image features, a photo statistics representation designed to align the photographic statistics between the input and the target, and an embedding of historical actions. We assess the effectiveness of our framework through comprehensive experiments on both photo finishing tuning and photo stylization tuning. The experimental results affirm that our goal-conditioned policy is an adept photo-finishing tuner, capable of exerting efficient and finegrained control over image processing pipeline parameters to fulfill a variety of objectives. Currently, our method exclusively supports conditional inputs in the form of images and lacks the capability to process non-image types such as textual inputs. Moving forward, we intend to broaden our research to include multi-modal conditional inputs. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: This follows the convention in photo-finishing tuning research, the same as previous works.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: We strictly adhere to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited all original papers and make sure that our usage is legal.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We show more photo finishing tuning visualization results on the FiveK-Target evaluation dataset in Fig. 7 , showcasing its superior performance in terms of color and brightness compared to all other methods.Cascaded Proxy Ours CMAES Target We show more photo stylization tuning visualization results in Fig. 8 , showcasing its superior performance in tuning input images to any style goal that is outside the training goal distribution.The detailed image processing pipeline in this paper encompasses various standard operations, including:(1) Exposure: Adjusts the overall brightness of the image, primarily affecting the amount of light captured. Increasing exposure makes the image brighter while decreasing it makes the image darker. This is useful for correcting overexposed or underexposed photos. This operation includes one slider parameter. (2) Color Balance: Used to adjust the color temperature and tint of an image, altering its overall color feel. Adjusting the color temperature (blue to yellow) and tint (green to magenta) can make the image appear warmer or cooler. This operation includes three slider parameters. (3) Saturation: Controls the intensity of colors in the image. Increasing saturation makes the colors more vivid and lively, whereas decreasing saturation reduces color intensity, making the image appear softer or closer to black and white. This operation includes one slider parameter. (4) Contrast: Adjusts the difference between the brightest and darkest parts of the image.Increasing contrast makes dark areas darker and bright areas brighter, enhancing the depth and dimension of the image. Decreasing contrast brings these areas closer to mid-grey, reducing the image's depth. This operation includes one slider parameter. ( 5) Tone Mapping: This operation compresses a high dynamic range input to a smaller dynamic range, which is affected by the Highlights and Shadows sliders, respectively. Highlights: Adjusts the brightness of the brightest areas of the image without affecting overall exposure. This helps recover details in overexposed areas. Shadows: Adjusts the brightness of the darkest areas, helping reveal details in shadowed regions without changing the overall exposure. This operation includes two slider parameters. (6) Texture: Enhances or reduces the detail and texture in the image without affecting colors.Enhancing texture can make details in the image clearer, while reducing texture can smooth out the image, often used in portrait photos for skin treatment. This operation includes one slider parameter.We state the details of TD3 [5] algorithm training in this subsection. TD3 is an off-policy RL algorithm. When the policy sample transition pair (s t , a t , r t , s t+1 , d) to form the replay buffer D, a gaussian noise is added to encourage exploration: a = clip (µ θ (s) + ϵ, a Low , a High ), where ϵ ∼ N (0, σ). We set σ = 0.1 for a trade-off between exploration and exploitation.To compute the target action in Q update target:EQUATIONThe next action of s t+1 comes from the target policy with a clipped noise, so that incorrect action peak produced by sub-optimal Q value estimation is smoothed out, as in the following equation:EQUATIONIn this implementation, we set ϵ 1 = 0.2, which is twice the value of ϵ. TD3 updates the policy less frequently than the Q-function. We set the policy to update half as frequently as the Q network update. The optimization objectives of the policy are already given in the main paper. To ensure a robust update of the policy and Q network, TD3 adopts an Exponential Moving Average (EMA) strategy to update the target network Q and policy network which is used to compute the Q value target above.EQUATIONEQUATIONWe set the EMA update rate ρ = 0.99 in all our experiments. The optimizer is performed using Adam optimizer [17] with (β 1 , β 2 ) = (0.9, 0.999). We set the learning rate to specific values for policy and value network, that is, 1e-4 for action and 2e-4 for Q network. We set batch size as 64 for both photo finishing tuning and photo stylization tuning experiments. We set the discount factor γ = 0.9. All experiments are conducted on NVIDIA RTX 4090 GPUs. Furthermore, we implement the CMAES method [28] based on open-source framework [1] .Reward design. For our detailed reward design, as described in Sec. 3.4, we utilize the PSNR metric of consecutive steps for photo finishing tuning. In photo stylization tuning, we adopt the style score as the main reward and add a content negative reward to prevent the policy from taking drastic actions that hinder photo stylization quality. Specifically, we set λ 0 and λ 1 in StyleScore t to 100 and 50, respectively, and set λ 2 = 0.5. For the VGG features selected to compute the style score, we follow [12] to set N S = N C = 4, selecting relu_{1...4}_1 features to form {F i } i=1...4 , which is then used to compute then gram matrix and the content regularization term.RL termination. Since our RL policy freely explores the image processing pipeline parameter space, we need to terminate the current episode if the state collapses, meaning the policy outputs parameters that render an abnormal image. Specifically, we calculate the average pixel intensity of the image I t as I t and set this value to be within the range of I min and I max . For each rollout, if I t is out of range, we terminate the state and do not save it to our replay buffer D.Network architecture. Our approach learns an end-to-end policy that maps from input images and goal specifications to the next action to take (the next parameters of the image processing pipeline.) The state representation is s t as described in Sec. 3.3, which is fed into a 4-layer multi-layer perceptron network (MLP), with a continuous action space of 9 parameters. Each hidden layer of the MLP network is of width 512.Implementation details for CMAES [9, 20] . CMAES is a gradient-free search (zeroth order optimization) method using an evolution strategy. As [20] does not provide code, we implemented CMAES using the pymoo library [1] , enabling parallel execution on multi-core CPUs and achieving reasonable performance. This baseline does not require training.Implementation details for Cascaded Proxy [29] . For proxy network training, we followed the architecture in [29] , using 3 consecutive 1 × 1 convolutions for pointwise ISP operations and 5 consecutive 3 × 3 convolutions for areawise operations. We trained with the Adam optimizer, a learning rate of 1e-4, a batch size of 512, and 100 epochs, as recommended. Camera metadata was extracted from DNG files, as described in [29] . Since [29] does not release its dataset, we used the MIT-Adobe FiveK dataset, as in our method. Following Section 5 of [29] , we used 1,000 raw images from FiveK and sampled 100 points for each ISP hyperparameter.Implementation details for Monolithic Proxy [28] . The architecture uses a single UNet to approximate the ISP pipeline, with hyperparameters conditioned by concatenating extra planes to the features. We trained the proxy with the Adam optimizer, a learning rate of 1e-4, a batch size of 512, and 100 epochs. The training set generation and first-order optimization method are the same as for [29] .We conduct a human subject study on the user preference among results of methods for photo stylization tuning given target style images. In this appendix section, we provide the original questions that we used to ask for users' feedback.For each of the 20 questions in our user study, the order of the methods was randomized and labeled as A, B, C, and D to ensure the anonymity of the techniques used. Participants were not aware of which method corresponded to each label, eliminating any potential bias in their selections. The prompt for all questions remained consistent: "Among the four images on the right (labeled A, B, C, and D), which one(s) most closely resembles the target image on the left? You may select up to two images if you believe they are equally similar to the target image. Otherwise, please choose only one."Detailed below are the options provided for each of the 20 questions. Justification: The abstract and introduction accurately reflect the contributions and scope.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: There is no direct negative societal impact for photo-finishing tuning.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).https://lightroom.adobe.com/learn/discover 


Decision: After a robust rebuttal period, the paper has been assigned the following ratings: weak accept, borderline accept (2x), and borderline reject. Two reviewers increased their ratings during the rebuttal process. The main lingering concerns from the borderline reject review relate to the motivation/justification for the selected RL algorithm and the value of including additional evaluation with more random seeds. The ACs agree that addressing these issues would strengthen the paper but do not believe they are sufficient to warrant rejection given the other strengths, including the magnitude of improvement over other approaches. The authors are encouraged to take the reviews and the post-rebuttal discussion into consideration as they prepare the camera-ready copy.

Reviews: review: paper_summary: paper_summary: This paper applies goal-conditioned reinforcement learning (RL) to photo finishing tuning. With only 10 queries, it demonstrates that goal-conditioned RL can achieve better performance than zeroth-order approaches that require 500 queries. Additionally, this method is non-differentiable, which might make it more suitable for commercial scenarios. Experiments show that this work can achieve better performance than previous efforts.
strength_weakness: strength_weakness: - Some details are missing: The methods used in this work were trained on the training set, but it is unknown whether the compared methods also underwent the training stage.
- More evaluation on additional datasets is preferred.
questions: questions: Please refer to the weakness section.
limitations: limitations: Please refer to the weakness section.
paper_summary: paper_summary: Proposed Goal-Conditioned Reinforcement Learning for Photo Finishing Tuning. Specifically, the authors introduce a novel goal-conditioned reinforcement learning framework for parameter tuning in photo processing pipelines. Unlike existing methods, the proposed approach operates without relying on proxies and treats the pipeline as a black box. By leveraging a trained RL policy, it efficiently identifies optimal parameters in just 10 queries, contrasting with 500 queries typically required by zeroth-order methods. The proposed framework utilizes a goal image to guide iterative parameter tuning, allowing adaptation to diverse target images and styles. Experiments on photo finishing and stylization tasks validate the effectiveness and versatility of the proposed approach.
strength_weakness: strength_weakness: The motivations behind some algorithmic choices are not clear. 

Additional ablation studies would enhance the persuasiveness of the findings.

It would be interesting to see the performance of the proposed method on other datasets.
questions: questions: (1) Why TD3? why not any other off-policy RL algorithm? Are there any specific reasons behind this choice?

(2) Enhancing the paper with more ablation studies would elevate its quality. For instance, exploring the impact of RL versus a greedy algorithm on performance would provide valuable insights.
limitations: limitations: Yes, the authors adequately addressed the limitations.
paper_summary: paper_summary: This paper presents a method by which RL is used to drive the optimization of ISP hyperparameters for two tasks: (1) recovering photo finishing parameters and (2) mimicking reference style photo characteristics.
strength_weakness: strength_weakness: I'm a little skeptical of whether the baselines for [26] and [27] were implemented correctly.  What I don't really understand is that [26] does not provide code, [27] does provide code, but there's no mention as to whether they used the code from [27] and why the errors in the two methods look similar (almost all examples have a hue shift in a seemingly weird direction).  I also don't see the terms "monolithic proxy" or "cascaded proxy" in either of these two references.  I think these details at least need clarification for reproducibility.

I don't really understand how CMAES was implemented.  The method is cited as being from [9][18], but then the appendix states that it was implemented from [26] using pymoo.  Some examples like the green image in Figure 4 make me skeptical that this baseline was implemented correctly.
questions: questions: Can the authors provide full details on how the baselines were implemented?  For me, acceptance is hinges on whether I can be convinced that the baselines were implemented correctly and are indeed that bad.
limitations: limitations: seems fine
paper_summary: paper_summary: This paper proposes a goal-conditioned reinforcement learning framework for photo finishing tuning. They introduce a novel state representation and treat the image processing pipeline as a black box, avoiding the need for differentiable proxies. The method can efficiently tune parameters to match various goals, including pixel-aligned target images and style images.
strength_weakness: strength_weakness: 1. Compared to search-based methods, this proposed RL-based framework may not generalize well to unseen datasets.
2. The efficiency comparison in Table 2 may not be entirely fair. While the CMAES method runs on CPUs as required, the paper doesn't explore potential speed-ups through multi-processing.
questions: questions: Is the search-based method (CMAES) run on a single CPU core to obtain the inference time?
limitations: limitations: The paper's proposed method may require training a new model for each new incoming dataset. This limitation could potentially impact the framework's adaptability and efficiency when applied to diverse or frequently changing data sources.

Comments: comment: Dear Reviewer syDC,

Thank you for your reply. We would like to further discuss the questions with you below:

### Q1. why TD3 performs better

In our task, the RL algorithm is only a tool for optimizing the policy network. We analyze why TD3 performs better as follows:

TD3 is an off-policy RL algorithm with a deterministic policy. The deterministic policy allows it to explore the entire action space, including actions near the boundary, which is crucial for recovering boundary ISP parameters in our photo finishing task. TD3 also employs tricks like target policy smoothing and delayed policy updates to enhance robustness. In our task, actions like significantly brightening an image can cause large changes in the reward function, so these robustness tricks are crucial to the stability of the RL algorithm.

In contrast, SAC uses a stochastic policy, sampling actions from a Gaussian distribution, which reduces the likelihood of exploring boundary actions, impacting performance. PPO also faces this issue with its stochastic policy. Both SAC and PPO also lack robustness tricks like target policy smoothing. Moreover, PPO is an on-policy algorithm, making it less sample-efficient in our complex photo finishing environment, potentially leading to suboptimal results.

|      | deterministic policy | robustness tricks | off-policy |
| ---- | -------------------- | ----------------- | ---------- |
| TD3  | ✔                    | ✔                 | ✔          |
| SAC  | ✗                    | ✗                 | ✔          |
| PPO  | ✗                    | ✗                 | ✗          |

These differences explain the variation in performance, but the effect of the RL algorithm is still minor compared to state representation. As shown in the first table of the rebuttal, TD3, SAC, and PPO all yield a PSNR of around 37 **with state representation**, whereas TD3 without state representation achieves only 32.17. 

### Q2. multiple seeds

We set the random seed to 1 in all our experiments in the paper and the rebuttal, ensuring fair comparison. In our main comparisons with all baselines, our method outperforms them by a large margin (over 4 dB in PSNR). Even in the ablation study over different RL algorithms, TD3 still outperforms SAC by 1.16 dB in PSNR. As discussed earlier, the RL algorithm is just a tool within our framework, and we are confident that we have selected the RL algorithm that best fits our task. Additionally, we plan to include the average scores over multiple seeds in the camera-ready version of this paper.

### Q3. greedy tuning performs well & stronger baseline with conditioning

**There are no stronger baselines available.** We followed the setup from [27] for baselines in the photo finishing task. We compared our method to existing work in the field of photo finishing tuning, including CMAES [18] and proxy-based methods [26, 27]. Currently, there are no stronger baselines for the photo finishing tuning task.

**Search-based methods (Greedy & CMAES) perform well because they optimize over more iterations.** Both the greedy tuning algorithm and CMAES are search-based methods. The search-based methods, including the greedy algorithm added in the rebuttal, only performs well when using 200 searching iterations, while our method can achieve better results with only 10 iterations. The main advantage of our RL method is its fast convergence speed, as shown in Table 2 in our paper.

**Conditioning is not possible for greedy algorithm and is part of our contribution.** Traditional search-based methods, including greedy algorithms, are driven purely by the loss function's output relative to the parameters being adjusted. These methods lack the ability to condition on complex features like the characteristics of the input image, the target image, or prior processing operations. Such conditioning requires an understanding capability and the ability to generalize from past experiences, which is beyond the capabilities of traditional search-based algorithms.

Developing a novel search algorithm conditioned on these complex characteristics is one of our contributions. Our proposed state representation encodes information about the input image, target image, and prior image processing operations. The policy network can then leverage this information to make decisions that are conditioned on these inputs, which leads to more efficient and effective optimization.

### Q4. why the proposed method generalizes well

We have analyzed why our method generalizes better in Q1 (cross-dataset generalization) of the **To All** section in our rebuttal.
The rebuttal and implementation details are sufficient.  I will raise my score to borderline accept.
I appreciate the authors' response and the additional results provided in the rebuttal. However, I remain unconvinced by the results presented in Table 1, which actually raise more concerns.
1. While I agree with the authors that state representation is crucial, the results showing variations with different RL algorithms suggest that the choice of RL algorithm is also important. It is unclear why TD3 performs better than the others. A stronger justification for the superiority of TD3 would be valuable. 
2. In RL experiments, it's essential to evaluate each method across multiple seeds since results can vary significantly. The table does not provide enough information to determine the best method conclusively. Reporting average results across multiple random seeds would be beneficial. 
3. The second table suggests that a naive greedy tuning algorithm performs well, though not as well as the RL-based method. It would be interesting to explore the possibility of developing a greedy tuning algorithm conditioned on the input image, target image, or prior image processing operations as a baseline.
4. Although the authors have added more ablation studies in the rebuttal, they remain insufficient.
5. While the authors have empirically demonstrated the method's generalization across different datasets, a detailed discussion on why the proposed method has superior generalization capability is necessary.
In summary, the problem is intriguing, and the proposed method appears effective, but it requires more in-depth analysis, additional ablation studies, and comparisons against stronger baselines to validate its effectiveness. Consequently, I have updated my score to borderline reject.
Thank you for your response. It has successfully addressed all of my minor concerns. Based on this, I am willing to upgrade my score.
Dear Reviewer WirW,

In our rebuttal, we test our model on an unseen dataset, HDR+. Results demonstrate that our method generalizes well to unseen datasets, outperforming all baselines by a large margin. We also explained that the CMAES baseline is implemented with a multi-core CPU with parallel acceleration.

We want to follow up to see if our responses address your concerns. We would be very grateful to hear additional feedback from you and will provide further clarification if needed.

Thank you again for your time and effort.
Dear Reviewer kK87,

In our rebuttal, we address the implementation details of three baselines, explain the code used, and clarify terms in the paper, along with citations of CMAES (`Q1`). We also point out that certain behaviors of the baselines (such as the similar hue shift and green image in Figure 4) are due to the inherent flaws of the baselines (`Q2` `Q3`). Additionally, we provide more detailed experiments of the baselines to prove the correctness of our reproduction (`Q4`).

We want to follow up to see if our responses address your concerns. We would be very grateful to hear additional feedback from you and will provide further clarification if needed.

Thank you again for your time and effort.
Dear Reviewer syDC,

In our rebuttal, we explain the reason for choosing TD3 and include further ablation studies to show that the choice of RL algorithm does not significantly impact performance. We also conduct more ablation over the greedy algorithm as suggested by the reviewer. Additionally, we test our method on the HDR+ dataset. Results demonstrate that our method generalizes well to unseen datasets.

We want to follow up to see if our responses address your concerns. We would be very grateful to hear additional feedback from you and will provide further clarification if needed.

Thank you again for your time and effort.
Dear Reviewer ASBs,

In our rebuttal, we explain whether each baseline requires training. We also provide more evaluation on the HDR+ dataset. Results demonstrate that our method generalizes well to unseen datasets, outperforming all baselines by a large margin.

We want to follow up to see if our responses address your concerns. We would be very grateful to hear additional feedback from you and will provide further clarification if needed.

Thank you again for your time and effort.