AdaptiveISP: Learning an Adaptive Image Signal Processor for Object Detection

Authors: yujin  wang
tianyi  xu
fan  zhang
tianfan  xue
jinwei  gu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 07:23:48

Abstract: Image Signal Processors (ISPs) convert raw sensor signals into digital images, which significantly influence the image quality and the performance of downstream computer vision tasks. 
    Designing ISP pipeline and tuning ISP parameters are two key steps for building an imaging and vision system.
    To find optimal ISP configurations, recent works use deep neural networks as a proxy to search for ISP parameters or ISP pipelines. However, these methods are primarily designed to maximize the image quality, which are sub-optimal in the performance of high-level computer vision tasks such as detection, recognition, and tracking. Moreover, after training, the learned ISP pipelines are mostly fixed at the inference time, whose performance degrades in dynamic scenes. 
    To jointly optimize ISP structures and parameters, we propose AdaptiveISP, a task-driven and scene-adaptive ISP. 
    One key observation is that for the majority of input images, only a few processing modules are needed to improve the performance of downstream recognition tasks, and only a few inputs require more processing.
    Based on this, AdaptiveISP utilizes deep reinforcement learning to automatically generate an optimal ISP pipeline and the associated ISP parameters to maximize the detection performance. Experimental results show that AdaptiveISP not only surpasses the prior state-of-the-art methods for object detection but also dynamically manages the trade-off between detection performance and computational cost, especially suitable for scenes with large dynamic range variations.
Project website: https://openimaginglab.github.io/AdaptiveISP/.

Main paper: 
 introduction: 
 Image Signal Processors (ISPs) play a fundamental role in camera systems. Originally, ISPs aimed at enhancing the perceptual quality, focusing on photography-related applications. Recently, machine vision cameras also optimized the ISP pipeline for downstream recognition tasks [34, 38, 33, 35] . For recognition tasks, studies have shown a specially designed ISP can significantly enhance their performance. However, most machine vision cameras [13] still prefer a static hand-designed ISP and manually-tuned parameters, making it sub-optimal for downstream recognition tasks and also inflexible for dynamic scenes. Another solution is to directly train detection networks that take raw files as input, skipping the entire ISP processing. However, this may require re-training the detection network for each camera sensor, as the raw format varies between cameras, and studies [22] also show that the raw detection network still performs worse than the detection network runs on ISP-processed images. Therefore, a well-tuned ISP is important for downstream recognition tasks.In this work, we aim to design an ISP that can dynamically adapt its pipeline and parameters for different inputs, tailored for a given high-level computer vision task. This problem faces two challenges: complexity and efficiency. First, it is a complicated optimization problem to jointly Figure 1 : AdaptiveISP takes a raw image as input and automatically generates an optimal ISP pipeline {M i } and the associated ISP parameters {Θ i } to maximize the detection performance for any given pre-trained object detection network with deep reinforcement learning. AdapativeISP achieved mAP@0.5 of 71.4 on the dataset LOD dataset, while a baseline method [34] with a fixed ISP pipeline and optimized parameters can only achieve mAP@0.5 of 70.1. Note that AdaptiveISP predicts the ISP for the image captured under normal light requires a CCM module, while the ISP for the image captured under low light requires a Desaturation module.re-organize the ISP modules, update their parameters, and also improve the performance of the downstream recognition module. Because of this complexity, previous works only update the parameters [34, 24, 29, 30, 37, 16, 25] . Some recent works jointly optimize the ISP pipeline and parameters [10, 27] , but they are not designed for downstream recognition tasks. Second, ISP optimization must be efficient enough to adapt to dynamically changing scenes, which is particularly important in real-time applications such as autonomous driving and robotics. The majority of ISP pipeline optimization methods use searching strategies, such as the neural architecture search (NAS) method [38] and the multi-object optimization search method [33] , which often takes several hours, making them infeasible for real-time applications across dynamically changing scenes.To solve this challenge, we observe that the majority of input images only require a few ISP operations to increase the downstream recognition accuracy. As shown in Figure 1 on the right, only the first two stages of ISP already boost the detection accuracy (mAP) from 67.8 to 70.9, and the rest of the three stages only further boost it to 71.4. Only challenging examples require more complicated pipelines. Therefore, we can model the ISP configuration process as a Markov Decision Process [28] , and our AdaptiveISP only selects one ISP module at each stage, as shown in Figure 1 on the left. This greatly reduces the search space at inference time and enables adaptively changing the ISP length, spending less time on easy inputs.Based on this idea, we introduce AdaptiveISP, a real-time reconfigurable ISP based on reinforcement learning (RL). As shown in Figure 1 , AdaptiveISP takes a linear image as input and generates an optimal ISP pipeline with associated parameters that best fit this image for object detection. Unlike previous neural architecture search (NAS) [38] that generates the entire pipeline at once, AdaptiveISP takes a greedy approach to only generate one module at each iteration, greatly reducing the searching time. At each iteration, a lightweight RL agent takes the processing output from the previous stage as input and finds the optimal module for the next iteration. Because of its efficiency, AdaptiveISP only takes 1 ms to predict one stage and can generate different pipelines for different scenes on the fly in real-time, as illustrated in Figure 1 .Furthermore, we design a reinforcement learning scheme tailored for ISP configuration. First, we integrate a pre-trained fixed object detection network, YOLO-v3 [31] , into the optimization system as a loss function, guiding our model to prioritize specific high-level computer vision tasks. Second, with the observation that many later-stage ISP modules may only bring little improvement to detection, we introduce a new cost penalty mechanism so that AdaptiveISP supports dynamically trading off object detection accuracy and ISP latency.The proposed AdaptiveISP have several distinctive properties compared to a typical ISP for visual quality. First, ISP for detection can be much simpler. As shown on the right side of Figure 1 , with only 4 stages, it achieves the best detection accuracy, while traditional ISP may need more than 10 stages to enhance image quality. Second, while color processing is important in both types of ISPs for both detection and viewing, their behaviors are different. For instance, ISPs for detection often completely desaturate color in low-light scenarios to boost the detection rate, as shown in the second-row of Figure 1 , which rarely happens for traditional ISPs. Third, a simple sharpening or blurring module may greatly enhance detection accuracy, while a more widely used denoising module in traditional ISPs is not super helpful for detection.We have evaluated the proposed AdaptiveISP on the LOD [9] , OnePlus [38] , and synthetic raw COCO [19] datasets and showed that it outperforms prior state-of-the-art methods under different challenging conditions and different downstream tasks. Experimental results also demonstrate the ability to dynamically switch from a high-accuracy ISP pipeline to a low-latency one.2 Related Work ISP Parameter Tuning. Recent studies have explored various methods for optimizing Image Signal Processing (ISP) hyper-parameters, particularly those based on handcrafted designs and tailored to meet the demands of downstream evaluation metrics. One category of methods focuses on derivative-free optimization techniques. Nishimura et al. [25] introduced an automatic image quality tuning method that employs nonlinear optimization and automatic reference generation algorithms.Another category utilizes gradient-based optimization techniques. Tseng et al. [34] introduced a gradient optimization method that relies on differentiable approximations, allowing for efficient hyper-parameter tuning but the ISP parameters are fixed during the inference stage. Immediately afterward, some researchers realized that one set of parameters was not necessarily suitable for different scenarios. Qin et al. [29] proposed an attention-based CNN method, but it does not consider sequence-specific prior knowledge. Then, Qin et al. [30] proposed a sequential ISP hyper-parameter prediction framework, which optimizes the ISP parameters by leveraging sequential relationships and parameter similarities. Additionally, Yoshimura et al. [37] proposed DynamicISP, which can causally and smoothly control the parameters of the current frame according to the recognition result of the previous frame. Departing from approximation methods, Mosleh et al. [24] introduced a hardwarein-the-loop approach. This approach directly optimizes hardware-based image processing pipelines to meet specific end-to-end objectives by using a novel 0th-order stochastic solver. Furthermore, an image enhancement method based on reinforcement learning, proposed by Kosugi et al. [16] , leverages reinforcement learning techniques to optimize hyper-parameters in an unpaired manner.ISP Pipeline Design. The conventional ISP pipelines are crafted to adhere to human visual perception, and this alignment might not always be conducive to fulfilling the demands of downstream highlevel tasks. The prior research works [33, 38] have proved that handcrafted ISP configuration does not necessarily benefit the downstream high-level vision tasks. ReconfigISP [38] proposed a novel Reconfigurable ISP whose architecture and parameters can be automatically tailored to specific data and tasks by using neural architecture search (Darts) [20] . RefactoringISP [33] jointly optimized ISP structure and parameters with task-specific loss and ISP computation budgets through multi-objective optimization algorithm (NSGA-II) [4] . However, these approaches maintain the ISP pipeline and parameters fixed during inference, regardless of the distinct characteristics of the input. An innovative study addresses the challenge of photo retouching by leveraging deep learning on unpaired data, allowing users to emulate their preferred retouching style [10] . Additionally, recent studies have explored replacing traditional ISP pipelines or modules with deep learning models to enhance image quality. PyNET [11] proposed a unified model that directly learns the RAWto-RGB mapping to improve mobile photography, while CycleISP [39] introduced a noise-aware denoising approach. ParamISP [14] developed forward and inverse ISP models conditioned on camera parameters, enabling a more accurate emulation of real-world ISP behavior to enhance image processing performance. DualDn [17] further employs a differentiable ISP to improve denoising capabilities. Although these methods improve image quality, they do not consistently translate into better performance for downstream high-level vision tasks. 
our method: 
 Image Signal Processors (ISPs) often consist of a pipeline of image processing modules that primarily transform raw sensor pixel data into RGB images suitable for human viewing [2] . A typical camera ISP pipeline includes two parts: raw-domain and RGB-domain. Compared to raw-domain processing, the RGB-domain processing is normally image-dependent and requires more dedicated design and tuning efforts. Details of ISP pipeline and modules can be found in the appendix. is employed to calculate the reward for the current policy. The entire system is optimized using the actor-critic algorithm [15, 23] .𝝅 ℱ 𝑀 ! , Θ ! 𝒟 𝒟"𝑠 ! $ -𝒟"𝑠 !"# $ Reward s ! 𝒂 𝑴 𝒕 𝒂 " 𝒕 s !"# s $ Terminated 𝑠 % Policy Network Value Network 𝑉 & "s ! $, 𝑉 & "s !"# $ State Value Input YOLO-v3 ❄ Based on Actor-Critic AlgorithmIn this work, we focus on the RGB-domain processing. We assume that the captured raw sensor data is already converted to linear RGB images using a simple static raw-domain processing, and our tuning mainly focuses on sRGB-domain processing, similar to [9, 37] .Given an input linear RGB image I, the goal of AdaptiveISP is to find an optimal ISP pipeline and parameters, with which the processed output image will result in high performance for object detection. Let F (M,Θ) denote one ISP module, where M is the ISP module type and Θ is the set of parameters for that module. An ISP pipeline consists of a sequence of ISP modules F (Mt,Θt) T t=0 , which transforms an input image I to an output image I t with t stages of ISP processing asEQUATIONThe goal of AdaptiveISP is to predict an optimal ISP pipeline (with T stages) and its parameters, i.e., {M t , Θ t } T t=0 for an input image, in order to maximize its performance for a given computer vision task (e.g. object detection), which can be formulated asEQUATION)where D is the detection error with a given object detector. This optimization problem can thus be modeled as a Markov Decision Process [28] , which can be solved efficiently via deep reinforcement learning.Our formulation is similar to [5, 6, 10, 27] . Let us denote the problem as P = (S, A), where S is a state space and A is an action space. Specifically, in our task, S is the space of images, which includes the input images and all the intermediate results in the ISP process, while A is the set of all ISP modules. Since each action includes the selection of ISP modules and the prediction of ISP module parameters, we can decompose the action space A into two parts: a discrete selection of ISP modules a M and a continuous prediction of ISP module parameters a Θ . At stage i, with a selected ISP module and its parameters (a M i , a Θ i ), the input image at state s i is mapped to state s i+1 . Applying a sequence of T selected ISP modules to an input image corresponds to a trajectory τ of states and actions:EQUATIONwhere s T is the stopping state. Our goal is to find a policy π that maximizes the accumulated reward during the decision-making process. The policy π consists of two sub-policies (π M and π Θ ), where π M takes a state and returns a probability distribution over ISP modules, and π Θ predicts the parameters a Θ of the selected ISP module. In this paper, the reward function with the i-th action (i.e., corresponding to the i-th stage of ISP processing) is thus written as:EQUATIONwhere s i+1 = p(s i , a M i , a Θ i ), and D is the error of object detection. Given a trajectory τ , we define the return g t as the summation of the discounted rewards after s t :EQUATIONwhere γ ∈ [0, 1] is a discount factor that places greater importance on rewards in the near future. We can thus define the value of state function V π (s) as:EQUATIONand the value of action function Q π (s, a M , a Θ ) as:EQUATION)Our goal is to select a policy π = (π M , π Θ ) that maximizes the expected accumulated reward during the decision-making process:EQUATIONwhere S 0 denotes the entire image dataset.Similar to [10, 27, 16, 5] , we employ deep neural networks to approximate the value function V π (s) and the policy π. As shown in Figure 2 , convolutional neural networks (CNNs) and a fully-connected layer are used as the policy network, which maps the image s into action probabilities π M (s, Φ M ) (after softmax) and ISP module parameters π Θ (s, Φ Θ ) (after tanh), where (Φ M , Φ Θ ) are the network parameters. The value network V π (s, Φ V ) uses a similar architecture with parameters Φ V . By maximizing the objective J(π) Φ with training these two networks Φ = (Φ M , Φ Θ , Φ V ), we can learn to predict the optimal policy π(s) for an input image s. Specifically, to train the policy network and the value network, as shown in Figure 2 , we apply the actor-critic algorithm [15, 23] , where the actor is represented by the policy network and the critic is the value network. Details of network architectures and training are provided in the appendix.To ensure stable and effective reinforcement learning for the ISP pipeline and parameter prediction, we augment the network input and implement several penalty functions to better constrain the training process. Specifically, we modify the reward function Equation 4 to:EQUATIONwhere P i stands for the penalties as explained below.Exploitation and Exploration. To avoid the same ISP module being selected consecutively multiple times by the policy network, we augment the network input with the module usage record (which is represented as N channels where N is the size of the ISP module pool) and one additional "stage" channel. At each stage, if an ISP module is being used, the corresponding "use" channel and penalty of reusing will be set to 1 and 0 otherwise. The "stage" channel is set with the index of the stage.During training, these additional inputs and the penalty of reusing constrain the policy network picking the same ISP module no more than once, which can effectively narrow down the solution space and improve the performance. Details of network input are provided in the appendix.In addition, we want to encourage the policy network π M (s) to explore different ISP modules to prevent the parameter prediction network from insufficient learning. Specifically, we introduce a penalty term P e on the output entropy of the policy network π M (s) to ensure that the action distribution is not overly concentrated.EQUATIONwhere p(m) is the probability of an ISP module m in the softmax output, and λ e ∈ [0, 1] is a penalty coefficient and gradually decreases from 1 to 0 with the progress of the training process.Penalty of Computational Time. ISP pipelines typically consist of multiple modules responsible for specific image processing tasks, such as denoising, sharpening, white balance, and more. These modules exhibit varying computational costs, some being faster and others slower, as shown in Table 4 . Consequently, when designing ISP pipelines, it is crucial to consider the distinct computational costs associated with these modules. Particularly, in scenarios requiring swift responses like autonomous driving, optimizing the computational efficiency of ISP pipelines becomes paramount. In order to make our method automatically select the appropriate modules and their sequence, we should account for the computational time of each module during the design of the ISP pipeline. Specifically, we collect the run time of all modules, marked as M c , which can be found in Table 4 , and the penalty of cost P c is defined as:EQUATIONwhere I m represents the one-hot encoding whether the module m is used, and λ c is the penalty coefficient. 
experiments: 
 Datasets. In line with prior research [9, 26, 37, 38, 29, 30] , we train and evaluate our models on widely used real low-light detection datasets and synthetic normal-light datasets, including:• LOD. LOD Dataset [9] is a real-world low-light object detection dataset, which contains 2,230 14-bit low-light raw images with eight categories of objects. This dataset aims to systematically assess the low-light detection performance. There are 1,830 data pairs for training and 400 data pairs for validation. Additionally, it provides accompanying metadata, including ISO, shutter speed, aperture settings, and more, which greatly facilitates our experimental analysis.• OnePlus. OnePlus Dataset [38] is a real-world low-light object detection dataset collected by the OnePlus 6T A6010 smartphone at driving scenes, which contains 141 raw images with three classes of objects in the street scenes. There are 50 pairs for training and 91 pairs for validation. Given the limited size of the dataset, all raw images were utilized as the validation set in our experiments.• Raw COCO. COCO [19] is a large-scale object detection, segmentation, and captioning dataset. To evaluate the generalization ability of our method, we convert the COCO validate dataset (5,000 images) to a synthetic raw-like dataset as our evaluate dataset by using UPI [1] , similar with [9, 37, 29, 30] .Experiment Details. Similar to [38] , YOLOv3 [31] is utilized as the detection model in all methods unless explicitly stated otherwise. YOLOv3 is a robust and fast object detection algorithm. Its real-time applicability, speed, and high-performance capabilities make it a powerful choice for various applications in numerous classic object detection algorithms [4, 32, 8, 21, 18, 5] . In terms of back-propagation, YOLOv3 passes gradients back to our method for optimizing both structure and parameters. It is important to note that the pre-trained YOLOv3 model remains unaltered throughout the training process. During training and inference, we follow [9] to use a fixed input resolution of 512 × 512. In addition, unless otherwise stated, our methods and comparison methods are all experiments conducted on handcrafted ISP.Evaluation Metrics. We utilize the mean Average Precision (mAP) across all Intersection over Union (IoU) thresholds to assess the performance of our method, following a similar approach as used in object detection algorithms [4, 32, 8, 21, 18, 5] .Results on LOD Dataset. The LOD dataset provides JPEG images with accompanying metadata, which serves as a convenience for the analysis of our experimental results. On this dataset, we choose several baseline methods for evaluation. First, we select two network-based ISP methods, namely Crafting [9] and NeuralAE [26] . Also, we evaluate our method against the static handcrafted pipeline and parameter methods, specifically, Hyperparameter Optimization [34] . Furthermore, we compare our approach with static handcrafted pipelines combined with dynamic parameters techniques, such Table 1 : Experimental results of LOD [9] , OnePlus [38] , and raw COCO Dataset [19] .Raw COCO mAP @0.5 mAP @0.75 mAP @0.5:0.95 mAP @0.5 mAP @0.75 mAP @0.5:0.95 mAP @0.5 mAP @0.75 mAP @0.5:0.95 Crafting [9] 67.9 49.0 44. Figure 3 : Object detection visualization results on LOD dataset. Our method outperforms the state-ofthe-art methods [34, 29, 38, 33] in terms of missed detection and false detection. The methods with fixed pipelines or fixed parameters struggle to effectively handle varying noise levels and brightness scenarios.as Attention-aware Learning [29] and DynamicISP [37] . Finally, we compare our method with the optimization method for ISP pipelines and parameters, as exemplified by ReconfigISP [38] and Refactoring ISP [33] . ReconfigISP utilizes a neural architecture search algorithm (Darts) [20] , while Refactoring ISP is based on Non-dominated Sorting Genetic Algorithm II (NSGA-II) [4] .As shown in the first column of Table 1 , our method achieves the best performance across all object detection metrics on the LOD dataset. Note that static pipelines with dynamic parameter methods outperform static pipeline and parameter approaches, dynamic pipelines and parameters yield superior results across all methods. We further show the detection results of our method in Figure 3 , showcasing its superior performance in terms of both missed detection and false detection compared to all other methods. Cross Datasets Test. To evaluate the generalization ability of our method on different datasets, we utilize a model trained on the LOD dataset and conduct testing on both the OnePlus and raw COCO datasets. As displayed in the central and rightmost column of Table 1 , our method achieves the best performance, which verifies that our method also has the best generalization ability. In terms of the mAP@0.75 evaluation metric, our method exhibits an improvement of approximately 1 point compared to alternative approaches on both Oneplus and raw COCO datasets. Furthermore, our method outperforms Refactoring ISP [33] and ReconfigISP [38] by 2 points on the raw COCO dataset and by 4 points on the OnePlus dataset in terms of mAP@0.5:0.95 evaluation metrics.Cross Detectors Test. To evaluate the generalization ability of our method on different detectors, we use the detection results from the RGB (existing ISP) as a baseline and conduct comparative experiments on DDQ [40] , representing the Transformer-based approach, and YOLOX [7] , representing the CNN-based approach. As shown in Table 3 , all detectors using our AdaptiveISP demonstrate improved detection performance, demonstrating that our method does not overfit one detector, but is suitable for other detectors. Note that DDQ [40] and YOLOX [7] are not used in the training process, but our ISP can still generalize to these detectors at testing time.Results for Image Segmentation. To show our approach can generalize to other tasks, in Table 2 , we also conduct an experiment using image segmentation as the downstream task. Note that all models are trained on the LOD dataset using the pre-trained YOLOv3 [31] detector and evaluatedAttention-aware Learning Ours Refactoring ISP ReconfigISP Figure 4 : Image segmentation visualization results on raw COCO dataset. Our method detects all the object, while the state-of-the-art methods [34, 29, 38, 33] may miss some.Table 2 : Image Segmentation results on raw COCO datasets [19] .Methods mAP@0.5 mAP@0.5:0.95Hyperparameter Optimization [34] 46.4 28.4 Attention-aware Learning [29] 45.5 27.9 ReconfigISP [38] 42.1 25.2 Refactoring ISP [33] 40.6 24.7Ours 47.0 28.8on the synthetic raw COCO datasets with the pre-trained YOLO-v5 [12] segmentor. Our method performs better than all baselines, further showing its generalizability for different downstream tasks and algorithms. We further present the segmentation results of our method in Figure 4 , highlighting its superior performance in terms of missed detection compared to all other methods.Accuracy-Efficiency Trade-off. Our approach can control λ c to regulate whether the optimization process also takes the computational time of each ISP module into consideration, where λ c is the weight computational cost. By tuning up λ c , we can generate a more efficient ISP with a minor drop of the recognition accuracy.We conducted analysis experiments on the LOD dataset, where λ c = 0.0 represents the accuracyoriented results and λ c = 0.01 stands for efficiency-oriented results. We calculated the proportions of the occurrence of each module in the test results. As shown in Table 4 , the efficiency-oriented method has a significant reduction in the average running time for each sample, which is accompanied by a slight decrease in performance. In addition, the frequency of appearance of modules with higher computational time, such as Sharpen/Blur and Tone Mapping, decreased by more than 50%. Conversely, modules with lower computational time, like Exposure and White Balance, saw a substantial increase in selection frequency.Runtime. To verify the practicality of our approach, we conducted speed tests using the NVIDIA GTX1660Ti GPU, which offers a computational capability of 11 TOPS-markedly lower than that of the NVIDIA DRIVE Orin TM SoC at 254 TOPS. Our method only takes 1.2 ms per stage during inference, this efficiency is attributed to just need to utilize the light-weight policy network to predict the modules and parameters during inference.Adaptive ISP. We perform a study to show that different data require different ISP pipelines to achieve the best performance. To accomplish this, we begin by gathering the various ISP pipelines predicted by our method on the LOD dataset. Next, we select the three most representative ISP pipelines along with their corresponding input raw images, resulting in three subsets of the LOD dataset. The specifics of these three distinct ISP pipelines can be found in Figure 5 . Finally, we conduct cross-testing on the three sets of ISP pipelines generated by our method using the three subsets, as depicted in Figure 5 . By analyzing the experimental results, we can see that only the Table 3 : Experimental results of different detectors on LOD dataset. Note that the DDQ [40] and YOLOX [7] do not participate in our training process. All detectors using our method demonstrate improved detection performance.Detectors Methods mAP@0.5 mAP@0.75 mAP@0. Representative Images Metrics mAP@0.5 mAP@0.75 mAP@0.5 mAP@0.75 mAP@0.5 mAP@0.75 Only the most matching pipeline can achieve the best results, which proves that a different pipeline is necessary.most matching pipeline can achieve the best results, otherwise, it will lead to varying degrees of performance degradation, which also proves that there are different pipeline requirements in different scenarios.Module preferences of different images. Furthermore, we analyze the reasons why different ISP pipelines are necessary for various scenarios. The LOD dataset provides JPEG images with accompanying metadata, which serves as a convenience for the analysis of our experimental results.It is noticed that there are two different choices at the first stage of the three ISP pipelines: CCM and Desaturation. The High ISO (ISO6400, ISO3200) and high noise level cases tend to favor the Desaturation module, whereas low ISO (ISO800, ISO1600) and low noise level cases tend to prefer the CCM module, as shown in Figure 5 . Because Desaturation can reduce the color noise and saturation, high ISO images with high-level noise prefer it. CCM can remove color casts and enhance color saturation. Therefore, it is the best choice is CCM for low-ISO images.Subsequently, we observe a divergence between the second and third ISP pipelines during the third stage. While the second ISP pipeline opts for the Tone Mapping module, the third ISP pipeline favors the White Balance module. Upon the analysis of the images subsequent to the second stage, we observe a pronounced color cast problem, particularly prevalent when the overall brightness was relatively high. In such scenarios, the most suitable course of action is to opt for the White Balance module. Conversely, when dealing with images characterized by a high dynamic range, such as those containing electric lights or direct sunlight, it becomes evident that the superior choice is to employ the Tone Mapping module.Finally, when summarizing the commonalities among the three ISP pipelines, several key conclusions emerge: i) The color correction module notably enhances detection performance (distinct from color correction for image quality tasks). However, optimal choices vary for images with different brightness and noise levels. ii) The Sharpen/Blur module holds a significant position, either enhancing or blurring the image to align with the detection network. iii) Tone Mapping also plays a crucial role, Table 4 : Experimental results considering computational cost on LOD dataset [9] . λ c = 0.0 represents the accuracy-oriented, λ c = 0.1 stands for efficiency-oriented. The total time represents the average running time of each sample. The efficiency-oriented method has a significant reduction in the average running time for each sample, which is accompanied by a slight decrease in performance. As λ c increases, our method tends to favor faster-executing modules. Our approach attains optimal performance with just two stages and dynamically achieves a trade-off between object performance and computational cost.Stages mAP@0.5 mAP@0.75 mAP@0. enhancing detection accuracy by adjusting the overall color and brightness. iv) Denoising, contrary to previous research conclusions, is not deemed crucial. This finding contributes to substantial computational cost savings for the ISP. These analyses and conclusions provide valuable insights for the future designs of ISPs tailored to specific downstream tasks.Adaptive Trade-off. To demonstrate that our model can achieve an adaptive trade-off between efficiency and accuracy during the inference phase, we use stages to represent the number of ISP modules. Recall that none of the previous algorithms support dynamic efficiency-accuracy tradeoff. Hyper-parameter Optimization [34] and Attention-aware Learning [29] are optimized based on handcrafted ISP, so the time consumption cannot be changed during the inference phase. Although ReconfigISP [38] and Refactoring ISP [33] have optimized ISP pipelines and parameters for specific tasks, the ISP pipelines and parameters remain fixed during inference. As shown in Table 5 , our method only requires 3 stages to achieve the best performance on the LOD dataset and reaches a good trade-off between efficiency and accuracy. Moreover, this trade-off happens at inference time, without any retraining, and supports the dynamic update of the trade-off strategy. 
conclusion: 
 In this paper, we introduce AdaptiveISP, a novel approach that leverages deep reinforcement learning to automatically generate an optimized ISP pipeline and associated parameters, maximizing detection performance with a pre-trained object detection network. Our method incorporates several key innovations. Firstly, we formulate the ISP configuration process as a Markov Decision Process, allowing reinforcement learning to autonomously discover an optimal pipeline and parameters for specific high-level computer vision tasks. Secondly, to account for computational costs associated with different ISP modules, we introduce a penalty of computational time. Comprehensive experiments demonstrate that AdaptiveISP surpasses existing state-of-the-art methods, and dynamically manages the trade-off between performance and computational cost. Furthermore, we conduct a detailed analysis of individual modules within ISP configurations, offering valuable insights for future ISP designs tailored to specific downstream tasks. The present approach relies on the utilization of differentiable ISP modules in research, with future endeavors aimed at exploring non-differentiable ISP methodologies. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Details are provided in Section 4 and Appendix A.1.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: This follows the convention in ISP tuning and design research, the same as previous works.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: The information on the computer resources is shown in Appendix A.1.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited all original papers and make sure that our usage is legal.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: We utilize the Adam optimizer with an initial learning rate of 3e -5 and a batch size of 8. The learning rate gradually decreases by a factor of λ = 0.1 3•iter/itertotal . Note that both the policy network and value network use the same initial learning rate. Our training comprises 100,000 iterations on one NVIDIA RTX 3090 (24G) GPU for the LOD dataset [9] , which is completed in around 24 hours. Additionally, we re-implement four methods, namely Hyperparameter Optimization [34] , Attention-aware Learning [29] , ReconfigISP [38] , and Refactoring ISP [33] , following the original papers. The ISP pipelines for these methods are illustrated in the RGB domain of Fig. 2 of the main paper. The results for Crafting [9] , NeuralAE [26] , and DynamicISP [37] are obtained from their respective original papers.The OnePlus dataset [38] comprises DNG files containing raw images and metadata. Given that our method operates on linear RGB, we employ "rawpy" to perform demosaicing operations on the original raw images, converting them into linear RGB images for the creation of our training and testing datasets.The LOD [9] , OnePlus [38] , and raw COCO [19] datasets are commonly used in ISP research. The LOD dataset provides accompanying metadata, which greatly facilitates our experimental analysis. The OnePlus dataset is a real-world dataset collected by smartphones. The COCO dataset is a well-known object detection and segmentation dataset. The ROD [36] dataset is a 24-bit HDR raw dataset collected by the SONY IMX490 sensor. The IMX490 sensor is rare in everyday life, therefore, we do not use ROD as our benchmark dataset.To validate the necessity of Image Signal Processing (ISP), we conducted a series of experiments on a raw COCO dataset [19] . For these experiments, we randomly selected 5,000 images from the COCO training set as our training dataset. We utilized a pre-trained YOLO-v3 model [31] as our starting model and trained it on the raw COCO dataset, setting the batch size to 128 and the learning rate to 0.01, with the training extending over 100 epochs. Additionally, we trained our method on the same dataset. It is important to note that within our methodology, the YOLO-v3 model [31] remained in a frozen state. The final experimental results were compared on the raw COCO validation set, as detailed in Table 6 . These results convincingly demonstrate that the inclusion of the ISP module is crucial and can significantly enhance detection performance.Table 6 : Ablation experiment of the necessity of ISP on raw COCO dataset [19] . Methods mAP@0.5 mAP@0.75 mAP@0. We conduct additional experiments on the ROD dataset. Note that the released ROD dataset differs from the one described in the published paper. Additionally, the released results (AP 28.1) on the new version of the dataset are lower than those reported in the published paper, according to the open-source code released on GitHub, indicating that the released version is more challenging.Because the released dataset is only a training dataset that provides paired raw images and annotations, we randomly split 80% of the dataset (12,800) for training, with the remainder as our validation dataset (3200). The dataset processing pipeline is similar to the original paper and released codes. Since our method emphasizes using training-well models, we selected only three categories (person, car, truck) belonging to COCO from the ROD dataset for a fair comparison. Due to time constraints, we select the previously best-performing method, Attention-Aware Learning [29] , and the state-ofthe-art method on the ROD dataset, Toward RAW Object Detection [36] , as our comparison methods. Each method was trained for 100 epochs.As shown in Table 7 , our method achieves the best performance, even though the detector we used is not trained on this input (Toward RAW Object Detection method [36] does).Table 7 : Experimental results on ROD dataset. * refers to a detector that is trained on raw input, which is normally better than detectors only trained on RGB input (like ours). The "Toward RAW Object Detection" is an end-to-end raw detection method, that updates its parameters during training time. Other methods use a pre-trained YOLO-v3 detector and freeze its parameters during training time. Methods mAP@0.5 mAP@0.75 mAP@0. To assess the impact of the penalty of reusing, we perform ablation experiments on the LOD dataset [9] . As shown in Table 8 , if the penalty of reusing is not applied, there is a noticeable decline in detection performance, accompanied by an occurrence frequency exceeding 100% for both CCM and Sharpen/Blur modules.To validate the applicability of our method with limited training data, we conducted experiments on the OnePlus dataset [38] , which comprises only 50 training images. The results are presented in Table 9 , demonstrating that our approach outperforms ReconfigISP.Table 9 : Experimental results on limited datasets (OnePlus dataset [38] ). mAP@0.5 mAP@0.75 mAP0. To validate the distinct requirements of ISP between image quality tasks and object detection tasks, we select Exposure [10] as the representative method for image quality tasks. Given that the Exposure [10] method requires paired raw-RGB data, and computing object detection results necessitate bounding box labels, we utilize 1,000 simulated raw-like images with the most bounding boxes converted from the COCO training dataset [19] using UPI [1] as our training dataset. Additionally, we employ the LOD Dataset, comprising all real raw images, as our test dataset, similar to prior works [9, 37, 29, 30] . We train both our approach and the Exposure method on the synthetic raw COCO dataset.As shown in Figure 6 , image quality and object detection tasks have distinct requirements for ISP. Image quality tasks primarily emphasize color and brightness, intending to produce images that closely align with human perception. In contrast, the results obtained from processing images for target detection tasks better meet the demands of machine-based systems. We further compare the object detection results with the image quality method [10] on the LOD dataset with all real raw images, as shown in Table 10 , the results of our method are much higher than the image quality method. Table 10 : Comparison results with image quality methods on all LOD datasets. mAP@0.5 mAP@0.75 mAP@0.5:0.95 We show more object detection and image segmentation visualization results in Figure 7 and Figure 8 , showcasing its superior performance in terms of both missed detection and false detection compared to all other methods.As shown in Figure 9 , a typical camera ISP pipeline includes two parts: raw-domain and RGBdomain. The raw-domain processing converts a raw sensor signal to a linear RGB image, which includes Defective Pixel Correction (DPC), Black Level Correction (BLC), Lens Shading Correction (LSC), and Demosaicking. The RGB-domain processing further applies customized rendering and post-processing to generate the final image. This processing includes tone mapping, color correction, denoising, sharpening, etc. We focus on RGB-domain processing, the ISP detailed in this paper encompasses various standard differentiable modules, including:(1) Exposure: Exposure control regulates the amount of light reaching the sensor to ensure the image's overall brightness matches the desired level. This module helps in adapting to varying lighting conditions. This module can be expressed as:EQUATION)where I is the input image, p is the exposure parameter and p ∈ [-3.5, 3.5]. (2) White Balance: White balance adjustment aims to rectify color temperature discrepancies, ensuring that colors in the final image appear natural and consistent with the observed scene. This module can be expressed as: EQUATIONEQUATIONwhere p is the CCM parameter and the sum of elements in each row equals 1. (4) Gamma: Gamma correction alters the image's tonal curve, affecting the distribution of lightness levels. This module is essential for ensuring luminance and contrast consistency. This module can be expressed as:EQUATIONwhere p is the gamma parameter and p ∈ [1/3, 3].(5) Denoise: Denoise techniques are applied to mitigate image noise, particularly in low-light or high ISO settings, preserving image quality by reducing noise. In this paper, we implement a differentiable Non-Local Means Denoising algorithm (NLM [3] ) based on PyTorch. This module can be expressed as:EQUATIONwhere p ∈ [0, 1] is filter-strength, which represents the strength of denoising. (6) Sharpen/Blur: This module enhances or blurs image details and edges, thus improving image clarity and visual appeal. It is useful for accentuating finer image structures. This module can be expressed as:EQUATIONwhere p ∈ [0, 2] is the factor and I blurred is the blurred image, which can be obtained by using the blur kernel, and the blur kernel can be expressed as:EQUATIONTone Mapping: Tone mapping regulates the dynamic range of an image, ensuring that details in both bright and dark areas are discernible. It is especially valuable for rendering high dynamic range scenes on standard displays. Following [10] , we approximate curves as monotonic and piecewise-linear functions. We use L parameters represent tone mapping curve, denoted as {p 0 , p 1 , . . . , p L-1 }. With the prefix-sum of parameters defined as P k = k-1 l=0 p l , the points on the curves are represented as (k/L, P k /P L ). Given an input image I ∈ [0, 1], the mapped result can be expressed as:EQUATIONwhere the slope of each segment in the curve is in [0.5, 2.0] and L = 8. (8) Contrast: This module modulates the contrast of an image, defining the variations between light and dark areas. It contributes to image aesthetics and enhances perceptual quality. This module can be expressed as:EQUATIONwhere p ∈ [-1, 1] is adjustment factor, and I lum can be expressed as:EQUATION(9) Saturation: Saturation adjustment manipulates the intensity of colors in an image, allowing for vivid or muted color representation according to the desired artistic effect. This module can be expressed as: EQUATIONwhere p ∈ [0, 1] is adjustment factor, and I lum can be obtained through Equation 20. (10) Desaturation: The desaturation module, conversely, reduces the intensity of colors in an image, leading to a more muted or grayscale appearance. It is often employed for specific artistic or visual effects, including creating black-and-white imagery or subtle color emphasis. This module can be expressed as:EQUATIONwhere p ∈ [0, 1] is adjustment factor, (I lum , I lum , I lum ) represents R-G-B channels of the image are I lum , and I lum can be obtained through Equation 20. Conversely, an overly small gain value leads to a completely black image, as depicted in (c).The process of reinforcement learning can be abstracted as an agent continually gathering experiences from the environment and learning from them. However, during the experience collection process, issues such as termination and truncation can significantly impact experience collection and training, necessitating special design considerations.We can limit the total number of ISP configuration steps to enable the policy network to autonomously learn the optimal module selection for downstream tasks under resource constraints. Specifically, the number of ISP module options in our ISP module pool is denoted as N, for the policy network selecting the modules more stably, we intentionally "terminated" once the execution stage reaches the maximum T (T ≤ N ). Besides, we incorporate step information into the policy networks by introducing an additional channel for providing stage-related data, as shown in the stage channel of Fig. 4 of the main paper. The terminated coefficient λ terminated can be defined as:EQUATION)where t is the number of ISP configuration stages, and T is the maximum stages and T is 5 in our experiments.Additionally, we intentionally truncate the execution when the network diverges to output abnormal images, as shown in Figure 11 , for example, "truncated" once the image mean gets out of range (lum min , lum max ). The truncated coefficient λ truncated can be defined as:EQUATION)where I is the mean of processed image.Ultimately, the state value function V π (s) can be defined asV π (s) = (1 -λ terminated ) × (1 - λ truncated )V π (s).C.3 Design for Object Detection ISP Modules. In the ISP pipelines for visual quality [10, 16, 27] , the ISP modules primarily focused on pixel operations related to color and brightness adjustment. To better adapt the ISP pipeline for object detection, following the traditional ISP modules, we extend the existing pipeline by incorporating differentiable Sharpen/Blur, Denoise, and CCM modules onto the foundation established in [10] . Empirically, we find these changes increase the detection performance.Input Resolution. For object detection networks, larger pixel images are commonly employed for training and testing [4, 32, 8, 21, 18, 5] . To seamlessly integrate our policy network and detection network into end-to-end training, the input resolution to our ISP is configured based on the input resolution of the detection network. This ensures the back-propagation from the detection network to the differentiable ISP and subsequently to the policy network. The policy network and value network can continue to utilize down-sampled input resolutions, such as 64 × 64 pixels, during training and testing, as illustrated in Figure 12 . To enhance the value network's assessment of the current state, we include the average luminance, contrast, and saturation of the input image as additional input features.Justification: Codes and datasets will be made publicly available upon acceptance.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Justification: We strictly adhere to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no direct negative societal impact for ISP tuning and design.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: The paper presents an approach to image signal processing (ISP) tailored for object detection tasks, leveraging deep reinforcement learning to optimize both ISP structures and parameters. The work introduces AdaptiveISP, which dynamically adjusts the ISP pipeline in response to varying scene requirements, enhancing detection performance while managing computational costs. The reviews are mixed, with some highlighting the paper's potential and others expressing concerns about its limitations. The submission received mixed ratings, namely, one accept, two borderline accept, and one reject. For the reviewer voting for rejection, the reviewer did not provide feedback on author rebuttal. The AC checked all the materials, and believes the rebuttal is reasonable. Thus, the AC decides to accept the paper based on the review comments of the other three reviewers.

However, the concerns raised regarding the evaluation scope, related work discussion, and generalization need to be addressed in more detail. The paper would benefit from a revision that includes a broader evaluation across different models and datasets, as well as a more comprehensive discussion of related work.

Reviews: review: paper_summary: paper_summary: A new perspective of designing ISP pipeline. Good results with some problems should be addressed.
strength_weakness: strength_weakness: 1/ Discussion on related works especially for ISP pipeline is not sufficient. Not only the ISP for task performance [1], but also for image quality [2,3].
2/ The datasets used for comparison are not actually real raw detection data. It is doubtful whether the real scene performance is good.
3/ Also, the compared methods are for designing special ISP params or orders to get better performance on  raw downstream tasks. However, what about using an existing ISP and detect on RGB images? For datasets such as LOD and COCO can do it. The results must be better than existing RGB detection SOTAs, so that this work can be meaningful.
4/ The order of ISP modules is still not fixed. There have been many works finding that it can be various and change its order according to tasks or image quality. Also, the pipelines can be different according to manufacturers. 
5/ Some new benchmarks such as [1] should be used for evaluation.

[1] Ruikang Xu, Chang Chen, Jingyang Peng, Cheng Li, Yibin Huang, Fenglong Song, Youliang Yan, Zhiwei Xiong: Toward RAW Object Detection: A New Benchmark and A New Model. CVPR 2023: 13384-13393
[2] Woohyeok Kim, Geonu Kim, Junyong Lee, Seungyong Lee, Seung-Hwan Baek, Sunghyun Cho: ParamISP: Learned Forward and Inverse ISPs using Camera Parameters. CoRR abs/2312.13313 (2023)
[3] Syed Waqas Zamir, Aditya Arora, Salman H. Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, Ling Shao: CycleISP: Real Image Restoration via Improved Data Synthesis. CVPR 2020: 2693-2702
questions: questions: Please refer to the weaknesses section.
limitations: limitations: 1/ Lack of new benchmark, new methods.
2/ Must compare with RGB SOTAs.
paper_summary: paper_summary: This paper proposes a novel approach to image signal processing (ISP) specifically tailored for object detection tasks, leveraging deep reinforcement learning to optimize both ISP structures and parameters. This method dynamically adjusts the ISP pipeline in response to different scene requirements, which enhances detection performance.
strength_weakness: strength_weakness: 1.	The system's performance heavily relies on the quality of the pre-trained object detection models. There is a potential limitation in cases where these models do not generalize well or when transitioning to different object detection tasks that were not part of the initial training set.
2.	The challenges and contributions are too general and not prepared objectively. It should briefly highlight the paper's novelty as what is the main problem, how has it been resolved and where the novelty lies?
3.	Although the paper conducted experiments on multiple datasets, the limited diversity and coverage of these datasets may not be sufficient to fully validate the performance of AdaptiveISP in various real-world application scenarios. For example, there is a lack of testing on the DAWN dataset, a dataset that covers multiple weather scenarios and is well-suited for dynamic testing.
questions: questions: The author mentions "Our method only takes 1.2 ms per stage during inference" , what does "per stage" mean, is it the end-to-end inference time for each image?
limitations: limitations: YOLOv3 is an older model, how does the method in this paper perform under newer YOLO models or other models? It is suggested that the authors add other models to the experiment.
paper_summary: paper_summary: This paper proposes AdaptiveISP, a task-driven and scene-adaptive ISP, which uses deep reinforcement learning to automatically generate an optimal ISP pipeline and associated ISP parameters, aiming to maximize the detection performance. Experimental results show that AdaptiveISP outperforms prior state-of-the-art methods for object detection, and it effectively manages the trade-off between detection performance and computational cost, demonstrating its effectiveness in dynamic scenes.
strength_weakness: strength_weakness: 1. The experiments are performed only on YOLO detectors. The conclusion and findings could be more solid and convincing if experiments are available on other architectures. 
2. It would be better to analyze and validate the generalization ability with some 3rd-party datasets.
questions: questions: 1. The tuning process could be costly when applied to real scenes since there are more complex factors in real scenes than in the experimental datasets. For example, the light condition can change rapidly in urban scenes during the nighttime. It is doubtful whether adaptiveISP can react to the change.  
2. Is it stable to tune adaptiveISP using RL? Will the tuning lead to even worse results than traditional ISPs? How can you evaluate the risk of your work in applications.?
limitations: limitations: See weakness
paper_summary: paper_summary: Image Signal Processors (ISP) are software pipelines that aim to improve images for their visual quality or application-dependent downstream tasks. This work presents AdaptiveISP, a method to simultaneously optimize an ISP pipeline, consisting of individual functions such as image sharpening or color correction and the functions' parameters. Learning policies that map an image into the optimal ISP structure and parameters while considering computation cost and the specific downstream task improves performance compared to the prior art and allows for real-time application, adapting to newly shot images in time.
strength_weakness: strength_weakness: - Since real-time applicability is presented as a key motivation for the paper, this work could benefit from an expanded evaluation of the method's computational flexibility. For example, Table 3 shows the average running time for two settings of the $\lambda_c$ parameter. It would be more informative to see the results for a range of values for $\lambda_c$ instead, not just showing that the expected impact is achieved but what the limitations and behavior of the system are when trying to tune for accuracy or speed. Similarly, one could experimentally consider how the method's time and memory demands change with varying pools of ISP modules to choose from.
questions: questions: - Figure 1 shows how different ISP modules are placed in the pipeline to modify the image's color values. Have you considered further calibration steps to be part of the adaptive pipeline, e.g., corrections for fish eye effects and other optical deficiencies?
limitations: limitations: The authors have adequately addressed the limitations.

Comments: title: After Rebuttal
comment: The rebuttal has addressed most of my concerns. I will retain my initial rating.
Thank you for your response. As discussed in our paper, the requirements for ISP differ significantly between high-level computer vision tasks and image quality tasks. Our primary focus is on optimizing ISP specifically for advanced computer vision tasks, which is critically important in scenarios such as autonomous driving.
The rebuttal has addressed most of my concerns. I will keep my initial rating. I strongly recommend the authors to add more discussions on image quality tasks, as pointed out by Reviewer Ln32, which is particularly important for an ISP.