AdaptiveISP: Learning an Adaptive Image Signal Processor for Object Detection
Authors: Yujin Wang
Xu Tian yi
Zhang Fan
Tianfan Xue
Jinwei Gu
Venue: NeurIPS 2024 poster (2024)
Keywords: Image Signal Processor
Image Processing
Reinforcement Learning
Primary area: machine_vision

Abstract: Image Signal Processors (ISPs) convert raw sensor signals into digital images, which significantly influence the image quality and the performance of downstream computer vision tasks. 
    Designing ISP pipeline and tuning ISP parameters are two key steps for building an imaging and vision system.
    To find optimal ISP configurations, recent works use deep neural networks as a proxy to search for ISP parameters or ISP pipelines. However, these methods are primarily designed to maximize the image quality, which are sub-optimal in the performance of high-level computer vision tasks such as detection, recognition, and tracking. Moreover, after training, the learned ISP pipelines are mostly fixed at the inference time, whose performance degrades in dynamic scenes. 
    To jointly optimize ISP structures and parameters, we propose AdaptiveISP, a task-driven and scene-adaptive ISP. 
    One key observation is that for the majority of input images, only a few processing modules are needed to improve the performance of downstream recognition tasks, and only a few inputs require more processing.
    Based on this, AdaptiveISP utilizes deep reinforcement learning to automatically generate an optimal ISP pipeline and the associated ISP parameters to maximize the detection performance. Experimental results show that AdaptiveISP not only surpasses the prior state-of-the-art methods for object detection but also dynamically manages the trade-off between detection performance and computational cost, especially suitable for scenes with large dynamic range variations.
Project website: https://openimaginglab.github.io/AdaptiveISP/.

AdaptiveISP: Learning an Adaptive Image Signal Processor for Object Detection
Yujin Wang
Tianyi Xu 1,2
Fan Zhang
Tianfan Xue 3,1
Jinwei Gu
Shanghai AI Laboratory
Peking University
(wangyujin, Bhangfan)@pjlaan org. en, photondstu. puu edu
3 The Chinese University of Hong Kong
(tfxuelie, jugu0cse). cuhk. edu. hk
Abstract
Image Signal Processors (ISPs) convert sensor signals into digital images, which significantly influence the image quality and the performance of downstream computer vision tasks. Designing ISP pipeline and tuning ISP parameters key steps for building an imaging and vision system. To find optimal ISP configura- tions, recent works use deep neural networks or ISP pipelines. However, these methods are primarily designed to maximize the image quality, which are sub-optimal in the performance of high-level computer vision tasks such as detection, recognition, and tracking. Moreover, after training, the learned ISP pipelines are mostly fixed at the inference time, whose performance degrades in dynamic scenes. To jointly optimize ISP structures and parameters, we propose AdaptiveISP. task-driven and scene-adaptive ISP. One key observation is that for the majority of input images, only few processing modules are needed to improve the performance of downstream recognition tasks, and only a few inputs require more processing. Based on this, AdaptivelSP utilizes deep reinforcement learning to automatically generate an optimal ISP pipeline and the associated ISP parameters to maximize the detection performance. Experimental results show that AdaptiveISP not only surpasses the prior statc-of-the-ed methods for object detec- tion but also dynamically manages the trade-off between detection performance and computational cost, especially suitable for scenes with large dynamic range variations. Project website: https:/openimmgiiglaagggbbbbbbb htw.abbithhuu.oo/daaiiiieee
two
proxy to search for ISP parameters
Introduction
Image Signal Processors (ISPs) play fundamental role in camera systems. Originally, ISPs aimed at enhancing the perceptual quality, focusing on photography-rellee relattd applications. Recently, machine
vision cameras also optimized the ISP pipeline for downstream recognition tasks [35
39 34 36).
For recognition tasks, studies have shown specially designed ISP can significantly enhance their performance. However, most machine vision cameras [13] still prefer static hand-designed ISP and manually-tunee parameters, making it sub-optimal for downstream recognition tasks and also inflexible for dynamic scenes. Another solution is to directly train detection networks that take raw files as input, skipping the entire ISP processing. However, this may require re-training the detection network for each camera sensor, as the raw format varies between cameras, and studies (23) also show that the raw detection network still performs worse than the detection network runs on ISP-processed images. Therefore, well-tuned ISP is important for downstream recognition tasks.
In this work, we aim to design an ISP that can dynamically adapt its pipeline and parameters for different inputs, tailored for given high-level computer vision task. This problem faces two challenges: complexity and efficiency. First, it is a complicated optimization problem to jointly
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

M
71.0
Boseting
Nocral Eygh
Surpoillt CUM Deeii GTanntaasiel Sinerion
Vinn Butieza O Ganra C Expearz Cressa
bonc 70.0
Module
Steges
Modile
Sagal
Stage
Sugrr
Sugel
VOLO-43
Model
ISP Pincinn
Figure 1: AdaptiveISP takes raw image as input and automatically generates optimal ISP pipeline {M,} and the associated ISP parameters {θ,} to maximize the detection performance for any given pre-trained object detection network with deep reinforcement learning. AdapativeISP achieved mAP@0.5 of 71.4 on the dataset LOD dataset, while a baseline method (35) with a fixed ISP pipeline and optimized parameters can only achieve mAP@0.5 of 70.1. Note that AdaptiveISP predicts the ISP for the image captured under normal light requires a CCM module, while the ISP for the image captured under low light requires Desaturation module.
re-organize the ISP modules, update their parameters, and also improve the performance of the downstream recognition module. Because of this complexity, previous works only update the parameters [35 25 30 31 38 [6 26). Some recent works jointly optimize the ISP pipeline and parameters [10 28), but they are not designed for downstream recognition tasks. Second, ISP optimization must be efficient enough to adapt dynamically changing scenes, which is particularly important in real-time applications such as autonomous driving and robotics. The majority ISP pipeline optimization methods searching strategies, such the neural architecture search (NAS) method (39 and the multi-object optimization search method [34]. which often takes several hours, making them infeasible for real-time applications across dynamically changing scenes.
To solve this challenge, we observe that the majority of input images only require a few ISP operations to increase the downstream recognition accuracy. As shown in Figure Ton on the right, only the first two stages of ISP already boost the detection accuracy (mAP) from 67.8 to 70.9, and the rest of the three stages only further boost it to 71.4. Only challenging examples require more complicated pipelines. Therefore, we can model the ISP configuration process as a Markov Decision Process [29]. and our AdaptivelSP only selects one ISP module at each stage, as shown in Figure I on the left. This greatly reduces the search space at inference time and enables adaptively changing the ISP length, spending less time easy inputs.
Based on this idea, we introduce AdaptivelSP, a real-time reconfigurable ISP based on reinforcement learning (RL). As shown in Figure AdaptivelSP takes linear image input and generates an optimal ISP pipeline with associated parameters that best this image for object detection. Unlike previous neural architecture search (NAS) [39] that generates the entire pipeline at once, AdaptiveISP takes a greedy approach to only generate one module at each iteration, greatly reducing the searching time. At each iteration, a lightweight RL agent takes the processing output from the previous stage as input and finds the optimal module for the next iteration. Because of its efficiency, AdaptiveISP only takes ms to predict one stage and can generate different pipelines for different scenes the fly in real-time, as illustrated in Figure [
Furthermore, we design reinforcement learning scheme tailored for ISP configuration. First, we integrate a pre-trained fixed object detection network, YOLO-v3 (32], into the optimization system as a loss function, guiding our model to prioritize specific high-level computer vision tasks. Second, with the observation that many later-stage ISP modules may only bring little improvement to detection, we introduce new cost penalty mechanism that AdaptiveISP supports dynamically trading off object detection accuracy and ISP latency.
The proposed AdaptivelSP have several distinctive properties compared to typical ISP for visual quality. First, ISP for detection can be much simpler. As shown on the right side of Figure with only 4 stages, it achieves the best detection accuracy, while traditional ISP may need more than 10 stages to enhance image quality. Second, while color processing is important in both types of ISPs for both detection and viewing, their behaviors are different. For instance, ISPs for detection often completely desaturate color in low-light scenarios to boost the detection rate, as shown the second-row of Figure which rarely happens for traditional ISPs. Third, simple sharpening or

---

blurring module may greatly enhance detection accuracy, while more widely used denoising module in traditional ISPs not super helpful for detection.
We have evaluated the proposed AdaptiveISP on the LOD [9] OnePlus 33 and synthetic raw COCO [9]] datasets and showed that it outperforms prior state-of-the-art methods under different challenging conditions and different downstream tasks. Experimental results also demonstrate the ability dynamically switch from high-accuracy ISP pipeline to low-latency one.
2 Related Work
ISP Parameter Tuning. Recent studies have explored various methods for optimizing Image Signal Processing (ISP) hyper-parameters, particularly those based on handcrafted designs and tailored to meet the demands of downstream evaluation metrics. One category of methods focuses on derivative-free optimization techniques. Nishimura et al. [26] introduced an automatic image quality tuning method that employs nonlinear optimization and automatic reference generation algorithms. Another category utilizes gradient-based optimization techniques. Tseng el al. (35) introduced a gradient optimization method that relies on differentiable approximations, allowing for efficient pprammer tuning but the ISP parameters are fixed during the inference stage. Immediately afterward, some researchers realized that one set of parameters was not necessarily suitable for different scenarios. Qin et al. [30] proposed an attention based CNN method, but it does not consider sequence-specific prior knowledge. Then, Qin et al. [3 proposed sequential ISP ryper-paramete prediction framework, which optimizes the ISP parameters by leveraging sequential relationships and parameter similarities. Additionally, Liu et al. adaptively process images under both normal and adverse weather conditions. Yoshimura et al. (38) proposed DynamicISP, which can causally and smoothly control the parameters of the current frame according to the recognition result of the previous frame. Departing from approximation methods, Mosleh al. (25) introduced hardware-in-the--oon approach. This approach directly optimizes hardware-basse image processing pipelines to meet specific end-to-end objectives by using a novel Oth-order stochastic solver. Furthermore, an image enhancement method based on reinforcement learning, proposed by Kosugi et al. [16), leverages reinforcement learning techniques to optimize hyper-parameters in an unpaired manner.
[22] proposed the IO-YOLO approach, which can
ISP Pipeline Design. The conventional ISP pipelines are crafted to adhere to human visual perception, and this alignment might not always be conducive fulfilling the demands of downstream high-
level tasks. The prior research works [34] [3] have proved that handcrafted ISP configuration
does not necessarily benefit the downstream high-level vision tasks. ReconfigISP [39 proposed a novel Reconfigurable ISP whose architecture and parameters can be automatically tailored to
specific data and tasks by using neural architecture search (Darts) (00) RefactoringISP (34] jointly optimized ISP structure and parameters with task-specific loss and ISP computation budgets through multi-objective optimization algorithm (NSGA-II) 4 However, these approaches maintain the ISP pipeline and parameters fixed during inference. regardless of the distinct characteristics of the input. An innovative study addresses the challenge photo retouching by leveraging deep learning on unpaired data, allowing users to emulate their preferred retouching style [10]. Additionally, recent studies have explored replacing traditional ISP pipelines or modules with deep learning models to enhance image quality. PyNET 111 proposed a unified model that directly learns the RAW- to-RGB mapping to improve mobile photography, while CycleISP [40] introduced a noise-aware denoising approach. ParamISP [14] developed forward and inverse ISP models conditioned on camera parameters, enabling a more accurate emulation of real-world ISP behavior to enhance image processing performance. DualDn [17 further employs differentiable ISP to improve denoising capabilities. Although these methods improve image quality, they do not consistently translate into better performance for downstream high-level vision tasks.
3
Our Method
Image Signal Processors (ISPs) often consist of pipeline of image processing modules that primarily transform raw sensor pixel data into RGB images suitable for human viewing 22] typical camera ISP pipeline includes two parts: raw-domain and RGB-domain. Compared to raw-domain processing, the RGB-domain processing normally image-dependent and requires more dedicated design and tuning efforts. Details of ISP pipeline and modules can be found in the appendix.
3

---

Input
F FMM, 0,)
Su
Mr
Seaz
ST
((s)) D(S.1)
Policy Network
(T,),, (=(41) State Value
Reward
D
Based on Actor-Critic Algorithm
YOLO-v3
Value Network
Figure 2: Overview of our method. The ISP configuration process is conceptualized Markov Decision Process, where a CNN-based policy network predicts the selection ISP modules and their parameters. Concurrently, a CNN-based value network estimates the state value. The YOLO-v3 [32 employed to calculate the reward for the current policy. The entire system is optimized using the actor-critic algorithm [5 24
In this work, we focus on the RGB-domain processing. We assume that the captured raw sensor data already converted to linear RGB images using simple static raw-domain processing, and our tuning mainly focuses sRGB-domaie processing, similar to [8]
3.1 Problem Formulation of AdaptiveISP
Given an input linear RGB image I, the goal of AdaptiveISP is find an optimal ISP pipeline and parameters, with which the processed output image will result in high performance for object detection. Let F(M,,) denote one ISP module, where M the ISP module type and the set of parameters for that module. An ISP pipeline consists of sequence of ISP modules {F (F) which transforms an input image I to an output image I, with stages of ISP processing as
I
(F (.) .... F(M).e.) (I).
(1)
The goal of AdaptivelSP is to predict an optimal ISP pipeline (with T stages) and its parameters, i.e., {M, Or for an input image, in order to maximize its performance for a given computer vision task (e.g. object detection), which can be formulated
[M, 0z) T
argmin (D(T)),
(M,
where 'D the detection with given object detector. This optimization problem can thus be modeled as Markov Decision Process 29 which can be solved efficiently via deep reinforcement learning.
3.2 Optimization Objectives
Our formulation similar to 5 state space and A is action space. Specifically, in our task, S is the space of images, which includes input images and all the intermediate results the ISP process, while A is the set of all ISP modules. Since each action includes the selection of ISP modules and the prediction of ISP module parameters, we can decompose the action space A into two parts: a discrete selection of ISP modules aM and continuous prediction of ISP module parameters a At stage with selected ISP module and its parameters (a." a the input image at state s; is mapped to state 8,41. Applying a sequence of T selected ISP modules
10 28
Let us denote the problem as P (S, A), where S is a
input image corresponds
trajectory of states and actions:
T T= (so, ap ag ...
8T-1, a a7
where 5T is the stopping state. Our goal is to find a policy स that maximizes the accumulated reward during the decision-making process. The policy consists of two sub-policies (T and 7 9), where π M takes a state and returns a probability distribution over ISP modules, and 77 predicts the parameters a of the selected ISP module. In this paper, the reward function with the i-th action (i.e., corresponding to the i-th stage of ISP processing) is thus written as:
r(s,, 0 a, D (s.) D (8141),
(4)

---

where 8++1 = p(s;, M a. and D is the error of object detection. Given a trajectory 7, we define the return 9t. as the summation of the discounted rewards after sç:
T =
r(stkk aitk a(t)),
(5)
where 7 E [0, 1] is discount factor that places greater importance rewards in the near future. We can thus define the value of state function V (s) as:
(*(
Loolsu s],
(6)
and the value of action function Q (s, aM , as:
(" (()) =
[ool00 8, 8.a aM aa..
Our goal to select policy л (M , )) that maximizes the expected accumulated reward during the decision-making process:
J(x) E [s)), S
(8)
where So denotes the entire image dataset.
we employ deep neural networks to approximate the value function (*(s) and the policy т. As shown in Figure 2 convolutional neural networks (CNNs) and a fully-connected layer are used as the policy network, which maps the image into action probabilities n (s, & (after softmax) and ISP module parameters π (s, Le) (after tanh), where (Mar &e) are the network parameters. The value network (T 2v) uses similar architecture with parameters dv. By maximizing the objective J(x) with training these two networks & = (2yr, & Lv), we can learn to predict the optimal policy π T(s) for input image s. Specifically, to train the policy network and the value network, as shown Figure 2 we apply the actor-critic algorithm [5] 24], where the actor is represented by the policy network and the critic is the value network. Details of network architectures and training are provided in the appendix.
Similar to 10 28
16 5
3.3 Implementation Details
To ensure stable and effective reinforcement learning for the ISP pipeline and parameter prediction, we augment the network input and implement several penalty functions to better constrain the training process. Specifically, modify the reward function Equation to:
r(s,, az a. = D (s;) - D (5++1) P..
where P3 stands for the penalties as explained below.
Exploitation and Exploration. To avoid the same ISP module being selected consecutively multiple times by the policy network, we augment the network input with the module usage record (which is represented as N channels where N is the size of the ISP module pool) and one additional "stage" channel. At each stage, if an ISP module is being used, the corresponding "use" channel and penalty of reusing will be set to and otherwise. The "stage" channel is set with the index of the stage. During training, these additional inputs and the penalty of reusing constrain the policy network picking the same ISP module no more than once, which can effectively narrow down the solution space and improve the performance. Details of network input are provided the appendix.
In addition, we want to encourage the policy network 7 <M(s (s) to explore different ISP modules to prevent the parameter prediction network from insufficient learning. Specifically, we introduce a penalty term Pe on the output entropy of the policy network x. (s) to ensure that the action distribution is not overly concentrated.
P. - d Σp(m) p(m) logp(m),
(10)
mEM
where p(m) the probability of an ISP module m in the softmax output, and X E [0, 1] is penalty coefficient and gradually decreases from I to 0 with the progress of the training process.
5

---

Penalty of Computational Time. ISP pipelines typically consist of multiple modules responsible for specific image processing tasks, such as denoising, sharpening. white balance, and more. These modules exhibit varying computational costs, some being faster and others slower, as shown in Table 4 Consequently, when designing ISP pipelines, is crucial to consider the distinct computational costs associated with these modules. Particularly, in scenarios requiring swift responses like autonomous driving, optimizing the computational efficiency of ISP pipelines becomes paramount. In order to make our method automatically select the appropriate modules and their sequence, we should account for the computational time of each module during the design the ISP pipeline. Specifically, we collect the time of all modules, marked Mc, which can be found in Table 4 and the penalty of cost Pe is defined as:
A Σ 4m MM.
(11)
MEM
where I. represents the one-hot encoding whether the module m is used, and Xa is the penalty coefficient.
4 Experiments
Datasets, In line with prior research 9 27 38
39
31I. we train and evaluate our models on 30
widely used real low-light detection datasets and synthetic normal-light datasets, including:
LOD. LOD Dataset [9] is real-world low-light object detection dataset, which contains 2,230 14-bit low-light raw images with eight categories of objects. This dataset aims to systematically assess the low-light detection performance. There are 1,830 data pairs for training and 400 data pairs for validation. Additionally, it provides accompanying metadata, including ISO, shutter speed, aperture settings, and more, which greatly facilitates our experimental analysis.
OnePlus. OnePlus Dataset 39 is real-world low-light object detection dataset collected by the OnePlus 6T A6010 smartphone at driving scenes, which contains 141 raw images with three classes of objects in the street scenes. There are 50 pairs for training and 91 pairs for validation. Given the limited size of the dataset, all raw images were utilized as the validation set in our experiments.
Raw COCO. COCO [9 is a large-scale object detection, segmentation, and captioning dataset. To evaluate the generalization ability of our method, convert the COCO validate dataset (5,000 images) to a synthetic raw-like dataset as our evaluate dataset by using UPI (I]
similar with 9 38 30
Experiment Details. Similar to [39. YOLOv3 32 is utilized as the detection model in all methods unless explicitly stated otherwise. YOLOv3 is robust and fast object detection algorithm. Its real-time applicability, speed, and high-performance capabilities make it a powerful choice for various applications in numerous classic object detection algorithms 4
In terms of back-propagation, YOLOv3 passes gradients back to our method for optimizing both structure and parameters. It is important to note that the pre-trained YOLOv3 model remains unaltered throughout the training process. During training and inference, we follow [ to use fixed input resolution of 512 512. In addition, unless otherwise stated, our methods and comparison methods are all experiments conducted on handcrafted ISP.
33
3 21
T8 5)
Evaluation Metrics. We utilize the mean Average Precision (mAP) across all Intersection over Union (IoU) thresholds to assess the performance of our method, following a similar approach as used in object detection algorithms 33
8 21 18 5].
4.1 Results
Results on LOD Dataset. The LOD dataset provides JPEG images with accompanying metadata, which serves as convenience for the analysis of our experimental results. On this dataset, we choose several baseline methods for evaluation. First, we select two network-based ISP methods, namely Crafting [2] and NeuralAF 277 Also, we evaluate our method against the static handcrafted pipeline and parameter methods, specifically, Hyperparameter Optimization (35). Furthermore, we compare approach with static handcrafted pipelines combined with dynamic parameters techniques, such

---

Table 1: Experimental results of LOD [9. OnePlus (39
and raw COCO Dataset 19]
LOD
OnePlus
Raw COCO
Methods
mAP
mAP
mAP
mAP
mAP
mAP
mAP @0.5
mAP
mAP @0.5:0.95
@0.75
@0.5:0.95
80.75
@0.5:0.95
@0.75
Crafting
67.9
49.0
44.7
NeuralAE
27
45.5
DSP
Hyperparaneter Optimization
35
70.1
49.7
46.1
69.8
48.7
43.8
53.8
38.7
36.6
Attention aware Leaminng
30
70.9
51.0
46.6
70.9
48.9
44.7
53.6
38.7
36.6
ReconfigISP
69.4
49.6
45.6
42.1
40.4
38.0
35.8
Refactoring ISP 34
68.3
47.6
44.1
66.7
44.3
40.9
52.4
38.0
35.7
Ours
71.4
51.7
47.1
70.1
49.7
45.0
54.9
40.1
37.7
MM
Hyperparameter Optimization
Attention-aware I. Leorning
RecoafigiSP
Refactoring
ISP
Ours
Figure 3: Object detection visualization results on LOD dataset. Our method outperforms the state-of- in terms of missed detection and false detection. The methods with
the-art methods 33 fixed pipelines or fixed parameters struggle to effectively handle varying noise levels and brightness scenarios.
30
39
as Attention-awaatttt aware Learning 30 and DynamicISP 38
Finally, we compare our method with the
optimization method for ISP pipelines and parameters, as exemplified by ReconfigISP (39 and
Refactoring ISP (3). ReconfigLSP utilizes neural architecture search algorithm (Darts) [20] while
Refactoring ISP is based on Non-dominated Sorting Genetic Algorithm (NSGA-II)
As shown in the first column of Table 1 our method achieves the best performance across all object detection metrics on the LOD dataset. Note that static pipelines with dynamic parameter methods outperform static pipeline and parameter approaches, dynamic pipelines and parameters yield superior results across all methods. We further show the detection results of our method in Figure 3 showcasing its superior performance in terms of both missed detection and false detection compared all other methods.
Cross Datasets Test. To evaluate the generalization ability of our method different datasets, utilize a model trained on the LOD dataset and conduct testing on both the OnePlus and raw COCO datasets. As displayed in the central and rightmost column of Table our method achieves the best performance, which verifies that our method also has the best generalization ability. In terms of the mAP@0.75 evaluation metric, our method exhibits an improvement of approximately 1 point compared to alternative approaches on both Oneplus and raw COCO datasets. Furthermore, our method outperforms Refactoring ISP 34 and ReconfigISP 39 by points on the raw COCO dataset and by 4 points on the OnePlus dataset in terms of mAP 0.0.5:0.95 evaluation metrics.
Cross Detectors Test. To evaluate the generalization ability of our method on different detectors, we use the detection results from the RGB (existing ISP) as a baseline and conduct comparative experi- ments on DDQ 41,, representing the Transformer-based approach, and YOLOX 77, representing the CNN-based approach. As shown in Table 3 all detectors using our AdaptivelSP demonstrate improved detection performance, demonstrating that our method does not overfit one detector, but is suitable for other detectors. Note that DDQ [41 and YOLOX [7] are not used in the training process, but our ISP can still generalize to these detectors at testing time.
Results for Image Segmentation. To show approach can generalize to other tasks, in Table 2 we also conduct an experiment using image segmentation as the downstream task. Note that all models are trained on the LOD dataset using the pre-trained YOLOv3 [32] detector and evaluated

---

Hyperparameter Optimization
Atteation Learning
RecoufigtSP
Refactering
Ours
Figure Image segmentation visualization results on raw COCO dataset. Our method detects all the object, while the state-of-the-art methods [35
30
39
may miss some.
Table 2: Image Segmentation results
COCO datasets 19
Methods
mAP@0.5
mAP@ 0.500995
Hyperparameter Optimization
35
46.4
28.4
Attention-awarc aware Learning
45.5
27.9
ReconfigISP 33
42.1
25.2
Refactoring ISP (34
40.6
24.7
Ours
47.0
28.8
on the synthetic raw COCO datasets with the pre-trained YOLO-v5 12] segmentor. Our method performs better than baselines, further showing its generalizability for different downstream tasks and algorithms. We further present the segmentation results of our method in Figure 4 highlighting its superior performance in terms missed detection compared all other methods.
Accuracy-Efficienee Trade-off. Our approach can control A₂ to regulate whether the optimization process also takes the computational time of each ISP module into consideration, where A is the weight computational cost. By tuning up Ao. we can generate a more efficient ISP with minor drop of the recognition accuracy.
We conducted analysis experiments on the LOD dataset, where A 0.0 represents the accuracy- oriented results and A = 0.01 stands for ffficiency-orientee results. We calculated the proportions of the occurrence of each module in the test results. As shown in Table { the efficiency-oriente method has significant reduction in the average running time for each sample, which is accompanied by slight decrease performance. In addition, the frequency of appearance of modules with higher computational time, such as Sharpen/Blur and Tone Mapping, decreased by more than 50%. Conversely, modules with lower computational time, like Exposure and White Balance, saw substantial increase in selection frequency.
Runtime. To verify the practicality of our approach, we conducted speed tests using the NVIDIA GTX1660Ti GPU, which offers a computational capability of TOPS -markely lower than that of the NVIDIA DRIVE Orin SoC at 254 TOPS. Our method only takes 1.2 ms per stage during inference, this efficiency attributed to just need to utilize the weight policy network to predict the modules and parameters during inference.
4.2 Ablation Study and Analysis
We perform study to show that different data require different ISP pipelines to achieve the best performance. To accomplish this, we begin by gathering the various ISP pipelines predicted by our method on the LOD dataset. Next, we select the three most representative ISP pipelines along with their corresponding input raw images, resulting in three subsets the LOD dataset. The specifics of these three distinct ISP pipelines can be found in Figure 5 Finally, we conduct cross-testing on the three sets of ISP pipelines generated by our method using the three that only the
Adaptive ISP.
subsets, as depicted in Figure 5 By analyzing the experimental results,

---

Table 3: Experimental results of different detectors LOD dataset. Note that the DDQ 41 and YOLOX [7] do not participate our training process. All detectors using our method demonstrate improved detection performance.
Detectors
Methods
mAP@0.5
mAP@0.75
mAP@0.5:0.95
RGB
55.6
40.9
37.4
YOLO-v3
32
Ours
71.4
$1.7
47.1
RGB
$7.0
43.5
39.2
YOLOX 7
Ours
69.7
51.4
47.2
RGB
50.3
42.0
35.9
DDQ
Ours
74.0
58.1
52.0
LOD
Subset
Subset
Subset 3
Representative Images
Metrics
mAP@00.5
mAP@00.7
mAP@0.55
mAP@0.75
mAP@0..5
mAP@0075
Pipeline
D-C-0-G-A
63.0
39.8
82.1
66.8
78.2
64.5
Pipeline
O-L+G~0-A
62.1
79.2
63.2
Pipeline 3 C-C-A-C-A
62.4
39.5
82.4
67.8
79.1
65.4
|Sharpen/Rlur
AWhite Balance
CCM
Tooee Mapping
Saturation
Desttraation
Figure 5: The cross-validated result of different ISP pipelines and its sub-dataset on LOD datasets. Only the most matching pipeline can achieve the best results, which proves that different pipeline is necessary.
most matching pipeline can achieve the best results, otherwise, it will lead varying degrees of performance degradation, which also proves that there are different pipeline requirements in different scenarios.
Module preferences of different images. Furthermore, we analyze the reasons why different ISP pipelines are necessary for various scenarios. The LOD dataset provides JPEG images with accompanying metadata, which serves as a convenience for the analysis of our experimental results.
It is noticed that there two different choices at the first stage of the three ISP pipelines: CCM and Desaturation. The High ISO (ISO6400, IS03200) and high noise level cases tend to favor the Desaturation module, whereas low ISO (ISO800, ISO1600) and low noise level cases tend to prefer the CCM module, as shown in Figure 5 Because Desaturation can reduce the color noise and saturation, high ISO images with high-level noise prefer it. CCM remove color casts and enhance color saturation. Therefore, is the best choice is CCM for low-ISO images.
Subsequently, we observe divergence between the second and third ISP pipelines during the third stage. While the second ISP pipeline opts for the Tone Mapping module, the third ISP pipeline favors the White Balance module. Upon the analysis of the images subsequent to the second stage, we observe a pronounced color cast problem, particularly prevalent when the overall brightness was relatively high. In such scenarios, the most suitable course of action is to opt for the White Balance module. Conversely, when dealing with images characterized by high dynamic range, such as those containing electric lights direct sunlight, it becomes evident that the superior choice is to employ the Tone Mapping module.
Finally, when summarizing the commonalities among the three ISP pipelines, several key conclusions emerge: The color correction module notably enhances detection performance (distinct from color correction for image quality tasks). However, optimal choices vary for images with different brightness and noise levels. ii) The Sharpen/Blur module holds significant position, either enhancing or blurring the image to align with the detection network. in) Tone Mapping also plays crucial role,

---

Table 4: Experimental results considering computational cost LOD dataset 9 A 0.0 represents the accuracy-oriented, A, = 0.1 stands for afficiency-oriennee The total time represents the average running time of each sample. The efficiency-orientee method has significant reduction in the average running time for each sample, which is accompanied by a slight decrease in performance. As D increases, our method tends favor faster-executing modules.
Exposme
Sharpes
Magping
Desatiaation
00.50005
Total
095
7516
10005
(79
10075
0.25%
77.355
10075
22.55
51.7
14.73
4.55
17g
55. 55.5%
1098
(9%-
98-35%
45.299
50 00.35%
10375
06.55
71.1
51.5
47.0
14.30
57.598
$6.25%
100%:
42.5%
(9%
43.35%
17.9%
2168
42.55%
1007
71.0
51.6
47.0
11.54
50 1008
75
100%
055
1316
49.75%
05.25%
1008
70.0
69.9
46.0
9.26
le 1008
0.75%
99.55
085
05
99.7536
10055
1008
08.9
30.1
45.9
9.39
Table 5: Comparison of object detection performance at different stages on the LOD dataset [
Our approach attains optimal performance with just two stages and dynamically achieves a trade-off between object performance and computational cost.
Methods
Stages
m.P@0.5
mAP@0.75
mAP@0.5:0.95
Hyperparameter Optimization (35
10
70.1
49.7
46.1
Attention-aware Learning (30]
10
70.9
51.0
46.6
ReconfigISP [39
5
69.4
49.6
45.6
Refactoring ISP
84
6
68.3
47.6
44.1
1
69.6
49.8
45.7
70.9
51.1
46.6
Ours
3
71.3
50.7
46.9
71.4
51.6
47.1
5
71.4
51.7
47.1
enhancing detection accuracy by adjusting the overall color and brightness. iv) Denoising, contrary to previous research conclusions, is not deemed crucial. This finding contributes to substantial computational cost savings for the ISP. These analyses and conclusions provide valuable insights for the future designs of ISPs tailored to specific downstream tasks.
Adaptive Trade-off. To demonstrate that model achieve adaptive trade-off between efficiency and accuracy during the inference phase, we use stages to represent the number of ISP modules. Recall that none of the previous algorithms support dynamic efficiency-accuraan trade- off. Hyper-parameter Optimization (35] and Attention-aware Learning (30 are optimized based on handcrafted ISP, so the time consumption cannot be changed during the inference phase. Although ReconfigLSP (39 and Refactoring ISP 34 have optimized ISP pipelines and parameters for specific tasks, the ISP pipelines and parameters remain fixed during inference. As shown in Table [5] our method only requires 3 stages to achieve the best performance on the LOD dataset and reaches good trade-off between efficiency and accuracy. Moreover, this trade-off happens inference time, without any retraining, and supports the dynamic update of the trade-off strategy.
Conclusion
In this paper, we introduce AdaptiveISP, a novel approach that leverages deep reinforcement learning automatically generate optimized ISP pipeline and associated parameters, maximizing detection performance with a pre-trained object detection network. Our method incorporates several key innovations. Firstly, formulate the ISP configuration process as a Markov Decision Process, allowing reinforeement learning to autonomously discover an optimal pipeline and parameters for specific high-level computer vision tasks. Secondly, to account for computational costs associated with different ISP modules, we introduce a penalty of computational time. Comprehensive experiments demonstrate that AdaptiveISP surpasses existing state-of the-art methods, and dynamically manages the trade-off between performance and computational cost. Furthermore, we conduct detailed analysis of individual modules within ISP configurations, offering valuable insights for future ISP designs tailored to specific downstream tasks. The present approach relies on the utilization of differentiable ISP modules in research, with future endeavors aimed at exploring non-differentiable ISP methodologies.
10

---

Acknowledgements
This work is supported by Shanghai Artificial Intelligence Laboratory and RGC Early Career Scheme (ECS) No. 24209224. We also extend our gratitude to Quanyi Li for his insightful discussions and valuable comments.
References
[1] Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron. Unprocessing images for learned raw denoising. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1.036-11043 2019.
[2] Michael S Brown and SJ Kim. Understanding the in-camera image processing pipeline for computer vision. In IEEE International Conference on Computer Vision (ICCV)-Tutorial,, volume 3, pages 1-354, 2019.
[3] Antoni Buades, Bartomeu Coll, and Jean-Michel Morel. Non-local means denoising. Image Processing On Line, 1:208-212, 2011.
[4] Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and TAMT Meyarivan. A fast and elitist multiobjective genetic algorithm: NSGA-II, IEEE transactions on evolutionary computation, 6(2):182-197, 2002.
[5] Ryosuke Furuta, Naoto Inoue, and Toshihiko Yamasaki. Fully convolutional network with multi-step reinforcement learning for image processing. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 3598-3605, 2019.
[6] Ryosuke Furuta, Naoto Inoue, and Toshihiko Yamasaki. Pixelrl: Fully convolutional net- work with reinforcement learning for image processing. IEEE Transactions on Multimedia, 2277):170411719 2019.
[7] Ge. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107.08430. 2021.
[8] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2961-2969, 2017.
[9] Yang Hong. Kaixuan Wei, Linwei Chen, and Ying Fu. Crafting object detection in very low light. In BMVC, volume 1, page 3, 2021.
[10] Yuanming Hu, Hao He, Chenxi Xu, Baoyuan Wang, and Stephen Lin. Exposure: white-box photo post-processing framework. ACM Transactions on Graphics (TOG), 37(2):1-17, 2018.
[11] Andrey Ignatov, Lue Van Gool, and Radu Timofte. Replacing mobile camera ISP with single deep learning model. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 536-537, 2020.
[12] Glenn Jocher, Alex Stoken, Jirka Borovec, NanoCodeO12, ChristopherSTAN, Liu Changyu, Laughing, tkianai, Adam Hogan, !orenzomammana. yxNONG, AlexWang 1900, Laurentiu Dia- conu, Marc, wanghaoyang01066 ml5ah, Doug Francisco Ingham, Frederik, Guilhen, Hatovix, Jake Poznanski, Jiacong Fang, Lijun Yu, changyu98, Mingyu Wang, Naman Gupta, Osama Akhtar, vetrDvoracek, and Prashant Rai. ultralytiss/oolov:: v3.1 Bug Fixes and Performance Improvements, October 2020.
[13] Hakki Can Karaimer and Michael S Brown. A software platform for manipulating the camera imaging pipeline. In Computer Vision 2016: 14th European Conference, Amsterdam, The Netherlands, October 1-14, 2016, Proceedings, Purt 14, pages 444. 444. Springer, 201
[14] Woohyeok Kim, Geonu Kim, Junyong Lee, Seungyong Lee, Seung-Hwan Baek, and Sunghyun Cho. Paramisp: learned forward and inverse isps using camera parameters. In 2024 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26067-26076. IEEE Computer Society, 2024.

---

[15] Vijay Konda and John Tsitsiklis. Actor-critic algorithms. Advances in neural information processing systems, 12, 1999.
[16] Satoshi Kosugi and Toshihiko Yamasaki. Unpaired image enhancement featuring reinforcement- caaming-controlles image editing software. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1299-11 1303, 2020.
[17] Ruikang Li, Yujin Wang, Shiqi Chen, Fan Zhang, Jinwei Gu, and Tianfan Xue. Dualdn: Dual Dual-domain denoising differentiable isp. arXiv preprint (rXiv:24091187.8 2024.
[18] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[19] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Purt 13, pages 740-755. Springer, 2014.
[20] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: 1806. 09055, 2018
[21] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yarne Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 21-37. Springer, 2016.
[22] Wenyu Liu, Gaofeng Ren, Runsheng Yu, Shi Guo, Jianke Zhu, and Lei Zhang, image-adaptive yolo for object detection in adverse weather conditions. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 1792-1800, 2022.
[23] William Ljungbergh, Joakim Johnander, Christoffer Petersson, and Michael Felsberg. Raw or cooked? object detection raw images. In Scandinavian Conference on Image Analysis, pages 374-385, Springer, 2023.
[24] Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforce- ment learning. In International conference on machine learning, pages 2016.
PMLR,
[25] Ali Moslch, Avinash Sharma, Emmanuel Onzon, Fahim Mannan, Nicolas Robidoux, and Felix Heide. Hardware-in-the-lee end-to-end optimization of camera image processing pipelines. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2020.
[26] Jun Nishimura, Timo Gerasimow, Rao Sushma, Aleksandar Sutie, Chyuan-Tyaa -ynn Wu, and Gilad Michael. Automatic ISP image quality tuning using nonlinear optimization. In 2018 25th IEEE International Conference on Image Processing (ICIP), pages 2471-2475 IEEE, 2018
[27] Emmanuel Onzon, Fahim Mannan, and Felix Heide. Neural auto-exposure for high-dynamic range object detection. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 7710-7720, 2021.
[28] Jongchan Park, Joon-Young Lee, Donggeun Yoo, and In So Kweon. Distort-and- Color enhancement using deep reinforcement learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5928-5936. 2018.
[29] Martin I. Puterman. Markov decision processes. Handbooks in operations research and management science, -334, 1990.
[30] Haina Qin, Longfei Han, Juan Wang, Congxuan Zhang, Yanwei Li, Bing Li, and Weiming Hu. Attention-aware learning for hyperparameter prediction in image processing pipelines. In European Conference on Computer Vision, pages 271 271-287. Springer, 2022.
12

---

[31] Haina Qin, Longfei Han, Weihua Xiong, Juan Wang. Wentao Ma, Bing Li, and Weiming Hu. Learning to exploit the sequence-specific prior knowledge for image processing pipelines optimization. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 22314-22323, 2023.
[32] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767, 2018.
[33] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[34] Yongjie Shi, Songjiang Li, Xu Jia, and Jianzhuang Liu. Refactoring ISP for high-level vision tasks. In 2022 International Conference on Robotics and Automation (ICRA), pages IEEE, 2022.
[35] Ethan Tseng, Felix Yu, Yuting Yang. Fahim Mannan, Karl ST Arnaud, Derek Nowrouzezahrai, Jean-François Lalonde, and Felix Heide. Hyperparameter optimization in black-box image processing using differentiable proxies. ACM Trans. Graph., 38(4):77-1, 2019.
[36] Chyuan-Tyng Wu, Leo F Isikdogan, Sushma Rao, Bhavin Nayak, Timo Gerasimow, Aleksandar Sutie, Liron Ain-kedem, and Gilad Michael. VisionISP: Repurposing the image signal processor for computer vision applications. In 2019 IEEE International Conference on Image Processing (ICIP), pages 4624 4628. IEEE, 2019.
[37] Ruikang Xu, Chang Chen, Jingyang Peng, Cheng Li, Yibin Huang, Fenglong Song. Youliang Yan, and Zhiwei Xiong. Toward raw object detection: A new benchmark and a new model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13384 13393, 2023.
[38] Masakazu Yoshimura, Junji Otsuka, Atsushi Trie, and Takeshi Ohashi. DynamicISP: dynamically controlled image signal processor for image recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 12866-12876, 2023.
[39] Ke Yu, Zexian Li, Yue Peng, Chen Change Loy, and Jinwei Gu. ReconfigISP: Reconfigurable camera image processing pipeline. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4248-4257, 2021.
[40] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming- Hsuan Yang, and Ling Shao. Cycleisp: Real image restoration via improved data synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2696-2705, 2020.
[41] Shilong Zhang, Xinjiang Wang, Jiaqi Wang, Jiangmiao Pang, Chenggi Lyu, Wenwei Zhang, Ping Luo, and Kai Chen. Dense distinct query for end-to-end object detection. Proceedings of the IEFF/CVF conference computer vision and pattern recognition, pages 1339-73338 2023.
13

---

A Additional Experiments
A.1 Experiments Details
We utilize the Adam optimizer with initial learning rate of 3e and batch size of 8. The learning rate gradually decreases by factor of X = 0.1 3iter/t//, Note that both the policy network and value network use the same initial learning rate. Our training comprises 100,000 iterations on one NVIDIA RTX 3090 (24G) GPU for the LOD dataset (9]. which is completed in around 24 hours. Additionally, we re- re-implement four methods, namely Hyperparameter Optimization (35), Attention-awant Learning [30), ReconfigISP (39), and Refactoring ISP [34, following the original papers. The ISP pipelines for these methods are illustrated in the RGB domain of Fig. 2 of the main paper. The results for Crafting [9]. NeuralAE 27 and DynamicISP [3 are obtained from their respective original papers.
The OnePlus dataset [39] comprises DNG files containing raw images and metadata. Given that our method operates on linear RGB, we employ "rawpy" to perform demosaicing operations on the original raw images, converting them into linear RGB images for the creation of our training and testing datasets.
The LOD 191. OnePlus [39], and raw COCO [19] datasets are commonly used in ISP research. The LOD dataset provides accompanying metadata, which greatly facilitates our experimental analysis. The OnePlus dataset is a real-world dataset collected by smartphones. The COCO dataset is well-known object detection and segmentation dataset. The ROD 37 dataset is a 24-bit HDR raw dataset collected by the SONY IMX490 sensor. The IMX490 sensor is rare in everyday life, therefore, we do not use ROD as our benchmark dataset.
A.2 Ablation Experiments on the Necessity of ISP
To validate the necessity of Image Signal Processing (ISP), we conducted a series of experiments on a raw COCO dataset [19). For these experiments, we randomly selected 5,000 images from the COCO training
our training dataset. We utilized a pre-trained YOLO-v3 model [ as our
starting model and trained it on the raw COCO dataset, setting the batch size to 128 and the learning rate to 0.01, with the training extending over 100 epochs. Additionally, we trained our method on the same dataset. It is important to note that within our methodology, the YOLO-v3 model (32) remained in frozen state. The final experimental results were compared on the raw COCO validation set, as detailed in Table 6 These results convincingly demonstrate that the inclusion of the ISP module is crucial and can significantly enhance detection performance.
Table 6: Ablation experiment of the necessity of ISP on raw COCO dataset 19
Methods
mAP@0.5
mAP@0.75
mAP@0..:0..5
Raw + YOLO-v3 32
34.5
22.9
21.8
Ours
56.2
41.2
38.6
A.3 Additional Experiments on ROD dataset
We conduct additional experiments on the ROD dataset. Note that the released ROD dataset differs from the one described in the published paper. Additionally, the released results (AP 28.1) 1) on the new version of the dataset are lower than those reported in the published paper, according to the open-source code released on GitHub, indicating that the released version is more challenging.
Because the released dataset is only training dataset that provides paired raw images and annotations, we randomly split 80% of the dataset (12,800) for training, with the remainder as our validation dataset (3200). The dataset processing pipeline is similar the original paper and released codes. Since our method emphasizes using training-well models, we selected only three categories (person, car, truck) belonging to COCO from the ROD dataset for a fair comparison. Due to time constraints, we select the previously best-performing method, Attention-Aware Learning [30), and the state-of- the-art method on the ROD dataset, Toward RAW Object Detection (37. as our comparison methods. Each method was trained for 100 epochs.
14

---

As shown in Table 7 our method achieves the best performance, even though the detector we used is trained on this input (Toward RAW Object Detection method 337 does).
Table 7: Experimental results on ROD dataset. refers to a detector that is trained on raw input, which is normally better than detectors only trained on RGB input (like ours). The "Toward RAW Object Detection" is an end-to-end raw detection method, that updates its parameters during training time. Other methods use pre-trained YOLO-v3 detector and freeze its parameters during training time.
Methods
mAP@0.5
mAP@0.5:0.. mAP@0.75
Attention-Aware Learning [30]
49.8
34.0
31.6
Toward RAW Object Detection 37
54.1
30.9
31.3
Ours
51.6
35.7
33.2
A.4
Ablation Experiments on Penalty of Reusing
Table 8: Ablation experiment of the penalty of reusing LOD dataset 9
Penalty of Rousing
Exposure
Gamma
CCM
Sharpen Blur
Denoise
Mapping
Tonc
Contrast
Saturation
Desaturation
White Balunce
mAP
605
2,25%
US
164%
139% 100%
0% 0%
46.5%
0% 0.25%
0%
99.5%
48. 75%
70.9
0%
0%
99.75%
100%
77.5%
100%
22.8%
71.4
To assess the impact of the penalty of reusing, we perform ablation experiments on the LOD dataset [ As shown in Table if the penalty of reusing is not applied, there is noticeable decline in detection performance, accompanied by an occurrence frequency exceeding 100% for both CCM and Sharpen/Blur modules.
A.5 Experiment on Limited Datasets
To validate the applicability our method with limited training data, we conducted experiments on the OnePlus dataset (39), which comprises only 50 training images. The results are presented in Table demonstrating that our approach outperforms ReconfigISP.
Table 9: Experimental results on limited datasets (OnePlus dataset
mAP@0.5
mAP@0.75
mAP0.5:0095
ReconfigISP 39
60.1 75.6
Ours
50.5
47.3
A.6
Comparison with Image Quality Task
To validate the distinct requirements of ISP between image quality tasks and object detection tasks, we select Exposure [10] as the representative method for image quality tasks. Given that the Expo- sure [10) method requires paired raw-RGB data, and computing object detection results necessitate bounding box labels, we utilize 1,000 simulated raw-like images with the most bounding boxes converted from the COCO training dataset [19] using UPI [I our training dataset. Additionally, we employ the LOD Dataset, comprising all real raw images, as our test dataset, similar to prior We train both our approach and the Exposure method the synthetic raw
works [9 38 30 31
COCO dataset.
As shown in Figure 6. image quality and object detection tasks have distinct requirements for ISP. Image quality tasks primarily emphasize color and brightness, intending to produce images that closely align with human perception. In contrast, the results obtained from processing images for target detection tasks better meet the demands of machine-based systems. We further compare object detection results with the image quality method [10] on the LOD dataset with all real raw images, as shown in Table TO the results of our method are much higher than the image quality method.
15

---

Image Quality Tank
Sngo Mappleg
Stage
Object Betection
Iepur
Stage Garana
Stage CCM
Stage 3 Time Moppleg
Stage Sharpoelller
Figure 6: Visualization results for image quality and object detection tasks on the raw COCO dataset. Image quality tasks and object detection tasks have distinct requirements for ISP.
Table 10: Comparison results with image quality methods all LOD datasets.
mAP@0.5
mAP@0.75
mAP@0.5:0.95
Hu et al. [T0]
54.3
35.4
33.7
Ours
62.1
42.1
39.6
A.7 More Visualization Results
Hy perpaarameter Optimization
Attention aware Learning
ReconligtSP
Refactoring
ISP
Ours
Figure 7: Object detection visualization results LOD datasets.
We show object detection and image segmentation visualization results in Figure Figure g showcasing its superior performance in terms of both missed detection and false detection compared all other methods.
Details of Implemented ISP
As shown in Figure a typical camera ISP pipeline includes two parts: raw-domain and RGB- domain. The raw-domain processing converts a raw sensor signal to a linear RGB image, which includes Defective Pixel Correction (DPC), Black Level Correction (BLC), Lens Shading Correction (LSC), and Demosaicking, The RGB domain processing further applies customized rendering and post-processing to generate the final image. This processing includes tone mapping, color correction. denoising, sharpening, etc. We focus on RGB-don (GB-domain processing, the ISP detailed in this paper encompasses various standard differentiable modules, including:
(1) Exposure: Exposure control regulates the amount of light reaching the sensor ensure the image's 's overall brightness matches the desired level. This module helps in adapting to varying lighting conditions. This module can be expressed as:
lesposure I 2"
(12)
where is the input image, is the exposure parameter and p 3-5.5,3.5..
16

---

Hyperparameter Optimization
Atfentlonsware Learning
ReconfigtSP
Refactoring
ISP
Ours
Figure 8: Image segmentation visualization results COCO datasets.
DLC
L.SC Rew Domain
Demoisaicing
Linear
--A---------------
O-A-0-
RGB Image
Image
RGB
RGB Domain
Sharpen Blur A White Dalance OOCM O Gamma O Denoise Eapposuee
L
Tone Mapping
O
Contrast
L Saturation Desaturation
Module Pool
Figure 9: typical camera ISP pipeline consists of different modules in the raw domain and RGB domain, which transforms raw sensor pixel data into RGB images suitable for viewing. In this paper, we focus on the ISP modules in the RGB domain.
(2) White Balance: White balance adjustment aims rectify color temperature discrepancies, ensuring that colors in the final image appear natural and consistent with the observed scene. This module can be expressed as:
Tw
PR
0
0
I
Ic
=
PG
0
Ic
(13)
0
PB.
[
where PR. PG, PB are gain value of input image TR, Ia, In and pr. PG: PB E [e-1/22, e/2)). (3) Color Correction Matrix (CCM): The CCM module fine-tunes color representations by mapping the sensor's 's color response to the desired color space. It plays pivotal role in achieving accurate color rendering, the original image data IR. Ig. IB is multiplied with CCM to obtain Ix, Ia IB
I
Poo
Po1
P02
[
Ic
P10
P11
P12
IG
(14)
L220
P21
P22.
[[_]
where p is the CCM parameter and the sum of elements each row equals
(4) Gamma: Gamma correction alters the image's tonal curve, affecting the distribution of lightness levels. This module is essential for ensuring luminance and contrast consistency. This module can be expressed as:
I Igamma I",
(15)
where is the gamma parameter and p E [1/3, 3).
(5) Denoise:
Denoise techniques are applied mitigate image noise, particularly low-light or high ISO settings, preserving image quality by reducing noise. In this paper, we implement differentiable Non-Local Means Denoising algorithm (NLM based PyTorch. This module can be expressed as:
Ldenoise NLM(I. p).
(16)
where E [0, 1] is filter-strength, which represents the strength of denoising.
(6) Sharpen/Blur: This module enhances blurs image details and edges, thus improving image clarity and visual appeal. It is useful for accentuating finer image structures. This module can be expressed as:
Lehorpen/blee /bbur =p. 1 p) Inturreds
(17)
17

---

where E [0, is the factor and Issurre is the blurred image, which can be obtained by
using the blur kernel, and the blur kernel can be expressed as: 1/13
5
1 1
Tone Mapping: Tone mapping regulates the dynamic range of image, ensuring that details in both bright and dark areas are discernible. It is especially valuable for rendering high dynamic range scenes on standard displays. Following [TO], we approximate curves as monotonic and piecewise-linear functions. We use L parameters represent tone mapping eurve, denoted as (Pu. P1, PL-11). With the prefix-sune of parameters defined as Pkc 1=0 the points on the curves are represented as (k/ L,, Pk/ PL). Given an input image I 0, the mapped result can be expressed as:
toope_napingg
1 PL
Σ
clip(L 1,, 0,0,1)pks 1)pk.
(18)
where the slope of each segment in the curve in [0.5, 2.0] and =
(8) Contrast: This module modulates the contrast of image, defining the variations between light and dark areas. contributes to image aesthetics and enhances perceptual quality. This module can be expressed as:
Icontrost (1-p)-I+p) {[1 (1-cos Itum)
(19)
4tm
where E [-1, 1] is adjustment factor, and Iuum can be expressed as:
Jrum 0.27 IR + 0.67 IG - 0.06 In-
(20)
(9) Saturation: Saturation adjustment manipulates the intensity of colors in an image, allowing for vivid muted color representation according to the desired artistic effect. This module can be expressed as:
(LH Is. Iv) (4,
Is = Is + (1 Is) (0.5 |0.5 Ivl) 0.8,
= HsvToRgb(In, 1st, Iv);
(21)
21 - cos(os(TT I'))
I saturation (1 p) p.
Iuam
where E [0, 1] is adjustment factor, and Ium can be obtained through Equation 20
(10) Desaturation:
The desaturation module, conversely, reduces the intensity of colors in an image, leading to a more muted grayscale appearance. It is often employed for specific artistic or visual effects, including creating black-and-whin imagery or subtle color emphasis. This module can be expressed as:
Idesaturation = (1-p) I (4tum Iunn 4um)
(22)
where [0, 1] adjustment factor, (Itum Itums Itum) represents R-G-B channels of the image are Iumns and Itm can be obtained through Equation 20
Additional Details of Method
C.1 Network Architecture
All of these networks share the same architecture depicted in Figure 10 while the extra channel (EC) represents different additional input information, more detail can be found in Fig. 4 of the main paper. For each feature extraction, we use four Conv-BN-LRee BN-LRelu layers and fully connected layer with 128 dimensions. Each Conv-BN-LRelu layer consists convolution layer with kernels and 2 x 2 strides, a batch normalization, and an LRelu activation of leak 0.2, the first layer has 32 channels, and this number progressively doubles the subsequent layers. The policy network includes dropout layer (with dropout rate of 0.5) preceding the fully connected layer
18

---

32
Channel:
Channel: the number modules
Activation: softmax
64x64x(3 + EC)
π
Channel: the number parameters
Feuture Extraction
Activation: tanh'sigmoid
EC: Extra Channel
Cony-BN-LRela
Conv: kemel sizem4; strid=22
IRRelu: leale-0.2
PC
Figure 10: The network architecture of the policy and value network in our method. The extra channel (EC) represents additional input that needs to be supplemented.
with 128 dimensions. The activation function for the module selection network is softmax, with the number outputs from the module selection network corresponding to the number of ISP modules. Parameter prediction networks share common feature extraction backbone. The activation function and the number of outputs from parameter prediction networks are specific to each module, reflecting variations in both the number and range of parameters for each ISP module.
C.2 Terminated and Truncated
(a) Input.
(b) Overexposure.
(c) Underexposure.
Figure 11:
Failure cases during the training stage. When the system predicts an excessively high gain value for the exposure module, the resulting image becomes overexposed, as illustrated in (b). Conversely, overly small gain value leads completely black image, depicted in (c).
The process of reinforcement learning be abstracted agent continually gathering experiences from the environment and learning from them. However, during the experience collection process, issues such termination and truncation significantly impact experience collection and training, necessitating special design considerations.
We can limit the total number of ISP configuration steps enable the policy network to autonomously learn the optimal module selection for downstream tasks under resource constraints. Specifically, the number of ISP module options in our ISP module pool is denoted as N, for the policy network selecting the modules more stably, we intentionally "terminated" once the execution stage reaches the maximum T (T N). Besides, introducing an additional channel for providing stage-related data, as shown the stage channel of Fig. 4 of the main paper. The terminated coefficient Atermnntted can be defined as:
incorporate step information into the policy networks by
(0, if T,
Aserminated
(23)
1, otherwise,
where is the number of ISP configuration stages, and is the maximum stages and T is 5 in our experiments.
Additionally, we intentionally truncate the execution when the network diverges to output abnormal images, as shown in Figure [I for example, "truncated" once the image mean gets out of range
19

---

Policy Network Iaput
Value Network Input
O. 09000
0.333
02.02
U.0AA
1000 0
0.3.3
02.02
0.444
1000
0.3
02203
0.0AA
64x64
Use (N) Stage
64x64
Use (M) Stage
64x64 Luminance Contrast Saturation
50
52
Figure 12:
The network imput of the policy and value network in our method. The number of use channels is the same the number of ISP modules (N). The stage channel represents the current stage. The extra 3 channels of value network input respectively represent the luminance, contrast, and saturation of the input image.
(lummins LLmmmoz). The truncated coefficient Atrancated can defined as:
Atruncated = [0, lurmmin I
Lumamnas
(24)
1 otherwise
where 7 the mean of processed image.
Ultimately, the state value function (*(s) can be defined as (*(s) = Aterminated) X (1 Atrancated) (s).
C.3 Design for Object Detection
ISP Modules. In the ISP pipelines for visual quality [00 16 28 the ISP modules primarily focused on pixel operations related color and brightness adjustment. To better adapt the ISP pipeline for object detection, following the traditional ISP modules, we extend the existing pipeline by incorporating differentiable Sharpen/Blur, Denoise, and CCM modules onto the foundation established in [10]. Empirically, we find these changes increase the detection performance.
Input Resolution. For object detection networks, larger pixel images are commonly employed for 18 5). To seamlessly integrate our policy network and detection network into end-to-end training, the input resolution to our ISP is configured based on the input resolution of the detection network. This ensures the back-propagation from the detection network the differentiable ISP and subsequently to the policy network. The policy network and value network can continue to utilize down- sampled input resolutions, such as 64 64 pixels, during training and testing, as illustrated in Figure [2] To enhance the value network's assessment of the current state, we include the average luminance, contrast, saturation of input image as additional input features.
training and testing 4 33 21
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect the contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Experimental details are provided.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [No]
Justification: Codes and datasets will be made publicly available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeeee //nips.com htte://alllllgiiiee///oeeeuaaass ssiniiiiccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Details are provided Section 4 and Appendix A.I Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: This follows the convention in ISP tuning and design research, the same as previous works.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on the computer resources is shown in Appendix A.1 A.1
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We strictly adhere to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no direct negative societal impact for ISP tuning and design.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited all original papers and make sure that usage is legal.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We construct experiments based public datasets and models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "0S5OgabNzD", "reviewer": "Reviewer_Ln32", "summary": "A new perspective of designing ISP pipeline. Good results with some problems should be addressed.", "questions": "Please refer to the weaknesses section.", "limitations": "1/ Lack of new benchmark, new methods.\n2/ Must compare with RGB SOTAs.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1/ One method for raw detection which is still a new subarea waiting more discovery.\n2/ Good performance compared with some methods.\n3/ Discuss some orders in ISP pipeline.", "weaknesses": "1/ Discussion on related works especially for ISP pipeline is not sufficient. Not only the ISP for task performance [1], but also for image quality [2,3].\n2/ The datasets used for comparison are not actually real raw detection data. It is doubtful whether the real scene performance is good.\n3/ Also, the compared methods are for designing special ISP params or orders to get better performance on  raw downstream tasks. However, what about using an existing ISP and detect on RGB images? For datasets such as LOD and COCO can do it. The results must be better than existing RGB detection SOTAs, so that this work can be meaningful.\n4/ The order of ISP modules is still not fixed. There have been many works finding that it can be various and change its order according to tasks or image quality. Also, the pipelines can be different according to manufacturers. \n5/ Some new benchmarks such as [1] should be used for evaluation.\n\n[1] Ruikang Xu, Chang Chen, Jingyang Peng, Cheng Li, Yibin Huang, Fenglong Song, Youliang Yan, Zhiwei Xiong: Toward RAW Object Detection: A New Benchmark and A New Model. CVPR 2023: 13384-13393\n[2] Woohyeok Kim, Geonu Kim, Junyong Lee, Seungyong Lee, Seung-Hwan Baek, Sunghyun Cho: ParamISP: Learned Forward and Inverse ISPs using Camera Parameters. CoRR abs/2312.13313 (2023)\n[3] Syed Waqas Zamir, Aditya Arora, Salman H. Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, Ling Shao: CycleISP: Real Image Restoration via Improved Data Synthesis. CVPR 2020: 2693-2702", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Related works, especially for ISP pipelines, are not sufficient.** In this paper, we primarily discuss how to design an ISP tailored for a specific high-level computer vision task. In most ISP tuning or design topics, the ISP consists of multiple modules with distinct roles. Using a network to mimic an end-to-end whole ISP or one of the modules of an ISP for computer vision or image quality tasks significantly differs from the objectives discussed in our paper. In addition, we would be happy to include these related works in the final version of our paper.\n\n2. **The datasets used for comparison are not real raw detection data.** The LOD and OnePlus datasets are real raw detection datasets collected in the real world. Since the demosaicking module lacks parameters and does not alter the number of channels, most ISP-related research tasks, including the mentioned paper \"Toward RAW Object Detection\", use post-demosaicking results as input.\n\n3. **Comparison with RGB (existing ISP) detection SOTAs.**  Please refer to part 1 of the Author Rebuttal.\n\n4. **New benchmarks.** Please refer to part 2 of the Author Rebuttal.\n\n5. **Many works are finding that ISP modules can change their order according to  computer version tasks or image quality.**  In the introduction and related work sections, we mention works such as [9, 23, 29, 33], which demonstrate that ISP modules can adapt their order for specific computer vision tasks or to enhance image quality. Specifically, methods [9, 23] optimize ISPs for image quality, method [29] focuses on object detection tasks, and method [33] addresses both image quality and object detection tasks.\n\n6. **The pipelines can be different.** This is one of the novelty of our method. As described in Section 4.2 of the main paper,  there are different pipeline requirements in different scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fRBRDIcCbO", "reviewer": "Reviewer_Sot8", "summary": "This paper proposes a novel approach to image signal processing (ISP) specifically tailored for object detection tasks, leveraging deep reinforcement learning to optimize both ISP structures and parameters. This method dynamically adjusts the ISP pipeline in response to different scene requirements, which enhances detection performance.", "questions": "The author mentions \"Our method only takes 1.2 ms per stage during inference\" , what does \"per stage\" mean, is it the end-to-end inference time for each image?", "limitations": "YOLOv3 is an older model, how does the method in this paper perform under newer YOLO models or other models? It is suggested that the authors add other models to the experiment.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe figures in this paper are of good quality and easy to understand.\n2.\tAdaptiveISP can dynamically adjust the ISP pipeline according to different input images to adapt to different scene changes.", "weaknesses": "1.\tThe system's performance heavily relies on the quality of the pre-trained object detection models. There is a potential limitation in cases where these models do not generalize well or when transitioning to different object detection tasks that were not part of the initial training set.\n2.\tThe challenges and contributions are too general and not prepared objectively. It should briefly highlight the paper's novelty as what is the main problem, how has it been resolved and where the novelty lies?\n3.\tAlthough the paper conducted experiments on multiple datasets, the limited diversity and coverage of these datasets may not be sufficient to fully validate the performance of AdaptiveISP in various real-world application scenarios. For example, there is a lack of testing on the DAWN dataset, a dataset that covers multiple weather scenarios and is well-suited for dynamic testing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **The performance of different detectors.** Please refer to part 2 of the Author Rebuttal.\n\n2. **Briefly highlight the paper's novelty.**  \n  \n     We aim to design the first adaptive ISP tailored for detection. There are two main challenges: complexity and efficiency. First, optimizing ISP modules, updating their parameters, and enhancing downstream recognition is a complex task, so prior works have only updated parameters. Second, ISP optimization must be efficient enough for real-time applications like autonomous driving and robotics, but most methods rely on searching strategies, making them impractical for dynamically changing scenes.\n  \n    To solve these challenges, we proposed three main innovations: (1) we model ISP configuration as a Markov Decision Process, integrate a pre-trained detector, and design a real-time reconfigurable ISP system based on reinforcement learning; (2) we introduce a new cost penalty mechanism, enabling AdaptiveISP  to dynamically trading off object detection accuracy and ISP latency; (3) we analyze the ISP pipeline predicted by our method to provide some insights for future ISP design work.\n  \n3. **The performance of AdaptiveISP in various real-world application scenarios.** We conduct additional experiments on real-world HDR raw datasets, our method achieves the best performance, please refer to part 2 of the Author Rebuttal. Since DAWN is not a raw image dataset, which may not fit into our setup, we only test the ROD dataset.\n\n4. **The meaning of \"per stage\".**  The ISP comprises multiple modules, such as exposure, white balance, and gamma correction. Each running ISP module represents a stage. We model the ISP configuration process as a Markov Decision Process, allowing our method to sequentially predict the ISP’s modules at inference time.\n\n5. **Performance on a newer detector.**  Please refer to part 1 of the Author Rebuttal.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kq4fiJ38nG", "reviewer": "Reviewer_z6yj", "summary": "This paper proposes AdaptiveISP, a task-driven and scene-adaptive ISP, which uses deep reinforcement learning to automatically generate an optimal ISP pipeline and associated ISP parameters, aiming to maximize the detection performance. Experimental results show that AdaptiveISP outperforms prior state-of-the-art methods for object detection, and it effectively manages the trade-off between detection performance and computational cost, demonstrating its effectiveness in dynamic scenes.", "questions": "1. The tuning process could be costly when applied to real scenes since there are more complex factors in real scenes than in the experimental datasets. For example, the light condition can change rapidly in urban scenes during the nighttime. It is doubtful whether adaptiveISP can react to the change.  \n2. Is it stable to tune adaptiveISP using RL? Will the tuning lead to even worse results than traditional ISPs? How can you evaluate the risk of your work in applications.?", "limitations": "See weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The AdaptiveISP pipeline combines ISP and detection together and turns the fixed process into a task-oriented tuning problem, which demonstrates greater potential for specific tasks.   \n2. The results of the algorithm on some datasets look good.", "weaknesses": "1. The experiments are performed only on YOLO detectors. The conclusion and findings could be more solid and convincing if experiments are available on other architectures. \n2. It would be better to analyze and validate the generalization ability with some 3rd-party datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. **Performance on a newer detector.**  Please refer to part 1 of the Author Rebuttal.\n\n2. **Generalization ability.** We utilized a model trained on the LOD dataset and conducted testing on the OnePlus dataset. As shown in Table 1 of the main paper, our method demonstrates the best generalization ability.\n\n3. **The tuning process is cheap.** Our method tunes an ISP pipeline in just 20ms (6ms for the tuning process + 14ms for ISP execution). We believe it can quickly adapt to rapidly changing light conditions, and the tuning process is efficient enough for ISP processing.\n\n4. **The stability of adaptiveISP.** We selected the worst 10% of samples from our method on the LOD validation dataset and tested them using the traditional ISP method. Our method achieved 35.6 mAP@50:95, while traditional ISPs only achieved 24.0 mAP@50:95, demonstrating that our method is stable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3fQ8OG9ZKd", "reviewer": "Reviewer_bosW", "summary": "Image Signal Processors (ISP) are software pipelines that aim to improve images for their visual quality or application-dependent downstream tasks. This work presents AdaptiveISP, a method to simultaneously optimize an ISP pipeline, consisting of individual functions such as image sharpening or color correction and the functions' parameters. Learning policies that map an image into the optimal ISP structure and parameters while considering computation cost and the specific downstream task improves performance compared to the prior art and allows for real-time application, adapting to newly shot images in time.", "questions": "- Figure 1 shows how different ISP modules are placed in the pipeline to modify the image's color values. Have you considered further calibration steps to be part of the adaptive pipeline, e.g., corrections for fish eye effects and other optical deficiencies?", "limitations": "The authors have adequately addressed the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- When creating novel ML methods, it is important to consider not only accuracy on pre-recorded datasets but also how these methods can be applied in the field, including adaptations of parameters and cost of computation. This work tackles these questions concerning image signal processing tasks, a significant endeavor for any robotic system equipped with a camera.\n- The presented method incorporates the specific downstream task, e.g., object detection, together with an adaptive trade-off for computation time, providing a comprehensive framework that has been presented with high clarity and incorporates a good level of originality, e.g., not aiming for visual clarity but seeking to obtain images best suited for the employed network.", "weaknesses": "- Since real-time applicability is presented as a key motivation for the paper, this work could benefit from an expanded evaluation of the method's computational flexibility. For example, Table 3 shows the average running time for two settings of the $\\lambda_c$ parameter. It would be more informative to see the results for a range of values for $\\lambda_c$ instead, not just showing that the expected impact is achieved but what the limitations and behavior of the system are when trying to tune for accuracy or speed. Similarly, one could experimentally consider how the method's time and memory demands change with varying pools of ISP modules to choose from.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback, we provide the results for a range of values for 𝜆𝑐, as shown in Table 3 on the global PDF, the efficiency-oriented method significantly reduces the average running time for each sample, with only a slight decrease in performance. As 𝜆𝑐 increases, our method tends to favor faster-executing modules. We will consider memory demands in the future. And we will consider including the calibration module as part of the adaptive pipeline in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their feedback. We will revise the manuscript as suggested. Below are responses to common questions. We hope this can address your concerns. If you have other concerns, we will reply as soon as possible.

1. **Additional experiments on different detectors and comparison with RGB (existing ISP) detection SOTAs.** (Reviewer Ln32, Sot8, z6yj)

    We use the detection results from the RGB (existing ISP) as a baseline and conduct comparative experiments on DDQ [1] and YOLOX [2] detectors. As shown in Table 1 of the global PDF, all detectors using our AdaptiveISP demonstrate improved detection performance, demonstrating that our method does not overfit one detector, but is suitable for other detectors. It is important to note that DDQ and YOLOX are not used in the training process, but our ISP can still generalize to these detectors at testing time.

    [1] Zhang, Shilong, et al. "Dense distinct query for end-to-end object detection." CVPR 2023.

    [2] Ge, Zheng, et al. "YOLOX: Exceeding yolo series in 2021." arXiv 2021.

2. **New benchmark or evaluation dataset.** (Reviewer Ln32, Sot8, z6yj)

      We conduct new experiments on the ROD dataset. As shown in Table 2 of the global PDF, our method achieves the best performance, even though the detector we used is not trained on this input (Toward RAW Object Detection method does).

      The LOD, OnePlus, and raw COCO datasets are commonly used in ISP research. The LOD dataset provides accompanying metadata, which greatly facilitates our experimental analysis. The OnePlus dataset is a real-world dataset collected by smartphones. The COCO dataset is a well-known object detection and segmentation dataset. The ROD dataset is a 24-bit HDR raw dataset collected by the SONY IMX490 sensor.  The IMX490 sensor is rare in everyday life; therefore, we do not use ROD as our benchmark dataset.

      Following the reviewers' suggestions, we also conducted comparison experiments on the ROD dataset. Note that the released ROD dataset differs from the one described in the published paper. Additionally, the released results (AP 28.1) on the new version of the dataset are lower than those reported in the published paper, according to the open-source code released on GitHub, indicating that the released version is more challenging.

    Because the released dataset is only a training dataset that provides paired raw images and annotations, we randomly split 80% of the dataset (12,800) for training, with the remainder as our validation dataset (3200). The dataset processing pipeline is similar to the original paper and released codes. Since our method emphasizes using training-well models, we selected only three categories (person, car, truck) belonging to COCO from the ROD dataset for a fair comparison. Due to time constraints, we select the previously best-performing method, Attention-Aware Learning, and the state-of-the-art method on the ROD dataset, Toward RAW Object Detection, as our comparison methods. Each method was trained for 100 epochs.

Meta-review: The paper presents an approach to image signal processing (ISP) tailored for object detection tasks, leveraging deep reinforcement learning to optimize both ISP structures and parameters. The work introduces AdaptiveISP, which dynamically adjusts the ISP pipeline in response to varying scene requirements, enhancing detection performance while managing computational costs. The reviews are mixed, with some highlighting the paper's potential and others expressing concerns about its limitations. The submission received mixed ratings, namely, one accept, two borderline accept, and one reject. For the reviewer voting for rejection, the reviewer did not provide feedback on author rebuttal. The AC checked all the materials, and believes the rebuttal is reasonable. Thus, the AC decides to accept the paper based on the review comments of the other three reviewers.

However, the concerns raised regarding the evaluation scope, related work discussion, and generalization need to be addressed in more detail. The paper would benefit from a revision that includes a broader evaluation across different models and datasets, as well as a more comprehensive discussion of related work.
Decision: Accept (poster)