RGB-Event ISP: The Dataset and Benchmark
Authors: Yunfan LU
Yanlin Qian
Ziyang Rao
Junren Xiao
Liming Chen
Hui Xiong
Venue: ICLR 2025 Poster (2025)
Keywords: event camera
image signal processor
color correction
denoising
Primary area: datasets and benchmarks

Abstract: Event-guided imaging has received significant attention due to its potential to revolutionize instant imaging systems. However, the prior methods primarily focus on enhancing RGB images in a post-processing manner, neglecting the challenges of image signal processor (ISP) dealing with event sensor and the benefits events provide for reforming the ISP process. To achieve this, we conduct the first research on event-guided ISP. First, we present a new event-RAW paired dataset, collected with a novel but still confidential sensor that records pixel-level aligned events and RAW images. This dataset includes 3373 RAW images with $2248\times 3264$ resolution and their corresponding events, spanning 24 scenes with 3 exposure modes and 3 lenses. Second, we propose a convential ISP pipeline to generate good RGB frames as reference. This convential ISP pipleline performs basic ISP operations, e.g., demosaicing, white balancing, denoising and color space transforming, with a ColorChecker as reference. Third, we classify the existing learnable ISP methods into 3 classes, and select multiple methods to train and evaluate on our new dataset. Lastly, since there is no prior work for reference, we propose a simple event-guided ISP method and test it on our dataset. We further put forward key technical challenges and future directions in RGB-Event ISP. In summary, to the best of our knowledge, this is the very first research focusing on event-guided ISP, and we hope it will inspire the community.

Published as a conference paper at ICLR 2025
RGB-EVENT ISP: THE DATASET AND BENCHMARK
Yunfan Lu, Yanlin Qian, Ziyang Rao, Junren Xiao, Liming Chen', Hui Xiong" AI Thrust, Hong Kong University of Science and Technology (Guangzhou); AlpsenTek? ylu06f@ccnnect. hkust -za gz .ddu. en, qianyan1in6 iel.iinnn11111111 gma com (zrac538, 3xiao767/8connect hwt.:/ecnnnncccuuttteeeeeeeee www.ttgz..ttc...... edu. 1 wew.ing.shneaaassseete com, xionghuieust.
ABSTRACT
Event-guided imaging has received significant attention due its potential rev- olutionize instant imaging systems. However, the prior methods primarily fo- cus on enhancing RGB images in a post-processing manner, neglecting the chal- lenges of image signal processor (ISP) dealing with event sensor and the bene- fits events provide for reforming the ISP process. To achieve this, we conduct the first research on event-guided ISP. First, we present a new event-RAW paired dataset, collected with a novel but still confidential sensor that records pixel-level aligned events and RAW images. This dataset includes 3373 RAW images with 2248 x 3264 resolution and their corresponding events, spanning 24 scenes with 3 exposure modes and 3 lenses. Second, propose conventional ISP pipeline generate good RGB frames as reference. This conventional ISP pipleline performs basic ISP operations, e.g. demosaicin, white balancing, denoising and color space transforming, with a ColorChecker as reference. Third, we classify the existing learnable ISP methods into 3 classes, and select multiple methods train and evaluate on our new dataset. Lastly, since there is no prior work for reference, we propose a simple event event-guided ISP method and test on our dataset. We further put forward key technical challenges and future directions in RGB-Event ISP. In summary, to the best of our knowledge, this is the very first research focusing on event-guided ISP, and we hope it will inspire the community. The code and dataset are available at: https: //gitt//ttti gighub. com/ funfaanuu/////cc L Eventt-e-t--. ISP.
INTRODUCTION
Since their invention in 1975, digital cameras have profoundly impacted various aspects of mod- emn society (Delbracio al., 2021; Kyung et al., 2016). Active pixel sensors (APS) (Liebe et al., 1998) are used as the core of cameras to capture RGB color signals, recording images or videos. This technology forms the foundation for widespread applications smartphones (Delbracio al., 2021), autopilot systems (Ingle & Phute, 2016), drones (Zhu et al., 2018), virtual reality (Huang et al., 2017). and more. However, nowadays APS has reached a bottleneck wrt. power consump- tion, frame rate, and dynamic range due to its global recording characteristics (Gallego et al., 2020). Event vision sensors (EVS), with their inherent asynchronous recording property, achieve lower power consumption (< 10mW), lower latency (< Lms), and higher dynamic range (> 120dB) (Gallego et al., 2020). As a result, integrating EVS as a significant enhancement to APS imaging system has received considerable attention in recent years (Lu et al., 2023b; Tulyakov et al., 2021; Gallego et al., 2020; Tulyakov et al., 2022). Heavy efforts have been put on developing new imag- ing system combining EVS and APS (Shariff al., 2024; Lu al., 2023b:a). The introduction of EVS has nearly reshaped the entire framework of imaging formation and enhancement, impacting almost all relevant areas e.g., video super-resolution (Lu et al., 2023b; Jing et al., 2021). video frame interpolation (Tulyakov et al.. 2021; 2022; Lu et al., 2023a), deblurring (Yuan et al., 2007; Zhang al., 2022; Yunfan et al., 2023), high dynamic range imaging (Xiaopeng al., 2024; Messikom- mer et al., 2022), liw-light image enhancement (Wang et al., 2020b; Liang et al., 2024), and rolling shutter correction (Zhou et al., 2022: Lu et al., 2023a). However, the majority of previous work fo- cuses on using events auxiliary information to boost the performance of classical RGB imaging
"corresponding author

---

Published as a conference paper at ICLR 2025
RAW
Mation
(6) RAW
(b) Events
(c) RGB Frame
(d) ISP
(e) Event-guided ISP
Figure 1: (a), (b), and (c) display RAW, Events, and RGB frame captured by the hybrid vision sensor (HHS), respectively. The RAW image follows quad-Bayer pattern (Yang al., 2022), while the events positioned the lower-right corner each color pixel block, making the RAW resolution twice that of the events. (d) illustrates the traditional ISP process. (c) shows the potential event- guided ISP process, where the higher temporal resolution of events can captures motion information for ISP.
systems, while methods and benchmarks that considering the challenges and opportunities of events the APS ISP process, are lacking.
Merging APS and EVS in ISP is non-trivial on the implementation level. Prism spectrometer is an early stage attempt and it needs the corresponding optical mechanic setting (Tulyakov al., 2022). However, this miism-based approach is very cumbersome, requiring additional optical prisms and failing to ensure the alignment between APS and EVS. Sensors that integrate both APS and EVS on the photodiode level are referred to hybrid-vision sensors (HVS) (Yaqi et al., 2024; MIPI Chal- lenge 2024, 2024), which represent cutting-edge' technology, offering significant advancements in camera imaging. Due the manufacturing complexity and eπror- error-prone design process of HVS, the RAW data generated by APS in HVS exhibits higher noise, missing values at fixed positions, and is more sensitive to defects (MIPI Challenge 2024, 2024; Yaqi et al., 2024). Recent works have acknowledged this challenge and proposed datasets for demosaicing, denoising, or defect correc- tion for APS RAW, where the challenges in APS of HVS take precedence over the potential benefits events signal could provide. With the inherent higher dynamic range and lower latency, events can perceive a broader spectrum and capture more-instant motion information (Shekhar Tripathi al., 2022; I Liang et al., 2021), allowing significant potential for boosting the denoising and color correc- tion of ISP processing of APS RAW, as shown Fig.
To better explore the benefits of events on the ISP process of HVS, we propose a new dataset with pixel-wise aligned events and APS RAW image. This dataset uses the unveredeeelopmmen HVS- ALPIX-Eiger sensor (Alpsentek, 2024), which rearranges event and APS in quad-Bayer pattern (a quarter photodiodes are dedicated for event, as in Fig. 1). This sensor has a high resolution with 1224 x 1632 for events and 2248 3264 for RAW, and offers superior color and noise profiles compared to the DVS346 (Scheerlinck et al., 2019). These features make it promising for various applications (Lu et al., 2023b). We ensure the dataset diversity in ways: photographic setting and scenes. For photographic setting, we adopt various values of aperture, focal length and exposure time. For the scene diversity, we cover 12 categories of scenes, across wide range of color scenes, including flowers, buildings, under different weather and lighting conditions. In total, 3373 APS frames and the corresponding events are captured. A standard 24-color ColorChecker (Goto al., 2003) is applied at certain frames as the color correction reference, as shown in Fig. 1 (c).
To generate the ground truth RGB images for the dataset, we propose controllable ISP framework based on MATLAB (Poon & Banerjee, 2001). This ISP framework, using the ColorChecker as prior, performs tasks such as black level calculation (Li et al., 2010), demosaicing (Hirakawa & Parks, 2006), white balance (Weng et al., 2005), denoising (Abdelhamed et al., 201 8), and color cor- rection (McElvain & Gish, 2013), resulting in high-quality RGB images with controllable errors as the reference ground truth. Since the controllable framework requires the ColorChecker information as a prior, it cannot generalize to arbitrary scenes. The color accuracy and temporal stability of this ISP are also analyzed. We categorize the existing ISP methods with RAW input into three categories and benchmark their performances on our dataset. We compare their performances across various scenarios and further conduct analysis certain phenomena we have observed. Additionally,
2

---

Published as a conference paper at ICLR 2025
propose simple UNet-like (Ronneberger al., 2015) event-guided ISP neural network fuse events with RAW images. This simple network can effectively improve the outdoor performance of ISP compared to the original UNet (Ronneberger et al., 2015). We also identify key contributions and challenges of events in the ISP process, providing a foundation and direction for future research.
RELATED WORKS
Event-guided Imaging Datasets: Event camera- cannerr-guieed imaging enhancement is an emerging field where the contribution of real datasets is crucial (Gallego et al., 2020). Currently, event cam- eras have made significant progress in areas such as frame interpolation (Tulyakov et al., 2021; Lu et al., 2023a; Niklaus al., 2017; Bao al., 2019), video super-resolution (Lu et al., 2023b; Jing et al., 2021), low-light enhancement (Liang et al., 2024; 2023), and deblurring (Xu et al., 2021; Lin et al., 2020; Jiang et al., 2020). These advancements are supported by many founda- tional datasets (Tulyakov et al., 2021; Scheerlinck et al., 2019; Lu et al., 2023b). For example, BS-REGB (Tulyakov et al., 2022) is a frame interpolation dataset using a beamsplitter to pair event cameras and RGB cameras. The CED (Scheerlinck et al., 2019) dataset and APLEX-VSR (Lu et al., 2023b) dataset have been used in research on event camera-guided video super-resolution. Overall, these datasets serve as the cornerstone and pioneers in research on related tasks. However, these datasets assume that event cameras can obtain high-quality RGB images through the ISP process, an assumption that is often too idealistic. Recognizing this, the MIPI (Yaqi et al., 2024; MIPI Challenge 2024, 2024) challenge introduced a RAW demosaic dataset for HVS in event cameras, addressing challenges like high noise and missing values in RAW from HVS. Although this dataset is the first to focus on the RAW domain ISP process in event cameras, it lacks real event streams, thereby overlooking the potential role of events in the ISP process. To address this gap, we propose the first dataset with aligned RAW and events from new HVS, aiming at exploring the potential value and role of event data in the ISP process.
Learning-based ISP: Traditional ISPs (Schwartz et al., 2018) consist of long pipelines. In re- cent years deep learning has brought new insights to ISPs (da Silva et al., 2023a) and has achieved higher performance. These methods can be roughly categorized into three types. The first type is full pipeline replacement methods, such as PyNet (Ignatov al., 2020b) which use CNN architec- tures to replace the entire ISP pipeline. The second type is stage-wise enhancement methods, like CameraNet (Liang et al., 2021) and AWNet (Dai et al., 2020), which divide the ISP pipeline into restoration and enhancement stages. The third type is image enhancement network- based methods, which utilze state-of-the-un image proessing backbone models such as UNet (Ronneberger et al., 2015) and Swin-Transformer (Liu al., 2021) to deal with ISP tasks. Though these methods have proven effective for RAW to RGB conversion, the potential of events in this process is not explored.
Event-guided Image/Video Enhancement: Due to their high dynamic range and high temporal res- olution (Gallego et al., 2020; Shariff et al., 2024), event cameras have garnered significant attention in the field of image/video enhancement and restoration (Gallego et al., 2020; Shariff et al., 2024), including many applications. Initially, the use of events focused primarily on single-task enhance- ments of RGB images or videos (Tulyakov et al., 2021: Pan et al., 2019; Lu et al., 2023b). Recently, researchers recognized image enhancement tasks are inherently coupled with various degradations interwoven (Zhang & Yu, 2022; Song et al., 2022; Yunfan et al., 2023), suggesting a trend towards using events for unified solutions in camera computational imaging for multiple tasks. However, existing methods focus solely on enhancing RGB images or videos using events, overlooking the ISP pipeline, which generate RGB images from RAW images. Additionally, existing methods neglect the potential value that events could provide in the ISP process.
DATASET COLLECTION
As the first dataset, which we call HVS-ISP Dataset, featuring paired raw-event data collected using a HVS, our aim is to facilitate research event-guided RAW ISP. We selected the HVS-Eiger sensor developed by ALPIX (Alpsentek, 2024), which can output both APS and EVS signals that align in both time and space, show in Fig. 2 (b). More parameter details of APS and EVS are shown in Tab. 1. Compared to the Prophesee sensor (Tulyakov et al., 2021), which can only output event signals, and the DV6346 sensor (Scheerlinck et al., 2019), which has lower resolution (260 346)
3

---

Published as a conference paper at ICLR 2025
APS Frames Checker Postion
Treu
Flower
Event Visin Sensor
Rish
Resolution Chart
Hybrid Vision Sensor
Events
(a) Examples of Dataset Scenes
(b) Camera
Examples of Our Dataset
Figure 2: Overview of dataset collection. (a) illustrates the variety of scenes in the dataset, including buildings, plants, animals, and calibration boards. (b) presents schematic of the HVS sensor, composed of a stacked active pixel sensor (APS) and an event vision sensor (EVS). (c) displays dataset samples.
Table I: Comparison between active pixel sensor (APS) and event vision sensor (EVS) (Alpsentek, 2024) in dataset collection. APS and EVS are stacked together to form hybrid-vision sensor (HVS).
Frame Rate
Power Consumption
Redundant
Dynamic Range
Sensor
Resolution
Data Rate
APS
2248
3264
10~60 fps
100 mW ~0 mW
10 MB/s
60 dB 120 dB
EVS
1124 1632
2 800 fps
40-180 KB/s
and higher noise, our choice offers significant advantages. Hence our dataset, captured with this advanced new sensor, holds significant value for the event vision research, providing a foundation resource for advanced exploration in event event-guided RAW ISP.
The collection of the dataset focuses on two main aspects: (1) the diversity of the dataset, ensuring it has broad representativeness to cover a wide range of real-world scenarios; (2) the inclusion of ColorChecker for ISP calibration, which helps the ISP accurately restore scene colors to generate high-quality RGB frames as references.
(1) Dataset Diversity: constructing our dataset, we paid particular attention two types of di- versity: camera parameter diversity and scene diversity. Camera Parameter Diversity: To ensure that our dataset encompasses variety photographic conditions, we made extensive adjustments to the camera parameters. This included aperture values ranging from F1.0 to F6.0, focal lengths extending from 8mm to 52mm, and exposure times varying from Ims to 100ms. Scene Diversity: diversity: Light Source Diversity:
We focused on three key aspects to ensure comprehensive We distinguished between indoor artificial light and outdoor natural light, with special consideration for different weather conditions. Data collection was performed under various lighting conditions, including sunny and cloudy days. Motion Diversity: We captured both dynamic and static videos, ensuring a mix of scenes with and without motion blur. This variety helps in testing and enhancing the performance of image processing algorithms under different motion conditions. Material Diver- siry: We included wide array of scenes such trees, plants, buildings, fish, dolls, and more. These scenes exhibit broad spectrum of colors and textures, providing a rich dataset for comprehensive testing and improvement of image processing techniques.
(2) ColorChecker as ISP Reference: To ensure precise color correction and white balance in ISP pipeline, we utilized standard 24-color ColorChecker (Tian et al., 2002) as critical references. At the start of each video shoot, we captured frames containing the ColorChecker and gradually removed the chart from subsequent frames. We meticulously annotated the position of the Col- orChecker in each frame using the LabelMe tool (Russell al., 2008), shown in Fig. (c). For frames without the ColorChecker, we applied previously determined ColorChecker parameters as references. This approach guarantees reliable color correction data in dataset. Incorporating the
4

---

Published as a conference paper at ICLR 2025
(e) Quad bayer pattern
(b) Black pattern, Fixed-pattem
Demosacing
(d) White Balancing
[el Denoise
( Color space transform
Figure 3: Flows in controllable ISP process. (a) Quad-hayer pattern raw image, which serves as the initial input. (b) Black pattern and fixed-pattem noise removal to suppress sensor-induced artifacts. (c) Demosaicing to reconstruct a rgb image from the raw data. (d) White balancing using ColorChecker for accurate color reproduction. Denoising filter out spatial noise from the image. (f) Color space transformation and Gamma convert image into desired color space for final output.
ColorChecker allows generating high-quality RGB values, enhancing color fidelity. This method ensures robustness for applications requiring accurate color restoration. Additionally, conducted a thorough manual review of the ColorChecker annotations to validate their accuracy, further im- proving dataset's reliability for ISP algorithms.
In summary, based on these two main objectives, we captured a total 24 videos. Each video contains 80 140 frames, resulting in total of 3373 APS RAW frames and their corresponding events. Additionally, the dataset includes the positions of the ColorCheckers within the APS images. We divided the dataset into training and test sets, with 3/4 of the data used for training and 1/4 for testing. The testing set includes 3 indoor scenes and outdoor scenes to ensure sufficient diversity. For more details on data collection and visualizations, please refer the supplementary material.
CONTROLLABLE ISP
The controllable ISP aims provide mrodule-based and analytically mea- surable RGB frames based on the APS RAW. With the support of the contained ColorChecker, the result- ing frames have good color accuracy and low noise, serving as the refer- ence for APS. Requirement of the ColorChecker prevents from gener- alizing to other arbitrary scenes. In this section, we introduce each mod- ule, followed by quality evaluation and coos-and-cons discussion, with the hope that this ISP pipeline will be beneficial for the community.
0.95
0.85
0.00
0.70
0.65
(8) (FPN)
(b) w/o
(c) w
Figure 4: Fixed pattern noise (FPN) removal. (a) Visualizes the camera's 's fixed pattern noise. (b) and (c) show the RGB images without and with fixed pattern noise removal, respectively. The age (c) demonstrates lower noise and more accurate white bal- ance after the removal of fixed pattern noisc.
4.1 CONTROLLABLE ISP PIPELINE
Fig. 3 depicts that how image is processed via a conventional ISP pipeline, making the reference for the APS data. Level and Fixed Pattern Subtraction: Taking arbitrary unprocessed bayer calibrated global black level value ble is subtracted, following by subtracting a fixed pattern vector fpn 1. blc the min of raw image taken under pure-black environment while fpn vector that records the per-row average value as the used sensor is only with horizontal fixed pattern, as shown
(1) Black input, pre-
blc and for calibrated in pure-dark laboratory setting. Over five frames captured and averaged to increase the calibration accuracy.

---

Published as a conference paper at ICLR 2025
C EDE00 Error Probability Density Distribution
CIEDELah Error Prbbbillity Density Distribution
Time-Domain Fluctuations of RGB Color Values
1.40
Aog iour
Aug Enor
And
140
1.5
123
Probeah by Deeetyy
Aug 5.84
Prduubiity Danuty
0.25
Med Mean: 7.00
2.20
Med Meari
0025
Probubiiity
155
0.28
43
Maz 24.52
22
Flccuutioon 0,322
Error Value
(a) CIEDEOO Error
(b) CIELAB Aab Error
(c) Time Domain Fluctuations
Figure 5: Color errors and fluctuations our ISP method, computed using a ColorChecker. (a) CIEDE 2000 Error Probability Density Distribation: Displays CIEDE 2000 error values distribution with annotations for average (5.84), median (5.07), and maximum error means (14.62). (b) CIEDE Lab Error Probability Density Distribution: Shows CIEDE Lab error values distribution, indicating average (8.99), median (7.0), and maxi- mum error means (24.37), (c) Time-Domain Fluctuations RGB Color Values: Illustrates RGB color values fluctuations over time, representing temporal stability and variations in color accuracy.
in Fig. 4. (2) Demosaicing: Given bayer pattern, the well-adopted demosaicing method (Rainbow- Bechnny-lmage-rreeeee
2022) is used. The resolution is preserved while the channel number is tripled. Note that this method is still prone to generating false color in very high frequency area, as shown in Fig. 3(c). (3) Manual White Balancing: On a RGB image (greenish due to no white balance), we use LabelMe (Russell et al., 2008) to extract the mean colors of 24 ColorChecker patches. The 21 patch is used the groundtruth illumination for manual white balance Qian et al. (2017; 2019). (4) Spatial Denoising: We use a milestone denoising method BM3D (Dabov et al., 2009) to perform spatial denoising with the setting of = 50. (5) Color Space transform: Fol- lowing Finlayson et.al. (Finlayson et al., 2015), given the retrieved ColorChecker values and the predefined oracle ColorChecker values, nal color correction matrix com of the shape (3, 3). A linear sRGB image is then computed from the input image I: Liinargh I cem. (6) Gamma: Following sRGB standard (Anderson al., 1996), piecewise gamma curve is applied for brightness perception. Due to space limitations, please refer to the supplementary material for more details and hyperparameters of controllable ISP.
optimize towards the CIEDEOO error and obtain the fi-
4.2 CONTROLLABLE ISP EVALUATION
We evaluated the controllable ISP two main aspects: the color accuracy of individual images and the temporal stability of color recovery in continuous videos. For color accuracy, used the CIEDEO0 (Luo et al., 2001) and CIELAB Aat (Lee & Powers, 2005) metrics to evaluate color accuracy. CIEDEO0 is widely used metric for color matching, considering the nonlinear char- acteristics of color differences and the human eye's sensitivity to colors, which accurately reflects human visual perception of color differences. CIELAB A.b is color difference metric based on the CIELAB color space (Mahy et al., 1994). Specifically, as shown in Fig. 5 (a) (b), we conducted a ColorChecker-based evaluation on 100 randomly selected samples. In CIEDEO0 (Luo al., 2001), we obtained an average value of 5.84 and median value of 5.07; For CIELAB Aab. we obtained an average value of 8.99 and median value of 7.00, demonstrating that our method can generally restore colors up to an accurate level. We displayed the maximum error distribution per image, showing that in CIEDEOO it is around 14, and in CIELAB Aab around 24, affected by color filter sensitivity and photodiode layout. For temporal stability in frame estimation differences, as shown in Fig. (c). We selected a 140-frame video, marking the ColorChecker in each frame. After gen- erating colors frame by frame, we observed that differences for the 24 ColorChecker colors are all under 0.01, mostly within 0.005. This confirms our algorithm's 's temporal stability.
In summary, presented controllable ISP pipeline and analyzed its performance. However, the ISP contains numerous controllable variables and hyperparameters. We hope that future researchers will focus optimizing these controllable aspects of the ISP further enhance its performance.

---

Published as a conference paper at ICLR 2025
Table 2: Comparison on Parameters, FL.PPS, and Time.
Top two models are highlighted in red and green.
Swin Transformer
Unet
PyNet CameraNet AWNet PyNetCA MW-ISPNet InvertISP
eSL
Ev-UNet
Params]
16.64
47.55
25.79
96.07
29.27
7.22
92.44
8.87
0.737
21.51
GFLOPS
4.52
111.96
19.19
120.21
51.27
29.22
1.41
14.24
48.49
6.89
Time (s).
0,0100
0,0775
0.0300
0.2138
0.0308
0,0459
0,0436
0.0868
0.063
0.012
Table 3: Comparison of Methods on HVS ISP Datasct outdoor scenes. Top two models are highlighted in red and green. refer to the results obtained by the same model with different hyperparameters.
2-Out 1-Out-Treer Tree-2
3-Out-Flewer" Flewer-2
4-Cut-
Buiidiin--
Avernge SSIM
PSNR
SSIM
PSNR1
SSIM
PSNR
SSIM
Ag4
PSNR
PyNET
31.70
0.9818
0.0190
35.12
0.9784
0.0127
30.60
0.9752
0.0223
32.47
0.9785
0.0180
PyNET"
27.56
0.9711
0,0310
32.35
0.9646
0.0175
28.20
0.960U
0.0311
29.37
0,9652
0.0265
PANe
31.86
0.9788
0.0202
34.19
0.9773
0.0139
29.22
0.9725
0.0280
31.76
0.9762
0.0207
InvertISP
28.56
0.9487
0.0243
25.59
0.9298
0.0313
28.62
0.9307
0.0287
27.59
0,9364
0.0281
MV-ISPNet
27.05
0.9680
0.0256
33.61
0.9648
0.0137
28.62
0.9657
0.0304
29.76
0.9662
0.0232
CameraNet
11.18
0.2580
0.2289
12.39
0.2741
0.1899
10.52
0.2534
0,2609
11.36
0.2618
0.2266
CameraNet"
13.26
0.637
0.2044
13.59
0.2736
0,1770
10.06
0.2753
0,2474
12.30
0.3953
0,2096
AWNet
14.33
0.8836
0,1166
20.10
0.9316
0.0519
16.70
0.939U
0,0951
17.04
0,9180
0.0879
Transffroer
25.02
0.9539
0.0308
29.14
0.9555
0.0231
21.57
0.9295
0.0523
25.24
0.9463
0.0354
UNet
21.97
0,9583
0,0393
29.43
0.9717
0,0208
22.12
0.9603
0,0460
24.51
0,9634
0.0354
UNet
29.52
0.9752
0.0206
25.75
0.9623
0.0323
29.24
0.9680
0.0265
28.17
0.9685
0.0265
eSL-Net
25.67
0.9424
0.0342
19.39
0.9130
0.0876
24.01
0.9277
0,0302
23.02
033234
0.0473
EV-UNet
32.86
0.9795
0.0148
32.87
0.9698
0.0157
24.59
0.9600
0.0369
30.11
0.9698
0.0225
5 BENCHMARK AND DIRECTION
Based on the RGB frames obtained from the controllable ISP, we evaluate the performance of four types of ISP methods, particularly in outdoor and indoor scenarios. The experiments are conducted in the same environment and framework. Additionally, we will discuss the potential reasons behind these results and propose future research directions. Implementation Details: All our models were trained and tested on the same machine with single A40 GPU with 48GB of GPU memory. We used PyTorch (Paszke et al., 2017) for all experiments, applying random cropping and rotation for data augmentation. The training batch size was 1, with each patch sized 1024 x 1024. The learning rate was 0.0001, and all models were trained for 50 epochs. Evaluation Metrics: We evaluate model performances in two aspects: resource consumption, including parameters in millions (M), ), GFLOPS, and average inference time (s); and image reconstruction for indoor and outdoor scenes, measured by PSNR (Hore& & Ziou, 2010), SSIM (Brunet et al., 2011), and L distance.
5.1 ISP BENCHMARK METHONS
Inspired by the prior ISP survey study (da Silva et al., 2023b), we categorize learning-based ISP models into three classes: full pipeline, stage-wise, image enhancement network-based. We selected two four open-source models from each category for training and evaluation on our dataset. Fur- thermore, we put forward another new category of event fusion method, and since there is no prior research to refer to, we design a simple event-guided ISP neural network test on our dataset. For more details on ISP methods, please refer to the supplementary material.
Full Pipeline ISP: These models utilize CNN architectures to integrate traditional ISP processes into an end-to-end conversion from RAW to RGB images. Notable models in this category include PyNet (Ignatov et al., 2020b), PyNeICA (Kim et al., 2020), InvertISP (Xing et al., 2021), and MV- ISPNet (Ignatov et al., 2020a).
They employ multiple specialized modules to handle different ISP tasks, either sequentially in parallel, to produce the final image. In our benchmark, we selected Camer- aNet (Liang et al., 2021) and AWNet (Dai et al., 2020) for their distinct approaches. Note that due to the unavailability of PyTorch version of CameraNet (Liang et al., 2021), we experimented converted version. The modules in the original AWNet (Dai et al., 2020) are trained independently, however in experiment trained them end-to-end.
Stage-wise ISP:
Image Enhancement Network- Based ISP: There have been numbers of high performance back- bone models for image enhancement in image enhancement tasks like deblurring (Zhang et al.,
7

---

Published as a conference paper at ICLR 2025
(a)
Good RGB
UNet
(e) PyNet
eSL-Net
(g) EV-UNeet
(h)
AWNet
PyNetCA
Figure 6: Visualization results of different methods on HVS-ISP Dataset outdoor scenes.
Table 4: Comparison of Methods on HVS ISP Dataset indoor scenes. refer to the results obtained by the model with different hyperparameters.
In Fruit-2
3
ColChecker SSIM
10
4 In RLChart -10 SSIM
Average SSIM
Methods
PSNR
SSIM
PSNR
PSNR
PSNR
L1
PyNET
13.09
0.7970
0.2182
11.38
0.7922
0.2489
11.42
0.7100
0.2563
11.97
0.7664
0.2412
PyNET
14.46
0.8068
0.2008
24.02
0,9550
0.0497
13.58
0.7694
0.1978
17.36
0.8437
0.1494
PyNeICA
18.13
0.8843
0.1253
29.53
0.9723
0.0246
35.51
0.9727
0.0121
27.72
0.9431
0.0540
InvertISP
25.83
0.9098
0.0346
28.33
0,9500
0.0235
30.65
0.9578
0.0183
28.27
0.9392
0.0254
MV-ISPNet
31.91
0.9594
0.0185
29.56
0.9729
0.0265
31,88
0.9670
0.0170
31.12
0.9664
0.0207
CameraNet
13.06
0,2660
0.1947
13.58
0.2722
0.1836
12.47
0.2391
0.2257
13.04
0.259]
0,2013
CameraNet"
14.18
0.290
0.1630
10.60
U,2667
0.2545
13.26
0.2636
0.2044
12.68
0.2672
0.2073
AWNet
17.95
0.8665
0.1302
32.17
0,9807
0.0184
30.98
0.9596
0.0215
27.03
0.9356
0.0567
Trin-Transfrmmrr
25.73
0.9397
0.0301
25.50
0.9561
0,0359
26.18
0.9486
0,0252
25.80
0.9481
0,0304
UNet
17.62
0.9161
0.0747
13.96
0.8828
0.1454
15.53
0.8750
0.1170
15.70
0.8913
0.1124
UNet"
32.52
0.9639
0.0161
29.04
0.9740
0.0257
33.72
0,9716
0.0146
31.76
0.9705
0.0188
eSL
27.09
0.9428
0.0331
24.79
0.9548
0.0434
26.52
0.9415
0.0379
26.
0.3464
0.0381
EV-UNet
14.16
0.8706
0.1533
31.64
0.9779
0.0214
32.33
0.9678
0.0173
26.04
0.9388
0.0640
2022) and super-resolution (Chen et al., 2022). Though not initially designed for ISPs, minor modi- fications can adapt these models for ISP tasks. For our benchmark, we selected UNet (Ronneberger et al., 2015) and Swin-Transformee (Liu et al., 2021; Lu al., 2024).
Event Fusion Method: As the first research event-guided ISP, we have no prior research for reference. Therefore, we selected eSL-Net (Wang et al., 2020a), an event-based backbone network used in various tasks (Lu al., 2023b), Additionally, we merged events voxel-grid (Liu et al., 2023) with UNet's encoder as EV-UNet to verify events effectiveness and challenges.
5.2 COMPARATIVE EXPERIMENTS AND VISUALIZATION ANALYSIS
Computational Performance: In Tab. 2, InvertISP (Xing et al., 2021) excels in computational ef- ficiency with 1.41 GFLOPS, significantly lower than the over 100 GFLOPS of AWNet (Dai al., 2020) and PyNet (Kim et al., 2020), which is suitable for limited computing resources. UNet sur- passes CameraNet (Liang et al., 2021) in processing speed with a response time of 0.01 s, preferable for real-time performance. Overall, UNet demonstrates balanced performance with low GFLOPS and the fastest processing speed, due to straightforward design.
Outdoor Performance: Tab. 3 shows the superior performance of PyNet across three outdoor back- grounds. PyNet (Kim et al., 2020) achieves the best PSNR (Hore Ziou, 2010), SSIM (Brunet al., 2011), and L1 with an overall average PSNR (Hore & Ziou, 2010) of 32.47, significantly higher than other models. Specifically, EV-UNet shows significant improvement in outdoor scenes with UNet after incorporating events gain, increasing from 28.17 to 30.11. In contrast, the commonly used bvent-based method eSL-Net performs poorly with a PSNR (Hore Ziou, 2010) of only 23. This poor performance mainly results from the limited receptive field of eSI., which is insufficient for
8

---

Published as a conference paper at ICLR 2025
(a)
Transformer
AWNNe
MW ISPNet
(e)
(g) PyNetCA
PyNet
Figure 7: Visualizations HVS-ISP Dataset indoor scenes,
estimating the global illumination information, and thus failing achieve consistent global illumi- nation enhancement. We further discuss on this issue in Sec. 5.3. we also visualize the results in Fig. 6. PyNet has achieved the highest PSNR (Hore & Ziou, 2010) but exhibits edge artifacts, this is likely due to the overfitting of the model. In outdoor scenes, event-enhanced outputs of EV-UNet show good global consistency. Fig. 6 shows that AWNet (Dai al., 2020) struggles with fine texture restoration, explaining its inferior performance to other methods.
Indoor Performance: Tab, 4 shows that UNet* excels in indoor environments, especially when handling multiple colored fruits and scenes with complex lighting and details. The output of AWNet (Dai al., 2020) overall excessive brightness, as illustrated in Fig. 7, explaining its low PSNR values. PyNet exhibits noticeable artifacts, consistent with the good RGB edge but with significantly different brightness, likely due the ill-posed nature of brightness recovery in the ISP process, resulting in its poor indoor performance. Event-fusion methods perform poorly indoors, primarily due to flickering light sources that complicate event characteristics. For more analysis about these issues, please refer to Sec. 5.3.
Summary: These sections show that the performance of numerous ISP methods on HVS sen- datasets varies significantly across different scenes. For instance, PyNet and AWNet (Dai et al., 2020) exhibit great variability between indoor and outdoor environments, underscoring that learning-based based ISP methods highly scene-dependent. This highlights the necessity for future work to analyze different scenes individually to fully understand the performance of a network. Furthermore, adding events UNet significantly improves performance in outdoor scenarios but not indoors, mainly due to the flickering indoor lighting. Addressing this issue remains a crucial challenge for future research.
5.3 DISCUSSION AND FUTURE DIRECTION
Through the comprehensive and objective evaluation of various models on our dataset, we have also observed number of findings that can bring insights for future work.
Significant Indoor-Outdoor Performance Gap PyNet and AWNet (Dai et al., 2020): We observed significant indoor-outdoor performance gap on PyNet (Ignatov et al., 2020b) and AWNet (Dai al., 2020). PyNet performs better in outdoor scenes than indoor, ranking the top of all models, while AWNet (Dai et al., 2020) shows quite the opposite behavior. Generally, outdoor scenes have more dynamic and varied lighting compared to indoor environments, which are difficult for models to learn. The original AWNet (Dai et al., 2020) is designed to be trained in multi-stage manner with different loss functions. Therefore might have fallen into sub-optima when trained end-to-end in experiment, resulting in the poor performance in modeling the harder outdoor scenes.
Local Brightness Artifacts: Artifacts occur when the brightness in certain image areas sig- nificantly deviates from the overall luminance (see Fig. 7). We investigated this by examin- ing the relationship between brightness of pixel and the RAW data within its 5 vicin-

---

Published as a conference paper at ICLR 2025
ity. We treat the neighboring RAW data as 25-dimensional vector, and apply t-SNE to project onto a 2D plane, recording the (z, coordinates. We then converted the RGB val- ues of the pixel to YUV, recording the Y (brightness) the coordinate, shown in Fig. 8. By plotting pixels from three ran- dom images in 3D (Fig. 8), show that pixel brightness and neighboring RAW data have non-injective relationship. Multi- ple brightness levels can emerge from the same RAW data, indicat- ing that global information, not just local RAW value, is essential for accurately determining pixel brightness avoid local artifacts.
L-SNE axis
Brignthiess
3NS-
(SSNE
The integration of events in our dataset signif- icantly enhances performance in outdoor scenes when comparing EV-UNET with UNet, primarily due the additional motion in- formation and dynamic range pro- vided by the events. However, simple fusion does not fully exploit these characteristics, highlighting the need for more sophis- ticated designs in future research. Conversely, performance decreases indoor scenes, primarily due to the flickerigg of artificial light sources.
(a)
(b)
Event Gains:
Figure 8: The ill-posedness of brightness estimation in the ISP pro- cess, We visualized the x region the RAW image and bright- ness of corresponding pixel the color image at the center this region. The results show that the same RAW region corresponds to different brightness levels in different images.
Flickering Artificial Lighting: Under certain indoor scenarios, some artificial light source (Xu et al., 2023), e.g. L.g.L.Ds. flicker because of the alternating current frequency. Given that the event frame rate of the sensor significantly exceeds the usual AC frequency (50 or 60 Hz), the flicker- ing lighting introduces considerable fluctuations in the event data over time. The distributions and features of events in these conditions are completely different from that in the natural lighting con- ditions, and could result in the model's failure in restoring the images from RAW data.
6 CONCLUSION
In this work, we present the first events-RAW paired dataset for event-guided ISP research. The dataset consists of 3373 high quality high resolution RAW images and corresponding pixel-level aligned events. Subsequently, good RGB frames are generated by controllable ISP pipeline we proposed. comprehensive evaluation and analysis of existing learnable ISPs and a simple event- guided ISP method are conducted on dataset. Based on this analysis, summarize some key points and challenges for guided ISP. We wish to emphasize the potential event data in ISP processes again. Event cameras have high dynamic range and high temporal resolution, which surpass the limits of human vision systems. In terms of dynamic range and temporal sampling, the information captured by event sensor is somehow a superset of that of human eye. Therefore, generating images perceptible to human vision is matter of downward compatibility. Limitations: Firstly, the scale of our dataset relatively small, because the HVS sensor we use is still in the prototype stage and the associated hardware is cumbersome and exhibits low stability, which has raised the cost in data collection and thus limited size dataset. And yet we are committed to expanding the dataset with more diverse real-world scenarios in future research. Secondly, our dataset has not thoroughly addressed the issue of flickering in artificial lighting caused by alternating current, especially in indoor scenarios. The flickering considerably impairs the performance of our method and further research should pay attention to this problem.
Acknowledgements: This work was supported in part by the National Key R&D Program of China (Grant No. 2023YFF0725001).ii part by the National Natural Science Foundation of China (Grant No. No.92370204), in part by the Guangdong Basic and Applied Basic Research Foundation (Grant No. 15 in part by Guangzhou- Guangzhou-HKUST(C HKUST(GZZ) Joint Funding Program (Grant No. No.2023A03100088, Education Bureau of Guangzhou Municipality.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1692-1700, 2018.
Alpsentek. Alpix-eiger product overview: https: www./oo/////.///e coon/product, 2024. URL https: www.tte.....eekeeeeeeeeea com/ prom/rrouct..... Accessed: 2024-05-19. 3,4
Matthew Anderson, Ricardo Motta, Srinivasan Chandrasekar, and Michael Stokes. Proposal for standard default color space for the internet- srgb. In Color and imaging conference, volume 4, pp. 238 245. Society of Imaging Science and Technology, 1996. 6
Wenbo Bao, Wei-Sheng Lai, Chao Ma, Xiaoyun Zhang, Zhiyong Gao, and Ming-Hsuan Yang. Depth-aware video frame interpolation. In Proceedings of the IEEF/CVF conference on com- puter vision and pattern recognition, pp. 3703-3712, 2019.
Dominique Brunet, Edward Vrscay, and Zhou Wang. On the mathematical properties of the 2011. 7,
structural similarity index. IEEE Transactions on Image Processing,
Prashant Chaudhari, Franziska Schirrmacher, Andreas Maier, Christian Riess, and Thomas Köhler. Merging-isp: Multi-exposure high dynamic range image signal processing, In DAGM German Conference on Pattern Recognition, pp. 328-342. Springer, 2021. 22
Honggang Chen, Xiaohai He, Linbo Qing. Yuanyuan Wu, Chao Ren, Ray E Sheriff, and Ce Zhu. review. Information Fusion, 79:124-145 2022.
Real-world single image super-resolution: reveew.
Matheus Henrique Marques da Silva, Jhessica Victoria Santos da Silva, Rodrigo Reis Arrais, Wladimir Barroso Guedes de Araújo Neto, Leonardo Tadeu Lopes, Guilherme Augusto Bileki, Iago Oliveira Lima, Lucas Borges Rondon, Bruno Melo de Souza, Mayara Costa Regazio, et al. Isp meets deep learning: A survey deep learning methods for image signal processing. arXiv preprint wwxXiv:3305.19990 2023a.
Matheus Henrique Marques da Silva, Jhessica Victoria Santos da Silva, Rodrigo Reis Arris, Wladimir Barroso Guedes de Araújo Neto, Leonardo Tadeu Lopes, Guilherme Augusto Bilcki, Iago Oliveira Lima, Lucas Borges Rondon, Bruno Melo de Souza, Mayara Costa Regazio, et al. Survey on software isp methods based on deep learning. arXiv preprint arXiv:2305. 1994 2023b.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Bm3d image de- noising with shape-adaptiie principal component analysis. In SPARS'09 09--ignna Processing with Adaptive Sparse Structured Representations, 2009.
Linhui Dai, Xiaohong Liu, Chengqi Li, and Jun Chen. Awnet: Attentive wavelet network for image isp. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceed- ings, Part III 16, pp. 185-201. Springer, 2020. 3, 7, 8, 8,9,
Mauricio Delbracio, Damien Kelly, Michael S Brown, and Peyman Milanfar. Mobile computational photography: A tour. Annual review vision science, 7:571-604, 2021.
Graham Finlayson, Michal Mackiewicz, and Anya Hurlbert. Color correction using root-
polynomial regression. IEEE Transactions on Image Processing, 24(5): 2015. 6
Guillermo Gallego, Tobi Delbrück, Garrick Orchard, Chiara Bartolozzi, Brian Taba, Andrea Censi, Stefan Leutenegger, Andrew Davison, Jörg Conradt, Kostas Danilidiis, et al. Event- vision: A survey. IEEE transactions on pattern analysis and machine intelligence, 44(1):154-180, 2020.
3,3
Eiki Goto, Murat Dogru, Takashi Kojima, and Kazuo Tsubota. Computer-synthesis of interfer- ence color chart human tear lipid layer, by colorimetric approach. Investigative ophthalmol- ogy & visual science, (1111)466334444 2003. 2

---

Published as a conference paper at ICLR 2025
Keigo Hirakawa and Thomas W Parks. Joint demosaicing and denoising. IEEE Transactions Image Processing, 15(8):2146225777 2006.
Alain Hore and Djemel Ziou. Image quality metrics: Psnr vs. ssim. In 2010 20th international conference on pattern recognition, pp. 2366-2369 IEEE, 2010. 7, 8, 9
Jingwei Huang, Zhili Chen, Duygu Ceylan, and Hailin Jin. 6-dof vr videos with a single 360-camera. In 2017 IEEE Virtual Reality (VR), pp. 37-44. IEEE, 2017. 1
Andrey Ignatov, Radu Timofte, Zhilu Zhang, Ming Liu, Haolin Wang, Wangmeng Zuo, Jiawei Zhang, Ruimao Zhang, Zhanglin Peng, Sijie Ren, et al. Aim 2020 challenge on learned image signal processing pipeline. In Computer Vision-ECCV 2020 Workshops: Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 152-170. Springer, 2020a. 7, 22, 23
Andrey Ignatov, Luc Van Gool, and Radu Timofte. Replacing mobile camera isp with single deep learning model. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition workshops. pp.
2020b. 3, 7, 9, 22, 23
Shantanu Ingle and Madhuri Phute. Tesla autopilot: semi autonomous driving, an uptick for future autonomy. International Research Journal of Engineering and Technology, 3(9):369-372 2016.
Zhe Jiang, Yu Zhang, Dongqing Zou, Jimmy Ren, Jiancheng Lv, and Yebin Liu. Learning event- based motion deblurring. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3320-3329, 2020. 3
Yongcheng Jing, Yiding Yang, Xinchao Wang, Mingli Song, and Dacheng Tao. Turning frequency to resolution: Video super-resolution via event cameras. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 7772-7781, 2021. 1,3
Byung-Hoon Kim, Joonyoung Song. Jong Chul Ye, and JacHyun Back. Pynet-ca: enhanced pynet with channel attention for end-to-end mobile image signal processing. In European Conference on Computer Vision, pp. 202-212. Springer, 2020. 7, 8, 22, 25
Chong-Min Kyung et al. Theory and applications of smart cameras. Springer, 2016.
Yong-Keun Lee and John M Powers. Comparison of cie lab, ciede 2000, and din 99 color differences between various shades of resin composites. International Journal of Prosthodontics, 18(2), 2005.
Zhaowen Li, Tingeun Wei, and Ran Zheng. Design of black level calibration system for cmos image sensor. In 2010 International Conference Computer Application and System Modeling (ICCASM 2010), volume 10, pp. V10-643. IEEE, 2010. 2
Zhihao Li, Si Yi, and Zhan Ma. Rendering nighttime image via cascaded color and brightness compensation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 897-905, 2022. 22
Guoqiang Liang, Kanghao Chen, Hangyu Li, Yunfan Lu, and Lin Wang. Towards robust event- guided low-light image enhancement: large-scale real-world event-image dataset and novel approach. arXiv preprint arXiv:2404.008.. 000344 2024. 1,
Jinxiu Liang, Yixin Yang, Boyu Li, Peiqi Duan, Yong Xu, and Boxin Shi. Coherent event guided low-light video enhancement. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 10615-10625, 2023. 3
Zhetong Liang, Jianrui Cai, Zisheng Cao, and Lei Zhang. Cameranet: A two-stage framework for effective camera isp learning. IEEE Transactions on Image Processing, 30:2248-2263 2021. 2, 3. 7, 8, 25
Carl C Liebe, Edwin W Dennison. Bruce Hancock, Robert C Stirbl, and Bedabrata Pain. Active pixel sensor (aps) based star tracker. In 1998 IEEE Aerospace Conference Proceedings (Cat. No. 987H8339), volume 1, pp. 119-127. IEFF, 1998.
12

---

Published as a conference paper at ICLR 2025
Songnan Lin, Jiawei Zhang, Jinshan Pan, Zhe Jiang, Dongqing Zou, Yongtian Wang, Jing Chen, and Jimmy Ren. Learning event-driven video deblurring and interpolation. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28 2020, Proceedings, Part VIII 16, pp. 695-710. Springer, 2020. 3
Daikun Liu, Teng Wang, and Changyin Sun. Voxel-based multi-scale transformer network for event stream processing. IEEE Transactions on Circuits and Systems for Video Technology, 2023. 8
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp- 10012-10022, 2021. 3,
Yunfan Lu, Guoqiang Liang, and Lin Wang. Self-supervised learning of event-guided video frame interpolation for rolling shutter frames. arXiv preprint urXiv:2306.15507 2023a. 1,
Yunfan Lu, Zipeng Wang, Minjie Liu, Hongjian Wang, and Lin Wang. Learning temporll implicit neural representations for event-guided video super-resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1557-1567, 2023b. 1, 2,8
Yunfan Lu, Yijie Xu, Wenzong Ma, Weiyu Guo, and Hui Xiong. Event camera demosaicing via swin transformer and pixel-focus loss. arXiv preprint arXiv:2404.0273,, 2024. 8, 26
M Ronnier Luo, Guihua Cui, and Bryan Rigg. The development of the cie 2000 colour-difference formula: Ciede2000. Color Research & Application: Endorsed by Inter-Society Color Council, The Colour Group (Great Britain), Canadian Society for Color, Color Science Association of Japan, Dutch Society for the Study of Color, The Swedish Colour Centre Foundation, Colour Society of Australia, Centre Français de la Couleur, 26(5):340-350, 2001.
Mare Mahy, Lue Van Eycken, and André Oosterlinck. Evaluation of uniform color spaces developed after the adoption cielab and cieluv. Color Research & Application, 122)1105-121, 1994. 6
Jon S McElvain and Walter Gish. Camera color correction using two-dimensional transforms. In Color and Imaging Conference, volume 21, pp. 250-256. Society for Imaging Science and Tech- nology, 2013. 2
Nico Messikommer, Stamatios Georgoulis, Daniel Gehrig, Stepan Tulyakov, Julius Erbach, Alfredo Bochicchio, Yuanyou Li, and Davide Scaramuzza. Multi-bracket high dynamic range imaging with event cameras. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 547-557, 2022.
MIPI Challenge 2024. Mobile intelligent photography and imaging workshop 2024. https: //mipi-chaallleee org Barg/MMPPP I2024/, 2024. 2,
Simon Niklaus, Long Mai, and Feng Liu. Video frame interpolation via adaptive separable convo- lution. In Proceedings of the IEEE international conference computer vision, pp. 261-270,
3
Liyuan Pan, Cedric Scheerlinck, Xin Yu, Richard Hartley, Miaomiao Liu, and Yuchao Dai. Bringing a blurry frame alive at high frame-rate with an event camera. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 6820-6829, 2019. 3
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
T-C Poon and Partha P Banerjee. Contemporary optical image processing with MATLAB. Elsevier, 2001. 2
Yanlin Qian, Ke Chen, Jamo Nikkanen, Joni-Kristian Kamarainen, and Jiri Matas. Recurrent color constancy. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017. 6
13

---

Published as a conference paper at ICLR 2025
Yanlin Qian, Joni- Kristian Kamarainen, Jarno Nikkanen, and Jiri Matas. On finding gray pixels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),
June 2019. 6
Rainbow-
Saainny---m--PPeee Siinge-Prooessing-Laee
Quadbayer
cfa
mod-
demosaicing, www.aaeettrrrailleeeeeeeeee 116085-quus SarSs-uuuaabbrrrrrereeaee cfaeeer--fa--oooo oas demosaicing Accessed: 2024-06-01.
ified maddeent-based
2022.
URL
https:
//www. www.mmmahworks...
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical image computing and computer-assisted intervention- MICCAI 2015; 18th international conference, Munich, Germany, October 5-9, 2015, proceed- ings, part III 18, pp. 234-241. Springer, 2015. 3, 8, 26
Bryan C Russell, Antonio Torralba, Kevin P Murphy, and William T Freeman. Labelme: a database and web-based tool for image annotation. International journal of computer vision, 77:157-173, 2008. 4, 6, 19, 23
Cedric Scheerlinck, Henri Rebecq, Timo Stoffregen, Nick Barnes, Robert Mahony, and Davide Scaramuzza. Ced: Color event camera dataset. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019. 3,
Eli Schwartz, Raja Giryes, and Alex M Bronstein. Deepisp: Toward learning an end-to-end image processing pipeline. IEEE Transactions on Image Processing, 28(2):912 923, 2018. 3, 22
Waseem Shariff, Mehdi Sefidgar Dilmaghani, Paul Kielty, Mohamed Moustafa, Joe Lemley, and Peter Corcoran. Event cameras in automotive sensing: A review. IEEE Access, 2024. 1,3
Ardhendu Shekhar Tripathi, Martin Danelljan, Samarth Shukla, Radu Timofte, and Luc Van Gool. Transform your smartphone into dsir camera: Learning the isp in the wild. In European Con- ference on Computer Vision, pp. 625 641. Springer, 2022. 2, 22
Chen Song, Qixing Huang, and Chandrajit Bajaj. E-cir: Event-enhanced continuous intensity recov- ery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7803-7812 2022. 3
Gui Yun Tian, Duke Gledhill, David Taylor, and David Clarke. Colour correction for panoramic imaging. In Proceedings Sixth International Conference on Information Visualisation, pp. 483-
488. IEEE, 2002. 4
Stepan Tulyakov, Daniel Gehrig, Stamatios Georgoulis, Julius Erbach, Mathias Gehrig, Yuanyou Li, and Davide Scaramuzza. Time lens: Event-based video frame interpolation. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16155-16164, 2021. 3.3
Stepan Tulyakov, Alfredo Bochicchio, Daniel Gehrig, Stamatios Georgoulis, Yuanyou Li, and Da- vide Scaramuzza. Time lens++: Event-based frame interpolation with parametric non-linear flow and multi-scale fusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17755-17764, 2022. 2,3
Bishan Wang, Jingwei He, Lei Yu, Gui-Song Xia, and Wen Yang. Event enhanced high-quality image recovery. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pp. 155-171. Springer, 2020a. 8
Wencheng Wang. Xiaojin Wu, Xiaohui Yuan, and Zairui Gao. An experiment-basee review of low- light image enhancement methods. leee Access, 1878844-791177 2020b.
Ching Chih Weng, Homer Chen, and Chiou-Shann Fuh. A novel automatic white balance method for digital still cameras. In 2005 IEEE International Symposium Circuits and Systems (ISCAS), pp. 3801-3804 IEEE, 2005.
Ching-Chih
Li Xiaopeng, Zeng Zhaoyuan, Fan Cien, Zhao Chen, Deng Lei, and Yu Lei. Hdr imaging for dynamic scenes with events. arXiv preprint 10X1v:2404.03210, 2024. 1
14

---

Published as a conference paper at ICLR 2025
Yazhou Xing, Zian Qian, and Qifeng Chen. Invertible image signal processing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 0287-6296 2021. 8,2.2
Fang Xu, Lei Yu, Bishan Wang, Wen Yang. Gui-Song Xia, Xu Jia, Zhendong Qiao, and Jianzhuang Liu. Motion deblurring with real events. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 2583-2592. 2021.
Lexuan Xu, Guang Hua, Haijian Zhang, Lei Yu, and Ning Qiao. seeing" electric network fre- quency from events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pp. 18022-18031, 2023. 10
Lexuan Xu, Guang Hua, Haijian Zhang, and Lei Yu. "seeing" enf from neuromorphic events: Mod- eling and robust estimation. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024a. 28
Senyan Xu, Zhijing Sun, Jiaying Zhu, Yurui Zhu, Xueyang Fu, and Zheng-Jun Zha. Demo- saicformer: Coarse-to-fine demosaicing network for hybridevs camera. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1pp.122 1126-1135, 2024b. 21
Qingyu Yang, Guang Yang. Jun Jiang, Chongyi Li, Ruicheng Feng, Shangchen Zhou, Wenxiu Sun, Qingpeng Zhu, Chen Change Loy. Jinwei Gu, et al. Mipi 2022 challenge on quad-bayer re- mosaic: Dataset and report. In European Conference Computer Vision, pp. 21-35. Springer, 2022. 2
Wu Yaqi, Fan Zhihao, Chu Xiaofeng, Ren Jimmy S., Li Xiaoming, Yue Zongsheng, Li Chongyi, Zhou Shangcheng, Feng Ruicheng, Dai Yuckun, Yang Peiqing, Loy Chen Change, et al. Mipi 2024 challenge on demosaic for hybridevs camera: Methods and results. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 2,
Ke Yu. Zexian Li, Yue Peng, Chen Change Loy, and Jinwei Gu. Reconfigisp: Reconfigurable camera image processing pipeline. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4248 4257, 2021. 22
Lu Yuan, Jian Sun, Long Quan, and Heung-Yenn Shum. Image deblurring with blurred/noisy image pairs. In ACM SIGGRAPH 2007 papers, pp. 1-es. 2007. 1
LU Yunfan, Guoqiang Liang, and Lin Wang. Uniinr: Unifying correction, deblur, and interpolation with an event camera. 2023. 1,
inr for is video
Kaihao Zhang, Wenqi Ren, Wenhan Luo, Wei-Sheng Lai, Bjöm Stenger, Ming-Hsuan Yang, and Hongdong Li. Deep image deblurring: A survey. International Journal of Computer Vision, 130 (9):2103-2133) 2130,, 2022. 1.7
Lin Zhang, Ying Shen, and Hongyu Li. Vsi:
visual saliency-induced index for perceptual image quality assessment. IEEE Transactions on Image processing, 23(10):4270-4281, 2014. 26
Lin Zhang, Lei Zhang, and Alan C Bovik. A feature-enriched completely blind image quality evaluator. IEEE Transactions on Image Processing. 24(8):2579-2555 2015. 26
Xiang Zhang and Lei Yu. Unifying motion deblurring and frame interpolation with events. Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17765- 17774, 2022.
Xinyu Zhou, Peiqi Duan, Yi Ma, and Boxin Shi. Evunroll: Neuromorphic events based rolling shutter image correction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17775-17784, 2022.
Pengfei Zhu, Longyin Wen, Xiao Bian, Haibin Ling, and Qinghua Hu. Vision meets drones: A challenge. arXiv preprint 180.:18804004433 2018.
15


Reviews:
[{"review_id": "CmoF7UE4cJ", "reviewer": "Reviewer_q8r6", "summary": "This paper presents a event-RAW paired dataset of pixel-level aligned events and RAW images, including 3373 images across 24 scenes, for ISP process reforming. The study presents a conventional ISP pipeline that generates high-quality RGB frames for reference, performing basic ISP operations like demosaicing and white balancing. Some existing learnable ISP methods are trained and evaluated on the dataset.", "questions": "1. The ISP method presented in this paper demonstrates varying effects across different tasks. It would be beneficial to analyze whether these effects are reflected in the proposed dataset.\n2. The authors should explain and analyze the reasons why the proposed method fails to surpass existing methods in performance, as well as highlight its advantages.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The present paper introduces a event-RAW paired dataset that addresses a gap in the field of ISP and facilitates further research on event-guided ISP. \n2. The trainable ISP methods are evaluated on this event-RAW paired dataset.", "weaknesses": "This paper, as a work primarily contributing a dataset, provides insufficient information about the dataset itself and focuses heavily on performance comparisons with existing methods, but the proposed method fails to outperform the current ones across all metrics. The overall logic and structure of the paper need improvement and refinement.\n1. As a dataset-centric paper, it presents too few sample images, making it difficult for readers to intuitively grasp the specific content of the dataset and the differences in the scenes, camera shots, and exposure modes mentioned.\n2. The introduction to the specific content of the dataset is insufficient. It is recommended to list statistical information about the dataset regarding the mentioned types and present them in tables.\n3. The ISP presented in the paper is used to handle demosaicing, white balance, denoising, and color space transformations, but its capabilities in handling these tasks are not reflected in the performance analysis results.\n4. Figures 6 and 7 have identical legends and can be combined into one figure, or one could be replaced with the visualization results of indoor data samples.\n5. The experimental section only presents quantitative analysis results for outdoor and indoor data separately, failing to reflect the overall performance of the dataset.\n6. The presented integrated improvement method does not enhance processing performance, and the paper does not provide sufficient explanation or analysis of this issue.\n7. The experimental visual results are too few, and the layout is too compact, making it difficult for readers to intuitively appreciate the value of the dataset.\n8. It is recommended to analyze the performance of different types of samples within the dataset using the same method in future work, rather than focusing primarily on comparing the performance of multiple ISP methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PbyaOqSqON", "reviewer": "Reviewer_qDHH", "summary": "This paper presents a novel dataset tailored for event-guided image signal processing (ISP), featuring 3,373 high-resolution RAW images paired with corresponding event data across various scenes, exposure modes, and lenses. This dataset aims to advance RGB-Event ISP research by enabling the development of methods that directly incorporate event information within the ISP pipeline. Besides, they introduce an event-guided ISP neural network as a baseline, fusing events with RAW data to optimize ISP tasks. Then, various ISP methods are evaluated on this dataset, establishing a foundation for future RGB-Event ISP advancements. Note that this work generated high-quality RGB images as ground truth by using a ColorChecker.", "questions": "My questions are highly overlapped with the weakness as follows:\n1.\tDataset Scale and Diversity: Could you provide more insights into the limitations that prevented a larger dataset collection, and provide more insights about how to scale up the data?\n2.\tEarly Data Release: Is there a timeline or plan for public release of the dataset, especially considering its importance for community validation and use? \n3.\tEvent-Guided ISP Baseline Evaluation: Given that the dataset is designed for RAW-Event ISP tasks, consider developing additional baseline methods that explicitly incorporate event data to better assess the dataset's strengths. For instance, could you explore a wider variety of event-guided architectures or fusion methods to better showcase the potential of the event data in ISP?", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1.\tDataset: This is the first dataset with aligned (in both time and spatial space) RAW and events from a new HVS equipment, which may provide some avenues for developing new ISP algorithms.\n2.\tBenchmark Task: This work evaluates the proposed dataset by testing various ISP baseline methods along with an event-guided ISP approach. Some conclusions and insights can be drawn from these experiments.", "weaknesses": "1.\tScale of the dataset: From my understanding, a robust dataset should have both scale and diversity. Although the authors mention that the dataset is relatively small due to the low stability of the HVS sensor, I am not completely convinced by it, and I still believe it would be beneficial to expand the dataset further to increase its richness and variety. Additionally, the authors could include suggestions in the future work section on strategies for scaling up the dataset.\n2.\tReproducibility and Usage of the dataset: Given that this is a dataset/benchmark paper, I strongly encourage the authors to release the data as soon as possible, even during the review stage. Early access is crucial for the community to begin evaluating and utilizing the dataset.\n3.\tBenchmark Task: This dataset is tailored for RAW-Event ISP tasks. So, I wonder if we should focus more on designing various Event-guided ISP baselines. In the current setup, most baselines do not utilize the event data, raising concerns about whether these limited event-guided baselines effectively demonstrate the dataset’s potential. I believe increasing the variety of event-driven baselines could better validate the usefulness of this dataset and highlight its unique contributions to event-guided ISP research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EIL0mxpl6N", "reviewer": "Reviewer_AvAS", "summary": "The authors introduce the first events-RAW paired dataset specifically designed for event-guided image signal processing (ISP) research. This dataset comprises 3,373 high-quality, high-resolution RAW images alongside corresponding pixel-level aligned events. Using a controllable ISP pipeline developed by the authors, high-quality RGB frames are generated. A thorough evaluation and analysis of existing learnable ISPs, as well as a straightforward event-guided ISP method, are performed on this dataset. From this analysis, the authors highlight several key points and challenges associated with event-guided ISP.", "questions": "Please see the weaknesses. I have assigned a preliminary score based on the initial manuscript and will consider adjustments depending on the authors' responses and feedback from other reviewers.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The relevant background knowledge of this paper is clearly explained.\n\n2. The topic of RGB-event ISP is very interesting topic and meanful for future camera.\n\n3. The writing is very well and easy to understand.", "weaknesses": "1. I greatly appreciate the effort to create a dataset for RGB-Event ISP, which opens up opportunities for event-assisted RGB ISP tasks. However, the dataset's scale is quite limited, with only 3,373 samples, which may not be sufficient to support data-driven learning methods. This raises concerns about the dataset's ability to serve as a professional, standardized, and challenging benchmark. If this is primarily a workload issue, could the authors consider generating some simulated datasets? I would appreciate an explanation regarding this.\n\n2. The authors use images generated from a controllable ISP framework based on MATLAB as ground truth. While they provide extensive explanations for this approach, it is difficult to trust software-generated images as ground truth, especially for a professional dataset. This practice differs significantly from the ground truth methods used by existing sensor or smartphone manufacturers for their ISPs.\n\n3. The integration of events into traditional ISP theoretically brings certain advantages, and the authors should elaborate on these benefits. Additionally, to demonstrate the advantages of event cameras, the authors should showcase scenes, particularly under high-speed motion or extreme lighting conditions, that highlight the potential benefits of using event-based approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Un0qE6hv5L", "reviewer": "Reviewer_fVak", "summary": "This paper introduces the RGB-Event ISP dataset and benchmark, a novel paired dataset that combines RAW and event data from hybrid vision sensors to support research in event-guided image signal processing (ISP). The authors design a customizable ISP pipeline, allowing for benchmarking ISP methods and exploring the benefits of integrating event data in RAW-level ISP tasks. The experiments demonstrate the effectiveness of several methods in both outdoor and indoor scenes, and the dataset offers unique insights into ISP improvements through event data.", "questions": "1. In Section 3.1, it would be helpful to elaborate on the role of event data in each ISP stage (e.g., how it improves demosaicing or color correction compared to using RAW data alone).\n2. There are some spelling errors in the paper. For instance, both \"convential ISP pipeline\" and \"conventional ISP pipeline\" appear, where \"convential\" is incorrect; it should be \"conventional ISP pipeline.\" Additionally, in Figure 5c, the y-axis should be labeled as \"probability density,\" but the word \"density\" seems to have been omitted. \n3. My concern is that some comparative algorithms have very low PSNR scores. From the varying performance of PyNET and UNet under different hyperparameters, I believe it would also be beneficial to test CameraNet with different hyperparameters. This would allow us to select a high-performance configuration as a baseline, ensuring a fairer and more comprehensive comparison.\n4. I have a question regarding the alignment between APS (Active Pixel Sensor) and EVS (Event-based Vision Sensor) data. As mentioned in the paper, there is a significant difference in their frame rates. How is the event data aligned with the RAW data given this disparity in FPS?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The dataset fills an existing gap by providing paired RAW and event data, which is the first of its kind specifically designed for ISP tasks.\n2. The proposed ISP pipeline is flexible, covering key ISP stages like black level adjustment, demosaicing, and color correction, making it versatile for testing various ISP algorithms.\n3. The experimental evaluation is thorough, comparing several state-of-the-art ISP methods and highlighting the advantages of using event data, particularly in dynamic outdoor scenes.\n4. The paper also addresses its limitations and suggests potential solutions. Specifically, it provides practical insights into the challenges of integrating event data with ISP, including issues like artifacts under artificial indoor lighting.", "weaknesses": "1. The dataset size is relatively small (3373 images), which might limit the generalizability of models trained on it. Additional data could enhance the applicability of the findings.\n2. The theoretical justification for how event data contributes to each ISP stage could be more developed. For instance, further details on how event data specifically enhances tasks like white balancing and noise reduction would make the contribution clearer.\n3. The writing lacks clarity, making the paper difficult to understand for readers not deeply familiar with the field. Enhancing the language clarity and providing more structured explanations would significantly improve the paper's readability and accessibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Nk5S5m0kv0", "reviewer": "Reviewer_vrds", "summary": "The paper presents a new event-RAW paired dataset collected with a novel but private sensor that records pixel-level aligned events and RAW images. 3373 RAW images with paired events spanning 24 scenes are captured. A convential ISP pipeline is proposed to generate RGB references and learnable ISP methods are used to train and evaluate on the dataset.", "questions": "Please explain what is 'event' (with examples) and why event camera is important in addition to RGB camera, maybe in the introduction/related work section. This is important for readers unfamiliar with this topic.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Event-guided ISP seems like a novel and interesting idea.\n2. A novel sensor is designed to capture the datasets, although it's private and confidential.\n3. The analysis and results looks good.", "weaknesses": "1. The proposed is rather small-scaled. There are 24 videos captured in total, with 80 to 140 frames for each video. Considering a FPS of 60, it's just 1-2 seconds. It would be better called an image dataset instead of a video dataset.\n2. I assume the dataset would be open-sourced? Although the paper does not explicitly state that.\n3. As the authors stated, the prototype is cumbersome with low stability and also private. This weakens the reproducibility of the work.\n4. The background of the dataset and issues need further clarifications. It is not clear to me at the moment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)