RGB-Event ISP: The Dataset and Benchmark

Authors: Yunfan LU, Yanlin Qian, Ziyang Rao, Junren Xiao, Liming Chen, Hui Xiong
Primary area: datasets and benchmarks
Keywords: event camera, image signal processor, color correction, denoising

Abstract:
Event-guided imaging has received significant attention due to its potential to revolutionize instant imaging systems. However, the prior methods primarily focus on enhancing RGB images in a post-processing manner, neglecting the challenges of image signal processor (ISP) dealing with event sensor and the benefits events provide for reforming the ISP process. To achieve this, we conduct the first research on event-guided ISP. First, we present a new event-RAW paired dataset, collected with a novel but still confidential sensor that records pixel-level aligned events and RAW images. This dataset includes 3373 RAW images with $2248\times 3264$ resolution and their corresponding events, spanning 24 scenes with 3 exposure modes and 3 lenses. Second, we propose a convential ISP pipeline to generate good RGB frames as reference. This convential ISP pipleline performs basic ISP operations, e.g., demosaicing, white balancing, denoising and color space transforming, with a ColorChecker as reference. Third, we classify the existing learnable ISP methods into 3 classes, and select multiple methods to train and evaluate on our new dataset. Lastly, since there is no prior work for reference, we propose a simple event-guided ISP method and test it on our dataset. We further put forward key technical challenges and future directions in RGB-Event ISP. In summary, to the best of our knowledge, this is the very first research focusing on event-guided ISP, and we hope it will inspire the community.

Decision: accept (poster)
Reviewer scores: 3, 5, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729047061296

thread_type:
official_review

<thread_object>
creator: reviewer_vrds

<content>
summary:
The paper presents a new event-RAW paired dataset collected with a novel but private sensor that records pixel-level aligned events and RAW images. 3373 RAW images with paired events spanning 24 scenes are captured. A convential ISP pipeline is proposed to generate RGB references and learnable ISP methods are used to train and evaluate on the dataset.

strengths:
1. Event-guided ISP seems like a novel and interesting idea.
2. A novel sensor is designed to capture the datasets, although it's private and confidential.
3. The analysis and results looks good.

weaknesses:
1. The proposed is rather small-scaled. There are 24 videos captured in total, with 80 to 140 frames for each video. Considering a FPS of 60, it's just 1-2 seconds. It would be better called an image dataset instead of a video dataset.
2. I assume the dataset would be open-sourced? Although the paper does not explicitly state that.
3. As the authors stated, the prototype is cumbersome with low stability and also private. This weakens the reproducibility of the work.
4. The background of the dataset and issues need further clarifications. It is not clear to me at the moment.

questions:
Please explain what is 'event' (with examples) and why event camera is important in addition to RGB camera, maybe in the introduction/related work section. This is important for readers unfamiliar with this topic.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730208744681

thread_type:
official_review

<thread_object>
creator: reviewer_fvak

<content>
summary:
This paper introduces the RGB-Event ISP dataset and benchmark, a novel paired dataset that combines RAW and event data from hybrid vision sensors to support research in event-guided image signal processing (ISP). The authors design a customizable ISP pipeline, allowing for benchmarking ISP methods and exploring the benefits of integrating event data in RAW-level ISP tasks. The experiments demonstrate the effectiveness of several methods in both outdoor and indoor scenes, and the dataset offers unique insights into ISP improvements through event data.

strengths:
1. The dataset fills an existing gap by providing paired RAW and event data, which is the first of its kind specifically designed for ISP tasks.
2. The proposed ISP pipeline is flexible, covering key ISP stages like black level adjustment, demosaicing, and color correction, making it versatile for testing various ISP algorithms.
3. The experimental evaluation is thorough, comparing several state-of-the-art ISP methods and highlighting the advantages of using event data, particularly in dynamic outdoor scenes.
4. The paper also addresses its limitations and suggests potential solutions. Specifically, it provides practical insights into the challenges of integrating event data with ISP, including issues like artifacts under artificial indoor lighting.

weaknesses:
1. The dataset size is relatively small (3373 images), which might limit the generalizability of models trained on it. Additional data could enhance the applicability of the findings.
2. The theoretical justification for how event data contributes to each ISP stage could be more developed. For instance, further details on how event data specifically enhances tasks like white balancing and noise reduction would make the contribution clearer.
3. The writing lacks clarity, making the paper difficult to understand for readers not deeply familiar with the field. Enhancing the language clarity and providing more structured explanations would significantly improve the paper's readability and accessibility.

questions:
1. In Section 3.1, it would be helpful to elaborate on the role of event data in each ISP stage (e.g., how it improves demosaicing or color correction compared to using RAW data alone).
2. There are some spelling errors in the paper. For instance, both "convential ISP pipeline" and "conventional ISP pipeline" appear, where "convential" is incorrect; it should be "conventional ISP pipeline." Additionally, in Figure 5c, the y-axis should be labeled as "probability density," but the word "density" seems to have been omitted. 
3. My concern is that some comparative algorithms have very low PSNR scores. From the varying performance of PyNET and UNet under different hyperparameters, I believe it would also be beneficial to test CameraNet with different hyperparameters. This would allow us to select a high-performance configuration as a baseline, ensuring a fairer and more comprehensive comparison.
4. I have a question regarding the alignment between APS (Active Pixel Sensor) and EVS (Event-based Vision Sensor) data. As mentioned in the paper, there is a significant difference in their frame rates. How is the event data aligned with the RAW data given this disparity in FPS?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697797895

thread_type:
official_review

<thread_object>
creator: reviewer_avas

<content>
summary:
The authors introduce the first events-RAW paired dataset specifically designed for event-guided image signal processing (ISP) research. This dataset comprises 3,373 high-quality, high-resolution RAW images alongside corresponding pixel-level aligned events. Using a controllable ISP pipeline developed by the authors, high-quality RGB frames are generated. A thorough evaluation and analysis of existing learnable ISPs, as well as a straightforward event-guided ISP method, are performed on this dataset. From this analysis, the authors highlight several key points and challenges associated with event-guided ISP.

strengths:
1. The relevant background knowledge of this paper is clearly explained.

2. The topic of RGB-event ISP is very interesting topic and meanful for future camera.

3. The writing is very well and easy to understand.

weaknesses:
1. I greatly appreciate the effort to create a dataset for RGB-Event ISP, which opens up opportunities for event-assisted RGB ISP tasks. However, the dataset's scale is quite limited, with only 3,373 samples, which may not be sufficient to support data-driven learning methods. This raises concerns about the dataset's ability to serve as a professional, standardized, and challenging benchmark. If this is primarily a workload issue, could the authors consider generating some simulated datasets? I would appreciate an explanation regarding this.

2. The authors use images generated from a controllable ISP framework based on MATLAB as ground truth. While they provide extensive explanations for this approach, it is difficult to trust software-generated images as ground truth, especially for a professional dataset. This practice differs significantly from the ground truth methods used by existing sensor or smartphone manufacturers for their ISPs.

3. The integration of events into traditional ISP theoretically brings certain advantages, and the authors should elaborate on these benefits. Additionally, to demonstrate the advantages of event cameras, the authors should showcase scenes, particularly under high-speed motion or extreme lighting conditions, that highlight the potential benefits of using event-based approaches.

questions:
Please see the weaknesses. I have assigned a preliminary score based on the initial manuscript and will consider adjustments depending on the authors' responses and feedback from other reviewers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730718300057

thread_type:
official_review

<thread_object>
creator: reviewer_qdhh

<content>
summary:
This paper presents a novel dataset tailored for event-guided image signal processing (ISP), featuring 3,373 high-resolution RAW images paired with corresponding event data across various scenes, exposure modes, and lenses. This dataset aims to advance RGB-Event ISP research by enabling the development of methods that directly incorporate event information within the ISP pipeline. Besides, they introduce an event-guided ISP neural network as a baseline, fusing events with RAW data to optimize ISP tasks. Then, various ISP methods are evaluated on this dataset, establishing a foundation for future RGB-Event ISP advancements. Note that this work generated high-quality RGB images as ground truth by using a ColorChecker.

strengths:
1.	Dataset: This is the first dataset with aligned (in both time and spatial space) RAW and events from a new HVS equipment, which may provide some avenues for developing new ISP algorithms.
2.	Benchmark Task: This work evaluates the proposed dataset by testing various ISP baseline methods along with an event-guided ISP approach. Some conclusions and insights can be drawn from these experiments.

weaknesses:
1.	Scale of the dataset: From my understanding, a robust dataset should have both scale and diversity. Although the authors mention that the dataset is relatively small due to the low stability of the HVS sensor, I am not completely convinced by it, and I still believe it would be beneficial to expand the dataset further to increase its richness and variety. Additionally, the authors could include suggestions in the future work section on strategies for scaling up the dataset.
2.	Reproducibility and Usage of the dataset: Given that this is a dataset/benchmark paper, I strongly encourage the authors to release the data as soon as possible, even during the review stage. Early access is crucial for the community to begin evaluating and utilizing the dataset.
3.	Benchmark Task: This dataset is tailored for RAW-Event ISP tasks. So, I wonder if we should focus more on designing various Event-guided ISP baselines. In the current setup, most baselines do not utilize the event data, raising concerns about whether these limited event-guided baselines effectively demonstrate the dataset’s potential. I believe increasing the variety of event-driven baselines could better validate the usefulness of this dataset and highlight its unique contributions to event-guided ISP research.

questions:
My questions are highly overlapped with the weakness as follows:
1.	Dataset Scale and Diversity: Could you provide more insights into the limitations that prevented a larger dataset collection, and provide more insights about how to scale up the data?
2.	Early Data Release: Is there a timeline or plan for public release of the dataset, especially considering its importance for community validation and use? 
3.	Event-Guided ISP Baseline Evaluation: Given that the dataset is designed for RAW-Event ISP tasks, consider developing additional baseline methods that explicitly incorporate event data to better assess the dataset's strengths. For instance, could you explore a wider variety of event-guided architectures or fusion methods to better showcase the potential of the event data in ISP?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731464790631

thread_type:
official_review

<thread_object>
creator: reviewer_q8r6

<content>
summary:
This paper presents a event-RAW paired dataset of pixel-level aligned events and RAW images, including 3373 images across 24 scenes, for ISP process reforming. The study presents a conventional ISP pipeline that generates high-quality RGB frames for reference, performing basic ISP operations like demosaicing and white balancing. Some existing learnable ISP methods are trained and evaluated on the dataset.

strengths:
1. The present paper introduces a event-RAW paired dataset that addresses a gap in the field of ISP and facilitates further research on event-guided ISP. 
2. The trainable ISP methods are evaluated on this event-RAW paired dataset.

weaknesses:
This paper, as a work primarily contributing a dataset, provides insufficient information about the dataset itself and focuses heavily on performance comparisons with existing methods, but the proposed method fails to outperform the current ones across all metrics. The overall logic and structure of the paper need improvement and refinement.
1. As a dataset-centric paper, it presents too few sample images, making it difficult for readers to intuitively grasp the specific content of the dataset and the differences in the scenes, camera shots, and exposure modes mentioned.
2. The introduction to the specific content of the dataset is insufficient. It is recommended to list statistical information about the dataset regarding the mentioned types and present them in tables.
3. The ISP presented in the paper is used to handle demosaicing, white balance, denoising, and color space transformations, but its capabilities in handling these tasks are not reflected in the performance analysis results.
4. Figures 6 and 7 have identical legends and can be combined into one figure, or one could be replaced with the visualization results of indoor data samples.
5. The experimental section only presents quantitative analysis results for outdoor and indoor data separately, failing to reflect the overall performance of the dataset.
6. The presented integrated improvement method does not enhance processing performance, and the paper does not provide sufficient explanation or analysis of this issue.
7. The experimental visual results are too few, and the layout is too compact, making it difficult for readers to intuitively appreciate the value of the dataset.
8. It is recommended to analyze the performance of different types of samples within the dataset using the same method in future work, rather than focusing primarily on comparing the performance of multiple ISP methods.

questions:
1. The ISP method presented in this paper demonstrates varying effects across different tasks. It would be beneficial to analyze whether these effects are reflected in the proposed dataset.
2. The authors should explain and analyze the reasons why the proposed method fails to surpass existing methods in performance, as well as highlight its advantages.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732018844010

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary and Answers of Official Reviews (1/2)

comment:
# Summary and Answers of Official Reviews (1/2)

Dear Reviewers,

We sincerely thank all reviewers for their thoughtful feedback and constructive comments. We deeply appreciate the recognition of our contributions and the valuable suggestions provided to improve our work. Below, we summarize the key strengths of our paper as highlighted by the reviewers in the Table A. We are grateful for the reviewers' recognition of the novelty, practical contributions, and thorough analysis presented in our work. These acknowledgments motivate us to further improve the quality and impact of our research.

Table A: Recognized Contributions.
| **Contribution**    | **Reviewer** | **Official Review**      |
| ---------------------------------------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| **1. Novel Research Question**    | vrds  | "Event-guided ISP seems like a **novel and interesting** idea."    |
|       | fVak  | "The paper presents a **new** event-RAW paired dataset..."  |
|       | AvAS  | "The relevant background knowledge of this paper is **clearly explained**."      |
|       |       | "The topic of RGB-event ISP is a very **interesting and meaningful** topic for future cameras."       |
|       | q8r6  | "This paper **addresses a gap** in the field of ISP and **facilitates further** research on event-guided ISP."      |
| **2. New Real-world Dataset**     | vrds  | "A **novel sensor** is designed to capture the datasets..."        |
|       | fVak  | "This paper introduces the RGB-Event ISP dataset and benchmark, **a novel paired dataset** that combines RAW and event data."     |
|       |       | "The dataset **fills an existing gap** by providing paired RAW and event data, the **first** of its kind specifically designed for ISP tasks."  |
|       |       | "The proposed ISP pipeline is **flexible**, making it versatile for testing various ISP algorithms."  |
|       | AvAS  | "I **greatly appreciate** the effort to create a dataset for RGB-Event ISP, which **opens up opportunities** for event-assisted RGB ISP tasks." |
|       | qDHH  | "This is the **first dataset** providing avenues for developing new ISP algorithms."    |
|       |       | "This paper presents a novel dataset **across various** scenes, exposure modes, and lenses."   |
| **3. Conventional ISP Pipeline**  | vrds  | "A **conventional ISP** pipeline is proposed to generate RGB references..."      |
|       | fVak  | "The authors design a **customizable ISP** pipeline, allowing for benchmarking..."      |
|       | AvAS  | "Using a controllable ISP pipeline developed by the authors, **high-quality RGB frames** are generated."     |
|       | qDHH  | "This work generated **high-quality RGB** images as ground truth by using a ColorChecker."     |
|       | q8r6  | "The study presents a conventional ISP pipeline that generates **high-quality RGB** frames for reference."   |
|       |       | "A **conventional ISP** pipeline is proposed and learnable ISP methods are used."       |
| **4. Thorough Analysis of Benchmarking** | vrds  | "The **analysis and results** look good."     |
|       |       | "The experiments **demonstrate the effectiveness** of several methods in both outdoor and indoor scenes."    |
|       | fVak  | "The experimental **evaluation is thorough**."       |
|       |       | "The paper also **addresses its limitations** and **suggests potential solutions**."    |
|       | qDHH  | "This work evaluates the proposed dataset by **testing various ISP** baseline methods along with an event-guided ISP approach."   |
|       |       | "Some **conclusions and insights** can be drawn from these experiments."  |
|       | q8r6  | "The trainable ISP methods are evaluated on this event-RAW paired dataset."      |

We sincerely appreciate the reviewers' valuable feedback and constructive suggestions, which have guided us in refining our work. In response to your insightful comments, we have carefully addressed each point and made substantial updates to the paper. These revisions have significantly strengthened the robustness and clarity of our research. Thank you again for your support and suggestions, which have greatly enhanced the quality of our paper.

Best

ICLR-4440 Authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1732018877182

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary and Answers of Official Reviews (2/2)

comment:
Thank you to all the reviewers for their hard work and careful review. Here we answer the questions of common concern.

## Concern about Dataset Size for ISP tasks:

We acknowledge the reviewer’s concern regarding the limited scale of our dataset. Below, we provide a detailed explanation to justify the sufficiency and significance of our dataset for ISP tasks:

### 1. ISP Tasks Focus on Pixel-Level Data·
Unlike traditional perception tasks, such as face detection, which often require datasets with tens of thousands of examples, ISP tasks focus on processing pixel-level data.
In ISP, every pixel with different neighbors can be considered an example, making the requirements of dataset size fundamentally different.
Therefore, our dataset is not only sufficient in terms of quantity (3,373) but also provides high-resolution (2248 × 3264) samples that are effective for training and testing ISP models.

### 2. Size Comparison with Related Dataset
Our work is inspired by the MIPI Demosaic Workshop [a], which represents the most closely related study in this field. Compared to the MIPI dataset, our dataset is significantly larger, containing over 3,373 real images. This is four times the size of the MIPI dataset, which includes only 800 images. Moreover, our dataset offers higher-resolution images, with dimensions of 2248 × 3264 pixels (approximately 7.3 million pixels per image), while the MIPI dataset has resolutions around 2K, such as 2040 × 1356 pixels.

Although the MIPI dataset is only about one-fourth the size of ours, it is still sufficient to train large networks, such as transformers [b]. Our dataset also supports more comprehensive training and testing.

More importantly, the MIPI dataset is entirely composed of simulated data [a], whereas our dataset is based on real-world data, providing a more realistic representation for ISP tasks. Additionally, our dataset includes authentic event streams, enabling research into event-guided ISP, which was not possible with the MIPI dataset.

Furthermore, the representative works [c,d,e] in ISP is summarized in Table B. These datasets all contain no more than 200 training samples. However, they feature high-resolution images, which are sufficient to support effective model training and evaluation.

In summary, in ISP tasks, the size of the dataset is not only related to the number of images but also to their resolution. Our dataset is sufficient in both aspects.

Table B, Size Comparison of Related Datasets
| Dataset | Resolution | Scalse | Real-World | Events | Tasks| Publication |
| --- | --- | --- | --- | --- | --- | --- |
| Ours | 2248 × 3264 | 3373 | Yes | Yes | hybrid sensor ISP | |
| MIPI[a,b] | 2K, e.g. 2040 × 1356 | 800 | No | No | hybrid sensor ISP | CVPR 2024 |
| ISPW[c] | 1368 × 1824, 4480×6720 | 197 | Yes | No | ISP | ECCV 2022 |
| NR2R[d] | 3464×5202 | 150 | Yes | No | ISP | CVPR 2022 |
| DeepISP [e] | 3024×4032 | 110 | Yes | No | ISP | IEEE TIP 2018 |



### 3. Diversity in Scenes and Conditions
Our dataset includes a wide variety of lenses, scenes, and lighting conditions. This diversity supports comprehensive testing across different scenarios, ensuring the dataset’s relevance and utility for a broad range of ISP tasks.

To further substantiate the sufficiency of our dataset, we have included a new section in the revised appendix of our paper.
This table highlights the scale, diversity, and resolution of our dataset compared to existing ISP-related benchmarks.

### 4. Long-term Maintenance
Our dataset provides a comprehensive resource for ISP research. Its real-world nature distinguishes it from existing datasets and makes it particularly well-suited for event-guided ISP tasks. Moreover, we are committed to the long-term maintenance of this dataset and plan to expand it in the future to accommodate larger and more complex tasks.


## Reference

- [a] Wu, Yaqi, et al. "MIPI 2024 Challenge on Demosaic for Hybridevs Camera: Methods and Results." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
- [b] Xu, Senyan, et al. "DemosaicFormer: Coarse-to-Fine Demosaicing Network for HybridEVS Camera." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
- [c] Shekhar Tripathi, Ardhendu, et al. "Transform your smartphone into a dslr camera: Learning the isp in the wild." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.
- [d] Li, Zhihao, Si Yi, and Zhan Ma. "Rendering nighttime image via cascaded color and brightness compensation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
- [e] Schwartz, Eli, Raja Giryes, and Alex M. Bronstein. "Deepisp: Toward learning an end-to-end image processing pipeline." IEEE Transactions on Image Processing 28.2 (2018): 912-923.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732029489008

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revision Paper

comment:
Dear Reviewers,

We thank all reviewers' thorough and careful evaluations. Here, we provide a summary of the revisions made to our paper.

**Please note that due to file size limitations, we have compressed the PDF document, which may result in blurry images. However, the original, clear PDF can be downloaded from the supplementary materials.**

Below, we outline the key revisions made to the paper.

**Section A**

We have added a new section titled "HYBRID SENSOR IMAGING PROCESS, PRINCIPLES, AND POTENTIAL."
- (1) Firstly, we included the imaging principles to help readers better understand the sensor's details.
- (2) Next, we analyzed the advantages of event data through theoretical analysis and practical demonstrations. For example, in Figure 9, we present cases of fast motion and low-light scenes.
- (3) Moreover, we examined the time alignment in imaging, which addresses the issue of synchronizing the two types of data.
- (4) Finally, we explored the potential of our approach in various ISP tasks, providing insights for future research.

**Section B**

We have added more dataset examples and a scale analysis.
- (1) To address the reviewers' concerns, we included additional examples from our dataset.
- (2) Furthermore, we analyzed the size of our dataset and compared it with studies most similar to ours. Compared to the latest research, our dataset is four times larger.
- (3) We also compared our dataset with classic ISP methods, demonstrating that its scale is sufficient to support training.
- (4) **Diversity Testing with GPT-4o**: We conducted a comprehensive analysis of dataset diversity using ChatGPT-4. This includes statistical evaluations of captured scenes, objects, lighting conditions, and weather variations, offering a clearer understanding of the dataset's richness.

**Section E**

We added a discussion on how hyperparameters affect model training results.

**Section F**

We included a discussion of the results of various methods on the entire dataset.

Best

ICLR-4440 Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces an event-RAW paired dataset and an event-guided ISP pipeline, addressing a clear gap in the ISP domain. The reviewers acknowledged the significance of the dataset and the approach but raised concerns about the dataset's scale, reproducibility, and the method's limited performance gains. The authors' revisions addressed these concerns effectively by expanding dataset details, clarifying event integration, and providing additional evaluations. While some limitations remain, the work lays a good foundation for future research in event-guided ISP. The AC recommends acceptance for its valuable contribution to the community and potential to inspire further advancements.