High Dynamic Range Modulo Imaging for Robust Object Detection in Autonomous Driving

Authors: kebin  contreras
brayan  monroy
jorge  bacca
Venue: thecvf_com_ECCV_2024_Workshop_ROAM
Published: 2024-07-31 23:00:39

Abstract: Object detection precision is crucial for ensuring the safety and efficacy of autonomous driving systems. The quality of acquired images directly influences the ability of autonomous driving systems to correctly recognize and respond to other vehicles, pedestrians, and obstacles in real-time. However, real environments present extreme variations in lighting, causing saturation problems and resulting in the loss of crucial details for detection. Traditionally, High Dynamic Range (HDR) images have been preferred for their ability to capture a broad spectrum of light intensities, but the need for multiple captures to construct HDR images is inefficient for real-time applications in autonomous vehicles. To address these issues, this work introduces the use of modulo sensors for robust object detection. The modulo sensor allows pixels to `reset/wrap' upon reaching saturation level by acquiring an irradiance encoding image which can then be recovered using unwrapping algorithms. The applied reconstruction techniques enable HDR recovery of color intensity and image details, ensuring better visual quality even under extreme lighting conditions at the cost of extra time. Experiments with the YOLOv10 model demonstrate that images processed using modulo images achieve performance comparable to HDR images and significantly surpass saturated images in terms of object detection accuracy. Moreover, the proposed modulo imaging step combined with HDR image reconstruction is shorter than the time required for conventional HDR image acquisition.

Main paper: 
 introduction: 
 Object detection is a critical component of autonomous driving. The ability to accurately identify and respond appropriately to other vehicles, pedestrians, and obstacles depends largely on the quality of the acquired images. Highquality imaging is vital for enabling precise object detection, which is essential for the safe operation of autonomous vehicles in both urban and highway environments [16] . One of the acquisition problems is saturation, which can be identified as areas where pixels have reached maximum intensity, losing important information about the scene [6] . Addressing light saturation issues is crucial for the accuracy of object detection systems. Figure 1 illustrates common challenges such as overexposure due to direct sunlight (a), reflections of artificial nightlights (b) and loss of road details due to solar saturation (c), encountered in autonomous driving environments. An alternative is the well-known High Dynamic Range (HDR) imaging technique, which combines multiple exposure images to capture a broader spectrum of light intensities [15] . This approach effectively addresses the issues of underexposure and overexposure. However, the time required to process these images makes them impractical for real-time applications in autonomous driving [9] .A solution to address these challenges is the introduction of modulo sensor technology, which employs an irradiance encoding scheme. This technology allows pixels to 'reset/wrap' upon reaching saturation, thereby enabling continuous data recording. Although modulo images do not directly maintain the integrity of color at the moment of capture, they preserve sufficient details to enhance object detection. Furthermore, due to similarities with phase unwrapping problems, certain unwrapping algorithms can be employed to obtain HDR images during post-processing, which is crucial for achieving high visual quality under variable lighting conditions [11, 19] . This technology is especially valuable in dynamic environments where lighting conditions can change rapidly and dramatically, providing a solution to preserve image information in cluttered areas.To the best of our knowledge, this is the first work to introduce the modulo sensor as an alternative for obtaining HDR images to enhance robust object detection in autonomous driving. We specifically evaluate the performance of object detection using YOLOv10 [14] models in various saturation scenarios and demonstrate that modulo measurements outperform commercial images under extreme conditions without the need for retraining. Furthermore, we propose to adapt the Simultaneous Phase Unwrapping and Denoising (SPUD) algorithm for direct HDR estimation from modulo images [10] . This approach significantly improves object detection results without significantly increasing processing time, making the modulo sensor a promising solution for autonomous driving systems. 2 BackgroundModulo Imaging. The image formation process in a modulo sensor involves the use of a modulo operator to reset the intensity values upon reaching a specific threshold of 2 b . This imaging process can be mathematically modeled as followsEQUATIONwhere y represents the acquired modulo image, Q b (•) represents the quantization process to b bits, x denotes the desired HDR image and mod(•) refers to the modulo operator. The implementation of this particular imaging process has been explored with self-reset Analog-to-Digital Converter (ADC) [18] and with programmable sensors [11] . More recently, with the advancements of the Unlimited Sampling Framework, it is now possible to implement it using modulo-ADCs in continuous time, i.e., prior sampling [2] . Once the modulo image y is captured, a recovery process is needed to get an estimation of the HDR image x. However, this recovery process is an optional step since the object detection network can be employed directly to raw modulo image.Modulo Recovery. Given that the modulo operator is a non-linear transformation, it is impractical to directly employ numerical optimization techniques to reconstruct from the modulo image y. However, similar to the unwrapping problem, the spatial differences in the HDR image, ∆x, matches with the wrapped spatial differences of the modulo image, expressed as ∆x = M b (∆y), where M b (∆y) = W b (∆y + 2 b-1 ) -2 b-1 is a centered modulo operator to align negative values [7, 12] . This assumption holds if the natural image discontinuities do not exceed the threshold 2 b-1 , namely, ∥∆x∥ ∞ < 2 b-1 , also referred to as Itoh's condition [7] . Consequently, the HDR image recovery can be mathematically formulated as the following optimization problemEQUATIONwhere the regularization operator R(•) is introduced to exploit additional knowledge about the structure of the HDR image x or to handle challenging recovery scenarios due to the presence of noise [1, 3, 13] . In the subsequent section, we introduce SPUD [10] , a non-interactive method to solve Equation 2 under sparsity assumption. 
methodology: 
 In this section, we detail our proposed methodology consisting of a two-step pipeline to process traffic images using modulo sensors: (A) Acquisition and Recovery with modulo sensors and (B) Object Detection on modulo images or over HDR recovered images.Step A. Acquisition and Recovery with Modulo Sensors: The first step consists of replacing the conventional CCD sensor with a modulo sensor to acquire a modulo image y following Equation (1) . Then, the acquired modulo image y can be fed into a modulo recovery algorithm to restore the HDR image x. We employ SPUD [10] which computes the HDR image recovery as followsEQUATIONIn ( 3), M and N represent the dimensions of the image, D denotes the discrete cosine transform (DCT) [10] , and T is the hard-thresholding operator with threshold parameter τ [10] . For simplicity, we denote the computation of Equation (3) by the reconstruction operator x = SPUD(y, b, τ ), effectively combining noise reduction and image unwrapping in a single step.Although new algorithms have been used to obtain HDR images from modulo images [1, 19] , the effectivity of the SPUD is mainly attributed to its non-iterative approach with an order of complexity of O (n log(n)). This method significantly improves the quality of the restored HDR image with a low latency time, which is crucial for real-time applications such as autonomous driving [10] .Step B. Object Detection on Modulo and Recovered Images: The second step involves performing an autonomous driving task on the modulo image and recovered HDR images. Consequently, a family of YOLOv10 detection models is used to identify the classes present in the autonomous driving datasets during the object detection task. This process can be represented asEQUATIONwhere C = {c (i) } N i is the set of boundary boxes provided by the YOLOv10 detection model for N different objects detected in the input image, and p is a boolean parameter indicating whether to use the restored HDR image x or the raw modulo image y. Here, each boundary box c is constructed as c = [j 1 , k 1 , j 2 , k 2 , z], where (j 1 , k 1 ) and (j 2 , k 2 ) are the top-left and bottom-right coordinates of the box, and z denotes the predicted object class. We evaluate object detection using various configurations of the YOLOv10 model, including the variants n, s, m, b, l, x, following an ascending order in terms of complexity and the number of model parameters. The proposed two-step methodology is summarized in Algorithm 1.  
simulation and results: 
 This section evaluates the robustness of ODMI compared to standard digital cameras. Our main goal is to analyze the object detection performance under varying saturation conditions for different versions of the YOLOv10 detection model without retraining. For this, we emulated the acquisition of modulo images, the reconstruction of the HDR image from modulo images using the SPUD algorithm [10] , and the acquisition of a saturated image in the KITTI database. Specifically, to simulate the different saturation conditions, we normalize the HDR image x raw to the range [0, 2 b ], with b = 8, and introduce a saturation factor α, which scales the HDR image to x = αx raw . The saturation scenario for a standard camera is modeled asEQUATIONwhere x sat is the saturated image, and α is the saturation factor that adjusts the image intensity to simulate different levels of light exposure. We evaluate the performance of different imaging methodologies from low to extreme saturation scenarios, by selected α values of 1.5, 2, 3, and 4. For emulating the modulo camera and HDR image reconstruction, we use Equation ( 1) and Equation ( 3), respectively, using the same scaled HDR image x as input.Dataset. The proposed methodology was evaluated using the KITTI dataset [5] , which is a benchmark in object detection focused on autonomous driving. This dataset contains 7,481 images, each with dimensions of 1242×375, and captures urban scenes featuring a variety of objects, including vehicles, pedestrians, cyclists, miscellaneous objects, sitting people, trams, trucks, and vans, making it a significant challenge for vision-based object detection systems.Model. The YOLOv10 model family, with configurations n, s, m, b, l, and x between 2.3M to 29.5M parameters, was used to assess object detection effectiveness. These models consist of a CNN backbone with detection layers and anchor boxes for predicting object-bound boxes. Additionally, the integration of partial self-attention modulo enhances feature extraction and localization accuracy [14] .Metrics.The object detection performance for all scenarios (image-saturated, modulo sensor, recovery, and ideal HDR) was evaluated using: Intersection over Union (IoU), the F1-score, and Accuracy. These metrics are standard benchmarks that enable an objective comparison of the performance of different YOLO-v10 variants under diverse image manipulation conditions. In the experiment, 100 testing images from the KITTI database were selected to represent different saturation conditions. The experiments were conducted on a computer equipped with an AMD Ryzen 7 5700X 8-Core Processor running at 3.40 GHz, 64.0 GB of RAM, and an NVIDIA GeForce RTX 4070 GPU with 12 GB of VRAM.The analysis of the different image techniques under varying levels of light saturation is presented in Table 1 . The models evaluated, from best to worst performance, are x, l, b, m, s, n, in relation to the number of trainable parameters described in Table 2 . As the value of α increases, the IOU, F1-Score, and accuracy metrics decrease due to the loss of information in saturated areas, as shown in 1 . Notice that the metric in Table 1 for the ideal HDR image is constant for different α values because the HDR image is not affected by α and does not suffer from saturation since we assume perfect recovery of the HDR image. This analysis shows that Modulo and Recovery image offer better spatial information and robust object detection under high saturation compared to the Saturated image. The Ideal HDR image, unaffected by saturation, serves as a reference for evaluation as it represents the ideal scene. By simulating the processes of saturation, modulo sensor, reconstruction, and ideal HDR, we can compare the inference times between different methodologies. For the Saturated and Modulo cases, we assume an ideal camera with an acquisition time of 33 ms, incorporating the inference time of the YOLOv10 models. For the Recovery case, the inference time includes the SPUD algorithm time of 5.1 ms ± 1.9 ms, using different of α value (1.5, 2, 3, 4). For ideal HDR images, we assume three captures of 33 ms each to obtain the final HDR image. This approach follows the standard procedure of using three images taken at different exposure times to create an HDR image presented in [8] . Table 2 summarizes the parameters and inference times for each model. As YOLO complexity decreases, the parameters and inference times is reduced. Recovery time from a modulo image is generally lower than the time for ideal HDR images using the conventional multiple exposure time technique. The Recovery time encompasses both image reconstruction and YOLO detection, whereas the remaining time refers exclusively to detection.  
conclusion: 
 This work proposes the use of modulo sensor combined with the SPUD reconstruction algorithm to enhance object detection in autonomous driving under high illumination conditions. Our results show a significant improvement over standard cameras, which are often affected by saturation issues. SPUD reconstructed images closely approximate HDR quality for object detection, demonstrating that phase unwrapping with modulo sensors achieves comparable results. Additionally, HDR recovery from modulo images is processed faster than multi-exposure HDR images, which benefits real-time processing in dynamic environments. The most complex version of the YOLOv10 model x) performed best, followed by versions l, b, m, s, and n, highlighting the advantage of increased model complexity for dynamic imaging. Interestingly, the object detection quality of the saturated images decreases significantly compared to the modulo sensor, confirming the effectiveness of this new sensor in preserving detail and maintaining higher object detection accuracy. 
 
 Appendix:  


Decision: The average score of the paper given all the reviews received before the deadline was higher than 5.5 (1 is lowest, 10 is highest), therefore the paper is accepted. The Authors are encouraged to consider feedback for the camera ready version of the paper due on August 31st.

**This was also the highest scoring paper of all submitted papers to the ROAM workshop of ECCV 2024. Congratulations to all the Authors!**

Reviews: review: title: title: HDR modulo imaging for robust object detection in autonomous driving
main_review: main_review: Presents an innovative method utilizing module sensors to improve object detection in autonomous vehicles, effectively tackling the challenge of light saturation. The approach is robust, and the experimental results show that this technique can surpass saturated images and compete with traditional HDR images in both accuracy and efficiency. The paper could strengthen its argument by providing more extensive experimental results, including diverse lighting conditions and a broader set of scenarios.
title: title: Module sensors enhances object detection in saturated images
main_review: main_review: The authors propose the use of modulo sensors for HDR imaging in autonomous driving, addressing critical issues with saturation in extreme lighting conditions. The experimental results demonstrate the effectiveness of the YOLOv10 model combined with SPUD reconstruction. The fact that modulo imaging outperforms conventional HDR methods in both detection accuracy and processing time is interesting and relevant to the workshop.


Pros:
Relevance: The method is directly applicable to autonomous vehicles, improving object detection accuracy under challenging conditions.
Performance Gains: Experimental results show significant improvements in detection accuracy and processing time compared to traditional HDR methods.

Cons:
Implementation Complexity: Incorporating modulo sensors and SPUD reconstruction adds complexity to the system, particularly it still requires the computation of the HDR image x.
Processing Overhead: The additional reconstruction step may introduce some processing delays, which could be a concern in real-time applications. The authors didn't mention how to accelerate efficient inference methods (quantization methods) to achieve near real-time object prediction while computing modulo imaging on the fly.

Recommendation:
Accept. The paper proposes a novel solution and clear experimental benefits show their benefits.
title: title: Modulo Sensor as potential alternative of HDR imaging for object detection.
main_review: main_review: The authors argue that, for the purpose of object detection, full HDR image capturing can be substituted with a Modulo sensor based camera. A mainstream object detection network could use the raw image with the wrapped around intensity values or the "unwrapped" HDR image as input. For unwrapping the authors use the SPUD algorithm.
To support these claims the paper compares the performance of a popular pretrained object detection model on the emulated raw modulo image, the reconstructed HDR and the original HDR image which was used  to produce the modulo images in the first place by artificially oversaturating them.

### **Strengths**: 
1. The paper is well structured and easy to follow
2. Tackles a relevant issue of information loss due to pixel saturation and runtime improvements.
3. Using both the raw and the reconstructed images for demonstration.

### **Opportunities for improvement**:
1. Expanded test set for emulated data.
2. Simulating the all too frequent local oversaturation (glare) would be welcome addition to the uniformly stretched intensity.
3. Fig 1 in the supplementary material is easier to follow than Fig 2 in the main paper as it uses the same image over the whole alpha range.
4. I understand the difficulties but acquiring real world modulo sensor images and putting them to the test would be convincing.
title: title: HDR modulo imaging for robust object detection in autonomous driving - Review
main_review: main_review: The authors have provided an excerpt from a paper that discusses the use of High Dynamic Range (HDR) modulo imaging for robust object detection in autonomous driving. The paper outlines the challenges of imaging under varying lighting conditions, where traditional HDR imaging methods are too slow for real-time applications. The authors propose using modulo sensors, which reset upon reaching saturation, allowing for HDR recovery through unwrapping algorithms.

The methodology is tested on object detection using various configurations of the YOLOv10 model, demonstrating that the modulo imaging technique improves detection performance compared to traditional saturated images, especially under extreme lighting conditions. The results highlight that modulo images combined with HDR recovery can be processed more efficiently than conventional HDR acquisition methods, making this approach suitable for real-time autonomous driving scenarios.

The paper appears to be thorough, including mathematical modeling, algorithmic details, and experimental evaluations, with promising results for real-world applications in autonomous driving.


### Pros:
1. **Better Dynamic Range:** Captures bright and dark areas clearly, enhancing visibility in tough lighting.
2. **Improved Object Detection:** Enhances accuracy, especially in challenging conditions.
3. **Less Noise:** Produces clearer images with reduced graininess in low light.
4. **Flexible Design:** Easier to integrate into existing systems.
5. **Effective in Difficult Scenarios:** Performs well in tricky environments like nighttime driving.

### Cons:
1. **High Processing Demand:** Requires significant computing power, possibly slowing performance.
2. **Expensive Hardware:** May need costly sensors or components.
3. **Integration Challenges:** Potential difficulties in fitting with current vehicle systems.
4. **Risk of Visual Errors:** Could produce inaccuracies in fast-moving settings.
5. **Limited Testing:** Might not be fully proven in diverse real-world conditions.
title: title: Modulo sensor for robust object detection
main_review: main_review: This paper proposed robust objection detection by processing through modulo image steps. Experiments are conducted using YOLO v10 model demonstrating higher object detection accuracy compared to HDR images. Overall the paper is well-motivated and clearly structured. The results shown are convincing. I have two comments that are worth the authors looking into:
 (1)  In Table 1, quantitative evaluation of saturated, modulo, and recovery imaging are conducted using accuracy metrics. I notice that YOLO v10 model "m" accuracy is higher than YOLO v10 "b". Considering YOLO v10 "b" contains more parameters, the accuracy should be higher than YOLO v10 "m". This is contradictory to other YOLO models as well as other metrics.

(2) It would be more convincing if authors could test the algorithm in more datasets not just KITTI.
title: title: Simulated modulo camera performs well on downstream object-detection task
main_review: main_review: Summary and Contributions

The Authors evaluated the zero-shot object-detection performance of pretrained YOLOv10 models on simulated modulo camera images created from the KITTI benchmark images using manual overexposure. The object-detection benchmark results were compared on raw overexposed images, simulated modulo images and HDR images recovered from the modulo images using the real-time SPUD algorithm. The Authors successfully demonstrate a significant performance increase by running the models on modulo images against the raw oversaturated images. Moreover, HDR recovery has been shown to further increase the performance for most of the models and overexposure rates at a minor computational cost.

Strengths

Saturation of images is a general limitation of conventional CCD cameras in highly dynamic scenes, which is especially relevant in autonomous driving. Sudden changes in brightness can cause overexposure, causing serious degradation of camera-based perception. This work examines whether using modulo-camera images along with a subsequent phase-unwrapping recovery algorithm could restore the quality of downstream perception tasks, by retrieving details lost by conventional cameras. As a generic perception task, object detection has been evaluated. The reported results are quite convincing, suggesting that using modulo cameras could be a viable choice to mitigate the degradation caued by overexposure. 

Opportunities for Improvement 

Whereas object detection is indeed a crucial part of the perception stack of most autonomous driving systems, other tasks should be evaluated as well. Especially dense pixel-level tasks such as segmentation or depth estimation could be affected by the false colors introduced by the modulo sampling or a noisy recovery. Nevertheless, the recovered images apparently exhibit much more details than the saturated images, which probably aids any downstream computer-vision task. In this manner the reported results confirm this naiive expectation. 
On the other hand, the domain gap between the recovered modulo images and the original unsaturated HDR images the models have been trained on seems to be still significant. A possible follow-up work could assess the usage of different recovery algorithms to close this domain gap. 
Furthermore the object-detection models could be trained on the KITTI dataset using simulated modulo images and the performance of the models could be re-assessed. This would establish a fair comparison between the performance on the modulo images and the raw HDR baseline. 

Limitations 

Even though modulo cameras have been proposed quite a while ago, I have not heard about them being used in autonomous driving systems. Even this work does not evaluate images obtained from such a camera, which makes me assume that there is some fundamental limitation why these hardwares are not so widespread. I am missing a more detailed discussion of strenghts and limitations of such cameras for industry use. Also, the results would have been even more convincing if real modulo-camera images had been used for evaluation instead of the simulated ones. However, I understand that it is straightforward to use a well-established benchmark and the artificially oversaturated images provide a good baseline for the comparison. 
Whereas I find the paper quite intriguing, I am missing the novelty component. The potential of modulo imaging to recover details of oversaturated images has already been demonstrated and several different HDR recovery algorithms have been proposed (Zhao et al ICCP 2015, Zhou et al NIPS 2020, Pineda J. et al Applied Optics 2020). The fact that object detection is also improved is quite expected due the recovered details in the image. On the other hand there is no discussion of the reduced performance with respect to the unsaturated baseline, which would really increase its impact.

Relation To Prior Work 

There is a short section titled Background, however a more extensive discussion of the pros and cons of modulo cameras is missing. Also there is no overview of the usage of such cameras in the industry. Some reconstruction algorithms are mentioned in the text, but they are not discussed extensively. 

Conclusion

Altogether, I think the main virtue of the paper is popularizing modulo imaging as a possible alternative to conventional cameras. With all the missing details and lack of discussion of the possible limitations of the technique, I still think that the paper is worth publishing at the ROAM workshop, as modulo imaging may be a novelty for many researchers in the computer-vision community.
title: title: Review: High Dynamic Range Modulo Imaging for Robust Object Detection in Autonomous Driving
main_review: main_review: Summary:
To address the problem of object detection in extreme lighting conditions, this paper proposes the novel usage of modulo sensors and adaption of Simultaneous Phase Unwrapping and Denoising (SPUD) algorithm. Such approaches to recover HDR images from modulo images improves results greatly with a reasonable impact to efficiency. The authors also evaluate their approach using YOLOv10 and confirm the effectiveness.

Pros:
- Novel idea
- Convincing and promising preliminary results
- Clear structure and easy to follow

Cons:
- Could have more comparisons with prior methods
- Limited model/dataset (sufficient for a workshop paper, but definitely benefits to have more)
title: title: Modulo imaging for handling extreme conditions
main_review: main_review: **Summary and Contributions**

The paper proposes the use of modulo sensors for enabling object detection in extreme lighting conditions. The authors utilize modulo imaging and the SPUD algorithm to recover an estimated HDR image in O(n log(n)) time. This way, the quality of the restored HDR image makes it possible to use conventional object detection methods that otherwise would fail on the saturated images.

The paper is well-written and easy to follow. The motivation is clear and the problem is important since saturation can severely decrease detection performance and hinder downstream tasks built on the perception stack. The experimental results are promising and seem to support the author's claims. Furthermore, the runtime analysis helps to evaluate whether the method can be further used in real time.
 
**Strengths**
- The problem addressed by the paper is important and relevant to the workshop.
- Introducing modulo sensing as an alternative for obtaining HDR images is a good idea.
- Well-designed experiments with promising initial results.
 
**Opportunities for Improvement**
- Increase the test set size and use other datasets for investigating the generalization ability of the proposed method
- It would be great to see how the proposed method works on real-world data instead of the emulated one. This would require the collection of a dataset which is a large effort but would aid the evaluation of the method.
- Extending the experiments with different models (e.g., segmentation, depth) would increase the value of the paper.