# Addressing Out-of-Label Hazard Detection in Dashcam Videos: Insights from the COOOL Challenge

Anh-Kiet Duong

L3i Laboratory, La Rochelle University
17042 La Rochelle Cedex 1 - France

anh.duong@univ-lr.fr

Petra Gomez-Krämer

L3i Laboratory, La Rochelle University
17042 La Rochelle Cedex 1 - France

petra.gomez@univ-lr.fr

## Abstract

This paper presents a novel approach for hazard analysis in dashcam footage, addressing the detection of driver reactions to hazards, the identification of hazardous objects, and the generation of descriptive captions. We first introduce a method for detecting driver reactions through speed and sound anomaly detection, leveraging unsupervised learning techniques. For hazard detection, we employ a set of heuristic rules as weak classifiers, which are combined using an ensemble method. This ensemble approach is further refined with differential privacy to mitigate overconfidence, ensuring robustness despite the lack of labeled data. Lastly, we use state-of-the-art vision-language models for hazard captioning, generating descriptive labels for the detected hazards. Our method achieved the highest scores in the Challenge on Out-of-Label in Autonomous Driving, demonstrating its effectiveness across all three tasks. Source codes are publicly available at https://github.com/ffyyytt/COOOL_2025.

## 1. Introduction

Detecting hazards in dynamic environments is a critical task in computer vision, essential for real-time applications like autonomous driving, surveillance, and human-computer interaction. Accurate hazard detection requires identifying potential dangers, understanding their context, and describing them meaningfully. The COOOL challenge [1] provides a unique benchmark by addressing the out-of-label problem, where annotations are limited to evaluation-only data. The challenge includes three tasks: the detection of driver state changes due to hazards, the identification of hazardous objects, and the generation of captions to describe these hazards. These tasks demand solutions that integrate temporal, spatial, and semantic reasoning, making it challenging and essential in computer vision.

Driver reaction detection and hazard detection can be

framed as vision classification problems, where advanced deep learning models have demonstrated state-of-the-art performance. These models often require large labeled datasets for effective training [8, 15]. Additionally, Zero-Shot Learning (ZSL), which leverages deep learning models, can perform classification on labels that were not present in the training data. However, while ZSL offers flexibility, it often struggles with fine-grained task performance and lacks the domain-specific knowledge necessary for accurate detection [11]. The generation of captions has seen significant advancements in recent years, particularly with the rise of Large Language Models (LLMs), which have shown promise in bridging the gap between vision and language understanding [13]. These developments have opened new possibilities for generating descriptive captions, although challenges remain in capturing key details throughout the entire video.

In this paper, we present our solution to the COOOL challenge [1], which achieved the first place. Our method addresses the out-of-label problem by leveraging advanced models and an optimized pipeline. By combining state-of-the-art vision and language models, we set new benchmarks on both the public and private leaderboards.

## 2. Method

In this section, we present our approach for hazard analysis in dashcam footage, addressing three tasks: detecting driver reactions (Section 2.1), identifying hazardous objects (Section 2.2), and generating hazard captions (Section 2.3). We combine anomaly detection, heuristic rules, and vision-language models for robust and accurate detection, securing first place in the COOOL [1, 2] challenge.

### 2.1. Driver reaction detection

To detect driver reactions to hazards, we propose two approaches: speed anomaly detection and sound anomaly detection. The first identifies sudden velocity changes, like abrupt braking or rapid acceleration, while the second cap-
tures anomalous sounds, such as shouting or emergency braking noises. Both methods rely on peak detection in unlabeled data [17], enabling effective anomaly detection without supervision. Together, they provide a robust solution for detecting driver reactions to hazards.

### 2.1.1 Speed anomaly detection

Driver reactions to hazards are often reflected in abrupt and irregular movements, such as sudden braking, rapid acceleration, or sharp changes in direction. These swift and unexpected maneuvers are critical indicators of hazardous events or challenging driving conditions. To effectively identify such reactions, we focus on detecting anomalies in the velocity profiles of objects and vehicles within the scene. However, directly analyzing object velocities within dashcam footage presents challenges due to the motion of the vehicle and the diverse movement patterns of surrounding objects, including those that are stationary, moving in the same direction, in the opposite direction, or other trajectories such as turning or crossing paths. Accounting for these varied behaviors is essential for robust and accurate anomaly detection in real-world driving environments.

To address these complexities, our method incorporates a two-stage approach. The steps of our approach are outlined in Algorithm 1. First, for each object in the scene, we compute the centroid of its bounding box and measure its frame-to-frame displacement. This displacement is modeled linearly over time, with the slope representing the object's velocity. Second, recognizing the instability of using velocity alone, we compute the vehicle's acceleration. By treating the velocity-time relationship of the vehicle as a linear model, we extract the slope as its acceleration. By identifying peaks in the computed acceleration values (Line 8 of Algorithm 1), we detect abrupt driver reactions indicative of hazardous events.

**Algorithm 1** Speed anomaly detection based on an object o

**Require:** $F_o$: list of frame numbers where the object o appears with annotated bounding boxes, *chunksize*: number of samples to compute the acceleration.

1. Compute the centroid of o in the first frame: $C_{F_o}[0]$

2. **for** $f = 1$ **to** length $(F_o)$ **do**

3. Compute the centroid of o: $C_{F_o}[f]$

4. Vehicle velocity to o<fnref n="1" />: $v_f \sim \frac{\|C_{F_o[f]} - C_{F_o[0]}\|}{f - F_o[0]}$

5. **if** $f \ge \text{chunksize}$ **then**

6. Acceleration<fnref n="1" />: $a_f \sim \frac{\|v_f - v_{F_o[f-\text{chunksize}]\|}{\text{chunksize}}$

7. **end if**

8. **if** found peak in a **then** return f

9. **end if**

10. **end for**

In Algorithm 1, the parameter *chunksize* smooths the data by reducing noise from fluctuating bounding box coordinates, which can cause inaccurate velocity and acceleration estimates. A larger value of *chunksize* improves the stability but may overlook brief, sudden events, while a smaller *chunksize* increases the sensitivity but can lead to false-positive detections due to noise.

### 2.1.2 Sound anomaly detection

In addition to analyzing speed, audio signals provide a valuable dimension for detecting driver reactions to hazards. Sudden and unusual sounds, such as the driver shouting, emergency braking noises, or the honking of a horn, often accompany critical events on the road [10]. These auditory cues are strong indicators of a driver's perception of danger or an imminent hazard. By leveraging the audio stream from dashcam recordings, anomalies in sound patterns can be identified and correlated with hazardous events, offering a complementary approach to speed-based detection methods. This multimodal analysis enhances the robustness of the system, particularly in scenarios where visual cues alone may not fully capture the driver's reactions.

To leverage sound data, we use an anomaly detection approach similar to speed analysis due to the lack of labels. Raw audio signals are preprocessed and normalized to reduce environmental noise (e.g., traffic sounds, background music) while preserving significant auditory cues like sudden loud noises or distinctive patterns. Peaks in the processed signal are identified to detect anomalies, such as shouting or emergency braking sounds, which indicate reactions to hazards. This unsupervised method enables the detection of critical auditory signals without relying on labeled data, enhancing the system's ability to capture diverse hazard scenarios.

### 2.2. Hazard detection

Detecting hazards in dashcam footage presents significant challenges due to the absence of labeled data. However, several heuristic rules can be employed to create weak classifiers that do not require labeled datasets. The following is a list of heuristic rules that we utilize as weak classifiers in our approach:

* Leverage pre-trained models on extensive datasets such as ImageNet [5, 8, 15] to classify objects and filter out those with labels that are unlikely to represent hazards, such as “car” or “traffic light”, which frequently appear but are less critical as hazards.

<footnote n="1">¹Instead of direct division, we use a linear regression model of the form $y = ax + b$, where a represents the desired value (velocity/acceleration each case). This approach reduces noise caused by inaccuracies in bounding box annotations, providing more reliable results.</footnote>
* Evaluate the proximity of an object to the center of the video frame, as hazards are more likely to be near the center point of the dashcam.

* Analyze the frame-by-frame position of objects to determine whether their movement direction differs from the vehicle's trajectory, as objects with differing movement directions are more likely to pose hazards.

* Assess whether the object is actively participating in traffic. For this step, road lane detection algorithms can be utilized. However, due to time constraints, we approximate traffic regions by defining fixed zones within the video and considering objects within these zones as traffic participants.

* Examine the number of frames in which an object appears and the area of the object's bounding box. Large objects that persist across many frames are less likely to represent sudden or unexpected hazards.

* Correlate the appearance of objects with moments when the driver exhibits reactions (identified in Section 2.1). Objects that appear close to these reaction points are more likely to be associated with hazards.

These individual rules are not entirely accurate in every situation, they offer a certain level of reliability. By combining these weak classifiers and leveraging diverse features, we construct a final model with improved performance and robustness. This ensemble approach mitigates the limitations of individual classifiers, enabling a more effective detection of hazards in complex driving scenarios.

To combine multiple weak classifiers, we employ a weighted ensemble approach [7], where the weights are estimated based on the performance of each heuristic rule. However, due to the absence of labeled data, these weight estimations are inherently uncertain and may only perform well for specific videos while lacking generalizability across diverse scenarios. To address this issue and prevent overconfidence in any particular rule, we incorporate differential privacy techniques. By perturbing the weights with noise drawn from a Gaussian distribution, controlled by a specified parameter $\epsilon$, we introduce robustness against overfitting to specific video contexts [3, 9].

After perturbing the weights, we aggregate predictions from multiple noisy weight configurations using a voting ensemble [7]. In this setup, each object is assigned a score based on the number of votes it receives from these ensemble predictions. Objects with the highest number of votes are deemed the most likely hazards. This dual-layer ensemble approach enhances the model's robustness, leveraging the strengths of individual classifiers while mitigating the impact of uncertain weight estimates.

## 2.3. Hazard captioning

With the rapid advancements in large language models (LLMs) and vision-language models, generating descriptive

captions for images and objects has become a powerful tool for understanding and interpreting visual data [6]. In this task, we leverage these developments to generate captions for identified hazards in dashcam footage. The objective is to provide meaningful and context-aware descriptions that help characterize the detected hazards, offering insights into their nature and potential risks.

To achieve this, we utilize state-of-the-art image captioning models such as BLIPv2 [13], BLIP [14], and CLIP [16], which are designed to generate captions based on the visual features of an image. These models are adept at recognizing a wide variety of objects and describing their attributes, making them ideal for our task. The process of generating captions is detailed in Algorithm 2, which illustrates how captions are assigned to each hazard based on bounding box areas across frames.

<caption>**Algorithm 2 Hazard captioning for object o**</caption>

**Require:** $F_o$: list of frame indices where the object o appears, $bbox_o[i]$: cropped bounding box image of the object o at the $i^{th}$ frame, $M$: list of captioning models, $R$: hash table initialized to 0.

1: **for** $f = 1$ **to** length($F_o$) **do**

2: &nbsp;&nbsp;&nbsp;&nbsp;**for each** captioning model $m$ in $M$ **do**

3: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Generate caption: text $\leftarrow m$ ($bbox_o[F_o[f]]$)

4: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Compute area: $A \leftarrow width$($bbox_o[F_o[f]]$) $\times$ height($bbox_o[F_o[f]]$)

5: &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Update score in hash table: $R[\text{text}] \leftarrow R[\text{text}] + A$

6: &nbsp;&nbsp;&nbsp;&nbsp;**end for**

7: &nbsp;&nbsp;&nbsp;&nbsp;**end for**

**return** arg max$_{\text{text} \in R}$ $R[\text{text}]$

In Algorithm 2, for each frame where the object appears, its bounding box is cropped, and a caption is generated using each captioning model $m$. The area of the bounding box is calculated and used as a weight to update a hash table that accumulates scores for each unique caption. At the end of the process, the caption with the highest score in the hash table is selected as the final description for the hazard. This scoring mechanism prioritizes captions associated with larger or more prominently visible bounding boxes, ensuring that the most relevant description is chosen.

The evaluation metric for this task only considers the first 35 characters of a caption and searches the ground-truth annotations for their presence. To improve we refine the scoring process in Algorithm 2 to operate at the word level instead of the entire text. Instead of assigning scores to entire captions, we distribute the bounding box area among each individual words in the text. Consequently, the hash table $R$ now stores word-score mappings rather than text-score mappings. Nouns and meaningful words are given extra by doubling their scores, while common stop words (e.g., "a",
"an", "the") are reduced [18]. Then, we double the score for objects that do not belong on the street (e.g., animals, trees). After scoring, we construct the final caption by selecting the highest-scoring words until the 35-character limit is met. This refinement ensures the selected captions are concise and aligned with the evaluation requirements. However, it may reduce the readability and interpretability of the generated captions for human users. As a result, we employ this refinement exclusively for metric optimization and do not integrate it into Algorithm 2. Instead, the algorithm retains its original design for broader applicability and clearer caption generation.

# 3. Experiments

In this section, we present the details of the COOOL dataset used for the challenge and summarize the experimental setup. We also provide the results of various methods applied to the three tasks, along with their performance on both the private and public leaderboards.

## 3.1. Challenge description

<figure><img src="image_1.png" /><figcaption>Frame 54 of video 0077</figcaption></figure>

<figure><img src="image_2.png" /><figcaption>Frame 98 of video 0115</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Frame 317 of video 0039</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Frame 55 of video 0104</figcaption></figure>

<caption>Figure 1. Sample frames from some videos of the COOOL dataset [1]. The red bounding box denotes the *challenge_object*, while the blue bounding box represents the *trafficScene* as labeled in the *annotations_public* file. The number of each bounding box corresponds to the tracking ID of the respective object.</caption>

The COOOL dataset [1] consists of 200 video clips with annotations in the *annotations_public* file, which includes bounding boxes for *challenge_objects* and *traffic_scenes*, each with a unique tracking ID. Sample frames from the dataset are shown in Figure 1, where the red and blue bounding boxes correspond to the *challenge_object* and *trafficScene* annotations, respectively. Unlike conventional datasets with fully labeled data, COOOL is an evaluation-only benchmark with sparse annotations, requiring participants to infer information without comprehensive labels. This setup increases task complexity and requires methods that generalize effectively. Participants are evaluated on three key challenges using the following metrics:

* Score for driver reaction detection:

$$ \frac{\text{correct.state.change prediction}}{\text{total frases}} $$

* Score of correctly identified bounding box(es) containing hazards (average across all frames):

$$ \frac{\text{correctPredicted hazards}}{\max(\text{known hazards}, \text{len (predicted hazards)})} $$

* Score for description (average across all frames):

$$ \frac{\text{correctPredicted caption}}{\max(\text{known caption}, \text{len (predicted caption)})} $$

## 3.2. Results

The results of the experiments are summarized in Table 1. The table shows the performance of various methods on the three tasks on both the private and public leaderboards. The public leaderboard uses 8% of the data, while the private leaderboard includes the remaining 92%. Notably, the method combining all tasks achieved the highest scores on both leaderboards, securing the first place overall in the challenge. As this is a new competition, there are limited existing methods available for direct comparison.

<table><caption>Table 1. Performance results for various methods on the COOOL challenge tasks. Where "False" or "-1" represent fixed values assigned to the predictions for that task.</caption><thead><tr><th></th><th colspan="2">Method</th><th colspan="2">Score</th></tr><tr><th></th><th>Task 1</th><th>Task 2</th><th>Task 3</th><th>Private</th><th>Public</th></tr></thead><tbody><tr><td>speed (2.1.1)</td><td>-1</td><td>-1</td><td>-1</td><td>0.25340</td><td>0.27579</td></tr><tr><td>sound (2.1.2) + speed</td><td>-1</td><td>-1</td><td>-1</td><td>0.27534</td><td>0.28458</td></tr><tr><td>False</td><td colspan="2">proposed method (2.2)</td><td>-1</td><td>0.29901</td><td>0.35704</td></tr><tr><td>sound + speed</td><td colspan="2">proposed method</td><td>blip2-opt-6.7b [13, 19]</td><td>0.51694</td><td>0.76830</td></tr><tr><td>sound + speed</td><td colspan="2">proposed method</td><td>blip2-flan-t5-xxl [4, 13]</td><td>0.51663</td><td>0.69252</td></tr><tr><td>sound + speed</td><td colspan="2">proposed method</td><td>blip [14]</td><td>0.49240</td><td>0.61384</td></tr><tr><td>sound + speed</td><td colspan="2">proposed method</td><td>vit-gpt2 [12]</td><td>0.47598</td><td>0.59966</td></tr><tr><td>Baseline</td><td colspan="2">Baseline</td><td>Baseline</td><td>0.25560</td><td>0.25681</td></tr><tr><td>sound + speed</td><td colspan="2">proposed method</td><td>all model</td><td>0.57261</td><td>0.78453</td></tr></tbody></table>

The baseline scores in Table 1 correspond to the default method provided by the challenge organizers. In this baseline approach, the first task is based on a model that detects if the velocity is negative compared to other objects, the second task identifies the object closest to the center, and the third task uses the CLIP [16] model for caption generation.

# 4. Conclusion

We have proposed an effective hazard analysis framework for dashcam footage, addressing three key tasks: detecting driver reactions to hazards, identifying hazardous objects, and generating descriptive captions for these hazards. By leveraging an ensemble of multiple methods for each task, our approach ensures robust and stable performance in hazard analysis. This method, tested on the COOOL dataset, demonstrates the effectiveness of combining various strategies to tackle hazard analysis in real-world driving scenarios.
## References

[1] Ali K AlShami, Ananya Kalita, Ryan Rabinowitz, Khang Lam, Rishabh Bezbarua, Terrance Boult, and Jugal Kalita. Coool: Challenge of out-of-label a novel benchmark for autonomous driving. arXiv preprint arXiv:2412.05462, 2024. 1, 4

[2] Ali K. AlShami and Ryan Rabinowitz. Challenge of out-of-label in autonomous driving. https://kaggle.com/competitions/coolwacv25, 2024. Kaggle. 1

[3] Zitao Chen and Karthik Pattabiraman. Overconfidence is a dangerous thing: Mitigating membership inference attacks by enforcing less confident prediction. arXiv preprint arXiv:2307.01610, 2023. 3

[4] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70):1–53, 2024. 4

[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, pages 248–255. Ieee, 2009. 2

[6] Quang Minh Dinh, Minh Khoi Ho, Anh Quan Dang, and Hung Phong Tran. Trafficvlm: A controllable visual language model for traffic video captioning. In Conference on Computer Vision and Pattern Recognition, pages 7134–7143, 2024. 3

[7] Xibin Dong, Zhiwen Yu, Wenming Cao, Yifan Shi, and Qianli Ma. A survey on ensemble learning. Frontiers of Computer Science, 14:241–258, 2020. 3

[8] Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. 1, 2

[9] Yann Fraboni, Martin Van Waerebeke, Kevin Scaman, Richard Vidal, Laetitia Kameni, and Marco Lorenzi. Sifu: Sequential informed federated unlearning for efficient and provable client unlearning in federated optimization. In International Conference on Artificial Intelligence and Statistics, pages 3457–3465. PMLR, 2024. 3

[10] Rubens Cruz Gatto and Carlos Henrique Quartucci Forster. Audio-based machine learning model for traffic congestion detection. IEEE Transactions on Intelligent Transportation Systems, 22(11):7200–7207, 2020. 2

[11] Jingcai Guo, Zhijie Rao, Zhi Chen, Jingren Zhou, and Dacheng Tao. Fine-grained zero-shot learning: Advances, challenges, and prospects. arXiv preprint arXiv:2401.17766, 2024. 1

[12] Ankur Kumar. The illustrated image captioning using transformers. ankur3107.github.io, 2022. 4

[13] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, pages 19730–19742. PMLR, 2023. 1, 3, 4

[14] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In International Conference on Machine Learning, pages 12888–12900. PMLR, 2022. 3, 4

[15] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Conference on Computer Vision and Pattern Recognition, pages 12009–12019, 2022. 1, 2

[16] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021. 3, 4

[17] Dominique T Shipmon, Jason M Gurevitch, Paolo M Piselli, and Stephen T Edwards. Time series anomaly detection; detection of anomalous drops with limited features and sparse examples in noisy highly periodic data. arXiv preprint arXiv:1708.03665, 2017. 2

[18] Catarina Silva and Bernardete Ribeiro. The importance of stop word removal on recall values in text categorization. In International Joint Conference on Neural Networks, volume 3, pages 1661–1666. IEEE, 2003. 4

[19] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. 4