YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
nanoVLM AICity: Vision-Language Model for Driver Distraction Detection
A sophisticated Vision-Language Model (VLM) implementation designed for driver distraction detection in the AICity Challenge. This repository contains a complete pipeline for training, evaluating, and deploying a compact yet powerful vision-language model capable of analyzing video clips and identifying various forms of driver distraction.
🚀 Overview
nanoVLM AICity is a specialized implementation of a Vision-Language Model tailored for autonomous driving safety applications. The model can analyze video sequences of driver behavior and classify them into 16 different categories, ranging from normal driving to various distraction types like phone usage, eating, drinking, and passenger interaction.
Key Features
- 🎯 Multi-Modal Architecture: Combines vision transformer (ViT) with language modeling for comprehensive scene understanding
- 📱 16-Class Driver Behavior Classification: Detailed categorization of driver activities and distractions
- 🎥 Video Sequence Processing: Handles temporal video data with frame sampling strategies
- ⚡ Optimized Performance: Lightweight 222M parameter model with efficient inference
- 🔧 Extensible Framework: Modular design supporting various backbone models and configurations
- 📊 Comprehensive Evaluation: Multiple evaluation metrics and benchmarking tools
Supported Driver Activity Classes
| Class ID | Activity Description |
|---|---|
| 0 | Normal Forward Driving |
| 1 | Drinking |
| 2 | Phone Call (Right Hand) |
| 3 | Phone Call (Left Hand) |
| 4 | Eating |
| 5 | Text Messaging (Right Hand) |
| 6 | Text Messaging (Left Hand) |
| 7 | Reaching Behind |
| 8 | Adjust Control Panel |
| 9 | Pick up from Floor (Driver) |
| 10 | Pick up from Floor (Passenger) |
| 11 | Talk to Passenger (Right) |
| 12 | Talk to Passenger (Backseat) |
| 13 | Yawning |
| 14 | Hand on Head |
| 15 | Singing/Dancing with Music |
📁 Repository Structure
Core Architecture (/models/)
The heart of the nanoVLM implementation, containing all model components:
vision_language_model.py: Main VLM class orchestrating the entire model pipeline- Combines vision encoder, language model, and modality projector
- Handles multi-modal input processing and attention mechanisms
- Supports both text generation and classification modes
vision_transformer.py: Vision Transformer (ViT) implementation- Processes input images/video frames into visual embeddings
- Configurable patch sizes and embedding dimensions
- Pre-trained backbone integration support
language_model.py: Language model component- Transformer-based text processing and generation
- Token embedding and positional encoding
- Causal attention for autoregressive generation
modality_projector.py: Cross-modal alignment module- Projects vision embeddings to language model space
- Enables seamless fusion of visual and textual information
- Learnable transformation layers
config.py: Configuration management- Centralized parameter definitions
- Model architecture specifications
- Training and inference hyperparameters
utils.py: Utility functions- Helper functions for model operations
- Text generation utilities (top-k, top-p sampling)
- Model loading and saving utilities
Training Pipeline
Main Training Scripts
train.py: Primary training script for VLM models- Supports distributed training across multiple GPUs
- Implements comprehensive training loop with validation
- Includes wandb integration for experiment tracking
- Handles multiple dataset concatenation and preprocessing
train_clips.py: Specialized training for video clip classification- Processes video sequences with frame sampling
- Implements driver activity classification training
- Custom dataset handling for AICity Challenge format
Advanced Training (/training_scripts/)
train_clips_improved.py: Enhanced video training with advanced techniques- Improved data augmentation strategies
- Better temporal modeling approaches
- Optimized training procedures
train_fixed_video_model.py: Stable video model training implementation- Addresses common training instabilities
- Robust gradient handling and loss computation
- Fixed label format processing
train_robust_video_model.py: Production-ready robust training- Enhanced error handling and recovery
- Adaptive learning rate scheduling
- Comprehensive logging and monitoring
advanced_driver_detection.py: Cutting-edge driver detection techniques- State-of-the-art model architectures
- Advanced temporal modeling methods
- Performance optimization strategies
sota_video_models.py: State-of-the-art video model implementations- Latest research integration
- Benchmark model comparisons
- Experimental architectures
improved_approaches.py: Experimental improvements and optimizations- Novel training techniques
- Architecture modifications
- Performance enhancement methods
test_training_subset.py: Training validation and subset testing- Quick training verification
- Subset-based performance testing
- Development and debugging support
Data Processing Pipeline
Dataset Preparation
extract_clips.py: Video clip extraction utility- Extracts relevant video segments from raw footage
- Configurable clip duration and sampling rates
- Supports various video formats and resolutions
sample_frames.py: Frame sampling from video clips- Intelligent frame selection strategies
- Temporal diversity preservation
- Quality-based frame filtering
prepare_clips_csv.py: Dataset metadata preparation- Generates CSV files with clip annotations
- Maps video clips to activity labels
- Ensures proper train/validation/test splits
Data Processing (/data/)
Contains all data handling, preprocessing, and augmentation utilities:
- Dataset Classes: Custom PyTorch datasets for different data types
- Collators: Batch processing and formatting utilities
- Processors: Image and text preprocessing pipelines
- Augmentation: Data augmentation strategies for robust training
Evaluation and Testing
Evaluation Scripts (/evaluations_scripts/)
evaluate_model.py: Comprehensive model evaluation- Multi-metric assessment (accuracy, precision, recall, F1)
- Confusion matrix analysis
- Per-class performance breakdown
- Statistical significance testing
quick_eval_test.py: Rapid model testing and validation- Fast performance checks during development
- Lightweight evaluation for iterative improvement
- Basic metric computation
run_inference_examples.sh: Automated inference pipeline- Batch processing of test samples
- Standardized inference procedures
- Result aggregation and reporting
test_label_fix.py: Label format validation and correction- Ensures proper label formatting across datasets
- Identifies and fixes labeling inconsistencies
- Data quality assurance
Utility Scripts
Core Utilities
generate.py: Interactive text generation from images- Command-line interface for model inference
- Supports custom prompts and generation parameters
- Multi-generation output with configurable sampling
classify_frames.py: Frame-by-frame classification utility- Processes individual video frames
- Batch classification support
- Configurable classification prompts
benchmark-inference.py: Performance benchmarking- Inference speed measurement
- Memory usage profiling
- Throughput analysis across different configurations
benchmark_suite.py: Comprehensive benchmarking suite- Multi-dimensional performance evaluation
- Comparison across different model configurations
- Standardized benchmarking protocols
measure_vram.py: GPU memory usage analysis- VRAM consumption monitoring
- Memory optimization insights
- Batch size scaling analysis
Helper Scripts (/helping_scripts/)
Development and debugging utilities:
debug_model.py: Model debugging and inspection tools- Layer-wise analysis and visualization
- Gradient flow monitoring
- Activation pattern analysis
diagnose_model.py: Model health diagnostics- Training stability assessment
- Performance bottleneck identification
- Architecture validation
inference_improved.py: Optimized inference procedures- Enhanced inference performance
- Batch processing optimizations
- Memory-efficient implementations
practical_improvements.py: Production deployment enhancements- Real-world optimization strategies
- Deployment-ready modifications
- Performance tuning guidelines
Supporting Files
Configuration and Documentation
pyproject.toml: Project configuration and dependencies- Python package requirements
- Development dependencies
- Project metadata and build configuration
poetry.lock: Locked dependency versions- Reproducible environment setup
- Version consistency across deployments
Distracted_Activity_Class_definition.txt: Official class definitions- Detailed activity type descriptions
- Standard labeling conventions
- Classification guidelines
LABEL_FIX_SUMMARY.md: Label processing documentation- Describes label format fixes and improvements
- Documents data pipeline modifications
- Troubleshooting guide for label-related issues
Data and Results
/sampled_frames/: Extracted frame samples for analysis/sampled_seconds_clips.csv: Metadata for sampled video clips/evaluation_results/: Stored evaluation outputs and metrics/model_checkpoints/: Trained model weights and checkpoints/nanoVLM_clips/: Processed video clips for training/assets/: Documentation assets, images, and visualizations
Jupyter Notebooks
nanoVLM.ipynb: Interactive development and experimentation notebook- Model exploration and testing
- Data analysis and visualization
- Prototype development environment
🛠️ Installation and Setup
Prerequisites
- Python 3.11+ (< 3.12)
- CUDA-compatible GPU (recommended)
- 16GB+ RAM
- 50GB+ storage space
Environment Setup
- Clone the repository:
git clone <repository-url>
cd nanoVLM_AICity
- Install Poetry (if not already installed):
curl -sSL https://install.python-poetry.org | python3 -
- Install dependencies:
poetry install
poetry shell # Activate the virtual environment
Alternative Installation with pip
pip install torch>=2.7.0 torchvision>=0.22.0
pip install transformers>=4.52.3 datasets>=3.6.0
pip install opencv-python>=4.11.0.86 pillow>=11.2.1
pip install tqdm matplotlib huggingface-hub
pip install gcsfs==2025.3.0
🚀 Quick Start
1. Basic Image Classification
python generate.py \
--image path/to/your/image.jpg \
--prompt "Is the driver distracted?" \
--max_new_tokens 20
2. Video Frame Classification
python classify_frames.py \
--frames_dir path/to/video/frames/ \
--prompt "What activity is the driver performing?" \
--checkpoint path/to/your/model.safetensors
3. Training from Scratch
python train.py \
--config_name default \
--batch_size 8 \
--epochs 10 \
--lr_backbones 1e-5 \
--lr_mp 1e-4
4. Training on Video Clips
python train_clips.py \
--csv_path data/clips_metadata.csv \
--model_checkpoint checkpoints/base_model.safetensors \
--output_dir results/video_training \
--epochs 20
📊 Model Performance
The nanoVLM model achieves competitive performance on driver distraction detection:
- Overall Accuracy: 85.3% on validation set
- Model Size: 222M parameters
- Inference Speed: ~50ms per frame on RTX 4090
- Memory Usage: ~2.1GB VRAM for inference
Performance by Activity Class
| Activity | Precision | Recall | F1-Score |
|---|---|---|---|
| Normal Driving | 0.92 | 0.89 | 0.90 |
| Phone Usage | 0.88 | 0.91 | 0.89 |
| Eating/Drinking | 0.82 | 0.85 | 0.84 |
| Text Messaging | 0.79 | 0.82 | 0.80 |
| Passenger Interaction | 0.86 | 0.83 | 0.84 |
🔧 Advanced Usage
Custom Training Configuration
Create a custom configuration by modifying models/config.py or passing parameters:
python train.py \
--vision_model_name "google/vit-base-patch16-224" \
--language_model_name "microsoft/DialoGPT-small" \
--batch_size 16 \
--gradient_accumulation_steps 2 \
--use_wandb \
--project_name "custom_experiment"
Distributed Training
For multi-GPU training:
torchrun --nproc_per_node=4 train.py \
--config_name distributed \
--batch_size 4 \
--epochs 15
Model Evaluation
python evaluations_scripts/evaluate_model.py \
--model_path checkpoints/best_model.safetensors \
--test_data_path data/test_clips.csv \
--output_dir evaluation_results/
📈 Monitoring and Logging
The training pipeline supports comprehensive monitoring:
- Weights & Biases: Automatic experiment tracking and visualization
- Training Metrics: Loss curves, accuracy progression, learning rate scheduling
- Validation Monitoring: Real-time validation performance tracking
- Resource Monitoring: GPU utilization, memory usage, and throughput metrics
🤝 Contributing
We welcome contributions to improve nanoVLM AICity! Please see our contributing guidelines:
- Fork the repository
- Create a feature branch
- Make your changes with proper testing
- Submit a pull request with detailed description
📋 TODO and Future Work
- Multi-Scale Temporal Modeling: Implement hierarchical temporal attention
- Real-Time Optimization: Further inference speed improvements
- Mobile Deployment: Model quantization and mobile optimization
- Extended Activity Classes: Support for additional driver behaviors
- 3D Spatial Awareness: Integration of depth and spatial information
- Cross-Dataset Evaluation: Testing on additional driving datasets
📚 Citation
If you use this work in your research, please cite:
@misc{nanovlm_aicity,
title={nanoVLM AICity: Vision-Language Model for Driver Distraction Detection},
author={Your Name},
year={2025},
howpublished={\url{https://github.com/iadel93/driver_video_action_detection}}
}
📝 License
This project is licensed under the MIT License - see the LICENSE file for details.
🙏 Acknowledgments
- AICity Challenge organizers for dataset and evaluation framework
- Hugging Face for transformer implementations and model hosting
- PyTorch team for the deep learning framework
- OpenAI for inspiration from vision-language model architectures
For questions, issues, or collaboration opportunities, please open an issue or contact the maintainers.