acd23/sam2-car-glass-segmentation

SAM 2.1 Car Glass Segmentation

Fine-tuned facebook/sam2.1-hiera-tiny for car interior glass segmentation (windshield, side windows, rear window).

Model Description

This model is fine-tuned on the Roboflow car-bg dataset (v3, COCO segmentation format) to segment glass regions in car interior images. The goal is to identify windshield and window glass areas so that exterior content visible through the glass can be removed while keeping interior elements (rear-view mirrors, side mirrors, dashboard, etc.) intact.

Training Details

Parameter	Value
Base model	facebook/sam2.1-hiera-tiny (39M params)
Dataset	Roboflow slz/car-bg v3 (340 train, 18 valid, 9 test images)
Trainable params	4,215,365 (13.4%) — mask decoder only
Frozen	vision_encoder, prompt_encoder
Loss	DiceCELoss (monai)
Optimizer	Adam (lr=1e-5)
Batch size	2
Epochs	100
Best val loss	0.0348
Prompt strategy	Bounding box from GT mask with random perturbation (0-20px)

Usage

Inference with point prompts (recommended)

from transformers import Sam2Model, Sam2Processor
import torch
from PIL import Image

model = Sam2Model.from_pretrained("acd23/sam2-car-glass-segmentation").to("cuda")
processor = Sam2Processor.from_pretrained("acd23/sam2-car-glass-segmentation")

image = Image.open("car_interior.jpg").convert("RGB")

# Use point prompts to segment glass
input_points = [[[[400, 300]]]]  # Point on the windshield
input_labels = [[[1]]]  # Positive click

inputs = processor(images=image, input_points=input_points, input_labels=input_labels, return_tensors="pt").to("cuda")
with torch.no_grad():
    outputs = model(**inputs, multimask_output=False)

masks = processor.post_process_masks(outputs.pred_masks.cpu(), inputs["original_sizes"])[0]

Full inference pipeline (auto-segmentation + glass removal)

# Install dependencies
pip install torch transformers opencv-python-headless pillow numpy

# Run on a single image
python inference_car_glass.py --image_path car_interior.jpg --output_dir ./output --remove_glass

# Run on a directory of images
python inference_car_glass.py --image_dir ./images --output_dir ./output --remove_glass

# Run with bbox prompts instead of point prompts
python inference_car_glass.py --image_path car.jpg --output_dir ./output --use_bbox

Using with bounding box prompts

# Provide a bounding box around the glass region
input_boxes = [[[75, 275, 1725, 850]]]  # [x_min, y_min, x_max, y_max]
inputs = processor(images=image, input_boxes=input_boxes, return_tensors="pt").to("cuda")
with torch.no_grad():
    outputs = model(**inputs, multimask_output=False)
masks = processor.post_process_masks(outputs.pred_masks.cpu(), inputs["original_sizes"])[0]

Pipeline for Car Interior Segmentation

1. Detect glass regions: Use the fine-tuned model with point or bbox prompts to segment windshield, side windows, and rear window glass
2. Remove exterior content: Fill the glass mask regions with black (or any fill value) to remove exterior views through the glass
3. Keep mirrors: Rear-view and side mirrors are separate objects — their masks are not filled, preserving their content

Trackio Dashboard

Training metrics are logged to: https://huggingface.co/spaces/acd23/ml-intern-sam2car

Limitations

• Trained on a small dataset (340 images) — may not generalize to all car interiors
• Single class ("glass") — does not distinguish between windshield, side windows, and rear window
• For best results, use with point or bounding box prompts on the glass region
• Mirrors are not explicitly segmented — they are preserved by not being included in the glass mask

Generated by ML Intern

This model repository was generated by ML Intern, an agent for machine learning research and development on the Hugging Face Hub.

• Try ML Intern: https://smolagents-ml-intern.hf.space
• Source code: https://github.com/huggingface/ml-intern