Image Segmentation
Image Segmentation divides an image into segments where each pixel in the image is mapped to an object. This task has multiple variants such as instance segmentation, panoptic segmentation and semantic segmentation.
About Image Segmentation
Use Cases
Autonomous Driving
Segmentation models are used to identify road patterns such as lanes and obstacles for safer driving.
Background Removal
Image Segmentation models are used in cameras to erase the background of certain objects and apply filters to them.
Medical Imaging
Image Segmentation models are used to distinguish organs or tissues, improving medical imaging workflows. Models are used to segment dental instances, analyze X-Ray scans or even segment cells for pathological diagnosis. This dataset contains images of lungs of healthy patients and patients with COVID-19 segmented with masks. Another segmentation dataset contains segmented MRI data of the lower spine to analyze the effect of spaceflight simulation.
Task Variants
Semantic Segmentation
Semantic Segmentation is the task of segmenting parts of an image that belong to the same class. Semantic Segmentation models make predictions for each pixel and return the probabilities of the classes for each pixel. These models are evaluated on Mean Intersection Over Union (Mean IoU).
Instance Segmentation
Instance Segmentation is the variant of Image Segmentation where every distinct object is segmented, instead of one segment per class.
Panoptic Segmentation
Panoptic Segmentation is the Image Segmentation task that segments the image both by instance and by class, assigning each pixel a different instance of the class.
Inference
You can infer with Image Segmentation models using the image-segmentation
pipeline. You need to install timm first.
!pip install timm
model = pipeline("image-segmentation")
model("cat.png")
#[{'label': 'cat',
# 'mask': mask_code,
# 'score': 0.999}
# ...]
You can use huggingface.js to infer image segmentation models on Hugging Face Hub.
import { HfInference } from "@huggingface/inference";
const inference = new HfInference(HF_TOKEN);
await inference.imageSegmentation({
data: await (await fetch("https://picsum.photos/300/300")).blob(),
model: "facebook/mask2former-swin-base-coco-panoptic",
});
Useful Resources
Would you like to learn more about image segmentation? Great! Here you can find some curated resources that you may find helpful!
Compatible libraries
Note Solid semantic segmentation model trained on ADE20k.
Note Background removal model.
Note A multipurpose image segmentation model for high resolution images.
Note Powerful human-centric image segmentation model.
Note Panoptic segmentation model trained on the COCO (common objects) dataset.
No example dataset is defined for this task.
Note Contribute by proposing a dataset for this task !
Note A semantic segmentation application that can predict unseen instances out of the box.
Note A human-centric segmentation model.
Note An instance segmentation application to predict neuronal cell types from microscopy images.
Note An panoptic segmentation application built for outdoor environments.
- Average Precision
- Average Precision (AP) is the Area Under the PR Curve (AUC-PR). It is calculated for each semantic class separately
- Mean Average Precision
- Mean Average Precision (mAP) is the overall average of the AP values
- Mean Intersection over Union
- Intersection over Union (IoU) is the overlap of segmentation masks. Mean IoU is the average of the IoU of all semantic classes
- APα
- APα is the Average Precision at the IoU threshold of a α value, for example, AP50 and AP75