Tasks

Image Segmentation

Image Segmentation divides an image into segments where each pixel in the image is mapped to an object. This task has multiple variants such as instance segmentation, panoptic segmentation and semantic segmentation.

Inputs
Image Segmentation Model
Output

About Image Segmentation

Use Cases

Autonomous Driving

Segmentation models are used to identify road patterns such as lanes and obstacles for safer driving.

Background Removal

Image Segmentation models are used in cameras to erase the background of certain objects and apply filters to them.

Medical Imaging

Image Segmentation models are used to distinguish organs or tissues, improving medical imaging workflows. Models are used to segment dental instances, analyze X-Ray scans or even segment cells for pathological diagnosis. This dataset contains images of lungs of healthy patients and patients with COVID-19 segmented with masks. Another segmentation dataset contains segmented MRI data of the lower spine to analyze the effect of spaceflight simulation.

Task Variants

Semantic Segmentation

Semantic Segmentation is the task of segmenting parts of an image that belong to the same class. Semantic Segmentation models make predictions for each pixel and return the probabilities of the classes for each pixel. These models are evaluated on Mean Intersection Over Union (Mean IoU).

Instance Segmentation

Instance Segmentation is the variant of Image Segmentation where every distinct object is segmented, instead of one segment per class.

Panoptic Segmentation

Panoptic Segmentation is the Image Segmentation task that segments the image both by instance and by class, assigning each pixel a different instance of the class.

Inference

You can infer with Image Segmentation models using the image-segmentation pipeline. You need to install timm first.

!pip install timm
model = pipeline("image-segmentation")
model("cat.png")
#[{'label': 'cat',
#  'mask': mask_code,
#  'score': 0.999}
# ...]

You can use huggingface.js to infer image segmentation models on Hugging Face Hub.

import { HfInference } from "@huggingface/inference";

const inference = new HfInference(HF_TOKEN);
await inference.imageSegmentation({
    data: await (await fetch("https://picsum.photos/300/300")).blob(),
    model: "facebook/mask2former-swin-base-coco-panoptic",
});

Useful Resources

Would you like to learn more about image segmentation? Great! Here you can find some curated resources that you may find helpful!

Compatible libraries

Image Segmentation demo
Models for Image Segmentation
Browse Models (988)

Note A multipurpose image segmentation model for high resolution images.

Note Panoptic segmentation model trained on the COCO (common objects) dataset.

Datasets for Image Segmentation
Browse Datasets (415)

No example dataset is defined for this task.

Note Contribute by proposing a dataset for this task !

Spaces using Image Segmentation

Note A semantic segmentation application that can predict unseen instances out of the box.

Note A human-centric segmentation model.

Note An instance segmentation application to predict neuronal cell types from microscopy images.

Note An panoptic segmentation application built for outdoor environments.

Metrics for Image Segmentation
Average Precision
Average Precision (AP) is the Area Under the PR Curve (AUC-PR). It is calculated for each semantic class separately
Mean Average Precision
Mean Average Precision (mAP) is the overall average of the AP values
Mean Intersection over Union
Intersection over Union (IoU) is the overlap of segmentation masks. Mean IoU is the average of the IoU of all semantic classes
APα
APα is the Average Precision at the IoU threshold of a α value, for example, AP50 and AP75