NanoVision — SEM Particle Segmentation (U-Net++)

Overview

NanoVision is a deep learning-based semantic segmentation model designed for automated particle identification and classification in Scanning Electron Microscopy (SEM) images.

The model uses a U-Net++ architecture to perform pixel-level segmentation of microscopic images, distinguishing between background regions, dark particles, and bright particles.

The project was developed as part of the NanoVision Bootcamp, organized in collaboration with Applied Materials and KamaTech.

The goal is to support accurate, efficient, and scalable SEM image analysis through deep learning.

Model Details

Property	Description
Model Name	NanoVision U-Net++
Architecture	U-Net++
Framework	PyTorch
Task	Semantic Segmentation
Domain	Scanning Electron Microscopy
Input	SEM images
Output	Pixel-wise classification masks
Number of Classes	3
Checkpoint	`unetplusplus_best.pt`
Development Team	NanoVision — AMAT-2 Efficiency

Segmentation Classes

The model predicts three semantic classes:

Class ID	Class Name	Description
0	Background	Non-particle regions
1	Dark	Dark particles
2	Bright	Bright particles

Important: The class mapping must remain consistent during training, inference, visualization, and evaluation.

The output mask contains integer class identifiers corresponding to these three categories.

Architecture

The model is based on U-Net++, a convolutional neural network architecture designed for image segmentation.

U-Net++ extends the original U-Net architecture through nested skip connections, enabling improved feature aggregation across multiple spatial resolutions.

Key architectural characteristics include:

Encoder-decoder segmentation architecture
Nested and dense skip connections
Multi-scale feature aggregation
Pixel-level semantic classification
Three-class segmentation output

The architecture is particularly relevant for microscopy applications where particles may differ in size, contrast, shape, and spatial distribution.

Dataset

The model was developed using SEM images containing particles with varying visual characteristics.

The dataset includes images with different magnifications, particle sizes, contrast levels, and microstructural patterns.

Data Preparation

The project incorporates:

Ground-truth segmentation masks
Dataset manifest and split management
Training and validation partitions
Image preprocessing
Crop-based training
Data augmentation
Class-consistent mask encoding

Class Encoding

All segmentation masks follow the convention:

0: Background
1: Dark particle
2: Bright particle

Invalid or ignored pixels may be represented by 255 in dataset annotations, according to the preprocessing and evaluation configuration.

Training

The model was trained for three-class semantic segmentation using supervised learning.

The training pipeline includes image preprocessing, crop-based sampling, data augmentation, model optimization, and validation-based checkpoint selection.

Training details such as the optimizer, learning rate, batch size, loss function, number of training steps, and checkpoint selection criterion should be taken from the final U-Net++ training configuration.

The repository contains the selected model checkpoint for inference.

Evaluation

Segmentation performance is evaluated using metrics including:

Metric	Purpose
Dice Score	Measures segmentation overlap
Intersection over Union (IoU)	Measures agreement between predicted and reference regions
Precision	Measures the proportion of predicted particle pixels that are correct
Recall	Measures the proportion of reference particle pixels successfully detected
Boundary F1 Score	Measures agreement between predicted and reference boundaries

Evaluation should distinguish between dark and bright particles and report results on the appropriate held-out dataset.

Validated numerical results can be added once the final benchmark configuration and checkpoint are confirmed.

Inference

The model receives a SEM image and produces a semantic segmentation mask.

Inference Workflow

Load the trained U-Net++ checkpoint.
Apply the preprocessing required by the training configuration.
Perform model inference.
Convert the predicted class scores into a segmentation mask.
Restore predictions to the original image dimensions when necessary.
Generate the final class mask or visualization overlay.

Output Format

The predicted segmentation mask contains the class IDs:

BACKGROUND = 0
DARK = 1
BRIGHT = 2

The model output can be used for visualization, quantitative particle analysis, or integration into downstream image-processing systems.

Deployment

The model is intended to support integration into a Python-based inference service.

A FastAPI inference server can load the model checkpoint during application startup and reuse the initialized model across prediction requests.

This approach avoids repeated model loading and supports a modular deployment architecture.

The model weights can be retrieved from Hugging Face Hub and loaded using the corresponding PyTorch architecture implementation.

Note: The checkpoint is not necessarily a standalone executable model. Loading may require the original architecture code, configuration, and compatible dependencies.

Intended Uses

Automated segmentation of SEM images
Identification of dark and bright particles
Microscopy image analysis
Materials science research
Particle distribution analysis
Experimental computer vision workflows

Limitations

The model may exhibit reduced performance under conditions that differ significantly from the training data.

Potential limitations include:

Very small particles
Low-contrast particle boundaries
Unseen magnification levels
Significant differences in image resolution
Imaging artifacts and noise
Domain shifts between SEM acquisition systems

Predictions should be validated before use in scientific or industrial decision-making.

Future Improvements

Potential directions for further research include:

Multi-scale inference
Improved segmentation of small particles
Magnification-aware preprocessing
Enhanced boundary refinement
Robustness evaluation across SEM datasets
Optimized inference latency
Automated particle statistics and measurements

Downloads last month: -; Downloads are not tracked for this model. How to track