Package: segmentation-models-pytorch
Python library with Neural Networks for ImageSegmentation based onPyTorch.The main features of this library are:High level API (just two lines to create a neural network)9 models architectures for binary and multi class segmentation (including legendary Unet)124 available encoders (and 500+ encoders fromtimm)All encoders have pre-trained weights for faster and better convergencePopular metrics and losses for training routines📚 Project Documentation 📚VisitRead The Docs Project Pageor read following README to know more about Segmentation Models Pytorch (SMP for short) library📋 Table of contentQuick startExamplesModelsArchitecturesEncodersTimm EncodersModels APIInput channelsAuxiliary classification outputDepthInstallationCompetitions won with the libraryContributingCitingLicense⏳ Quick start1. Create your first Segmentation model with SMPSegmentation model is just a PyTorch nn.Module, which can be created as easy as:importsegmentation_models_pytorchassmpmodel=smp.Unet(encoder_name="resnet34",# choose encoder, e.g. mobilenet_v2 or efficientnet-b7encoder_weights="imagenet",# use `imagenet` pre-trained weights for encoder initializationin_channels=1,# model input channels (1 for gray-scale images, 3 for RGB, etc.)classes=3,# model output channels (number of classes in your dataset))seetablewith available model architecturesseetablewith available encoders and their corresponding weights2. Configure data preprocessingAll encoders have pretrained weights. Preparing your data the same way as during weights pre-training may give you better results (higher metric score and faster convergence). It isnot necessaryin case you train the whole model, not only decoder.fromsegmentation_models_pytorch.encodersimportget_preprocessing_fnpreprocess_input=get_preprocessing_fn('resnet18',pretrained='imagenet')Congratulations! You are done! Now you can train your model with your favorite framework!💡 ExamplesTraining model for pets binary segmentation with Pytorch-LightningnotebookandTraining model for cars segmentation on CamVid datasethere.Training SMP model withCatalyst(high-level framework for PyTorch),TTAch(TTA library for PyTorch) andAlbumentations(fast image augmentation library) -hereTraining SMP model withPytorch-Lightningframework -here(clothes binary segmentation by@ternaus).📦 ModelsArchitecturesUnet [paper] [docs]Unet++ [paper] [docs]MAnet [paper] [docs]Linknet [paper] [docs]FPN [paper] [docs]PSPNet [paper] [docs]PAN [paper] [docs]DeepLabV3 [paper] [docs]DeepLabV3+ [paper] [docs]EncodersThe following is a list of supported encoders in the SMP. Select the appropriate family of encoders and click to expand the table and select a specific encoder and its pre-trained weights (encoder_nameandencoder_weightsparameters).ResNetEncoderWeightsParams, Mresnet18imagenet / ssl / swsl11Mresnet34imagenet21Mresnet50imagenet / ssl / swsl23Mresnet101imagenet42Mresnet152imagenet58MResNeXtEncoderWeightsParams, Mresnext50_32x4dimagenet / ssl / swsl22Mresnext101_32x4dssl / swsl42Mresnext101_32x8dimagenet / instagram / ssl / swsl86Mresnext101_32x16dinstagram / ssl / swsl191Mresnext101_32x32dinstagram466Mresnext101_32x48dinstagram826MResNeStEncoderWeightsParams, Mtimm-resnest14dimagenet8Mtimm-resnest26dimagenet15Mtimm-resnest50dimagenet25Mtimm-resnest101eimagenet46Mtimm-resnest200eimagenet68Mtimm-resnest269eimagenet108Mtimm-resnest50d_4s2x40dimagenet28Mtimm-resnest50d_1s4x24dimagenet23MRes2Ne(X)tEncoderWeightsParams, Mtimm-res2net50_26w_4simagenet23Mtimm-res2net101_26w_4simagenet43Mtimm-res2net50_26w_6simagenet35Mtimm-res2net50_26w_8simagenet46Mtimm-res2net50_48w_2simagenet23Mtimm-res2net50_14w_8simagenet23Mtimm-res2next50imagenet22MRegNet(x/y)EncoderWeightsParams, Mtimm-regnetx_002imagenet2Mtimm-regnetx_004imagenet4Mtimm-regnetx_006imagenet5Mtimm-regnetx_008imagenet6Mtimm-regnetx_016imagenet8Mtimm-regnetx_032imagenet14Mtimm-regnetx_040imagenet20Mtimm-regnetx_064imagenet24Mtimm-regnetx_080imagenet37Mtimm-regnetx_120imagenet43Mtimm-regnetx_160imagenet52Mtimm-regnetx_320imagenet105Mtimm-regnety_002imagenet2Mtimm-regnety_004imagenet3Mtimm-regnety_006imagenet5Mtimm-regnety_008imagenet5Mtimm-regnety_016imagenet10Mtimm-regnety_032imagenet17Mtimm-regnety_040imagenet19Mtimm-regnety_064imagenet29Mtimm-regnety_080imagenet37Mtimm-regnety_120imagenet49Mtimm-regnety_160imagenet80Mtimm-regnety_320imagenet141MGERNetEncoderWeightsParams, Mtimm-gernet_simagenet6Mtimm-gernet_mimagenet18Mtimm-gernet_limagenet28MSE-NetEncoderWeightsParams, Msenet154imagenet113Mse_resnet50imagenet26Mse_resnet101imagenet47Mse_resnet152imagenet64Mse_resnext50_32x4dimagenet25Mse_resnext101_32x4dimagenet46MSK-ResNe(X)tEncoderWeightsParams, Mtimm-skresnet18imagenet11Mtimm-skresnet34imagenet21Mtimm-skresnext50_32x4dimagenet25MDenseNetEncoderWeightsParams, Mdensenet121imagenet6Mdensenet169imagenet12Mdensenet201imagenet18Mdensenet161imagenet26MInceptionEncoderWeightsParams, Minceptionresnetv2imagenet /  imagenet+background54Minceptionv4imagenet /  imagenet+background41Mxceptionimagenet22MEfficientNetEncoderWeightsParams, Mefficientnet-b0imagenet4Mefficientnet-b1imagenet6Mefficientnet-b2imagenet7Mefficientnet-b3imagenet10Mefficientnet-b4imagenet17Mefficientnet-b5imagenet28Mefficientnet-b6imagenet40Mefficientnet-b7imagenet63Mtimm-efficientnet-b0imagenet / advprop / noisy-student4Mtimm-efficientnet-b1imagenet / advprop / noisy-student6Mtimm-efficientnet-b2imagenet / advprop / noisy-student7Mtimm-efficientnet-b3imagenet / advprop / noisy-student10Mtimm-efficientnet-b4imagenet / advprop / noisy-student17Mtimm-efficientnet-b5imagenet / advprop / noisy-student28Mtimm-efficientnet-b6imagenet / advprop / noisy-student40Mtimm-efficientnet-b7imagenet / advprop / noisy-student63Mtimm-efficientnet-b8imagenet / advprop84Mtimm-efficientnet-l2noisy-student474Mtimm-efficientnet-lite0imagenet4Mtimm-efficientnet-lite1imagenet5Mtimm-efficientnet-lite2imagenet6Mtimm-efficientnet-lite3imagenet8Mtimm-efficientnet-lite4imagenet13MMobileNetEncoderWeightsParams, Mmobilenet_v2imagenet2Mtimm-mobilenetv3_large_075imagenet1.78Mtimm-mobilenetv3_large_100imagenet2.97Mtimm-mobilenetv3_large_minimal_100imagenet1.41Mtimm-mobilenetv3_small_075imagenet0.57Mtimm-mobilenetv3_small_100imagenet0.93Mtimm-mobilenetv3_small_minimal_100imagenet0.43MDPNEncoderWeightsParams, Mdpn68imagenet11Mdpn68bimagenet+5k11Mdpn92imagenet+5k34Mdpn98imagenet58Mdpn107imagenet+5k84Mdpn131imagenet76MVGGEncoderWeightsParams, Mvgg11imagenet9Mvgg11_bnimagenet9Mvgg13imagenet9Mvgg13_bnimagenet9Mvgg16imagenet14Mvgg16_bnimagenet14Mvgg19imagenet20Mvgg19_bnimagenet20MMix Vision TransformerBackbone from SegFormer pretrained on Imagenet! Can be used with other decoders from package, you can combine Mix Vision Transformer with Unet, FPN and others!Limitations:encoder isnotsupported by Linknet, Unet++encoder is supported by FPN only for encoderdepth = 5EncoderWeightsParams, Mmit_b0imagenet3Mmit_b1imagenet13Mmit_b2imagenet24Mmit_b3imagenet44Mmit_b4imagenet60Mmit_b5imagenet81MMobileOneApple's "sub-one-ms" Backbone pretrained on Imagenet! Can be used with all decoders.Note: In the official github repo the s0 variant has additional num_conv_branches, leading to more params than s1.EncoderWeightsParams, Mmobileone_s0imagenet4.6Mmobileone_s1imagenet4.0Mmobileone_s2imagenet6.5Mmobileone_s3imagenet8.8Mmobileone_s4imagenet13.6M*ssl,swsl- semi-supervised and weakly-supervised learning on ImageNet (repo).Timm EncodersdocsPytorch Image Models (a.k.a. timm) has a lot of pretrained models and interface which allows using these models as encoders in smp, however, not all models are supportednot all transformer models havefeatures_onlyfunctionality implemented that is required for encodersome models have inappropriate stridesTotal number of supported encoders: 549table with available encoders🔁 Models APImodel.encoder- pretrained backbone to extract features of different spatial resolutionmodel.decoder- depends on models architecture (Unet/Linknet/PSPNet/FPN)model.segmentation_head- last block to produce required number of mask channels (include also optional upsampling and activation)model.classification_head- optional block which create classification head on top of encodermodel.forward(x)- sequentially passxthrough model`s encoder, decoder and segmentation head (and classification head if specified)Input channelsInput channels parameter allows you to create models, which process tensors with arbitrary number of channels.
If you use pretrained weights from imagenet - weights of first convolution will be reused. For
1-channel case it would be a sum of weights of first convolution layer, otherwise channels would be
populated with weights likenew_weight[:, i] = pretrained_weight[:, i % 3]and than scaled withnew_weight * 3 / new_in_channels.model=smp.FPN('resnet34',in_channels=1)mask=model(torch.ones([1,1,64,64]))Auxiliary classification outputAll models supportaux_paramsparameters, which is default set toNone.
Ifaux_params = Nonethen classification auxiliary output is not created, else
model produce not onlymask, but alsolabeloutput with shapeNC.
Classification head consists of GlobalPooling->Dropout(optional)->Linear->Activation(optional) layers, which can be
configured byaux_paramsas follows:aux_params=dict(pooling='avg',# one of 'avg', 'max'dropout=0.5,# dropout ratio, default is Noneactivation='sigmoid',# activation function, default is Noneclasses=4,# define number of output labels)model=smp.Unet('resnet34',classes=4,aux_params=aux_params)mask,label=model(x)DepthDepth parameter specify a number of downsampling operations in encoder, so you can make
your model lighter if specify smallerdepth.model=smp.Unet('resnet34',encoder_depth=4)🛠 InstallationPyPI version:$pipinstallsegmentation-models-pytorchLatest version from source:$pipinstallgit+https://github.com/qubvel/segmentation_models.pytorch🏆 Competitions won with the librarySegmentation Modelspackage is widely used in the image segmentation competitions.Hereyou can find competitions, names of the winners and links to their solutions.🤝 ContributingInstall SMPmakeinstall_dev# create .venv, install SMP in dev modeRun tests and code checksmakeall# run flake8, black, testsUpdate table with encodersmaketable# generate table with encoders and print to stdout📝 Citing@misc{Iakubovskii:2019,
  Author = {Pavel Iakubovskii},
  Title = {Segmentation Models Pytorch},
  Year = {2019},
  Publisher = {GitHub},
  Journal = {GitHub repository},
  Howpublished = {\url{https://github.com/qubvel/segmentation_models.pytorch}}
}🛡️ LicenseProject is distributed underMIT License