Package: segmentation-pipeline
Segmentation Training PipelineThis package is a part ofMusket MLframework.Reasons to use Segmentation PipelineSegmentation Pipeline was developed with a focus of enabling to make fast and
simply-declared experiments, which can be easily stored,
reproduced and compared to each other.Segmentation Pipeline has a lot of common parts withGeneric pipeline, but it is easier to define an architecture of the network.
Also there are a number of segmentation-specific features.The pipeline provides the following features:Allows to describe experiments in a compact and expressive wayProvides a way to store and compare experiments in order to methodically find the best deap learning solutionEasy to share experiments and their results to work in a teamExperiment configurations are separated from model definitionsIt is easy to configure network architectureProvides great flexibility and extensibility via support of custom substancesCommon blocks like an architecture, callbacks, model metrics, predictions vizualizers and others should be written once and be a part of a common libraryInstallationpip install segmentation_pipelineNote: this package requires python 3.6This package is a part ofMusket MLframework,
it is recommended to install the whole collection of the framework
packages at once using instructionshere.LaunchingLaunching experimentsfit.pyscript is designed to launch experiment training.In order to run the experiment or a number of experiments,A typical command line may look like this:musket fit --project "path/to/project" --name "experiment_name" --num_gpus=1 --gpus_per_net=1 --num_workers=1 --cache "path/to/cache/folder"--projectpoints to the root of theproject--nameis the name of the project sub-folder containing experiment yaml file.--num_gpussets number of GPUs to use during experiment launch.--gpus_per_netis a maximum number of GPUs to use per single experiment.--num_workerssets number of workers to use.--cachepoints to a cache folder to store the temporary data.Other parameters can be found in thefit script referenceLaunching taskstask.pyscript is designed to launch experiment training.Tasks must be defined in the project python scope and marked by an
annotation like this:frommusket_coreimporttasks,model@tasks.taskdefmeasure2(m:model.ConnectedModel):returnresultWorking directorymustpoint to themusket_coreroot folder.In order to run the experiment or a number of experiments,A typical command line may look like this:python -m musket_core.task --project "path/to/project" --name "experiment_name" --task "task_name" --num_gpus=1 --gpus_per_net=1 --num_workers=1 --cache "path/to/cache/folder"--projectpoints to the root of theproject--nameis the name of the project sub-folder containing experiment yaml file.--taskis the name of the task function.--num_gpussets number of GPUs to use during experiment launch.--gpus_per_netis a maximum number of GPUs to use per single experiment.--num_workerssets number of workers to use.--cachepoints to a cache folder to store the temporary data.Other parameters can be found in thetask script referenceLaunching project analysisanalize.pyscript is designed to launch project-scope analysis.Note that only experiments, which training is already finished will be covered.musket analize --inputFolder "path/to/project"--inputFolderpoints to a folder to search for finished experiments in. Typically, project root.Other parameters can be found in theanalyze script referenceUsage guideTraining a modelLet's start from the absolutely minimalistic example. Let's say that you have two folders, one of them contains
jpeg images, and another one - png files with segmentation masks for these images. And you need to train a neural network
that will do segmentation for you. In this extremely simple setup all that you need is to type following 5
lines of python code:fromsegmentation_pipeline.impl.datasetsimportSimplePNGMaskDataSetfromsegmentation_pipelineimportsegmentationds=SimplePNGMaskDataSet("./pics/train","./pics/train_mask")cfg=segmentation.parse("config.yaml")cfg.fit(ds)Looks simple, but there is aconfig.yamlfile in the code, and probably it is the place where everything actually happens.backbone:mobilenetv2#let's select classifier backbone for our networkarchitecture:DeepLabV3#let's select segmentation architecture that we would like to useaugmentation:Fliplr:0.5#let's define some minimal augmentations on imagesFlipud:0.5classes:1#we have just one class (mask or no mask)activation:sigmoid#one class means that our last layer should use sigmoid activationencoder_weights:pascal_voc#we would like to start from network pretrained on pascal_voc datasetshape:[320,320,3]#This is our desired input image and mask size, everything will be resized to fit.testSplit:0.4optimizer:Adam#Adam optimizer is a good default choicebatch:16#Our batch size will be 16metrics:#We would like to track some metrics-binary_accuracy-iouprimary_metric:val_binary_accuracy#and the most interesting metric is val_binary_accuracycallbacks:#Let's configure some minimal callbacksEarlyStopping:patience:15monitor:val_binary_accuracyverbose:1ReduceLROnPlateau:patience:4factor:0.5monitor:val_binary_accuracymode:autocooldown:5verbose:1loss:binary_crossentropy#We use simple binary_crossentropy lossstages:-epochs:100#Let's go for 100 epochsSo as you see, we have decomposed our task in two parts,code that actually trains the modelandexperiment configuration,
which determines the model and how it should be trained from the set of predefined building blocks.Moreover, the whole fitting and prediction process can be launched with built-in script,
the only really required python code is dataset definition to let the system know, which data to load.What does this code actually do behind the scenes?it splits your data into 5 folds, and trains one model per fold;it takes care of model checkpointing, generates example image/mask/segmentation triples, collects training metrics. All this data will
be stored in the folders just near yourconfig.yaml;All your folds are initialized from fixed default seed, so different experiments will use exactly the same train/validation splitsAlso, datasets can be specified directly in your config file in more generic way, see examples ds_1, ds_2, ds_3 in "segmentation_training_pipeline/examples/people" folder. In this case you can just call cfg.fit() without providing dataset programmatically.Lets discover what's going on in more details:General train propertiesLets take our standard example and check the following set of instructions:testSplit:0.4optimizer:Adam#Adam optimizer is a good default choicebatch:16#Our batch size will be 16metrics:#We would like to track some metrics-binary_accuracy-iouprimary_metric:val_binary_accuracy#and the most interesting metric is val_binary_accuracyloss:binary_crossentropy#We use simple binary_crossentropy losstestSplitSplits the train set into two parts, using one part for train and leaving the other untouched for a later testing.
The split is shuffled.optimizersets the optimizer.batchsets the training batch size.metricssets the metrics to track during the training process. Metric calculation results will be printed in the console and tometricsfolder of the experiment.primary_metricMetric to track during the training process. Metric calculation results will be printed in the console and tometricsfolder of the experiment.
Besides tracking, this metric will be also used by default for metric-related activity, in example, for decision regarding which epoch results are better.losssets the loss function. if your network has multiple outputs, you also may pass a list of loss functions (one per output)Framework supports composing loss as a weighted sum of predefined loss functions. For example, following constructionloss:binary_crossentropy+0.1*dice_losswill result in loss function which is composed frombinary_crossentropyanddice_lossfunctions.There are many more properties to check inReference of root propertiesDefining architectureLets take a look at the following part of our example:backbone:mobilenetv2#let's select classifier backbone for our networkarchitecture:DeepLabV3#let's select segmentation architecture that we would like to useclasses:1#we have just one class (mask or no mask)activation:sigmoid#one class means that our last layer should use sigmoid activationencoder_weights:pascal_voc#we would like to start from network pretrained on pascal_voc datasetshape:[320,320,3]#This is our desired input image and mask size, everything will be resized to fit.The following three properties are required to set:backboneThis property configures encoder that should be used. Different kinds ofFPN,PSP,Linkenet,UNetand more are supported.architectureThis property configures decoder architecture that should be used.net,Linknet,PSP,FPNand more are supported.classessets the number of classes that should be used.The following ones are optional, but commonly used:activationsets activation function that should be used in last layer.shapeset the desired shape of the input picture and mask, in the form heigth, width, number of channels. Input will be resized to fit.encoder_weightsconfigures initial weights of the encoder.Image and Mask AugmentationsFramework uses awesomeimgauglibrary for augmentation, so you only need to configure your augmentation process in declarative way like in the following example:augmentation:Fliplr:0.5Flipud:0.5Affine:scale:[0.8,1.5]#random scalingstranslate_percent:x:[-0.2,0.2]#random shiftsy:[-0.2,0.2]rotate:[-16,16]#random rotations on -16,16 degreesshear:[-16,16]#random shears on -16,16 degreesaugmentationproperty definesIMGAUGtransformations sequence.
Each object is mapped onIMGAUGtransformer by name, parameters are mapped too.In this example,FliplrandFlipudkeys are automatically mapped onFlip agugmenters,
their0.5parameter is mapped on the firstpparameter of the augmenter.
Named parameters are also mapped, in examplescalekey ofAffineis mapped onscaleparameter ofAffine augmenter.One interesting augementation option when doing background removal task is replacing backgrounds with random
images. We support this withBackgroundReplaceraugmenter:augmentation:BackgroundReplacer:path:./bg#path to folder with backgroundsrate:0.5#fraction of original backgrounds to preserveFreezing and Unfreezing encoderFreezing encoder is often used with transfer learning. If you want to start with frozen encoder just addfreeze_encoder:truestages:-epochs:10#Let's go for 10 epochs with frozen encoder-epochs:100#Now let's go for 100 epochs with trainable encoderunfreeze_encoder:truein your experiments configuration, then on some stage configuration just addunfreeze_encoder:trueto stage settings.Bothfreeze_encoderandunfreeze_encodercan be put into the root section and inside the stage.Note: This option is not supported for DeeplabV3 architecture.Custom datasetsTraining data and masks are not necessarily stored in files, so sometimes you need to declare your own dataset class,
for example, the following code was used to supportAirbus ship detection challengeto decode segmentation masks from rle encoded strings stored in csv filefromsegmentation_pipeline.impl.datasetsimportPredictionItemimportosfromsegmentation_pipeline.implimportrleimportimageioimportpandasaspdclassSegmentationRLE(datasets.DataSet):def__init__(self,path,imgPath):self.data=pd.read_csv(path);self.values=self.data.values;self.imgPath=imgPath;self.ship_groups=self.data.groupby('ImageId');self.masks=self.ship_groups['ImageId'];self.ids=list(self.ship_groups.groups.keys())passdef__len__(self):returnlen(self.masks)def__getitem__(self,item):pixels=self.ship_groups.get_group(self.ids[item])["EncodedPixels"]returnPredictionItem(self.ids[item]+str(),imageio.imread(os.path.join(self.imgPath,self.ids[item])),rle.masks_as_image(pixels)>0.5)defgetTrain()->datasets.DataSet:returnSegmentationRLE("train.csv","images/")Now, if this python code sits somewhere in python files located inmodulesfolder of the project, and that file is referred byimportsinstruction, following YAML can refer it:dataset:getTrain:[]datasetsets the main training dataset.datasetssets up a list of available data sets to be referred by other entities.Multistage trainingSometimes you need to split your training into several stages. You can easily do it by adding several stage entries
in your experiment configuration file.stagesinstruction allows to set up stages of the train process, where for each stage it is possible to set some specific training options like the number of epochs, learning rate, loss, callbacks, etc.
Full list of stage properties can be foundhere.stages:-epochs:100#Let's go for 100 epochs-epochs:100#Let's go for 100 epochs-epochs:100#Let's go for 100 epochsstages:-epochs:6#Train for 6 epochsnegatives:none#do not include negative examples in your training setvalidation_negatives:real#validation should contain all negative examples-lr:0.0001#let's use different starting learning rateepochs:6negatives:realvalidation_negatives:real-loss:lovasz_loss#let's override loss functionlr:0.00001epochs:6initial_weights:./fpn-resnext2/weights/best-0.1.weights#let's load weights from this fileBalancing your dataOne common case is the situation when part of your images does not contain any objects of interest, like inAirbus ship detection challenge. More over your data may
be to heavily inbalanced, so you may want to rebalance it. Alternatively you may want to inject some additional
images that do not contain objects of interest to decrease amount of false positives that will be produced by the framework.These scenarios are supported bynegativesandvalidation_negativessettings of training stage configuration,
these settings accept following values:none - exclude negative examples from the datareal - include all negative examplesinteger number(1 or 2 or anything), how many negative examples should be included per one positive examplestages:-epochs:6#Train for 6 epochsnegatives:none#do not include negative examples in your training setvalidation_negatives:real#validation should contain all negative examples-lr:0.0001#let's use different starting learning rateepochs:6negatives:realvalidation_negatives:real-loss:lovasz_loss#let's override loss functionlr:0.00001epochs:6initial_weights:./fpn-resnext2/weights/best-0.1.weights#let's load weights from this fileif you are using this setting your dataset class must supportisPositivemethod which returns true for indexes
which contain positive examples:defisPositive(self,item):pixels=self.ddd.get_group(self.ids[item])["EncodedPixels"]formaskinpixels:ifisinstance(mask,str):returnTrue;returnFalseAdvanced learning ratesDynamic learning ratesAs told inCyclical learning rates for training neural networksCLR policies can provide quicker converge for some neural network tasks and architectures.We support them by adopting Brad KenstlerCLR callbackfor Keras.If you want to use them, just addCyclicLRin your experiment configuration file as shown below:callbacks:EarlyStopping:patience:40monitor:val_binary_accuracyverbose:1CyclicLR:base_lr:0.0001max_lr:0.01mode:triangular2step_size:300There are alsoReduceLROnPlateauandLRVariatoroptions to modify learning rate on the fly.LR FinderEstimating optimal learning rate for your modelis an important thing, we support this by using slightly changed
version ofPavel Surmenok - Keras LR Findercfg=segmentation.parse(people-1.yaml)ds=SimplePNGMaskDataSet("./train","./train_mask")finder=cfg.lr_find(ds,start_lr=0.00001,end_lr=1,epochs=5)finder.plot_loss(n_skip_beginning=20,n_skip_end=5)plt.show()finder.plot_loss_change(sma=20,n_skip_beginning=20,n_skip_end=5,y_lim=(-0.01,0.01))plt.show()will result in this couple of helpful images:Training on cropsYour images can be too large to train model on them. In this case you probably want to train model on crops. All
that you need to do is to specify number of splits per axis. For example, following lines in configshape:[768,768,3]crops:3will lead to splitting each image/mask into 9 cells (3 horizontal splits and 3 vertical splits) and training model on these splits.
Augmentations will be run separately on each cell.cropsproperty sets the number of single dimension cells.During prediction time, your images will be split into these cells, prediction will be executed on each cell, and then results
will be assembled in single final mask. Thus the whole process of cropping will be invisible from a consumer perspective.Using trained modelOkey, our model is trained, now we need to actually do image segmentation. Let's say, we need to run image segmentation on
images in the directory and store results in csv file:fromsegmentation_pipelineimportsegmentationfromsegmentation_pipeline.impl.rleimportrle_encodefromskimage.morphologyimportremove_small_objects,remove_small_holesimportpandasaspd#this is our callback which is called for every imagedefonPredict(file_name,img,data):threshold=0.25predictions=data["pred"]imgs=data["images"]post_img=remove_small_holes(remove_small_objects(img.arr>threshold))rle=rle_encode(post_img)predictions.append(rle)imgs.append(file_name[:file_name.index(".")])passcfg=segmentation.parse("config.yaml")predictions=[]images=[]#Now let's use best model from fold 0 to do image segmentation on images from images_to_segmentcfg.predict_in_directory("./images_to_segment",0,0,onPredict,{"pred":predictions,"images":images})#Let's store results in csvdf=pd.DataFrame.from_dict({'image':images,'rle_mask':predictions})df.to_csv('baseline_submission.csv',index=False)Ensembling predictionsAnd what if you want to ensemble models from several folds? Just pass a list of fold numbers topredict_in_directorylike in the following example:cfg.predict_in_directory("./images_to_segment",[0,1,2,3,4],onPredict,{"pred":predictions,"images":images})Another supported option is to ensemble results from extra test time augmentation (flips) by adding keyword argttflips=True.Custom evaluation codeSometimes you need to run custom evaluation code. In such case you may use:evaluateAllmethod, which provides an iterator
on the batches containing original images, training masks and predicted masksforbatchincfg.evaluateAll(ds,2):foriinrange(len(batch.predicted_maps_aug)):masks=ds.get_masks(batch.data[i])fordinrange(1,20):cur_seg=binary_opening(batch.predicted_maps_aug[i].arr>d/20,np.expand_dims(disk(2),-1))cm=rle.masks_as_images(rle.multi_rle_encode(cur_seg))pr=f2(masks,cm);total[d]=total[d]+prAccessing modelYou may get trained keras model by calling:cfg.load_model(fold, stage).Analyzing experiments resultsOkey, we have done a lot of experiments and now we need to compare the results and understand what works better. This repository
containsscriptwhich may be used to analyze folder containing sub folders
with experiment configurations and results. This script gathers all configurations, diffs them by doing structural diff, then
for each configuration it averages metrics for all folds and  generates csv file containing metrics and parameters that
was actually changed in your experiment like in the followingexampleThis script accepts following arguments:inputFolder - root folder to search for experiments configurations and resultsoutput - file to store aggregated metricsonlyMetric - if you specify this option all other metrics will not be written in the report filesortBy - metric that should be used to sort resultsExample:python analize.py --inputFolder ./experiments --output ./result.pyWhat is supported?At this moment segmentation pipeline supports following architectures:UnetLinknetPSPFPNDeeplabV3FPN,PSP,Linkenet,UNetarchitectures support following backbones:VGGNetvgg16vgg19ResNetresnet18resnet34resnet50resnet101resnet152ResNextresnext50resnext101DenseNetdensenet121densenet169densenet201Inception-v3Inception-ResNet-v2All them support the weights pretrained onImageNet:encoder_weights:imagenetAt this momentDeeplabV3architecture supports following backbones:MobileNetV2XceptionDeeplab supports weights pretrained onPASCAL VOC:encoder_weights:pascal_vocEach architecture also supports some specific options, list of options is documented insegmentation RAML library.Supported augmentations are documented inaugmentation RAML library.Callbacks are documented incallbacks RAML library.Custom architectures, callbacks, metricsSegmentation pipeline uses keras custom objects registry to find entities, so if you need to use
custom loss function, activation or metric all that you need to do is to register it in Keras as:keras.utils.get_custom_objects()["my_loss"]=my_lossIf you want to inject new architecture, you should register it insegmentation.custom_modelsdictionary.For example:segmentation.custom.models['MyUnet']=MyUnetwhereMyUnetis a function that accepts architecture parameters as arguments and returns an instance
of keras model.ExamplesTraining background removal task(Pics Art Hackaton) in google collabFAQHow to continue training after crash?If you would like to continue training after crash, callsetAllowResumemethod before callingfitcfg=segmentation.parse("./people-1.yaml")cfg.setAllowResume(True)ds=SimplePNGMaskDataSet("./pics/train","./pics/train_mask")cfg.fit(ds)My notebooks constantly run out of memory, what can I do to reduce memory usage?One way to reduce memory usage is to limit augmentation queue limit which is 50 by default,
like in the following example:segmentation_pipeline.impl.datasets.AUGMENTER_QUEUE_LIMIT=3How can I run sepate set of augmenters on initial image/mask when replacing backgrounds with Background Augmenter?BackgroundReplacer:rate:0.5path:./bgaugmenters:#this augmenters will run on original image before replacing backgroundAffine:scale:[0.8,1.5]translate_percent:x:[-0.2,0.2]y:[-0.2,0.2]rotate:[-16,16]shear:[-16,16]erosion:[0,5]How can I visualize images that are used for training (after augmentations)?You should setshowDataExamplesto True like in the following samplecfg=segmentation.parse("./no_erosion_aug_on_masks/people-1.yaml")cfg.showDataExamples=Trueif will lead to generation of training images samples and storing them in examples folder at the end of each epochWhat I can do if i have some extra training data, that should not be included into validation, but should be used during the training?extra_data=NotzeroSimplePNGMaskDataSet("./phaces/all","./phaces/masks")#My dataset that should be added to trainingsegmentation.extra_train["people"]=extra_dataand in the config file:extra_train_data:peopleHow to get basic statistics across my folds/stagesThis code sample will return primary metric stats over folds/stagescfg= segmentation.parse("./no_erosion_aug_on_masks/people-1.yaml")
metrics = cfg.info()I have some callbacks that are configured globally, but I need some extra callbacks for my last training stage?There are two possible ways how you may configure callbacks on stage level:override all global callbacks withcallbackssetting.add your own custom callbacks withextra_callbackssetting.In the following sample CyclingRL callback is only appended to the sexond stage of training:loss:binary_crossentropystages:-epochs:20negatives:real-epochs:200extra_callbacks:CyclicLR:base_lr:0.000001max_lr:0.0001mode:triangularstep_size:800negatives:realWhat if I would like to build a really large ansemble of models?One option to do this, is to store predictions for each file and model in numpy array, and then sum these predictions
like in the following sample:cfg.predict_to_directory("./pics/test","./pics/arr1",[0,1,4,2],1,ttflips=True,binaryArray=True)cfg.predict_to_directory("./pics/test","./pics/arr",[0,1,4,2],2,ttflips=True,binaryArray=True)segmentation.ansemblePredictions("./pics/test",["./pics/arr/","./pics/arr1/"],onPredict,d)How to train on multiple gpus?cfg.gpus=4#or another number matching to the count of gpus that you have