Package: classification-pipeline
Classification training pipelineMy puny attempt to build reusable training pipeline for image classificationMotivationInstallationUsage guideTraining a modelImage/Mask AugmentationsFreezing/Unfreezing encoderCustom datasetsBalancing your dataMultistage trainingComposite lossesCyclical learning ratesLR FinderBackground AugmenterTraining on cropsUsing trained modelEnsembling predictions and test time augmentationCustom evaluation codeAccessing modelAnalyzing Experiments ResultsWhat is supported?Custom architectures, callbacks, metricsExamplesFaqMotivationIdea for this project came from my first attempts to participate in Kaggle competitions. My programmers heart was painfully damaged by looking on my own code as well as on other people kernels. Code was highly repetitive, suffering from numerous reimplementations of same or almost same things through the kernels, model/experiment configuration was often mixed with models code, in other words - from programmer perspective it all looked horrible.So I decided to extract repetitive things into framework that will work at least for me and will follow these statements:experiment configurations should be cleanly separated from model definitions;experiment configuration files should be easy to compare and should fully describe experiment that is being performed except for the dataset;common blocks like an architecture, callbacks, storing model metrics, visualizing network predictions, should be written once and be a part of common libraryInstallationpip install classification_pipelineNote: this package requires python 3.6Usage guideTraining a modelLet's start from a simple example of classification. Suppose, your data are structured as follows: a .cvs file with images ids and their labels and a folder with all these images. For training a neural network to classify these images all you need are few lines of python code:importmusket_corefromclassification_pipelineimportclassificationclassProteinDataGenerator:def__init__(self,paths,labels):self.paths,self.labels=paths,labelsdef__len__(self):returnlen(self.paths)def__getitem__(self,idx):X,y=self.__load_image(self.paths[idx]),self.labels[idx]returnPredictionItem(self.paths[idx],X,y)def__load_image(self,path):R=Image.open(path+'_red.png')G=Image.open(path+'_green.png')B=Image.open(path+'_blue.png')im=np.stack((np.array(R),np.array(G),np.array(B),),-1)returnimdataset=ProteinDataGenerator(paths,labels)cfg=classification.parse("config.yaml")cfg.fit(dataset)Looks simple, but there is aconfig.yamlfile in the code, and probably it is the place where everything actually happens.architecture:DenseNet201#pre-trained model we are going to usepooling:avgaugmentation:#define some minimal augmentations on imagesFliplr:0.5Flipud:0.5classes:28#define the number of classesactivation:sigmoid#as we have multilabel classification, the activation for last layer is sigmoidweights:imagenet#we would like to start from network pretrained on imagenet datasetshape:[224,224,3]#our desired input image size, everything will be resized to fitoptimizer:Adam#Adam optimizer is a good default choicebatch:16#our batch size will be 16lr:0.005copyWeights:truemetrics:#we would like to track some metrics-binary_accuracy-macro_f1primary_metric:val_binary_accuracy#the most interesting metric is val_binary_accuracyprimary_metric_mode:maxcallbacks:#configure some minimal callbacksEarlyStopping:patience:3monitor:val_macro_f1mode:maxverbose:1ReduceLROnPlateau:patience:2factor:0.3monitor:val_binary_accuracymode:maxcooldown:1verbose:1loss:binary_crossentropy#we use binary_crossentropy lossstages:-epochs:10#let's go for 100 epochsSo as you see, we have decomposed our task in two parts,code that actually trains the modelandexperiment configuration,
which determines the model and how it should be trained from the set of predefined building blocks.What does this code actually do behind the scenes?it splits your data into 5 folds, and trains one model per fold;it takes care of model checkpointing, generates example image/label tuples, collects training metrics. All this data will
be stored in the folders just near yourconfig.yaml;All your folds are initialized from fixed default seed, so different experiments will use exactly the same train/validation splits.Image AugmentationsFramework uses awesomeimgauglibrary for augmentation, so you only need to configure your augmentation process in declarative way like in the following example:augmentation:Fliplr:0.5Flipud:0.5Affine:scale:[0.8,1.5]#random scalingstranslate_percent:x:[-0.2,0.2]#random shiftsy:[-0.2,0.2]rotate:[-16,16]#random rotations on -16,16 degreesshear:[-16,16]#random shears on -16,16 degreesFreezing/Unfreezing encoderFreezing encoder is often used with transfer learning. If you want to start with frozen encoder just addfreeze_encoder:truestages:-epochs:10#Let's go for 10 epochs with frozen encoder-epochs:100#Now let's go for 100 epochs with trainable encoderunfreeze_encoder:truein your experiments configuration, then on some stage configuration just addunfreeze_encoder:trueto stage settings.Note: This option is not supported for DeeplabV3 architecture.Custom datasetsYou can declare your own dataset class as in this example:frommusket_core.datasetsimportPredictionItemimportosimportimageioimportpandasaspdimportnumpyasnpimportcv2classClassification:def__init__(self,imgPath):self.species=['Black-grass','Charlock','Cleavers','Common Chickweed','Common wheat','Fat Hen','Loose Silky-bent','Maize','Scentless Mayweed','Shepherds Purse','Small-flowered Cranesbill','Sugar beet']self.data=[]self.targets=[]self.ids=[]fors_id,sinenumerate(self.species):s_folder=os.path.join(imgPath,s)forfileinos.listdir(s_folder):self.data.append(os.path.join(s_folder,file))self.targets.append(s_id)self.ids.append(file)def__len__(self):returnlen(self.data)def__getitem__(self,item):item_file=self.data[item]target=self.targets[item]t=np.zeros(len(self.species))t[target]=1.0image=self.read_image(item_file,(224,224))returnPredictionItem(self.ids[item],image,t)defread_image(self,filepath,target_size=None):img=cv2.imread(filepath,cv2.IMREAD_COLOR)img=cv2.resize(img.copy(),target_size,interpolation=cv2.INTER_AREA)returnimgMulti output classificationSometimes you need to create network that performs several classification tasks at the same moment, in this situation
you need to declareclasses,activationandlossas the lists of class counts, activation functions and losses
like in the following snippet:classes:[4,4]#define the number of classesactivation:[sigmoid,sigmoid]loss:-binary_crossentropy-binary_crossentropyprimary_metric:val_loss#the most interesting metric is val_binary_accuracyprimary_metric_mode:minit is also very likely that you need to change primary metric toval_lossBalancing your dataOne common case is the situation when part of your images does not contain any objects of interest, like inAirbus ship detection challenge. More over your data may
be to heavily inbalanced, so you may want to rebalance it. Alternatively you may want to inject some additional
images that do not contain objects of interest to decrease amount of false positives that will be produced by the framework.These scenarios are supported bynegativesandvalidation_negativessettings of training stage configuration,
these settings accept following values:none - exclude negative examples from the datareal - include all negative examplesinteger number(1 or 2 or anything), how many negative examples should be included per one positive exampleif you are using this setting your dataset class must supportisPositivemethod which returns true for indexes
which contain positive examples:defisPositive(self,item):pixels=self.ddd.get_group(self.ids[item])["EncodedPixels"]formaskinpixels:ifisinstance(mask,str):returnTrue;returnFalseMultistage trainingSometimes you need to split your training into several stages. You can easily do it by adding several stage entries
in your experiment configuration file like in the following example:stages:-epochs:6#Train for 6 epochsnegatives:none#do not include negative examples in your training setvalidation_negatives:real#validation should contain all negative examples-lr:0.0001#let's use different starting learning rateepochs:6negatives:realvalidation_negatives:real-loss:lovasz_loss#let's override loss functionlr:0.00001epochs:6initial_weights:./fpn-resnext2/weights/best-0.1.weights#let's load weights from this fileStage entries allow you to configure custom learning rate, balance of negative examples, callbacks, loss function
and even initial weights which should be used on a particular stage.Composite lossesFramework supports composing loss as a weighted sum of predefined loss functions. For example, following constructionloss:binary_crossentropy+0.1*dice_losswill result in loss function which is composed frombinary_crossentropyanddice_lossfunctions.Cyclical learning ratesAs told inCyclical learning rates for training neural networksCLR policies can provide quicker converge for some neural network tasks and architectures.We support them by adopting Brad KenstlerCLR callbackfor Keras.If you want to use them, just addCyclicLRin your experiment configuration file as shown below:callbacks:EarlyStopping:patience:40monitor:val_binary_accuracyverbose:1CyclicLR:base_lr:0.0001max_lr:0.01mode:triangular2step_size:300LR FinderEstimating optimal learning rate for your modelis an important thing, we support this by using slightly changed
version ofPavel Surmenok - Keras LR Findercfg=classification.parse(config.yaml)ds=SimplePNGMaskDataSet("./train","./train_mask")-???????????????????finder=cfg.lr_find(ds,start_lr=0.00001,end_lr=1,epochs=5)finder.plot_loss(n_skip_beginning=20,n_skip_end=5)plt.show()finder.plot_loss_change(sma=20,n_skip_beginning=20,n_skip_end=5,y_lim=(-0.01,0.01))plt.show()will result in this couple of helpful images:Training on cropsYour images can be too large to train model on them. In this case you probably want to train model on crops. All
that you need to do is to specify number of splits per axis. For example, following lines in configshape:[768,768,3]crops:3will lead to splitting each image into 9 cells (3 horizontal splits and 3 vertical splits) and training model on these splits.
Augmentations will be run separately on each cell.During prediction time, your images will be split into these cells, prediction will be executed on each cell, and then results
will be assembled in single final mask. Thus the whole process of cropping will be invisible from a consumer perspective.Using trained modelOkey, our model is trained, now we need to actually do image classification. Let's say, we need to run image classification on
images in the directory and store results in csv file:predictions=[]images=[]#Now let's use best model from fold 0 to do image segmentation on images from images_to_segmentpreds=cfg.predict_all_to_array(dataset_test,0,0)fori,iteminenumerate(dataset_test):images.append(dataset_test.get_id(i))p=np.argmax(preds[i])predictions.append(dataset_test.get_label(p))#Let's store results in csvdf=pd.DataFrame.from_dict({'file':images,'species':predictions})df.to_csv('submission.csv',index=False)Ensembling predictionsAnd what if you want to ensemble models from several folds? Just pass a list of fold numbers topredict_all_to_arraylike in the following example:cfg.predict_all_to_array(dataset_test,[0,1,2,3,4],0)Another supported option is to ensemble results from extra test time augmentation (flips) by adding keyword argttflips=True.Custom evaluation codeSometimes you need to run custom evaluation code. In such case you may use:evaluateAllmethod, which provides an iterator
on the batches containing original images, training masks and predicted masksforbatchincfg.evaluateAll(ds,2):foriinrange(len(batch.predicted_maps_aug)):masks=ds.get_masks(batch.data[i])fordinrange(1,20):cur_seg=binary_opening(batch.predicted_maps_aug[i].arr>d/20,np.expand_dims(disk(2),-1))cm=rle.masks_as_images(rle.multi_rle_encode(cur_seg))pr=f2(masks,cm);total[d]=total[d]+prAccessing modelYou may get trained keras model by calling:cfg.load_model(fold, stage).Analyzing experiments resultsOkey, we have done a lot of experiments and now we need to compare the results and understand what works better. This repository
containsscriptwhich may be used to analyze folder containing sub folders
with experiment configurations and results. This script gathers all configurations, diffs them by doing structural diff, then
for each configuration it averages metrics for all folds and  generates csv file containing metrics and parameters that
was actually changed in your experiment like in the followingexampleThis script accepts following arguments:inputFolder - root folder to search for experiments configurations and resultsoutput - file to store aggregated metricsonlyMetric - if you specify this option all other metrics will not be written in the report filesortBy - metric that should be used to sort resultsExample:python analize.py --inputFolder ./experiments --output ./result.pyWhat is supported?At this moment classification pipeline supports following pre-trained models:ResnetResNet18ResNet34ResNet50ResNet101ResNet152ResNeXt50ResNeXt101VGG:VGG16VGG19InceptionV3InceptionResNetV2XceptionMobileNetMobileNetV2DenseNet:DenseNet121DenseNet169DenseNet201NasNet:NASNetMobileNASNetLargeEach architecture also supports some specific options, list of options is documented insegmentation RAML library.Supported augmentations are documented inaugmentation RAML library.Callbacks are documented incallbacks RAML library.Custom architectures, callbacks, metricsClassification pipeline uses keras custom objects registry to find entities, so if you need to use
custom loss function, activation or metric all that you need to do is to register it in Keras as:keras.utils.get_custom_objects()["my_loss"]=my_lossIf you want to inject new architecture, you should register it inclassification.custom_modelsdictionary.For example:classification.custom.models['MyUnet']=MyUnetwhereMyUnetis a function that accepts architecture parameters as arguments and returns an instance
of keras model.