Package: kit4dl
A quick way to start with machine and deep learning🖋️ AuthorsOpenGeokube Developers:Jakub WalczakMarco MaciniMirko StojiljkovicShahbaz Alvi📜 Cite Us@SOFTWARE{kit4dl,author={Walczak, Jakub andMancini, Marco andStojiljković, Mirko andAlvi, Shahbaz},title={Kit4DL},month=sep,year=2023,note={{Available in GitHub: https://github.com/opengeokube/kit4dl}},publisher={Zenodo},version={2023.9b1},doi={10.5281/zenodo.8328176},url={https://doi.org/10.5281/zenodo.8328176}}🚧 RoadmapWarning: Kit4DL is currently in its alpha stage. All recommendations are welcomed.add handling sklearn-like modelsadd functionality to serve the modelenable custom metricsenable using callbacks (also custom ones)write more unit tests🎬 QuickstartGetting startedInstallationpipinstallkit4dlorcondainstall-cconda-forgekit4dlFor contributing:Download and install themaketool unless it is already available in your system.git clone https://github.com/opengeokube/kit4dl
cd kit4dl
conda env create -f dev-env.yaml
pip install -e .Preparing simple projectTo start the new project in the current working directory, just run the following command:kit4dlinit--name=my-new-projectIt will create a directory with the namemy-new-projectwhere you'll find sample files.
Implement necessery methods for datamodule (dataset.py) and network (model.py).
Then, adjustconf.tomlaccording to your needs.
That's all 🎉Running the trainingTo run the training just type the following command:kit4dltrainNote: If you want to run also test for best saved weight, use flag--testIf theconf.tomlfile is present in your current working directory, the training will start.If you need to specify the path to the configuration file, use--confargument:kit4dltrain--conf=/path/to/your/conf.tomlServing the modelThe packuge does not yet support model serving.🪁 PlaygroundAt first, installkit4dlpackage as indicated in the SectionInstallation.Handwritten digit recognitionJust navigate to the directory/examples/cnn_mnist_classificationand runkit4dltrainPoint cloud instance segmentationJust navigate to the directory/examples/cnn_s3dis_segmentationand runkit4dltrain💡 InstructionConfiguring base setupConfiguring loggingDefining modelDefining datamoduleConfiguring trainingConfiguring optimizerConfiguring criterionConfiguring metricsConfiguring checkpointDefiningtargetSubstitutable symbolsContext constantsConfiguring base setupMost of the training/validation procedure is managed by a configuration file in the TOML format (recommended name isconf.toml).
Each aspect is covered by separate sections. The general one is called[base].
It has the following properties:PropertyTypeDetailsseedintseed of the random numbers generators forNumPyandPyTorchcuda_idintorNoneID of the cuda device (if available) orNoneforCPUexperiment_name*strname of the experimentNote: Arguments marked with*are obligatory!Warning: Remember to install the version ofpytorch-cudapackage compliant to your CUDA Toolkit version.✍️ Example[base]seed=0cuda_id=1experiment_name="point_clout_segmentation"Configuring loggingLogging section is optional but it provides you with some extra flexibility regarding the logging.
All configuration related to logging is included in the[logging]section of the configuration file.
You can define following properties:PropertyTypeDetailstypestrtype of metric logger (one of the value:"comet","csv","mlflow","neptune","tensorboard","wandb"- metric loggers supported by PyTorch Lightninghttps://lightning.ai/docs/pytorch/stable/api_references.html#loggers.DEFAULT:csv)levelstrPython-supported logging levels (i.e."DEBUG","INFO","WARN","ERROR","CRITICAL")DEFAULT:INFOformatstrlogging message format as defined for the Pythonloggingpackage (seehttps://docs.python.org/3/library/logging.html#logging.LogRecord)Warning: Loggerlevelandformatare related to the PythonloggingLoggers you can use in your model and datamodule classes with approperiate methodsself.debure,self.info, etc. Intype, in turn, you just specify the metric logger as used in PyTorch Lightning package!Note: All required arguments for metric logger can be specified as extra arguments in the[logging]section.✍️ Example[logging]# we gonna use CSVLoggertype="csv"# for CSVLogger, we need to define 'save_dir' argument and/or# other extra ones (https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.loggers.csv_logs.html#module-lightning.pytorch.loggers.csv_logs)save_dir="{{ PROJECT_DIR }}/my_metrics.csv"# then we define level and format for logging messageslevel="info"format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"Note: If you don't pass anameorexperiment_nameargument explicitly for the metric logger, theexperiment_namevalue defined in the[base]section will be applied as, respectively:nameargument forcsv,neptune,tensorboard,wandb, and asexperiment_nameforcometandmlflow.Defining modelThe machine learning/deep learning model definition is realized in two aspects.The definition of the model (e.g. PyTorch model) in the.pyfile.The configuration in the[model]section of the configuration file.The file with the model definition should contain a subclass ofKit4DLAbstractModuleabstract class of thekit4dlpackage.
The subclass should implement, at least, abstract methodsconfigureandrun_step.
In theconfiguremethod, the architecture of the network should be defined.
Inrun_stepmethod, it turn, the logic for single forward pass should be implemented.✍️ Exampleimporttorchfromtorchimportnnfromkit4dl.nn.baseimportKit4DLAbstractModuleclassSimpleCNN(Kit4DLAbstractModule):defconfigure(self,input_dims,output_dims)->None:self.l1=nn.Sequential(nn.Conv2d(input_dims,16,kernel_size=3,padding="same",bias=True),nn.ReLU(),)defrun_step(self,batch,batch_idx)->tuple[torch.Tensor,...]:x,label=batchlogits=self.l1(x)preds=logits.argmax(dim=-1)returnlabel,logits,predsNote:run_stepmethod should return a tuple of 2 (ground-truth, scores) or 3 (ground-truth, scores, loss) tensors.Note:batchargument can be unpacked depending on how you define your dataset for datamodule (seeDefining datamodule)In the configuration file, in the dedicated[model]section, at leasttargetproperty should be set. The extra arguments are treated as the arguments for theconfiguremethod.Note: Arguments' values of theconfiguremethod (i.e.input_dimsandoutput_dims) are taken from the configuration files. Those names can be arbitrary.✍️ Example[model]target="./model.py::SimpleCNN"input_dims=1output_dims=10Note:targetis a required parameter thatmustbe set. It contains a path to the class (a subclass ofKit4DLAbstractModule). To learn howtargetcould be defined, see SectionDefiningtarget.If a forward pass for your model differs for the training, validation, test, or prediction stages, you can define separate methods for them:✍️ Exampleimporttorchfromtorchimportnnfromkit4dl.nn.baseimportKit4DLAbstractModuleclassSimpleCNN(Kit4DLAbstractModule):...defrun_val_step(self,batch,batch_idx)->tuple[torch.Tensor,torch.Tensor]:passdefrun_test_step(self,batch,batch_idx)->tuple[torch.Tensor,torch.Tensor]:passdefrun_predict_step(self,batch,batch_idx)->torch.Tensor:passNote: If you need more customization of the process, you can always override the existing methods according to your needs.Defining datamoduleSimilarily to the model, datamodule instance is fully defined by the Python class and its configuration.
The datamodule need to be a subclass of theKit4DLAbstractDataModuleabstract class from thekit4dlpackage.
The class has to implement, at least,prepare_trainvaldataset(if preparing is the same for the train and validation splits) orprepare_traindatasetandprepare_valdataset(if preparing data differs). Besides those, you can defineprepare_testdatasetandprepare_predictdataset, for test and prediction, respectively.✍️ Examplefromtorch.utils.dataimportDataset,random_splitfromtorchvisionimporttransformsfromtorchvision.datasetsimportMNISTfromkit4dl.datasetimportKit4DLAbstractDataModuleclassMNISTCustomDatamodule(Kit4DLAbstractDataModule):defprepare_trainvaldataset(self,root_dir:str)->tuple[Dataset,Dataset]:dset=MNIST(root=root_dir,train=True,download=True,transform=transforms.ToTensor(),)train_dset,val_dset=random_split(dset,[0.8,0.2])return(train_dset,val_dset)defprepare_testdataset(self,root_dir:str)->Dataset:returnMNIST(root=root_dir,train=False,download=True,transform=transforms.ToTensor(),)If you need to acquire data or do some other processing, implementprepare_datamethod. In that method you can use extra attributes you defined in the[dataset]section of the configuration file.✍️ Example[dataset]target="./datamodule.py::MNISTCustomDatamodule"my_variable=10...classMNISTCustomDatamodule(Kit4DLAbstractDataModule):my_variable:int# NOTE: To make attribute visible, we can declare it heredefprepare_data(self):result=self.my_variable*2Warning:DO NOTset state insideprepare_datamethod (self.x = ...).If you need more customization, feel free to override the other methods ofKit4DLAbstractDataModulesuperclass.
To force custom batch collation, override selected methods out of the following ones. They should return the proper callable object!defsome_collate_func(samples:list):...classMNISTCustomDatamodule(Kit4DLAbstractDataModule):...defget_train_collate_fn(self):returnsome_collate_funcdefget_val_collate_fn(self):returnsome_collate_funcdefget_test_collate_fn(self):returnsome_collate_funcdefget_predict_collate_fn(self):returnsome_collate_funcWarning:DO NOTuse nested function as a callation callable. It will fail due to pickling nested function error.If you need a custom batch collation but the same for each stage (train/val/test/predict), implement the methodget_collate_fn():defget_collate_fn(self):returnsome_collate_funcIn the configuration file, there are dedicated[dataset]-related sections.✍️ Example[dataset]target="./datamodule.py::MNISTCustomDatamodule"[dataset.trainval]root_dir="./mnist"[dataset.train.loader]batch_size=150shuffle=truenum_workers=4[dataset.validation.loader]batch_size=150shuffle=falsenum_workers=4In the root[dataset]you should definetargetproperty being a path to the subclass of theKit4DLAbstractDataModulemodule (seeDefiningtarget).
Then, you need to define either[dataset.trainval]section or two separate sections:[dataset.train],[dataset.validation]. There are also optional sections:[dataset.test]and[dataset.predict].
In[dataset.trainval]you pass values for parameters of theprepare_trainvaldatasetmethod.
Respectively, in the[dataset.train]you pass values for the parameters of theprepare_traindatasetmethod, in[dataset.validation]—prepare_valdataset,[dataset.test]—prepare_testdataset,[dataset.predict]—prepare_predictdataset.Besides dataset configuration, you need to specify data loader arguments as indicated in the PyTorch docstorch.utils.data.DataLoader.Warning: Youcannotspecify loader arguments for in the[dataset.trainval.loader]. Loaders should be defined for each split separately.Configuring trainingTraining-related arguments should be defined in the[training]section of the configuration file.
You can define the following arguments.PropertyTypeDetailsepochs*int > 0number of epochscallbackslistlist of callbacksepoch_schedulerslist of dictlist of schedulers definitionsNote: Arguments marked with*are obligatory!You can define a list of custom callbacks applied in the training process. Your callbacks need to be subclasses oflightning.pytorch.callbacks.Callbackorkit4dl.Kit4DLCallback(for convenience) class and define one/some of the methods indicated in thePyTorch-Lightning callback API. You can always use one of thepredefined callbacks.[training]callbacks=[{target="./callbacks.py::SaveConfusionMatrixCallback",task="multiclass",num_classes=10,save_dir="{{ PROJECT_DIR }}/cm},{target = "lightning.pytorch.callbacks::DeviceStatsMonitor"}]Where the 1st callback is user-defined and the other - PyTorch-Loghtning built-in. For the custom callback we need to provide a class (here: located in thecallbacks.pyfile in the project directory, the class is namedSaveConfusionMatrixCallback).importosfromtypingimportAnyimportlightning.pytorchasplimporttorchmetricsastmfromkit4dlimportKit4DLCallbackclassSaveConfusionMatrixCallback(Kit4DLCallback):_cm:tm.ConfusionMatrix_num_classes:int_task:str_save_dir:strdef__init__(self,task:str,num_classes:int,save_dir:str)->None:super().__init__()self._num_classes=num_classesself._save_dir=save_dirself._task=taskos.makedirs(self._save_dir,exist_ok=True)defon_validation_epoch_start(self,trainer:pl.Trainer,pl_module:pl.LightningModule)->None:self._cm=tm.ConfusionMatrix(task=self._task,num_classes=self._num_classes)defon_validation_batch_end(self,trainer:pl.Trainer,pl_module:pl.LightningModule,outputs:dict,batch:Any,batch_idx:int,dataloader_idx:int=0,)->None:self._cm.update(outputs['pred'],outputs['true'])defon_validation_epoch_end(self,trainer:pl.Trainer,pl_module:pl.LightningModule)->None:"""Called when the val epoch ends."""fig,_=self._cm.plot()target_file=os.path.join(self._save_dir,f"confusion_matrix_for_epoch_{pl_module.current_epoch}",)fig.savefig(target_file)Besides those listed in the table above, you can specify PyTorch Lightning-relatedTrainerarguments, like:accumulate_grad_batchesgradient_clip_valgradient_clip_algorithm...✍️ Example[training]epochs=10epoch_schedulers=[{target="torch.optim.lr_scheduler::CosineAnnealingLR",T_max=100}]accumulate_grad_batches=2Configuring optimizerOptimizer configuration is located in the subsection[training.optimizer].
There, you should definetarget(seeDefiningtarget) and extra keyword arguments passed to the optimizer initializer.✍️ Example[training.optimizer]target="torch.optim::Adam"lr=0.001weight_decay=0.01Note: The section[training.optimizer]ismandatory.Note: You can always define the custom optimizer. Then, you just need to set the propertargetvalue.Configuring criterionSimilarily to the optimizer configuration, there is a subsection dedicated for the critarion.
You need to specify, at least, thetarget(seeDefiningtarget) and other mandatory or optional
properties of the selected critarion (loss function).✍️ Example[training.criterion]target="torch.nn::CrossEntropyLoss"weight=[0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1,0.1]Note: The section[training.criterion]ismandatory.Note: You can always define the custom optimizer. Then, you just need to set the propertargetvalue.Configuring metricsMetrics are configured in the section[metrics]of the configuration file. You can define several metrics (including the custom ones).
The only thing you need to do is to define all desired metrics. For each metric dictionary, you need to settarget(see SectionDefiningtarget) value and, eventually, extra arguments.REMEMBERto have metric names (hereMyPrecisionandFBetaScore) unique!✍️ Example[metrics]MyPrecision={target="torchmetrics::Precision",task="multiclass",num_classes=10}FBetaScore={target="torchmetrics::FBetaScore",task="multiclass",num_classes=10,beta=0.1}Note: You can define custom metrics. Just properly settargetvalue.REMEMBER!The custom metric need to be a subclass oftorchmetrics.Metricclass!importtorchimporttorchmetricsastmclassMyMetric(tm.Metric):def__init__(self):...defupdate(self,preds:torch.Tensor,target:torch.Tensor):...defcompute(self):...Configuring checkpointIf you need to save your intermediate weights (do checkpoints) you can configure the optional subsection[training.checkpoint].
In the section, you can define the following proeprties:PropertyTypeDetailspath*strpath to a directory where checkpoints should be storedmonitor*dicta dictionary with two keys:metricandstage.metricsis a metric name as defined in the[metrics]section (Configuring metrics),stageis one of the following: [train,val]filename*strfilename pattern of the checkpoint (see (PyTorch LightningModelCheckpoint)[https://lightning.ai/docs/pytorch/stable/api/lightning.pytorch.callbacks.ModelCheckpoint.html])	you can use value of the defined metric for the stage. if you wantMyPrecisionscore for the validation stage, use{val_myprecision}in the filenamemodeminmaxsave_top_kintsave checkepoints for the topkvalues of the metric.default:1save_weights_onlyboolif only weights should be saved (True) or other states (optimizer, scheduler) also (False).default:Trueevery_n_epochsintThe number of training epochs between saving sucessive checkpoints.default:1save_on_train_epoch_endboolifFalsecheckpointing is run at the end of the validation, otherwise - trainingdefault:FalseNote: Arguments marked with*are obligatory!✍️ Example[training.checkpoint]path="{{ PROJECT_DIR }}/chckpt"monitor={"metric"="Precision","stage"="val"}filename="{epoch}_{val_precision:.2f}_cnn"mode="max"save_top_k=1Note: You can see we used substitutable symbol{{ PROJECT_DIR }}. More about them in the SectionSubstitutable symbols.DefiningtargetTarget property in the Kit4DL package is kind of extended fully qualified name pointing to the classes supposed to use in the
given context, like for:neural network class (target = "./model.py::SimpleCNN")datamodule (target = "./datamodule.py::MNISTCustomDatamodule")optimizer (target = "torch.optim::Adam")criterion (target = "torch.nn::CrossEntropyLoss")schedulers (target = "torch.optim.lr_scheduler::CosineAnnealingLR")Note: As a package/module - class separator the double colon is used::!It might be set in several different ways:By using a built-and installed package. Then, you just need to specify the package/module name and the class name, liketarget = "torch.nn::CrossEntropyLoss"(we use moduletorch.nnand classCrossEntropyLossdefined within).By using a custom module in the project directory. The project directory, i.e. the directory where the confguration TOML file is located, is added to thePYTHONPATH, so you can freely use.pyfiles defined there as modules. Having the modulemodel.pywith theSimpleCNNclass definition, we can writetargetastarget = "model::SimpleCNN".By using a custom.pyfile.In this case, you specifytargetas an absolute or relative (w.r.t. the configuration file) to a.pyfile, liketarget = "./model.py::SimpleCNN"ortarget = "/usr/neural_nets/my_net/model.py::SimpleCNN".Note: Fortargetdefinition you can use substitutable symbols defined below.Substitutable symbolsIn the configuration file you can use symbols that will be substituted during the runtime.
The symbols should be surrended by single spaces and in double curly brackets (e.g.{{ PROJECT_DIR }}.)SymbolMeaning of the symbolExamplePROJECT_DIRthe home directory of the TOML configuration file (project directory)target = {{ PROJECT_DIR }}/model.pyNote: You can also use environmental variables. Just useenvdict, e.g.:{{ env['your_var_name'] }}.✍️ ExampleFirst, let's define some environmental variable: using Python or system tool.importosos.environ["MY_LOG_LEVEL"]="INFO"orexportMY_LOG_LEVEL="MY_LOG_LEVEL"Now, we can use the environmental variableMY_LOG_LEVELin our config file:[logging]level="{{ env['MY_LOG_LEVEL'] }}"format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"Warning: If you usedouble quotefor text values in TOML configuration file, then usesingle quoteto accessenvvalues.Context constantsWhen you run training usingkit4dl traincommand, all custom modules have access to context constant values (defined for the current Python interpreter session).
You can access them viacontextmodule:✍️ Examplefromkit4dlimportcontextprint(context.PROJECT_DIR)The constants currently available inkit4dlare the following:SymbolMeaning of the symbolExamplePROJECT_DIRthe home directory of the TOML configuration file (project directory)context.PROJECT_DIRLOG_LEVELlogging level as defined in the configuration TOML filecontext.LOG_LEVELLOG_FORMATlogging message format as defined in the configuration TOML filecontext.LOG_FORMATVERSIONthe current version of the packagecontext.VERSION