Package: lpd
lpdA Fast, Flexible Trainer with Callbacks and Extensions for PyTorchlpdderives from the Hebrew wordlapid(לפיד) which means "torch".For latest PyPI stable releaseThere are 2 types oflpdpackages availablelpdwhich brings dependencies for pytorch, numpy and tensorboardpipinstalllpdlpd-nodepswhichyou provideyour own dependencies for pytorch, numpy and tensorboardpipinstalllpd-nodepsv0.4.12-betaRelease - contains the following:ThresholdCheckeris updated to compute improvement according to last improved step and not to the best received metricSome minor cosmetic changesPreviously on lpd:Densecustom layer to support apply norm (configurable to before or after activation)StatsPrintcallback to support printing best confusion matrix when at least one of the metrics is of typeMetricConfusionMatrixBaseTransformerEncoderStackto support activation as inputPositionalEncodingto support more than 3 dimensions inputUpdated PipfileFixed confusion matrix cpu/gpu device errorBetter handling on callbacks where apply_on_states=None (apply on all states)Bug fix in case validation samples are emptyUsagelpdintended to properly structure your PyTorch model training.The main usages are given below.Training your modelfromlpd.trainerimportTrainerfromlpd.enumsimportPhase,State,MonitorType,MonitorMode,StatsTypefromlpd.callbacksimportLossOptimizerHandler,StatsPrint,ModelCheckPoint,Tensorboard,EarlyStopping,SchedulerStep,CallbackMonitorfromlpd.extensions.custom_schedulersimportKerasDecayfromlpd.metricsimportBinaryAccuracyWithLogits,FalsePositivesfromlpd.utils.torch_utilsimportget_gpu_device_if_availablefromlpd.utils.general_utilsimportseed_allfromlpd.utils.threshold_checkerimportAbsoluteThresholdCheckerseed_all(seed=42)# because its the answer to life and the universedevice=get_gpu_device_if_available()# with fallback to CPU if GPU not availablemodel=MyModel().to(device)# this is your model class, and its being sent to the relevant deviceoptimizer=torch.optim.SGD(params=model.parameters())scheduler=KerasDecay(optimizer,decay=0.01,last_step=-1)# decay scheduler using keras formulaloss_func=torch.nn.BCEWithLogitsLoss().to(device)# this is your loss class, already sent to the relevant devicemetrics=[BinaryAccuracyWithLogits(name='Accuracy'),FalsePositives(name='FP',num_class=2,threshold=0)]# define your metrics# you can use some of the defined callbacks, or you can create your owncallbacks=[LossOptimizerHandler(),SchedulerStep(apply_on_phase=Phase.BATCH_END,apply_on_states=State.TRAIN),ModelCheckPoint(checkpoint_dir,checkpoint_file_name,CallbackMonitor(monitor_type=MonitorType.LOSS,stats_type=StatsType.VAL,monitor_mode=MonitorMode.MIN),save_best_only=True),Tensorboard(summary_writer_dir=summary_writer_dir),EarlyStopping(CallbackMonitor(monitor_type=MonitorType.METRIC,stats_type=StatsType.VAL,monitor_mode=MonitorMode.MAX,patience=10,metric_name='Accuracy'),threshold_checker=AbsoluteThresholdChecker(monitor_mode=MonitorMode.MAX,threshold=0.01)),StatsPrint(train_metrics_monitors=[CallbackMonitor(monitor_type=MonitorType.METRIC,stats_type=StatsType.TRAIN,monitor_mode=MonitorMode.MAX,# <-- notice MAXmetric_name='Accuracy'),CallbackMonitor(monitor_type=MonitorType.METRIC,stats_type=StatsType.TRAIN,monitor_mode=MonitorMode.MIN,# <-- notice MINmetric_name='FP')],print_confusion_matrix=True)# since one of the metric (FalsePositives) is confusion matrix based, lets print the whole confusion matrix]trainer=Trainer(model,device,loss_func,optimizer,scheduler,metrics,train_data_loader,# DataLoader, Iterable or Generatorval_data_loader,# DataLoader, Iterable or Generatortrain_steps,val_steps,callbacks,name='Readme-Example')trainer.train(num_epochs)Evaluating your modeltrainer.evaluatewill returnStatsResultthat stores the loss and metrics results for the test setevaluation_result=trainer.evaluate(test_data_loader,test_steps)Making predictionsPredictorclass will generate output predictions from input samples.Predictorclass can be created fromTrainerpredictor_from_trainer=Predictor.from_trainer(trainer)predictions=predictor_from_trainer.predict_batch(batch)Predictorclass can also be created from saved checkpointpredictor_from_checkpoint=Predictor.from_checkpoint(checkpoint_dir,checkpoint_file_name,model,# nn.Module, weights will be loaded from checkpointdevice)prediction=predictor_from_checkpoint.predict_sample(sample)Lastly,Predictorclass can be initialized explicitlypredictor=Predictor(model,device,callbacks,# relevant only for prediction callbacks (see callbacks Phases and States)name='lpd predictor')predictions=predictor.predict_data_loader(data_loader,steps)Just to be fair, you can also predict directly fromTrainerclass# On single sample:prediction=trainer.predict_sample(sample)# On batch:predictions=trainer.predict_batch(batch)# On Dataloader/Iterable/Generator:predictions=trainer.predict_data_loader(data_loader,steps)TrainerStatsTrainertracks stats fortrain/validate/testand you can access them in your custom callbacks
or any other place that has access to your trainer.Here are some examplestrain_loss=trainer.train_stats.get_loss()# the mean of the last epoch's train lossesval_loss=trainer.val_stats.get_loss()# the mean of the last epoch's validation lossestest_loss=trainer.test_stats.get_loss()# the mean of the test losses (available only after calling evaluate)train_metrics=trainer.train_stats.get_metrics()# dict(metric_name, MetricMethod(values)) of the current epoch in train stateval_metrics=trainer.val_stats.get_metrics()# dict(metric_name, MetricMethod(values)) of the current epoch in validation statetest_metrics=trainer.test_stats.get_metrics()# dict(metric_name, MetricMethod(values)) of the test (available only after calling evaluate)CallbacksWill be used to perform actions at various stages.Some common callbacks are available underlpd.callbacks, and you can also create your own, more details below.In a callback,apply_on_phase(lpd.enums.Phase) will determine the execution phase,andapply_on_states(lpd.enums.Stateorlist(lpd.enums.State)) will determine the execution statesThese are the current available phases and states, more might be added in future releasesTraining and Validation phases and states will behave as followState.EXTERNALPhase.TRAIN_BEGIN# train loop:Phase.EPOCH_BEGINState.TRAIN# batches loop:Phase.BATCH_BEGIN# batchPhase.BATCH_ENDState.VAL# batches loop:Phase.BATCH_BEGIN# batchPhase.BATCH_ENDState.EXTERNALPhase.EPOCH_ENDPhase.TRAIN_ENDEvaluation phases and states will behave as followState.EXTERNALPhase.TEST_BEGINState.TEST# batches loop:Phase.BATCH_BEGIN# batchPhase.BATCH_ENDState.EXTERNALPhase.TEST_ENDPredict phases and states will behave as followState.EXTERNALPhase.PREDICT_BEGINState.PREDICT# batches loop:Phase.BATCH_BEGIN# batchPhase.BATCH_ENDState.EXTERNALPhase.PREDICT_ENDCallbacks will be executed under the relevant phase and state, and by their order.With phases and states, you have full control over the timing of your callbacks.Let's take a look at some of the callbackslpdprovides:LossOptimizerHandler CallbackDerives fromLossOptimizerHandlerBase, probably the most important callback during training 😎UseLossOptimizerHandlerto determine when to call:loss.backward(...)optimizer.step(...)optimizer.zero_grad(...)Or, you may choose to create your ownAwesomeLossOptimizerHandlerclass by deriving fromLossOptimizerHandlerBase.Trainer.train(...)will validate that at least oneLossOptimizerHandlerBasecallback was provided.LossOptimizerHandlerAccumulateBatches CallbackAs well asLossOptimizerHandlerAccumulateSampleswill call loss.backward() every batch, but invoke optimizer.step() and optimizer.zero_grad()only after the defined num of batches (or samples) were accumulatedStatsPrint CallbackStatsPrintcallback prints informative summary of the trainer stats including loss and metrics.CallbackMonitorcan add nicer look withIMPROVEDindication on improved loss or metric, see output example below.Loss (for all states) will be monitored asMonitorMode.MINFor train metrics, provide your own monitors viatrain_metrics_monitorsargumentValidation metrics monitors will be added automatically according totrain_metrics_monitorsargumentfromlpd.enumsimportPhase,State,MonitorType,StatsType,MonitorModeStatsPrint(apply_on_phase=Phase.EPOCH_END,apply_on_states=State.EXTERNAL,train_metrics_monitors=CallbackMonitor(monitor_type=MonitorType.METRIC,stats_type=StatsType.TRAIN,monitor_mode=MonitorMode.MAX,metric_name='TruePositives'),print_confusion_matrix_normalized=True)# in case you use one of the ConfusionMatrix metrics (e.g. TruePositives), you may also print the confusion matrixOutput example:ModelCheckPoint CallbackSaving a checkpoint when a monitored loss/metric has improved.The callback will save the model, optimizer, scheduler, and epoch number.You can also configure it to save Full Trainer.For example,ModelCheckPointthat will save a newfull trainer checkpointevery time the validation metric_namemy_metricis getting higher than the highest value so far.ModelCheckPoint(Phase.EPOCH_END,State.EXTERNAL,checkpoint_dir,checkpoint_file_name,CallbackMonitor(monitor_type=MonitorType.METRIC,# It's a Metric and not a Lossstats_type=StatsType.VAL,# check the value on the Validation setmonitor_mode=MonitorMode.MAX,# MAX indicates higher is bettermetric_name='my_metric'),# since it's a Metric, mention its namesave_best_only=False,save_full_trainer=True)EarlyStopping CallbackStops the trainer when a monitored loss/metric has stopped improving.For example, EarlyStopping that will monitor at the end of every epoch, and stop the trainer if the validation loss didn't improve (decrease) for the last 10 epochs.EarlyStopping(Phase.EPOCH_END,State.EXTERNAL,CallbackMonitor(monitor_type=MonitorType.LOSS,stats_type=StatsType.VAL,monitor_mode=MonitorMode.MIN,patience=10))SchedulerStep CallbackWill invokestep()on your scheduler in the desired phase and state.For example, SchedulerStep callback to invokescheduler.step()at the end of every batch, in train state (as opposed to validation and test):fromlpd.callbacksimportSchedulerStepfromlpd.enumsimportPhase,StateSchedulerStep(apply_on_phase=Phase.BATCH_END,apply_on_states=State.TRAIN)Tensorboard CallbackWill export the loss and the metrics at a given phase and state, in a format that can be viewed on Tensorboardfromlpd.callbacksimportTensorboardTensorboard(apply_on_phase=Phase.EPOCH_END,apply_on_states=State.EXTERNAL,summary_writer_dir=dir_path)TensorboardImage CallbackWill export images, in a format that can be viewed on Tensorboard.For example, a TensorboardImage callback that will output all the images generated in validationfromlpd.callbacksimportTensorboardImageTensorboardImage(apply_on_phase=Phase.BATCH_END,apply_on_states=State.VAL,summary_writer_dir=dir_path,description='Generated Images',outputs_parser=None)Lets pass outputs_parser that will change the range of the outputs from [-1,1] to [0,255]fromlpd.callbacksimportTensorboardImagedefoutputs_parser(input_output_label:InputOutputLabel):outputs_scaled=(input_output_label.outputs+1.0)/2.0*255outputs_scaled=torchvision.utils.make_grid(input_output_label.output)returnoutputs_scaledTensorboardImage(apply_on_phase=Phase.BATCH_END,apply_on_states=State.VAL,summary_writer_dir=dir_path,description='Generated Images',outputs_parser=outputs_parser)CollectOutputs CallbackWill collect model's outputs for the defined states.CollectOutputs is automatically used byTrainerto collect the predictions when calling one of thepredictmethods.CollectOutputs(apply_on_phase=Phase.BATCH_END,apply_on_states=State.VAL)Create your custom callbacksfromlpd.enumsimportPhase,Statefromlpd.callbacksimportCallbackBaseclassMyAwesomeCallback(CallbackBase):def__init__(self,apply_on_phase=Phase.BATCH_END,apply_on_states=[State.TRAIN,State.VAL]):# make sure to call init parent classsuper(MyAwesomeCallback,self).__init__(apply_on_phase,apply_on_states)def__call__(self,callback_context):# <=== implement this method!# your implementation here# using callback_context, you can access anything in your trainer# below are some examples to get the hang of itval_loss=callback_context.val_stats.get_loss()train_loss=callback_context.train_stats.get_loss()train_metrics=callback_context.train_stats.get_metrics()val_metrics=callback_context.val_stats.get_metrics()optimizer=callback_context.optimizerscheduler=callback_context.schedulertrainer=callback_context.trainerifval_loss<0.0001:# you can also mark the trainer to STOP training by calling stop()trainer.stop()Lets expandMyAwesomeCallbackwithCallbackMonitorto track if our validation loss is getting betterfromlpd.callbacksimportCallbackBase,CallbackMonitor# <== CallbackMonitor addedfromlpd.enumsimportPhase,State,MonitorType,StatsType,MonitorMode# <== added few needed enums to configure CallbackMonitorclassMyAwesomeCallback(CallbackBase):def__init__(self,apply_on_phase=Phase.BATCH_END,apply_on_states=[State.TRAIN,State.VAL]):super(MyAwesomeCallback,self).__init__(apply_on_phase,apply_on_states)# adding CallbackMonitor to track VAL LOSS with regards to MIN (lower is better) and patience of 20 epochsself.val_loss_monitor=CallbackMonitor(MonitorType.LOSS,StatsType.VAL,MonitorMode.MIN,patience=20)def__call__(self,callback_context:CallbackContext):# <=== implement this method!# same as before, using callback_context, you can access anything in your trainertrain_metrics=callback_context.train_stats.get_metrics()val_metrics=callback_context.val_stats.get_metrics()# invoke track() method on your monitor and pass callback_context as parameter# since you configured your val_loss_monitor, it will get the relevant parameters from callback_contextmonitor_result=self.val_loss_monitor.track(callback_context)# monitor_result (lpd.callbacks.CallbackMonitorResult) contains informative properties# for example lets check the status of the patience countdownifmonitor_result.has_patience():print(f'[MyAwesomeCallback] - patience left:{monitor_result.patience_left}')# Or, let's stop the trainer, by calling the trainer.stop()# if our monitored value did not improveifnotmonitor_result.has_improved():print(f'[MyAwesomeCallback] -{monitor_result.description}has stopped improving')callback_context.trainer.stop()CallbackMonitor, AbsoluteThresholdChecker and RelativeThresholdCheckerWhen using callbacks such asEarlyStopping, aCallbackMonitoris provided to tracka certain metric and reset/trigger the stopping event (or any event in other callbacks).CallbackMonitorwill internally useThresholdCheckerwhen comparing new value to old valuefor the tracked metric, andAbsoluteThresholdCheckerorRelativeThresholdCheckerwill be usedto check if the criteria was met.The following example creates aCallbackMonitorthat will track if the metric 'accuracy'has increased with more then 1% usingRelativeThresholdCheckerfromlpd.utils.threshold_checkerimportRelativeThresholdCheckerrelative_threshold_checker_1_percent=RelativeThresholdChecker(monitor_mode=MonitorMode.MAX,threshold=0.01)CallbackMonitor(monitor_type=MonitorType.METRIC,# It's a Metric and not a Lossstats_type=StatsType.VAL,# check the value on the Validation setmonitor_mode=MonitorMode.MAX,# MAX indicates higher is bettermetric_name='accuracy',# since it's a Metric, mention its namethreshold_checker=relative_threshold_checker_1_percent)# track 1% increase from last highest valueMetricslpd.metricsprovides metrics to check the accuracy of your model.Let's create a custom metric usingMetricBaseand also show the use ofBinaryAccuracyWithLogitsin this examplefromlpd.metricsimportBinaryAccuracyWithLogits,MetricBasefromlpd.enumsimportMetricMethod# our custom metricclassInaccuracyWithLogits(MetricBase):def__init__(self):super(InaccuracyWithLogits,self).__init__(MetricMethod.MEAN)# use mean over the batchesself.bawl=BinaryAccuracyWithLogits()# we exploit BinaryAccuracyWithLogits for the computationdef__call__(self,y_pred,y_true):# <=== implement this method!# your implementation hereacc=self.bawl(y_pred,y_true)return1-acc# return the inaccuracy# we can now define our metrics and pass them to the trainermetrics=[BinaryAccuracyWithLogits(name='accuracy'),InaccuracyWithLogits(name='inaccuracy')]Let's do another example, a custom metricTruthfulnessbased on confusion matrix usingMetricConfusionMatrixBasefromlpd.metricsimportMetricConfusionMatrixBase,TruePositives,TrueNegativesfromlpd.enumsimportConfusionMatrixBasedMetric# our custom metricclassTruthfulness(MetricConfusionMatrixBase):def__init__(self,num_classes,labels=None,predictions_to_classes_convertor=None,threshold=0.5):super(Truthfulness,self).__init__(num_classes,labels,predictions_to_classes_convertor,threshold)self.tp=TruePositives(num_classes,labels,predictions_to_classes_convertor,threshold)# we exploit TruePositives for the computationself.tn=TrueNegatives(num_classes,labels,predictions_to_classes_convertor,threshold)# we exploit TrueNegatives for the computationdef__call__(self,y_pred,y_true):# <=== implement this method!tp_per_class=self.tp(y_pred,y_true)tn_per_class=self.tn(y_pred,y_true)# you can also access more confusion matrix metrics such asf1score=self.get_stats(ConfusionMatrixBasedMetric.F1SCORE)precision=self.get_stats(ConfusionMatrixBasedMetric.PRECISION)recall=self.get_stats(ConfusionMatrixBasedMetric.RECALL)# see ConfusionMatrixBasedMetric enum for morereturntp_per_class+tn_per_classSave and Load full TrainerSometimes you just want to save everything so you can continue training where you left off.To do so, you may useModelCheckPointfor saving full trainer by setting parametersave_full_trainer=TrueOr, you can invoke it directly from your traineryour_trainer.save_trainer(dir_path,file_name)Loading a trainer from checkpoint is as simple as:loaded_trainer=Trainer.load_trainer(dir_path,# the folder where the saved trainer file existstrainer_file_name,# the saved trainer file namemodel,# state_dict will be loadeddevice,loss_func,# state_dict will be loadedoptimizer,# state_dict will be loadedscheduler,# state_dict will be loadedtrain_data_loader,# provide new/previous data_loaderval_data_loader,# provide new/previous data_loadertrain_steps,val_steps)Utilslpd.utilsprovidestorch_utils,file_utilsandgeneral_utilsFor example, a good practice is to useseed_allas early as possible in your code, to make sure that results are reproducible:importlpd.utils.general_utilsasgugu.seed_all(seed=42)# because its the answer to life and the universeExtensionslpd.extensionsprovides some custom PyTorch layers, and schedulers, these are just some stuff we like using when we create our models, to gain better flexibility.So you can use them at your own will, more extensions are added from time to time.Something is missing?! please share with usYou can open an issue, but also feel free to email us attorch.lpd@gmail.com