Package: mltb
Machine Learning Tool BoxThis is the machine learning tool box. A collection of userful machine learning tools intended for reuse and extension.
The toolbox contains the following modules:hyperopt - Hyperopt tool to save and restart evaluationskeras - Keras (tf.keras) callback for various metrics and various other Keras toolslightgbm - metric tool functions for LightGBMmetrics - several metric implementationsplot - plot and visualisation toolstools - various (i.a. statistical) toolsModule: hyperoptThis module contains a tool function to save and restart Hyperopt evaluations.
This is done by saving and loading thehyperopt.Trialsobjects.
The usage looks like this:from mltb.hyperopt import fmin
from hyperopt import tpe, hp, STATUS_OK


def objective(x):
    return {
        'loss': x ** 2,
        'status': STATUS_OK,
        'other_stuff': {'type': None, 'value': [0, 1, 2]},
        }


best, trials = fmin(objective,
    space=hp.uniform('x', -10, 10),
    algo=tpe.suggest,
    max_evals=100,
    filename='trials_file')

print('best:', best)
print('number of trials:', len(trials.trials))Output of first run:No trials file "trials_file" found. Created new trials object.
100%|██████████| 100/100 [00:00<00:00, 338.61it/s, best loss: 0.0007185087453453681]
best: {'x': 0.026805013436769026}
number of trials: 100Output of second run:100 evals loaded from trials file "trials_file".
100%|██████████| 100/100 [00:00<00:00, 219.65it/s, best loss: 0.00012259809712488858]
best: {'x': 0.011072402500130158}
number of trials: 200Module: lightgbmThis module implements metric functions that are not included in LightGBM.
At the moment this is the F1- and accuracy-score for binary and multi class problems.
The usage looks like this:bst = lgb.train(param,
                train_data,
                valid_sets=[validation_data]
                early_stopping_rounds=10,
                evals_result=evals_result,
                feval=mltb.lightgbm.multi_class_f1_score_factory(num_classes, 'macro'),
               )Module: keras (for tf.keras)BinaryClassifierMetricsCallbackThis module provides custom metrics in form of a callback.
Because the callback adds these values to the internallogsdictionary it is
possible to use theEarlyStoppingcallback to do early stopping on these metrics.ParametersParameterDescriptionTypeDefault valuesval_dataValidation inputlistval_labelValidation outputlistpos_labelWhich index is the positive labelOptional[int]1metricsList of supported metric names or custom metric functionsList[Union[str, Callable]]['val_roc_auc', 'val_average_precision', 'val_f1', 'val_acc']Available metricsval_roc_auc: ROC-AUCval_f1: F1-scoreval_acc: Accuracyval_average_precision: Average precisionval_mcc: Matthews correlation coefficientThe usage looks like this:bcm_callback=mltb.keras.BinaryClassifierMetricsCallback(val_data,val_labels)es_callback=callbacks.EarlyStopping(monitor='val_roc_auc',patience=5,mode='max')history=network.fit(train_data,train_labels,epochs=1000,batch_size=128,#do not give validation_data here or validation will be done twice#validation_data=(val_data, val_labels),#always provide BinaryClassifierMetricsCallback before the EarlyStopping callbackcallbacks=[bcm_callback,es_callback],)You can also define your own custom metric:defcustom_average_recall_score(y_true,y_pred,pos_label):rounded_pred=np.rint(y_pred)returnsklearn.metrics.recall_score(y_true,rounded_pred,pos_label)bcm_callback=mltb.keras.BinaryClassifierMetricsCallback(val_data,val_labels,metrics=[custom_average_recall_score])es_callback=callbacks.EarlyStopping(monitor='custom_average_recall_score',patience=5,mode='max')history=network.fit(train_data,train_labels,epochs=1000,batch_size=128,#do not give validation_data here or validation will be done twice#validation_data=(val_data, val_labels),#always provide BinaryClassifierMetricsCallback before the EarlyStopping callbackcallbacks=[bcm_callback,es_callback],)