Package: trava
🌿 Trava ( initially stands for TrainValidation )Framework that helps to train models, compare them and track parameters&metrics along the way. Works with tabular data only.pip install travaCompare models and keep track of metrics with ease!While working on a project, we often experiment with different models looking at the same metrics.
For example, we log those that can be represented as a single number, however some of them require graphs to make sense. It's also useful to save those metrics somewhere for future analysis, the list can go on.So why not to use some unified interface for that?Here is Trava's way:1). Declare metrics you want to calculate:# in this case, sk and sk_proba are just wrappers around sklearn's metrics# but you can use any metric implementation you wantscorers=[sk_proba(log_loss),sk_proba(roc_auc_score),sk(recall_score),sk(precision_score),]2). What do you want to do with the metrics?# let's log the metricslogger_handler=LoggerHandler(scorers=scorers)3). Initialize Travatrava=TravaSV(results_handlers=[logger_handler])4). Fit your model using Trava# prepare your dataX_train,X_test,y_train,y_test=...split_result=SplitResult(X_train=X_train,y_train=y_train,X_test=X_test,y_test=y_test)trava.fit_predict(raw_split_data=split_result,model_type=GaussianNB,# we pass model class and parameters separatelymodel_init_params={},# to be able to track them properlymodel_id='gnb')# just a unique identifier for this modelfit_predictcall does roughly the same as:gnb=GaussianNB()gnb.fit(split_result.X_train,split_result.y_train)gnb.predict(split_result.X_test)But now you don't care how the metrics you declared are calculated. You just get them in your console! Btw, those metrics definitely need to be improved. :]Modelevaluationnb
*Resultsforgnbmodel*
Trainmetrics:
log_loss:16.755867191506482
roc_auc_score:0.7746522424770221
recall_score:0.10468384074941452
precision_score:0.9122448979591836


Testmetrics:
log_loss:16.94514025416013
roc_auc_score:0.829444814485889
recall_score:0.026041666666666668
precision_score:0.7692307692307693After training multiple models you can get all metrics for all models by calling.trava.resultsGet the full picture and more examples by going throughthe guide notebook!Built-in handlers:LoggerHandler- logs metricsPlotHandler- plots metricsMetricsDictHandler- returns all metrics wrapped in a dictEnable metrics autotracking. How cool is that?Experiments tracking is a must in Data Science, so you shouldn't neglect that. You may integrate any tracking framework inTrava!Travacomes withMLFlowtracker ready-to-go.
It can autotrack:model's parametersany metricplotsserialized modelsMLFlow example:# get tracker's instancetracker=MLFlowTracker(scorers=scorers)# initialize Trava with ittrava=TravaSV(tracker=tracker)# fit your model as beforetrava.fit_predict(raw_split_data=split_result,model_type=GaussianNB,model_id='gnb')Done. All model parameters and metrics are now tracked!
Also supported tracking of:cross-validation case with nested trackingeval results for common boosting libraries (XGBoost,LightGBM,CatBoost)Checkout a detailed notebooks how totrack metrics & parametersandplots & serialized models.General informationhighly customizable training & evaluation processes ( seetrava.fit_predictor.py.FitPredictorclass and its subclasses )built-in train/test/validation split logiccommon boosting libraries extensions ( for early-stopping with validation sets )tracks model parameters, metrics, plots, serialized models. it's easy to integrate any tracking framework of your choiceyou are also able to evaluate metrics afterfit_predictcall, if you forgot to add some metricyou are able to evaluate metrics even when your data and even a trained model are already unloaded ( depends on a metric used, true most of the times )now only supervised learning problems are supported yet there is a potential to extend it to support unsupervised problemsunit-testedI use it every day for my needs thus I care about the quality and reliabilityOnly sklearn-style model are supported for the time being. ( it usesfit,predict,predict_probamethods )Requirementspandas
numpy
python 3.7It's also convenient to use the lib withsklearn( e.g. for taking metrics from there. ). Also couple of extensions are based on sklearn classes.