Package: mlcomposer
:robot: Applying Machine Learning in an easy and efficient way like you never did before! :robot:Table of contentsAbout mlcomposerPackage StructureTransformers ModuleTrainer ModuleExamplesUsage around the worldContributionSocial MediaAbout mlcomposerHave ever tried to apply Machine Learning for solving a problem and found yourself lost on how much code you needed in order to reach your goal? Have you ever though it was too hard to do it? Don't be afraid because this package certainly can help you improving your skills and your modeling flow. Meetmlcomposeras a useful python package for helping users to use some built in classes and functions for applying Machine Learning as easy as possible.Withmlcomposer, you can:Build data prep solutions with custom pipelines by using python classes that handle transformation stepsTrain and extract all information you need from Machine Learning basic tasks like classification and regressionBuild and visualize custom evaluation charts like performance reports, confusion matrix, ROC curves and othersSave models in pkl or joblib formats after applying hyperparameter optimization in only one method executionCompare performances for multiple models at onceMuch more...Package StructureThe package is built around two main modules calledtransformersandtrainer. The first one contains custom python classes written strategically for improving constructions of pipelines using native sklearn's classPipeline. The second one is a powerful tool for training and evaluating Machine Learning models with classes for each different task (binary classification, multiclass classification and regression at this time). We will dive deep into those pieces on this documentation and I'm sure you will like it!Transformers ModuleAs said on the top of this section, thetransformersmodule allocates custom classes with useful transformations to be applied on data prep pipelines. In order to provide the opportunity to integrate these classes into a preparation pipeline using sklearn'sPipeline, every class on transformers module inheritsBaseEstimatorandTransformerMixinfrom sklearn. This way, the code for data transformation itself is written inside atransform()method in each class, giving the chance for execute a sequence of steps outside the module in a more complex Pipeline and also the possibility to use user custom classes on this preparation flow.If things are still a little complicated, the table below contains all classes built insidetransformersmodule. After that, it will be placed an example of a data prep pipeline written using some of those classes.ClassShort DescriptionColumnFormatterApplies a custom column formatting in a pandas DataFrame to standardize column namesColumnSelectionFilters columns in a DataFrame based on a list passed as a class attributeBinaryTargetMappingTransforms a raw target column in a binary one (1 or 0) based on a positive class argumentDropDuplicatesDrops duplicated rows in a datasetDataSplitterApplies a separation on data and creates new sets usingtrain_test_split()functionCategoricalLimitterLimits entries in categorical columns and restrict them based on a "n_cat" argumentCategoricalMapperReceives a dictionary for mapping entries on categorical columnsDtypeModifierChanges dtypes for columns based on a "mod_dict" argumentDummiesEncodingApplies dummies encoding process (OrdinalEncoder usingpd.get_dummies()method)FillNullDataFills null data (at just only columns, if needed)FeatureSelectionUses feature importance analysis for selecting top k features for the modelLogTransformationApplieslog1p()from numpy in order to log transform all numerical dataDynamicLogTransformationContains a boolean flag for applying or not log transformation (can be tunned afterall)DynamicScalerApplies normalization on data (StandardScalerorMinMaxScalerbased on an application flagModelResultsReceives a set of features and an estimator for building a DataFrame with source data and predictionsWith table above, we can imagine a dot of custom transformations that can be applied one by one on data preparation pipelines. The snippet below simulates a pipeline construction with some of those classes. The idea is to create a block of code that automatically fills null data with a dummy number, applies log transformation and scaler on numerical features and finally applies dummies encoding on categorical features. Let's see how it can be built.frommlcomposer.transformersimportFillNulldata,DynamicLogTransformation,DynamicScalerfromsklearn.pipelineimportPipeline# Building a numerical pipelinenum_pipeline=Pipeline([('imputer',FillNullData(value_fill=-999)),('log_transformer',DynamicLogTransformation(application=True,num_features=NUM_FEATURES,cols_to_log=COLS_TO_LOG)),('scaler',DynamicScaler(scaler_type='Standard'))])# Building a categorical pipelinecat_pipeline=Pipeline([('encoder',DummiesEncoding(dummy_na=True,cat_features_final=CAT_FEATURES_FINAL))])# Building a complete pipelineprep_pipeline=ColumnTransformer([('num',num_pipeline,NUM_FEATURES),('cat',cat_pipeline,CAT_FEATURES)])Done! With few steps and some python tricks, it was possible to build a resilient pipeline for handling all transformation steps specified. The best from it is that it was not necessary to build up those custom classes as long as thetransformersmodule did it for us.Trainer ModuleWell, by now what we can do after designing a complete pipeline for data preparation step? The answear could be straightforward: train and evaluate Machine Learning models! And that's the point thetrainermodule frommlcomposergets in. Withtrainer, we can use classes already built for handling common steps for training and evaluating models considering basic ML approaches like binary and multiclass classification or linear regression.So, it's possible to define thetrainermodule as a extremely poweful tool that gives the user the hability to solve complex problems and generates useful analysis with few lines of codes wherever there's a basic ML problem. It provides classes that encapsulates most of all steps needed for completing the solution. For make it cleaner, the table below shows up what kind of problemstrainermodule can solves or, in other words, what classes are built into the module.ClassShort DescriptionBinaryClassifierSolves binary classification problems by delivering useful methods for training and evaluating multiple models at onceMulticlassClassifierSolves multiclass classification problems by delivering useful methods for training and evaluating multiple models at onceLinearRegressorSolves linear regression problems by delivering useful methods for training and evaluating multiple models at onceBelow it will be place an example oftrainermodule usage by passing through a complete flow of a binary classification problem. Two powerful methods ofBinaryClassifierclass will be applied in order to train, evaluate, extract feature importance, plotting confusion matrix, ROC Curve, score distribution and other visual analysis: thetraining_flow()andvisual_analysis()methods. Take a look:#[...] other imports, variable definition and data pre processfrommlcomposer.trainerimportBinaryClassifier# Creating a objecttrainer=BinaryClassifier()# Training and evaluating modelstrainer.training_flow(set_classifiers,X_train_prep,y_train,X_val_prep,y_val,features=MODEL_FEATURES,metrics_output_path=METRICS_OUTPUT_PATH,models_output_path=MODELS_OUTPUT_PATH)# Generating and saving figures for visual analysistrainer.visual_analysis(features=MODEL_FEATURES,model_shap=MODEL_SHAP,output_path=IMGS_OUTPUT_PATH)The simple execution of these two methods on binary classification problems can generate the following output:└──output├──imgs│├──confusion_matrix.png│├──feature_importances.png│├──learning_curve.png│├──metrics_comparison.png│├──roc_curve.png│├──score_bins_percent.png│├──score_bins.png│├──score_distribution.png│└──shap_analysis_modelname.png├──metrics│├──metrics.csv│└──top_features.csv└──models├──modelname_date.pklInstalling the PackageThe last version ofmlcomposerpackage are published and available onPyPI repository:pushpin:Note:as a good practice for every Python project, the creation of avirtual environmentis needed to get a full control of dependencies and third part packages on your code. By this way, the code below can be used for creating a new venv on your OS.# Creating and activating venv on Linux$python-mvenv<path_venv>/<name_venv>
$source<path_venv>/<nome_venv>/bin/activate# Creating and activating venv on Windows$python-mvenv<path_venv>/<name_venv>
$<path_venv>/<nome_venv>/Scripts/activateWith the new venv active, all you need is execute the code below using pip for installing xplotter package (upgrading pip is optional):$pipinstall--upgradepip
$pipinstallmlcomposerThe mlcomposer package is built in a layer above some other python packages like pandas, numpy, sklearn and shap. Because of that, when installing mlcomposer, the pip utility will also install all dependencies linked to the package.ExamplesFor making the package usage easy as possible for new users, it's placed on this Github repository a direcotry identified asexamples/with python scripts with complete examples usingtransformersandtrainermodules for solving differente Machine Learning problems with different approachs.Usage Around the WorldBy now, themlcomposerpackage was first introduced on Kaggle's Tabular Playground Series of May 2021 with thelinked notebookby userThiago Panini. This notebooks also uses thexplottermodule for making the Exploratory Data Analysis and that would be a good idea too for getting insights from data. By now, the notebook reaches a bronze model with around 15 upvotes.ContributionThe mlcomposer python package is an open source implementation and the more people use it, the more happy the developers will be. So if you want to contribute with mlcomposer, please feel free to follow the best practices for implementing coding on this github repository through creating new branches, making merge requests and pointig out whenever you think there is a new topic to explore or a bug to be fixed.Thank you very much for reaching this and it will be a pleasure to have you as mlcomposer user or developer.Social MediaFollow me on LinkedIn:https://www.linkedin.com/in/thiago-panini/See my other Python packages:https://github.com/ThiagoPanini