Package: tabml
TabML: a Machine Learning pipeline for tabular dataTabML: a Machine Learning pipeline for tabular dataIntroductionInstallationMain componentsExamplesSetup for developmentAdd path to this repoCreate the environmentCheck that everthing is workingAuthor's notesHow to release a new versionCommon errorsIntroductionThis is an active project that aims to create a general machine learning framework for working with tabular data.Key features:One of the most important tasks in working with tabular data is to hanlde feature extraction. TabML allow users to define multiple features isolatedly without worrying about other features. This helps reduce coding conflicts if your team have multiple members simultaneously developing different features. In addition, if one feature needs to be updated, unrelated features could be untouched. In this way, the computating cost is relatively small (compared with running a pipeline to re-generate all other features).Parameters are specified in a config file as a config file. This config file is automatically saved into an experiment folder after each training for the reproducibility purpose.Support multiple ML packages for tabular data:LightGBMXGBoostCatBoostScikit-learnKerasPytorchTabNet...InstallationpipinstalltabmlMain componentsIn TRAINING step,TheFeatureManagerclass is responsible for loading raw data and engineering it into relavent features for model training and analysis. If afitstep, e.g. imputation, is required for a feature, the fitted parameters will be stored for using later in thetransformstep. One such usage is in the serving step when there is onlytransformstep. For each project, there is onefeature_manager.pyfile which specifies how each feature is computed (example). The computation order as well as feature dependencies are specified in a yaml config file (example).TheDataLoaderloads training and validation data for model training and analysis. In a typical project, tabml already takes care of this class, users only need to specify configuration in the pipeline config file (example). In that file, features and label used for training need to be specified. In addition, a set of boolean features are used as conditions for selecting training and validation data. Only rows in the dataset that meet all training/validation conditions are selected.TheModelWrapperclass defines the model, how to train it and other methods for loading the model and making predictions.TheModelAnalysisanalyzes the model on different metrics at user-defined dimensions. Analyzing metrics at different slices of data could determine if the trained model is biased to some feature value or any slice of data that model performance could be improved.In SERVING step, raw data is fed into thefittedFeatureManager to get the transfomed features that the trained model could use. The model is then making predictions for the transformed features.ExamplesPlease check theexamplesfolder for several example projects. For each project:pythonfeature_manager.py# to generate featurespythonpipelines.py# to train the modelYou can change some parameters in the config file then runpython pipelines.pyagain.In most project, users only need to focus their efforts on designing features. The feature dependecy is defined in a yaml config file and the feature implementation is stored infeature_manager.py.Setup for developmentAdd path to this repoAdd the following lines to your shell config file (~/.bashrc,~/.zshrcor any shell config file of
your choice):exportTABML=<local_path_to_this_git_repo>alias2tabml='cd $TABML; source bashrc; source tabml_env/bin/activate; python3 setup.py install'Create the environmentcd$TABMLpython3-mvenvtabml_envsourcetabml_env/bin/activate
pip3install-rrequirements.txtSetuppre-committo auto format code when creating a git
commit:pre-commitinstallCheck that everthing is workingby running test2tabml
python3-mpytest./tests./examplesAuthor's notesHow to release a new versionIncreaseversioninsetup.pyas inthis PR example.Generate tar file:pythonsetup.pysdistUpload tar file:twineuploaddist/tabml-x.x.xx.tar.gzCommon errorsSHAPSHAP might not work for MacOS if Xcode version < 13, try to upgrade it to xcode 13.Related issue.LightGBMpip install lightgbmmight not work for MacOS, try to followofficial installation guide for mac.If you find a bug or want to request a feature, feel free to create an issue. Any Pull Request would be much appreciated.