Package: lazytransform
lazytransformAutomatically transform all categorical, date-time, NLP variables in your data set to numeric in a single line of code for any data set any size.Update (Aug 2022)lazytransform is very easy to install on Kaggle and Colab notebooks using this command:.!pip install lazytransform --ignore-installed --no-cache --no-depslazytransform as of version 0.91 has two Super Learning Optimized (SULO) Ensembles named "SuloClassifier" and "SuloRegressor". The estimators are "super-optimized" in the sense that they perform automatic GridSearchCV so you can use them for all kinds of multi-label multi-class and Imbalanced data set problems with just default parameters to get great results in Kaggle competitions.Take a look at the amazing benchmarking results notebook here for SuloClassifier:We ran a similar benchmarking result in SuloRegressor against XGBoost and LightGBM Regressors and it held its own against them. Take a look at the benchmarking result:Table of ContentsWhat is lazytransformHow to use lazytransformHow to install lazytransformUsageTipsAPIMaintainersContributingLicenseIntroductionWhat is lazytransform?lazytransformis a new python library for automatically transforming your entire dataset to numeric format using category encoders, NLP text vectorizers and pandas date time processing functions. All in a single line of code!Useslazytransformhas two important uses in the Data Science process. It can be used in Feature Engg to transform features or add features (see API below). It can also be used in MLOps to train and evaluate models in data pipelines with multiple models being trained simultaneusly using the same train/test split and the same feature engg steps. This ensures that there is absolutely zero or minimal data leakage in your MLOps pipelines.1.  Using lazytransform as a simple pandas data transformation pipelineThe first method is probably the most popular way to use lazytransform. The transformer within lazytransform can be used to transform and create new features from categorical, date-time and NLP (text) features in your dataset. This transformer pipeline is fully scikit-learn Pipeline compatible and can be used to build even more complex pipelines by you based on `make_pipeline` statement from `sklearn.pipeline` library.Let us see an example:2.  Using lazytransform as a sklearn pipeline with sklearn models or XGBoost or LightGBM modelsThe second method is a great way to create an entire data transform and model training pipeline with absolutely no data leakage. `lazytransform` allows you to send in a model object (only the following are supported) and it will automatically transform, create new features and train a model using sklearn pipelines.This method can be seen as follows:3.  Using lazytransform in GridSearchCV to find the best model pipelineThe third method is a great way to find the best data transformation and model training pipeline using GridSearchCV or RandomizedSearchCV along with a LightGBM or XGBoost or scikit-learn model. This is explained very clearly in theLazyTransformer_with_GridSearch_Pipeline.ipynbnotebook in the same github here. Make sure you check it out!The following models are currently supported:All sklearn modelsAll MultiOutput models from sklearn.multioutput libraryXGboost modelsLightGBM modelsHowever, you must install and import those models on your own and define them as model variables before passing those variables to lazytransform.InstallPrerequsites:lazytransform is built using pandas, numpy, scikit-learn, category_encoders and imbalanced-learn libraries.It should run on most Python3 Anaconda installations without additional installs. You won't have to import any special libraries other than "imbalanced-learn" and "category_encoders".The best method to install lazytransform is to use conda:conda install -c conda-forge lazytransformThe second best installation method is to use "pip install".pip install lazytransformAlert! When using Colab or Kaggle Notebooks, you must use a slightly modify installation process below. If you don't do this, you will get weird errors in those platforms!pip install lazytransform --ignore-installed --no-deps
pip install category-encoders --ignore-installed --no-depsTo install from source:cd <lazytransform_Destination>
git clone git@github.com:AutoViML/lazytransform.gitor download and unziphttps://github.com/AutoViML/lazytransform/archive/master.zipconda create -n <your_env_name> python=3.7 anaconda
conda activate <your_env_name> # ON WINDOWS: `source activate <your_env_name>`
cd lazytransform
pip install -r requirements.txtUsageYou can invoke `lazytransform` as a scikit-learn compatible fit and transform or a fit and predict pipeline. See syntax below.from lazytransform import LazyTransformer
lazy = LazyTransformer(model=None, encoders='auto', scalers=None, 
        date_to_string=False, transform_target=False, imbalanced=False,
        combine_rare=False, verbose=0)if you are not using a model in pipeline, you must use fit and transformX_trainm, y_trainm = lazy.fit_transform(X_train, y_train)
X_testm = lazy.transform(X_test)If you are using a model in pipeline, use must use fit and predict onlylazy = LazyTransformer(model=RandomForestClassifier(), encoders='auto', scalers=None, 
        date_to_string=False, transform_target=False, imbalanced=False,
        combine_rare=False, verbose=0)lazy.fit(X_train, y_train)
lazy.predict(X_test)TipsTips for using SuloClassifier and SuloRegressor for High Performance:First try it with base_estimator as None and all other params as either None or FalseCompare it against a competitor model such as XGBoost or RandomForest and see whether it beats them.If not, then set weights = True for Sulo models and then imbalanced=True and see whether that works.If a competitor is still beating Sulo, then use that model as base_estimator while leaving all other params above untouched.Finally change the n_estimators from default=None to 5.Finally increase n_estimators to 7 and then 10 and see. By now, Sulo should be beating all other models.The more you increase the number of estimators, the more performance boost you will get until at some point it drops off. Keep increasing until then.APIlazytransform has a very simple API with the following inputs. You need to create a sklearn-compatible transformer pipeline object by importing LazyTransformer from lazytransform library.Once you import it, you can define the object by giving several options such as:ArgumentsCaution:X_train and y_train must be pandas Dataframes or pandas Series. DO NOT send in numpy arrays. They won't work.model: default is None. Or it could be any scikit-learn model (including multioutput models) as well as the popular XGBoost and LightGBM libraries. You need to install those libraries if you want to use them.encoders: could be one more encoders in a string or a list. Each encoder string can be any one of the 10+ encoders fromcategory_encoderslibrary below.  Available encoders are listed here as strings so that you can input them in lazytransform:auto- It usesonehotencoding for low-cardinality variables andlabelencoding for high cardinality variables.onehot- One Hot encoding - it will be used for all categorical features irrespective of cardinalitylabel- Label Encoding - it will be used for all categorical features irrespective of cardinalityhashingorhash- Hashing (or Hash) Encoding - will be used for all categorical variableshelmert- Helmert Encoding - will be used for all categorical variablesbdc- BDC Encoding - will be used for all categorical variablessum- Sum Encoding - will be used for all categorical variablesloo- Leave one out Encoding - will be used for all categorical variablesbase- Base encoding - will be used for all categorical variableswoe- Weight of Evidence Encoding - will be used for all categorical variablesjames- James Encoding - will be used for all categorical variablestarget- Target Encoding - will be used for all categorical variablescount- Count Encoding - will be used for all categorical variablesglm,glmm- Generalized Linear Model EncodingHere is a description of various encoders and their uses from the excellentcategory_encoderspython library:HashingEncoder: HashingEncoder is a multivariate hashing implementation with configurable dimensionality/precision. The advantage of this encoder is that it does not maintain a dictionary of observed categories. Consequently, the encoder does not grow in size and accepts new values during data scoring by design.SumEncoder: SumEncoder is a Sum contrast coding for the encoding of categorical features.PolynomialEncoder: PolynomialEncoder is a Polynomial contrast coding for the encoding of categorical features.BackwardDifferenceEncoder: BackwardDifferenceEncoder is a Backward difference contrast coding for encoding categorical variables.OneHotEncoder: OneHotEncoder is the traditional Onehot (or dummy) coding for categorical features. It produces one feature per category, each being a binary.HelmertEncoder: HelmertEncoder uses the Helmert contrast coding for encoding categorical features.OrdinalEncoder: OrdinalEncoder uses Ordinal encoding to designate a single column of integers to represent the categories in your data. Integers however start in the same order in which the categories are found in your dataset. If you want to change the order, just sort the column and send it in for encoding.FrequencyEncoder: FrequencyEncoder is a count encoding technique for categorical features. For a given categorical feature, it replaces the names of the categories with the group counts of each category.BaseNEncoder: BaseNEncoder encodes the categories into arrays of their base-N representation. A base of 1 is equivalent to one-hot encoding (not really base-1, but useful), a base of 2 is equivalent to binary encoding. N=number of actual categories is equivalent to vanilla ordinal encoding.TargetEncoder: TargetEncoder performs Target encoding for categorical features. It supports following kinds of targets: binary and continuous. For multi-class targets it uses a PolynomialWrapper.CatBoostEncoder: CatBoostEncoder performs CatBoost coding for categorical features. It supports the following kinds of targets: binary and continuous. For polynomial target support, it uses a PolynomialWrapper. This is very similar to leave-one-out encoding, but calculates the values “on-the-fly”. Consequently, the values naturally vary during the training phase and it is not necessary to add random noise.WOEEncoder: WOEEncoder uses the Weight of Evidence technique for categorical features. It supports only one kind of target: binary. For polynomial target support, it uses a PolynomialWrapper. It cannot be used for Regression.JamesSteinEncoder: JamesSteinEncoder uses the James-Stein estimator. It supports 2 kinds of targets: binary and continuous. For polynomial target support, it uses PolynomialWrapper.
For feature value i, James-Stein estimator returns a weighted average of:
The mean target value for the observed feature value i.
The mean target value (regardless of the feature value).QuantileEncoder: This is a very good encoder for Regression tasks. See Paper and article:https://towardsdatascience.com/quantile-encoder-eb33c272411dscalers: could be one of three main scalers used in scikit-learn models to transform numeric features. Default is None. Scalers are used in the last step of the pipeline to scale all features that have transformed. However, you might want to avoid scaling in NLP datasets since after TFiDF vectorization, scaling them may not make sense. But it is up to you. The 4 options are:NoneNo scaler. Great for almost all datasets. Test it first and then try one of the scalers below.stdstandard scaler. Great for almost all datasets.minmaxminmax scaler. Great for datasets where you need to see the distribution between 0 and 1.robustRobust scaler. Great for datasets where you have outliers.maxabsmax absolute scaler. Great for scaling but leaves the negative values as they are (negative).date_to_string: default is False. If you want to use date variables as strings (categorical), then set it as True.You can use this option when there are very few dates in your dataset. If you set it as False, it will convert it into date time format and extract up to 20 features from your date time column. This is the default option and best option.transform_target: default is False. If you want to transform your target variable(s), then set it as True and we will transform your target(s) as numeric using Label Encoding as well as multi-label Binary classes. This is a great option when you have categorical target variables.imbalanced: default is False. If you have an imbalanced dataset, then set it to True and we will transform your train data using BorderlineSMOTE or SMOTENC which are both great options. We will select the right SMOTE function automatically.combine_rare: default is False. This is a great option if you have too many rare categories in your categorical variables. It will automatically combine those rare categories which are less than 1% of the dataset into one combined category called "rare_categories". You can also set it to False and we will not transform any of your categories in your categorical variables.verbose: This has 3 possible states:0silent output. Great for running this silently and getting fast results.1more verbiage. Great for knowing how results were and making changes to flags in input.2highly verbose output. Great for finding out what happens under the hood in lazytransform pipelines.To view the text pipeline, the default display is 'text', do:from sklearn import set_config
set_config(display="text")
lazy.xformerTo view the pipeline in a diagram (visual format), do:from sklearn import set_config
set_config(display="diagram")
lazy.xformer
# If you have a model in the pipeline, do:
lazy.modelformerTo view the feature importances of the model in the pipeline, you can do:lazy.plot_importance()Maintainers@AutoViMLContributingSeethe contributing file!PRs accepted.LicenseApache License 2.0 © 2020 Ram SeshadriNote of GratitudeThis libray would not have been possible without the following great libraries:Category Encoders library:Fantastic library https://contrib.scikit-learn.org/category_encoders/index.htmlImbalanced Learn library:Another fantastic library https://imbalanced-learn.org/stable/index.htmlThe amazing `lazypredict`was an inspiration for `lazytransform`. You can check out the library here:
https://github.com/shankarpandala/lazypredictThe amazing `Kevin Markham`was another inspiration for lazytransform. You can check out his classes here:
https://www.dataschool.io/about/DISCLAIMERThis project is not an official Google project. It is not supported by Google and Google specifically disclaims all warranties as to its quality, merchantability, or fitness for a particular purpose.