Package: prep-ml
Prep-MLWhat is Prep-ML?prep-ml is an open-source pre-processing library aimed at simplifying the data processing steps and streamlining the transformation techniques before feeding it to your choice of machine learning algorithm.Why Prep-ML?Production grade machine learning is quite different from the standard notebook building. Notebook building is aimed at fast development, interactive code, and visual feedback system. While the scripts aim to cater models to large groups of audience or companies.For eg, consider one of the key features of your model is DATE_OF_BIRTH, in real-time, due to various database schemas, the feature could be available in any of its synonyms (say, DOB, BIRTH_DATE). This is where prep-ml tries to fill in, like a heavily inspired from ETL tools and other design patterns.Installation$ pip install prep-mlDocumentationThis is the schema currently supported by the library. This can take python dict or JSON string.{
    "FEATURE_NAME": {
        "required": bool,
        "encoding": str,
        "alias": str,
        "imputation": str,
        "derived_eq": str
    }
}Schema Definitions:required: boolaccepted values:True,Falsedetermines if the feature is required for the model.Ifrequiredis set toFalse, theFEATURE_NAMEis discarded for further processing.encoding: straccepted values:label,oheperforms the given encoding strategy on theFEATURE_NAME.Ifencodingis set to "label", LabelEncoding or OrdinalEncoding is performed on theFEATURE_NAMEIfencodingis set to "ohe", OneHotEncoding is performed on theFEATURE_NAMEalias: straccepted values: any stringthis is a synonym or alias for the givenFEATURE_NAME.For eg, Ifaliasis set to"FEATURE_OTHER_NAME", the alias name will be mapped to theFEATURE_NAMEimputation: straccepted values:mean,median,most_frequentperforms the given imputation strategy on theFEATURE_NAME. This is a wrapper of SimpleImputer.If "mean", then replace missing values using the mean for theFEATURE_NAME. Can only be used with numeric data.If "median", then replace missing values using the median for theFEATURE_NAME. Can only be used with numeric data.If "most_frequent", then replace missing using the most frequent value for theFEATURE_NAME. Can be used with strings or numeric data. If there is more than one such value, only the smallest is returned.derived_eq: straccepted values: eval equation as a stringevaluated the given equation and then assigns the response toFEATURE_NAME. The reference to dataframe should bedfFor eg consider the above featureDOB, Ifderived_eqis set to "pd.to_datetime(df.DOB, format='%m/%d/%Y')", the expression will be evaluated and assigned toFEATURE_NAME. Note that,dfis reference to the provided input df.Methods:from_dict(mapper, dataframe)-- reads the dict and processes the input dataframe.from_json(json_mapper, dataframe)-- reads the json and processes the input dataframe.get_data()--  fetches the processed data.Usage Example:This is the input dataData Explanation:This is randomly generated data for the purposes of demo. All references are assumptions.This is a company employee data. We have various features, which are self explanatory.Ideally, we would want to remove the NAMES, as they are uniques and serve no purpose in model. Transform DOB to say a derived feature called AGE. Encode, GENDER, DESIGNATION and PROMOTED. Impute RATING.So, on using the driver code.from prep_ml.pre_processor import Prep
import pandas as pd

prep_ob = {
    "EMPLOYEE_ID": {
        "alias": "EMP_ID",
        "required": True
    },
    "FIRST_NAME": {
        "required": False,
    },
    "LAST_NAME": {
        "required": False,
    },
    "AGE": {
        "required": True,
        "alias": "DOB",
        "derived_eq": "(pd.Timestamp('now') - pd.to_datetime(df.AGE, format='%m/%d/%Y')).astype('<m8[Y]')"
    },
    "GENDER": {
        "required": True,
        "encoding": 'ohe'
    },
    "RATING": {
        "required": True,
        "imputation": 'most_frequent'
    },
    "DESIGNATION": {
        "required": True,
        "encoding": 'label'
    },
    "PROMOTED": {
        "required": True,
        "encoding": 'label'
    }
}

df = pd.read_csv('tests/MOCK_DATA.csv')

p = Prep.from_dict(prep_ob, df)
rdf = p.get_data()
print(rdf.columns.to_list())Output['EMPLOYEE_ID', 'DESIGNATION', 'PROMOTED', 'RATING', 'AGE', 'GENDER_1', 'GENDER_2', 'GENDER_3']The output in dataviewer is as follows.Future Development RoadmapPerformance improvements.Add support for more imputation and encoding strategies.Support for feature scaling.Support for multiple schemas.Support for multiple input sources.Support for enforcing column types.Feasibility for model training.Changelog2nd May, 2021 :: v0.1.0:This is a very early dev version. This further needs development and code optimization.