Package: kesh-utils
Chart + Util = Chartil (Click to expand)Chart + Util = ChartilData visualization: Simple, Single unified API for plotting and chartingDuring EDA/data preparation we use few common and fixed set of chart types to analyse the relation among various features.
Few are simple charts like univariate and some are complex 3D or even multiple features>3.This api is simple, single api to plot various type of relations which will hide all the technical/code details from Data Science task and approch.
This overcomes the difficulties of maintaining several api or libraries and avoid repeated codes.Using this approach we just need one api (Rest all decided by library)from KUtils.eda import chartil

chartil.plot(dataframe, [list of columns]) or
chartil.plot(dataframe, [list of columns], {optional_settings})Demo code:Load UCI Dataset. DownloadFrom hereheart_disease_df = pd.read_csv('../input/uci/heart.csv')Quick data preparationcolumn_to_convert_to_categorical = ['target', 'cp', 'fbs', 'exang', 'restecg', 'slope', 'ca', 'thal']
for col in column_to_convert_to_categorical:
	heart_disease_df[col] = heart_disease_df[col].astype('category')

heart_disease_df['age_bin'] = pd.cut(heart_disease_df['age'], [0, 32, 40, 50, 60, 70, 100], labels=['<32', '33-40','41-50','51-60','61-70', '71+'])   

heart_disease_df['sex'] = heart_disease_df['sex'].map({1:'Male', 0:'Female'})

heart_disease_df.info()Heatmapchartil.plot(heart_disease_df, heart_disease_df.columns) # Send all column nameschartil.plot(heart_disease_df, heart_disease_df.columns, optional_settings={'include_categorical':True} )chartil.plot(heart_disease_df, heart_disease_df.columns, optional_settings={'include_categorical':True, 'sort_by_column':'trestbps'} )# Force to plot heatmap when you have fewer columns, otherwise tool will decide as different chart
chartil.plot(heart_disease_df, ['chol', 'thalach', 'trestbps'], chart_type='heatmap')Uni-categoricalchartil.plot(heart_disease_df, ['target']) # Barchart as count plotUni-Continuouschartil.plot(heart_disease_df, ['age'])chartil.plot(heart_disease_df, ['age'], chart_type='barchart') # Force barchart on cntinuous by auto creating 10 equal binschartil.plot(heart_disease_df, ['age'], chart_type='barchart', optional_settings={'no_of_bins':5}) # Create custom number of binschartil.plot(heart_disease_df, ['age'], chart_type='distplot')Uni-categorical with optional_settingschartil.plot(heart_disease_df, ['age_bin']) # Barchart as count plotchartil.plot(heart_disease_df, ['age_bin'], optional_settings={'sort_by_value':True})chartil.plot(heart_disease_df, ['age_bin'], optional_settings={'sort_by_value':True, 'limit_bars_count_to':5})Bi Category vs Category (& Univariate Segmented)chartil.plot(heart_disease_df, ['sex', 'target'])chartil.plot(heart_disease_df, ['sex', 'target'], chart_type='crosstab')chartil.plot(heart_disease_df, ['sex', 'target'], chart_type='stacked_barchart')Bi Continuous vs Continuouschartil.plot(heart_disease_df, ['chol', 'thalach']) # Scatter plotBi Continuous vs Categorychartil.plot(heart_disease_df, ['thalach', 'sex']) # Grouped box plot (Segmented univariate)chartil.plot(heart_disease_df, ['thalach', 'sex'], chart_type='distplot') # DistplotMulti 3 Continuouschartil.plot(heart_disease_df, ['chol', 'thalach', 'trestbps']) # Colored 3D scatter plotMulti 3 Categoricalchartil.plot(heart_disease_df, ['sex', 'age_bin', 'target']) # Paired barchartMulti 2 Continuous, 1 Categorychartil.plot(heart_disease_df, ['chol', 'thalach', 'target']) # Scatter plot with colored groupsMulti 1 Continuous, 2 Categorychartil.plot(heart_disease_df, ['thalach', 'sex', 'target']) # Grouped boxplotchartil.plot(heart_disease_df, ['thalach', 'sex', 'target'], chart_type='violinplot') # Grouped violin plotMulti 3 Continuous, 1 categorychartil.plot(heart_disease_df, ['chol', 'thalach', 'trestbps', 'target']) # Group Color highlighted 3D plotMulti 3 category, 2 Continuouschartil.plot(heart_disease_df, ['sex','cp','target','thalach','trestbps']) # Paired scatter plotFull working demo available onkaggle hereAuto Linear Regression (Click to expand)Auto Linear RegressionWe have seen Auto ML like H2O which is a blackbox approach to generate models.During our model building process, we try with brute force/TrialnError/several combinations to come up with best model.
However trying these possibilities manually is a laborious process.
In order to overcome or atleast have a base model automatically I developed this auto linear regression using backward feature elimination technique.The library/package can be foundhereand source codehereHow Auto LR works?We throw the cleaned dataset to autolr.fit(<>)
The method willTreat categorical variable if applicable(dummy creation/One hot encoding)First model - Run the RFE on datasetFor remaining features elimination - it follows backward elimination - one feature at a timecombination of vif and p-values of coefficients (Eliminate with higher vif and p-value combinationvif only (or eliminate one with higher vif)p-values only (or eliminate one with higher p-value)Everytime when a feature is identified we build new model and repeat the processon every iteration if adjusted R2 affected significantly, we re-add/retain it and select next possible feature to eliminate.Repeat until program can't proceed further with above logic.Auto Linear Regression Package/Function detailsThe methodautolr.fit()has below parametersdf, (The full dataframe)dependent_column, (Target column)p_value_cutoff = 0.01, (Threashold p-values of features to use while filtering features during backward elimination step, Default 0.01)vif_cutoff = 5, (Threashold co-relation of vif values of features to use while filtering features during backward elimination step, Default 5)acceptable_r2_change = 0.02, (Restrict degradtion of model efficiency by controlling loss of change in R2, Default 0.02)scale_numerical = False, (Flag to convert/scale numerical fetures using StandardScaler)include_target_column_from_scaling = True, (Flag to indiacte weather to include target column from scaling)dummies_creation_drop_column_preference='dropFirst', (Available options dropFirst, dropMax, dropMin - While creating dummies which clum drop to convert to one hot)train_split_size = 0.7, (Train/Test split ration to be used)max_features_to_select = 0, (Set the number of features to be qualified from RFE before entring auto backward elimination)random_state_to_use=100, (Self explanatory)include_data_in_return = False, (Include the data generated/used in Auto LR which might have gobne thru scaling, dummy creation etc.)verbose=False (Enable to print detailed debug messgaes)Above method returns 'model_info' dictionary which will have all the details used while performing auto fit.Full working demo available onkaggle hereClustered Linear Regression (Click to expand)Clustered Linear RegressionFor a linear regression approach we try to fit a best model on entire dataset.
However often we have seen within dataset based on a particular feature the dataset behaves totally different and single model is not the best solutions,
instead have multiple model which applied on different subset or filtered data does better.How to find the feature which splits the dataset into multiple sub dataset (and there after build and apply different models)There is no easy solution, instead use trial and error or brute force to subset data on different feature and build multiple model.
This clustred or grouped Linear Regression does the same.
You send the entire dataset and specifiy list of columns to separate the dataset individually and return the kpi measures like rmse or r2 etc and then decide which way to go.How "Clustered Linear Regression" works?First it lists possible combinationsFor each possible combinations split the data into subsetFor each subset execute the Auto Linear Regression. Check previous kaggle post on this.Return summary or consolidated kpi measures at group level.The API clustlr.fit() has below parametersdata_df (Full dataset)feature_group_list (List of column on which filter and group the datadependent_column (The target column)max_level = 2 (When it is 2 it uses two feature combination to filter)min_leaf_in_filtered_dataset=1000 (Condition the minimum datapoints in subgroup without which autolr will not be executed)no_of_bins_for_continuous_feature=10 (number of bins to be created when you use continuous varibale for grouping)verbose (Use True if you want detailed debug/log message)Full working demo available onkaggle hereAuto Logistic Regression (Click to expand)Auto Logistic Regression