Binary Classification Example. Predicting Opioid Use | by Kamil Mysiak | Towards Data Science

This global crisis has impacted all of our lives in one way or another but this is a perfect opportunity to hone your craft. I happened to renew my Coursera account specifically trying my hand at network analysis. Cryptocurrency/blockchain medium articles have become part of my daily routine. Finally, I wanted to post an ML classification example to help those looking for a detailed ‘Beginning to End’ use case and call for constructive feedback from the community at large. This journey will be long and detailed, I hope you are ready for the ride.
We will utilize an insurance dataset which outlines a series of patient centered features with the ultimate goal to correctly predict whether or not opioid abuse has occurred.
ClaimID Unique: Identifier for a claimAccident DateID: Number of days since the accident occurred from an arbitrary dateClaim Setup DateID: Number of days since the Resolution Manager sets up the claim from an arbitrary dateReport To DateID: Number of days since the employer notifies insurance of a claim from an arbitrary dateEmployer Notification DateID: Number of days since the claimant notifies employer of an injury from an arbitrary dateBenefits State: The jurisdiction whose benefits are applied to a claimAccident State: State in which the accident occurredIndustry ID: Broad industry classification categoriesClaimant Age: Age of the injured worker Claimant Sex: Sex of the injured worker Claimant State: State in which the claimant residesClaimant Marital Status: Marital status of the injured worker Number Dependents: Number of dependents the claimant hasWeekly Wage: An average of the claimant’s weekly wages as of the injury date.Employment Status Flag: F — Regular full-time employee P — Part-time employee U — Unemployed S — On strike D — Disabled R — Retired O — Other L — Seasonal worker V — Volunteer worker A — Apprenticeship full-time B — Apprenticeship part-time C — Piece workerRTW Restriction Flag: A Y/N flag, used to indicate whether the employees responsibilities upon returning to work were limited as a result of his/her illness or injury.Max Medical Improvement DateID: DateID of Maximum Medical Improvement, after which further recovery from or lasting improvements to an injury or disease can no longer be anticipated based on reasonable medical probability.Percent Impairment: Indicates the percentage of anatomic or functional abnormality or loss, for the body as a whole, which resulted from the injury and exists after the date of maximum medical improvementPost Injury Weekly Wage: The weekly wage of the claimant after returning to work, post-injury, and/or the claim is closed.NCCI Job Code: A code that is established to identify and categorize jobs for workers’ compensation.Surgery Flag: Indicates if the claimant’s injury will require or did require surgeryDisability Status: — Temporary Total Disability (TTD) — Temporary Partial Disability (TPD) — Permanent Partial Disability (PPD) — Permanent Total Disability (PTD)SIC Group: Standard Industry Classification group for the clientNCCI BINatureOfLossDescription: Description of the end result of the bodily injury (BI) loss occurrenceAccident Source Code: A code identifying the object or source which inflicted the injury or damage.Accident Type Group: A code identifying the general action which occurred resulting in the lossNeurology Payment Flag: Indicates if there were any payments made for diagnosis and treatment of disorders of the nervous system without surgical interventionNeurosurgery Payment Flag: Indicates if there were any payments made for services by physicians specializing in the diagnosis and treatment of disorders of the nervous system, including surgical intervention if neededDentist Payment Flag: Indicates if there were any payments made for prevention, diagnosis, and treatment of diseases of the teeth and gumsOrthopedic Surgery Payment Flag: Indicates if there were any payments made for surgery dealing with the skeletal system and preservation and restoration of its articulations and structures.Psychiatry Payment Flag: Indicates if there were any payments made for treatment of mental, emotional, or behavioral disorders.Hand Surgery Payment Flag: Indicates if there were any payments made for surgery only addressing one or both hands.Optometrist Payment Flag: Indicates if there were any payments made to specialists who examine the eye for defects and faults of refraction and prescribe correctional lenses or exercises but not drugs or surgeryPodiatry Payment Flag: Indicates if there were any payments made for services from a specialist concerned with the care of the foot, including its anatomy, medical and surgical treatment, and its diseases.HCPCS A Codes — HCPCS Z Codes: Count of the number of HCPCS codes that appear on the claim within each respective code groupICD Group 1 — ICD Group 21: Count of the number of ICD codes that appear on the claim w/in each respective code groupCount of the number of codes on the claim — CPT Category — Anesthesia — CPT Category — Eval_Mgmt — CPT Category — Medicine — CPT Category — Path_Lab — CPT Category — Radiology — CPT Category — SurgeryCount of the number of NDC codes on the claim within each respective code class — NDC Class — Benzo — NDC Class — Misc (Zolpidem) — NDC Class — Muscle Relaxants — NDC Class — StimulantsOpioids Used: A True (1) or False (0) indicator for whether or not the claimant abused an opioid
Let us begin with importing all the required libraries along with our dataset.
from math import sqrtimport pandas as pdimport numpy as npimport scipy.stats as statsimport matplotlib.pyplot as pltimport seaborn as snsfrom IPython.display import displayfrom sklearn.model_selection import train_test_splitfrom sklearn.pipeline import Pipelinefrom feature_engine import missing_data_imputers as mdifrom feature_engine import categorical_encoders as cefrom sklearn.preprocessing import RobustScalerfrom imblearn.over_sampling import SMOTEfrom sklearn.model_selection import cross_val_scorefrom sklearn.metrics import roc_auc_scorefrom sklearn.metrics import roc_curvefrom sklearn.metrics import recall_scorefrom sklearn.metrics import classification_reportfrom sklearn.metrics import confusion_matrixfrom sklearn.model_selection import GridSearchCVfrom sklearn.ensemble import GradientBoostingClassifierfrom sklearn.linear_model import LogisticRegressionfrom sklearn.ensemble import RandomForestClassifierfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.naive_bayes import GaussianNBfrom sklearn import svm%matplotlib inlinepd.pandas.set_option('display.max_columns', None)with open('opioid_use_data.csv') as f:    df = pd.read_csv(f)f.close()SEED = 42df.shapedf.head(10)df.info()
Our dataset contains just over 16,000 observations along with 92 features including the target (ie. Opiods Used). We also have a variety of feature types including integers, floats, strings, booleans and mixed type.
Before we tackle missing data, outliers or cardinality, let’s see if we can quickly delete any features to simplify our further analysis.
for var in df.columns:    print(var, df[var].nunique(), len(df))df.drop('ClaimID', axis=1, inplace=True)
As we scroll through the output we can see the number of unique values for each feature along with the total length of the entire dataset. Features which have a similar number of unique values as the total length of the dataframe can be removed as they don’t provide much predictive ability (ie. variance). ‘ClaimID’ is the only feature which meet this criteria and can be removed.
df_copy = df.copy()corr_matrix = df_copy.corr().abs()upper = corr_matrix.where(np.triu(np.ones(corr_matrix.shape), k=1).astype(np.bool))upper
to_drop = [var for var in upper.columns if any(upper[var] > .90)]to_drop
df.drop(df[to_drop], axis=1, inplace=True)
Next, we can examine the correlations between our numerical features and delete features which are very highly correlated. It is up to you to determine what is considered “highly correlated” but in this case we will select a correlation of 90 and above. Notice in the code we have constructed a correlation matrix and converted the correlations to their absolute values in order to deal with negative correlations. ‘Claim Setup DateID’, ‘Report to DateID’, ‘Employer Notification DateID’ and ‘Max Medical Improvement DateID’ can be removed.
for var in df.columns:    print(var, 'percent of missing values', df[var].isnull().mean().round(3))
We can examine the percentage of missing values for the remaining features and remove any features with an excessive missing data. One feature ‘Accident Source Code’ has over 50% of missing values but that’s not enough to warrant deletion.
states = pd.DataFrame(df[['Claimant State', 'Accident State', 'Benefits State']])states
df.drop(['Claimant State', 'Accident State'], axis=1, inplace=True)
Examining ‘Claimant State’, ‘Accident State’ and ‘Benefits State’ we find that the vast majority of the values are the same. We will keep ‘Benefits State’ as it contains the least amount of missing values.
for var in df.columns:    print(var, df[var].unique(), '\n')
# Converting nulls to np.nanfor var in df.columns:    df[var].replace(to_replace='  ', value=np.nan, inplace=True)# Converting values of "X" to np.nanfor var in df.columns:    df[var].replace(to_replace='X', value=np.nan, inplace=True)# Splitting out "Accident Type Group" (ie. mixed type feature) into separate featuresdf['Accident Type Group num'] = df['Accident Type Group'].str.extract('(\d+)')df['Accident Type Group str'] = df['Accident Type Group'].str[0:5]df.drop(['Accident Type Group'], axis=1, inplace=True)
When we examine the unique values for each feature we can start to see some discrepancies which require our attention. First, we notice blank or null values which have not been converted to Np.nan. Next, we are seeing the value of ‘X’ for many features and this seems like a recording discrepancy where the individual recording the data recorded missing values with an ‘X’. Finally, the feature named ‘Accident Type Group’ is what we call a mixed-type which because it contains both string and numerical values. Let’s separate the string and numerical values into their own features and delete the original ‘Accident Type Group’ feature.
for var in df.columns:    print(var,'\n', df[var].value_counts()/len(df))
df.drop(['Neurology Payment Flag', 'Neurosurgery Payment Flag', 'Dentist Payment Flag',         'Psychiatry Payment Flag', 'Hand Surgery Payment Flag', 'Optometrist Payment Flag',         'Podiatry Payment Flag', 'Accident Type Group str', 'Post Injury Weekly Wage'], axis=1, inplace=True)df.drop(['HCPCS B Codes','HCPCS C Codes', 'HCPCS D Codes', 'HCPCS F Codes', 'HCPCS H Codes', 'HCPCS I Codes','HCPCS K Codes', 'HCPCS M Codes', 'HCPCS N Codes', 'HCPCS O Codes', 'HCPCS P Codes','HCPCS Q Codes', 'HCPCS R Codes', 'HCPCS S Codes', 'HCPCS T Codes', 'HCPCS U Codes', 'HCPCS V Codes', 'HCPCS X Codes', 'HCPCS Y Codes', 'HCPCS Z Codes', 'ICD Group 1','ICD Group 2', 'ICD Group 3', 'ICD Group 4', 'ICD Group 5', 'ICD Group 7', 'ICD Group 8', 'ICD Group 9', 'ICD Group 10', 'ICD Group 11', 'ICD Group 12', 'ICD Group 14', 'ICD Group 15', 'ICD Group 16', 'ICD Group 17', 'ICD Group 20', 'NDC Class - Benzo', 'NDC Class - Misc (Zolpidem)','NDC Class - Stimulants'], axis=1, inplace=True)
Let’s now turn our attention to cardinality or the number of unique values/categories for each feature. Continuous feature such as ‘Weekly Wage’ will no doubt have hundreds or even thousands of unique categories. Nominal and discrete features (ie. gender and number of dependents) will have a much smaller number of categories. The goal of this exercise is to determine if any categories hold the majority (90%+) of the values. If a feature contains one or two categories which hold 90%+ of the values there simply isn’t enough variability in the data to retain the feature. It is ultimately up to you to determine the cut off but we feel 90% or more is a safe assumption.
Now that we have successfully eliminated many of the features due to high correlations, duplicate values and lack of variability we can focus on examining feature characteristics and deciding how to tackle each problem.
You will notice that in this section we are simply identifying the issue and making a mental note. We wouldn’t be actually applying the discussed changes until the end of the notebook into a feature engineering pipeline.
categorical = ['Benefits State', 'Industry ID', 'Claimant Sex', 'Claimant Marital Status', 'Employment Status Flag', 'RTW Restriction Flag', 'NCCI Job Code', 'Surgery Flag', 'Disability Status', 'SIC Group', 'NCCI BINatureOfLossDescription', 'Accident Source Code','Orthopedic Surgery Payment Flag','Accident Type Group num']discrete = ['Claimant Age', 'Number Dependents', 'Percent Impairment', 'HCPCS A Codes','HCPCS E Codes', 'HCPCS G Codes', 'HCPCS J Codes','HCPCS L Codes', 'HCPCS W Codes', 'ICD Group 6', 'ICD Group 13','ICD Group 18', 'ICD Group 19', 'ICD Group 21', 'CPT Category - Anesthesia', 'CPT Category - Eval_Mgmt', 'CPT Category - Medicine', 'CPT Category - Path_Lab', 'CPT Category - Radiology', 'CPT Category - Surgery', 'NDC Class - Muscle Relaxants']continuous = ['Accident DateID', 'Weekly Wage']df.columns,'Number of Features:',len(df.columns)
for var in df.columns:    if df[var].isnull().sum() > 0:        print(var, df[var].isnull().mean())
From our previous look at missing values, we discovered that only one feature contained more than 50% of missing values and the vast majority did not contain any missing data. That said, we do have a number of features which do contain missing data and we need to determine how we will deal with this issue as many ML algorithms require full clean datasets. This is an extensive topic and we do not wish to cover the intricacies in this blog but the reader should get well acquainted with this topic. First, one must orient oneself with the various types of missing data such as ‘Missing Completely at Random’, ‘Missing at Random’ and ‘Missing not at Random’. These topics help to pinpoint how and when you should deal with your missing data. Additionally, a further reading into imputation techniques such as Mean/Median/Mode, Arbitrary Value Imputation, Adding Missing Data Indicator, Random Sample Imputation, ML imputation, etc. will provide a great overview.
We have 9 features with missing data. For features with less than 5% of missing values (ie. Claimant Sex, Claimant Marital Status, Employment Status Flag, RTW Restriction Flag) we will replace the missing values with the mode of their distributions. Due to their low percentage of missing values, a mode imputation wouldn’t change the distribution by very much. ‘Accident DateID’ is our only continuous feature with missing data and we’ll impute missing values with an arbitrary number of -99999. All other features are categorical in nature and since they have more than 5% of missing values we’ll impute the missing values with the string ‘missing’. This ultimately wouldn’t change the distribution and only add a new category to their distributions.
Accident DateID: continuous w/ -99999Claimant Sex: categorical w/modeClaimant Marital Status: categorical w/modeEmployment Status Flag: categorical w/modeRTW Restriction Flag: categorical w/modeDisability Status: categorical w/’missing’NCCI BINatureOfLossDescription: categorical w/’missing’Accident Source Code: categorical w/’missing’Accident Type Group num: categorical w/’missing’
for var in categorical:    print(var, 'has', df[var].nunique(), 'unique categories')
for var in discrete:    print(var, 'has', df[var].nunique(), 'unique categories')
for var in categorical:    print(df[var].nunique(),(df.groupby(var)[var].count()/len(df)))
for var in discrete:    print(var, 'has', df[var].nunique(), 'unique categories')
In the previous section, we looked at cardinality in order to remove features with low variability (ie. features with categories which contained the majority of the data). We need to examine cardinality a bit deeper and identify ‘rare’ categories. In other words, which categories contain only a very small percentage of the data (=<1%). We will aggregate all the categories into a ‘rare’ category thereby, reducing the cardinality of each feature and simplifying the model. This method will be very helpful when we discuss encoding categorical and discrete features.
As an example, the feature ‘Employment Status Flag’ currently has 13 categories (including np.nan) but as you can see the “F = Full-Time” and “P=Part-Time” categories make up almost 96% of the data. All other categories barely show up 0.5% of the time and they will all be aggregated as ‘rare’ categories. Any categorical or discrete feature with categories occurring less than 1% of the time will have those categories encoded as ‘rare’.
The dataset only contains two continuous features ‘Accident DateID’ and ‘Weekly Wage’. We need to determine whether nor not these features contain skewed distributions and if they contain any outliers.
for var in continuous:    plt.figure(figsize=(10,5))    plt.subplot(1,2,1)    fig = df.boxplot(column=var)    fig.set_title('outliers')    fig.set_ylabel(var)        plt.subplot(1,2,2)    fig = df[var].hist(bins=20)    fig.set_ylabel('number of cases')    fig.set_xlabel(var)        plt.show()
Both features certainly maintain skewed distributions but only ‘Weekly Wage’ contains any outliers. ML algorithms have certain assumptions about the data which we need to follow in other to increase their predictive ability. For example, linear regression assumes the relationship between your predictors and target is linear (linearity). It also assumes there are no outliers in the data. It has a particularly difficult time with highly correlated features (multicollinearity). Finally, it assumes your features are normally distributed.
Normally distributed features follow a Gaussian distribution which you probably remember from your high school statistics course resembles a bell shape. As you can see neither ‘Accident DateID’ nor ‘Weekly Wage’ are normally distributed. There are several commonly used methods to fix skewed distributions such as log, reciprocal, square root, box-cox and yeo-johnson transformations.
df_copy = df.copy()df_copy['Accident DateID'].skew(skipna = True)df_copy['Accident DateID_log'] = df_copy['Accident DateID'].map(lambda x: np.log(x))df_copy['Accident DateID_rec'] = df_copy['Accident DateID'].map(lambda x: np.reciprocal(x))df_copy['Accident DateID_sqrt'] = df_copy['Accident DateID'].map(lambda x: np.sqrt(x))df_copy['Accident DateID_log'].skew(skipna = True)df_copy['Accident DateID_rec'].skew(skipna = True)df_copy['Accident DateID_sqrt'].skew(skipna = True)df['Accident DateID_rec'] = df['Accident DateID'].map(lambda x: np.reciprocal(x))df.drop(['Accident DateID'], axis=1, inplace=True)
The corrected skews of ‘Accident DateID’ are as follows:
Accident DateID: 0.137
Accident DateID_log: 0.111
Accident DateID_rec: 0.00
Accident DateID_sqrt: 0.124
We can see the initial skew of ‘Accident DateID’ was 0.137 which technically speaking isn’t very skewed as a normal distribution has a skew of zero (0). That said, applying the reciprocal transformation adjusted our skew to zero (0).
def diagnostic_plot(df, variable):    plt.figure(figsize=(16, 4))    sns.distplot(df[variable], bins=30)    plt.title('Histogram')diagnostic_plot(df_copy, 'Accident DateID_rec')
df_copy['Weekly Wage'].skew(skipna = True)df_copy['Weekly Wage_log'] = df_copy['Weekly Wage'].map(lambda x: np.log(x) if x > 0 else 0)df_copy['Weekly Wage_rec'] = df_copy['Weekly Wage'].map(lambda x: np.reciprocal(x) if x > 0 else 0)df_copy['Weekly Wage_sqrt'] = df_copy['Weekly Wage'].map(lambda x: np.sqrt(x))df_copy['Weekly Wage_log'].skew(skipna = True)df_copy['Weekly Wage_rec'].skew(skipna = True)df_copy['Weekly Wage_sqrt'].skew(skipna = True)df['Weekly Wage_sqrt'] = df['Weekly Wage'].map(lambda x: np.sqrt(x))df.drop(['Weekly Wage'], axis=1, inplace=True)
The corrected skews of ‘Weekly Wage’ are as follows:
Weekly Wage: 2.57
Weekly Wage_log: -3.74
Weekly Wage_rec: 54.37
Weekly Wage_sqrt: 0.407
‘Weekly Wage’ had a much larger initial skew at 2.56 but a square root transformation brought the skew down significantly (0.40).
Now that we have fixed the skewness, let’s address the outliers located inside ‘Weekly Wage_sqrt’ as we have dropped the original feature. Since ‘Weekly Wage_sqrt’ is normally distributed we can use the ‘3 Standard Deviations from the Mean’ rule to identify the outliers. If your distribution was skewed you would be better off calculating the quantiles and then the IQR to identify your upper and lower boundaries.
It is important to note continuous features such as ‘Accident DateID’ and ‘Weekly Wage_sqrt’ can often benefit from discretization or binning. Discretization entails cutting the feature values into groups or bins. This method is also a valid way to deal with outliers as they are typically brought closer to a mean of the distribution. This will ultimately change the feature from continuous to discrete as the end result will be the number of observations in each bin (ie. 0–1000, 1000–2000, 2000–5000, etc.).
def find_boundaries(df, variable):# calculate the boundaries anything outside the upper and lower boundaries is an outlier    upper_boundary = df[variable].mean() + 3 * df[variable].std()    lower_boundary = df[variable].mean() - 3 * df[variable].std()return upper_boundary, lower_boundaryupper_boundary, lower_boundary = find_boundaries(df, 'Weekly Wage_sqrt')upper_boundary, lower_boundary
The upper boundary was 51.146 and the lower boundary was -0.763. In other words, any value which falls outside either of these boundaries will be considered as an outlier. Notice that we used the three (3) standard deviation rule to determine outliers. We could have changed this value to 2 and our boundaries would have shrunk resulting more outliers.
print('We have {} upper boundary outliers:'.format(len(df[df['Weekly Wage_sqrt'] > upper_boundary])))print('We have {} lower boundary outliers:'.format(len(df[df['Weekly Wage_sqrt'] < lower_boundary])))
# Identify the outliers in 'weekly wage_sqrt'outliers_weekly_wage = np.where(df['Weekly Wage_sqrt'] > upper_boundary, True, np.where(df['Weekly Wage_sqrt'] < lower_boundary, True, False))# trim the df of the outliersdf = df.loc[~(outliers_weekly_wage)]
df['Opiods Used'].value_counts()/len(df)
Our target is ‘Opiods Used’ and as with most classification problems the false class tends to be the majority in terms of sheer numbers. As you can see above, almost 90% of all the cases are False or did not abuse opioids. Some ML algorithms such as decision trees tend to bias their predictions towards the majority class (ie. False in our case). There are a number of techniques we can use to solve this issue. Oversampling is a technique which attempts to add random copies of the minority class to the dataset until the imbalance is eliminated. Undersampling is the opposite of oversampling as it entails removing majority class observations. The major drawback is the idea of removing data which can lead to underfitting of your model. Last but not least, synthetic minority oversampling technique (SMOTE) uses the KNN algorithm to generate new observations to eliminate the imbalance. We’ll use the SMOTE technique in this use case to generate new (synthetic) observations.
It is important to note any technique used should only be used to generate or eliminate observations in the training set.
We need to split our data into train and test datasets but first let’s convert the our features to proper format in order to coincide with our pipeline’s requirements.
df[discrete] = df[discrete].astype('O')df['Industry ID'] = df['Industry ID'].astype('O')df['Surgery Flag'] = df['Surgery Flag'].astype('O')df['Accident Source Code'] = df['Accident Source Code'].astype('O')df['Orthopedic Surgery Payment Flag'] = df['Orthopedic Surgery Payment Flag'].astype('O')df['Orthopedic Surgery Payment Flag'] *= 1df['Opiods Used'] *= 1X_train, X_test, y_train, y_test = train_test_split(    df.drop('Opiods Used', axis=1),    df['Opiods Used'],    test_size=0.3,    random_state=SEED)
In order to simplify the task of processing the data for missing data, rare values, cardinality, and encoding we will utilize Scikit-Learn’s make_pipeline library. A pipeline allows us to apply multiple processes into a single piece of code which will run each processes in series, one after another. Using a pipeline makes our code much easier to understand and much more reproducible.
# Imputing missing values for continuous features with more than 5% missing data with -99999.feature_transform = make_pipeline(mdi.ArbitraryNumberImputer(arbitrary_number = -99999, variables='Accident DateID_rec'),                                  # Imputing categorical (object) features w/ more than 5% of nulls as 'missing'mdi.CategoricalVariableImputer(variables=['Disability Status','NCCI BINatureOfLossDescription','Accident Source Code','Accident Type Group num'], imputation_method='missing'),                                  # Imputing categorical features w/less than 5% of missing values with the modemdi.CategoricalVariableImputer(variables=['Claimant Sex', 'Claimant Marital Status','Employment Status Flag','RTW Restriction Flag'],imputation_method='frequent'),                                  # Encoding rare categories for categorical and discrete features (Less than 1% is rare)ce.RareLabelCategoricalEncoder(tol=0.01, n_categories=6, variables=['Benefits State', 'Industry ID', 'NCCI Job Code','Employment Status Flag', 'SIC Group', 'NCCI BINatureOfLossDescription','Accident Type Group num', 'Claimant Age', 'Number Dependents', 'Percent Impairment', 'HCPCS A Codes', 'HCPCS E Codes', 'HCPCS G Codes', 'HCPCS J Codes', 'HCPCS L Codes','HCPCS W Codes', 'ICD Group 6', 'ICD Group 13','ICD Group 18', 'ICD Group 19', 'ICD Group 21','CPT Category - Anesthesia', 'CPT Category - Eval_Mgmt','CPT Category - Medicine', 'CPT Category - Path_Lab','CPT Category - Radiology', 'CPT Category - Surgery', 'NDC Class - Muscle Relaxants']),                                  # We will use one_hot_encoding for categorical featuresce.OneHotCategoricalEncoder(variables=['Benefits State', 'Industry ID', 'Claimant Sex','Claimant Marital Status','Employment Status Flag','RTW Restriction Flag','Disability Status','SIC Group','NCCI Job Code','NCCI BINatureOfLossDescription','Accident Source Code','Accident Type Group num'],drop_last=True),                                  # We are going to use ordinal encoding according to the target mean to the target featurece.OrdinalCategoricalEncoder(encoding_method='ordered',variables=['Claimant Age','Number Dependents','Percent Impairment','HCPCS A Codes', 'HCPCS E Codes','HCPCS G Codes','HCPCS J Codes', 'HCPCS L Codes', 'HCPCS W Codes','ICD Group 6', 'ICD Group 13', 'ICD Group 18', 'ICD Group 19','ICD Group 21', 'CPT Category - Anesthesia', 'CPT Category - Eval_Mgmt','CPT Category - Medicine','CPT Category - Path_Lab','CPT Category - Radiology','CPT Category - Surgery','NDC Class - Muscle Relaxants']))
Our data processing pipeline makes extensive use of the “Feature-Engine” library. We could have used Scikit-Learn to accomplish these tasks but feature-engine has certain advantages which we would like to point out. First, being built on top of scikit-learn, pandas, Numpy and SciPy, feature-engine is able to return pandas dataframes instead of numpy arrays like scikit-learn. Secondly, feature-engine transformers are able to learn and store training parameters and transform your test data using the stored parameters.
But enough about feature-engine, let’s discuss the pipeline in more detail. It is important to understand the steps in a pipeline are run in series, starting with the top transformer. Often students make the mistake of applying the first step which ultimately changes the structure of the data or the name of the feature which is not recognized by the second stop.
The first transformer “ArbitraryNumberImputer” imputes continuous features with more than 5% of missing data with the value -99999. The second transformer, “CategoricalVariableImputer”, imputes categorical data with more than 5% of missing data with the string value of ‘Missing”. The third transformer, “FrequentCategoryImputer”, imputes categorical data with less than 5% of missing data with the mode of the feature. The fourth transformer, “RareLabelCategoricalEncoder”, encodes categorical and discrete feature observations which appear less than 1% of the time into a new category named “rare”.
The fifth transformer, “OneHotCategoricalEncoder”, transforms each unique value for each categorical feature into binary form stored in a new feature. For example, “Gender” has the values of “M”, “F” and “U”. One hot encoding will produce three (or two “k-1” depending on your settings) new features (ie. Gender_M, Gender_F, Gender_U). If the observation has a value of “M” under the original “Gender” feature then “Gender_M” will have the value of 1 and “Gender_F” and “Gender_U” will have the values of 0. As this method greatly expands the feature space, now you understand why it was important to bin rare observations (<1%) as “rare”.
The final transformer, “OridinalCategoricalEncoder”, is specifically used to encode discrete features in order to maintain their ordered relationship with the target feature. Utilizing this encoder will generate missing values or throw an error for categories present in the test set which were not encoded in the training set. This is yet another reason to handle rare values before you encode ordinal/discrete features. To better understand this ordinal encoder let’s examine the “Claimant age” feature. We have three potential values of “F”, “M” and “U”. Let’s assume the average opioid abuse for “F” is 10%, “M” is 25% and “U” is 5%. The encoder will encode “M” as 1, “F” as 2, and “U” as 3 according to the magnitude of their average opioid abuse.
Notice the expanded feature space to 155 features
feat_transform.fit(X_train, y_train)X_train_clean = feat_transform.transform(X_train)X_test_clean = feat_transform.transform(X_test)X_train_clean.head()
Next, fit the pipeline onto X_train and y_train and transform X_train and X_test. Notice that our feature space has increased greatly to 155 features, this is due to the one-hot encoder we used on categorical features.
Finally, we have to scale the features in order to have all their values on the same range or magnitude. This step has to be done as some ML classifiers use Euclidean distance and features with higher magnitudes or range would have more influence on the prediction. For example temperature, 32 degree Fahrenheit is the same as 273.15 degrees Kelvin and if we were to use both features in a model Kelvin would have more weight or influence the prediction.
scaler = StandardScaler()scaler.fit(X_train_clean)X_train_std = scaler.transform(X_train_clean)X_test_std = scaler.transform(X_test_clean)
X_train_std_df = pd.DataFrame(X_train_std, columns=col_names)X_test_std_df = pd.DataFrame(X_test_std, columns=col_names)
models = []models.append(('log_reg', LogisticRegression(max_iter=10000, random_state=42)))models.append(('rf_classifer', RandomForestClassifier(random_state=42)))models.append(('bayes', GaussianNB()))models.append(('gbc', GradientBoostingClassifier()))base_model_train = []base_model_test = []for name, classifier in models:    scores = cross_val_score(classifier, X_train_std_df, y_train, cv=5, scoring='recall')    base_model_train.append(scores.mean().round(4))    print(scores)    print('{}: Avg CV recall using all features on training data: {}'.format(name, scores.mean().round(4)))        classifier.fit(X_train_std_df, y_train)    y_preds = classifier.predict(X_test_std_df)    test_recall = recall_score(y_test, y_preds, average='binary')    test_class = classification_report(y_test, y_preds)    cnf_matrix = confusion_matrix(y_test, y_preds)    base_model_test.append(test_recall.round(4))    print('{}: Recall w/all features on test data {}:'.format(name, test_recall.round(4)))    print(test_class)    print(cnf_matrix)    print('-------------------------------------------------------')
We will compare four different classifiers on their relative recall score. We are using recall as we want to minimize the number of false negatives (ie. abused opioids but predicted not to abuse). First, we want to establish a baseline to which we can compare against additional iterations of the classifiers to determine relative improvement. The baseline model includes the entire feature space with an imbalanced target. To properly evaluate our classifiers we’ll use 5-fold stratified cross-validation applied only to the training dataset thereby drastically reducing data leakage. In other words, each classifier will be trained and tested 5 times on five unique splits of training data. Five unique recall scores will be calculated for each classifier and averaged together to produce the final recall score. The classifiers will not “see” any of the test data during training. Finally, each classifier will be tested on the held-out test dataset to determine generalizability and overfitting.
LoLogistic regression, random forest, and gradient boosting classifier have achieved great overall accuracy. However, naive bayes managed to achieve the highest recall as it only had 331 false-negative predictions.
If you wish to read more about classification metrics (LINK)
sm = SMOTE(sampling_strategy='auto', k_neighbors=5, random_state=42)X_train_std_sm, y_train_sm = sm.fit_resample(X_train_std_df, y_train)model1_train = []model1_test = []for name, classifier in models:    pipeline = make_pipeline(sm, classifier)    scores = cross_val_score(pipeline, X_train_std_df, y_train, cv=5, scoring='recall')    model1_train.append(scores.mean().round(4))    print(scores)    print('{}: Avg CV Recall w/All Reatures: {}'.format(name, scores.mean().round(4)))        classifier.fit(X_train_std_sm, y_train_sm)    y_preds = classifier.predict(X_test_std_df)    test_recall = recall_score(y_test, y_preds)    test_class = classification_report(y_test, y_preds)    cnf_matrix = confusion_matrix(y_test, y_preds)    model1_test.append(test_recall.round(4))    print('{}: Recall w/All Features on test data {}:'.format(name, test_recall.round(4)))    print(test_class)    print(cnf_matrix)    print('-------------------------------------------------------')
The imbalanced target is imposing bias onto our prediction. As expected logistic regression improved greatly as the algorithm performs significantly better with balanced targets.
A few words on SMOTE and its methodology. If you recall from our initial examination of the dataset the target variable was imbalanced. We had significantly more observations which did not result in opioid abuse (89%) compared to those which resulted in opioid abuse (10%). We also decided to use the SMOTE method as it creates new synthetic observations of the minority class instead of copying existing observations. SMOTE uses KNN (typically k=5) where a random observation from the minority class is selected and k of the nearest neighbors are found. Then, one of the k neighbors is randomly selected and a synthetic sample is built from a randomly selected point between the original observation and the randomly selected neighbor.
Modern datasets in areas such as natural language processing and IoT are typically highly dimensional. It is not uncommon to see thousands even millions of features. Knowing how to narrow down the features to a selected few not only improves our chances of finding a generalizable model but also decreases our reliance on expensive computational power. By accurately reducing the number of features/dimensions in our data we are ultimately removing unnecessary noise from our data.
It is important to note feature selection consists of not only the reduction in the feature space but also feature creation. Understanding how to find trends in your dataset and relationships among features (ie. polynomial features) takes many years of practice but pays big dividends in predictive power. There are whole college courses dedicated to feature selection/engineering but those interested more in the topic please research Filter, Wrapper, and Embedded methods as an introduction.
RandomForestClassifier from Scikit-Learn has a “feature_importances_” attribute which is used to determine the relative importance of each feature in your dataset. As random forest tends to perform better under a balanced target we are going to use the SMOTE balanced X_train_std_sm and y_train_sm datasets.
rf_selector = RandomForestClassifier(n_estimators=100, random_state=SEED, n_jobs=-1)rf_selector.fit(X_train_std_sm, y_train_sm)feature_imp = pd.Series(rf_selector.feature_importances_, index=X_train_std_df.columns).sort_values(ascending=False)feature_imp[:30]
sum(feature_imp[0:30]
We were able to eliminate most of the original features down to just 30 which accounts for 91% of the performance variance. Any additional features would only add a very small additional predictive power.
X_train_rf = X_train_std_df[feature_imp[:30].index]X_test_rf = X_test_std_df[feature_imp[:30].index]plt.figure(figsize=(12,8))sns.barplot(x=feature_imp[0:30], y=feature_imp.index[0:30])plt.xlabel('Feature Importance Score')plt.ylabel('Features')plt.title("Visualizing Important Features")plt.grid(b=False, which='major', color='#666666', linestyle='-', alpha=0.2)plt.show()
models = []models.append(('log_reg', LogisticRegression(max_iter=10000, random_state=42)))models.append(('rf_classifer', RandomForestClassifier(random_state=42)))models.append(('bayes', GaussianNB()))models.append(('gbc', GradientBoostingClassifier()))model2_train = []model2_test = []for name, classifier in models:    scores = cross_val_score(classifier, X_train_rf, y_train, cv=5, scoring='recall')    model2_train.append(scores.mean().round(3))    print(scores)    print('{}: Avg CV Recall on RF Features: {}'.format(name, scores.mean().round(3)))        classifier.fit(X_train_rf, y_train)    y_preds = classifier.predict(X_test_rf)    test_recall = recall_score(y_test, y_preds, average='binary')    test_class = classification_report(y_test, y_preds)    cnf_matrix = confusion_matrix(y_test, y_preds)    model2_test.append(test_recall.round(3))    print('{}: Recall w/RF features on test data {}:'.format(name, test_recall.round(3)))    print(test_class)    print(cnf_matrix)    print('-------------------------------------------------------')
X_train_rf_sm, y_train_sm = sm.fit_resample(X_train_rf, y_train)models = []models.append(('log_reg', LogisticRegression(max_iter=10000, random_state=42)))models.append(('rf_classifer', RandomForestClassifier(random_state=42)))models.append(('bayes', GaussianNB()))models.append(('gbc', GradientBoostingClassifier()))sm = SMOTE(sampling_strategy='auto', k_neighbors=5, random_state=42)skf = StratifiedKFold(n_splits=5)model3_train = []model3_test = []for name, classifier in models:    pipeline = make_pipeline(sm, classifier)    scores = cross_val_score(pipeline, X_train_rf, y_train, cv=skf, scoring='recall')    model3_train.append(scores.mean().round(3))    print(scores)    print('{}: Avg CV Recall w/RF Reatures: {}'.format(name, scores.mean().round(3)))        classifier.fit(X_train_rf_sm, y_train_sm)    y_preds = classifier.predict(X_test_rf)    test_recall = recall_score(y_test, y_preds, average='binary')    test_class = classification_report(y_test, y_preds)    cnf_matrix = confusion_matrix(y_test, y_preds)    model3_test.append(test_recall.round(3))    print('{}: Recall w/RF on test data {}:'.format(name, test_recall.round(3)))    print(test_class)    print(cnf_matrix)    print('-------------------------------------------------------')
classifiers = ['Log_Regression', 'Random_Forest', 'Naive_Bayes', 'Gradient_Boosting_clf']idx = ['All_Feat_Imbalance_Train', 'All_Feat_Imbalance_Test','All_Feat_Smote_Train','All_Feat_Smote_Test','RF_Imbalance_Train', 'RF_Imbalance_Test','RF_Smote_Train','RF_Smote_Test']combined_results = pd.DataFrame([base_model_train,base_model_test,model1_train, model1_test, model2_train,model2_test, model3_train, model3_test],columns=classifiers, index=idx)test_results = pd.DataFrame([base_model_test, model1_test, model2_test, model3_test], columns=classifiers, index=idx[1:8:2])print(test_results)
plt.figure(figsize=(15,10))sns.lineplot(data=test_results[['Log_Regression', 'Random_Forest',                             'Naive_Bayes', 'Gradient_Boosting_clf']])plt.xlabel('Iterations', fontsize=17, labelpad=15)             plt.ylabel('Recall Scores', fontsize=17, labelpad=15)plt.title('Classifier Recall Scores on Test Data',fontsize=25, pad=15)plt.show()
The reduction in features resulted in a slight decrease in recall performance. This is expected as using only 30 features accounted for 91% of the performance variance. Increasing the number of features would have certainly increased the recall. Only naive bayes was not affected by the reduction in features. Next, the imbalance in our dataset also affected the recall performance of the classifiers. In general, most of the classifiers improved their performance once the imbalance was correct. Logistic regression was affected by the imbalance the most.
In terms of overall classifier performance, it must be said all classifiers performed best with a reduction in features along with a balanced target. Of course one could argue naive bayes had performed the best as it managed to achieve the best test recall (0.949) but I would argue it is logistic regression outperformed the field. It achieved very similar recall 0.945 compared to 0.949 of naive bayes which only accounted for an increase in the false negatives by 14. What is more impressive is the fact it had 2,276 additional correct true negative predictions compared to naive bayes. In the next section, we’ll attempt hyperparameter tuning to see if we can increase the classification recall of the logistic regression model.
Think of hyperparameters are ‘tuning’ knobs. Imagine you are editing a picture to achieve a certain effect. You can tune the picture using ‘knobs’ such as exposure, highlights, shadows, contrast, brightness, saturation, warmth, tint, etc. Well, hyperparameters are the same type of ‘knobs’ but for classifiers.
Logistic regression does not have many hyperparameters to tune besides the solver, penalty and C.
The solvers try to optimize the parameter weights or thetas which will elicit the least error or cost function. Sklearn comes with several solvers: newton-cg, lbfgs, liblinear.
The penalty parameter is also called “regularization” and its purpose is to help a model avoid overfitting the training data thereby producing a more generalizable model. It does this by “penalizing” features which are considered noise or contribute very little to the model.
L1 Penalty or Lasso: Adds the absolute magnitude of the penalty or the sum of weights of the features to the cost function. L1 reduces noisy features to zero thereby simplifying the model.
L2 Penalty or Ridge: Adds the square magnitude of the penalty or the sum of the square of the feature weights to the cost function. L2 also reduces the influence of the noisy features but their weights or thetas come very close to zero but not quite.
Finally, the “C” parameter determines the strength of the regularization penalty. The larger the C parameter the less regularization and the more complex the model becomes and over-fitting increases. The smaller the C parameter the more regularization is applied and under-fitting increases.
clf_lr = LogisticRegression(max_iter=10000, random_state=SEED)penalty = ['l1','l2']C = [0.001,0.002,0.003,0.005,1,10,100,1000]skf = StratifiedKFold(n_splits=5)pipeline = make_pipeline(sm, clf_lr)param_grid = dict(logisticregression__penalty=penalty,                  logisticregression__C=C)grid = GridSearchCV(pipeline,                     param_grid=param_grid,                    scoring='recall',                     verbose=1, cv=skf)grid_results = grid.fit(X_train_rf, y_train) print('Best Score: ', grid_results.best_score_)print('Best Params: ', grid_results.best_params_)
X_train_rf_sm, y_train_sm = sm.fit_resample(X_train_rf, y_train)clf_lr = LogisticRegression(max_iter=10000, penalty='l2', C=0.0001, random_state=42)clf_lr.fit(X_train_rf_sm, y_train_sm)y_preds = clf_lr.predict(X_test_rf)test_recall = recall_score(y_test, y_preds).round(3)test_class = classification_report(y_test, y_preds)cm = confusion_matrix(y_test, y_preds)print('Log Regression Recall w/RF on test data {}:'.format(test_recall.round(3)))print(test_class)print(cm)
To reduce training times and the number of warnings we’ll focus on tuning the “penalty” and “C” parameters. Understanding not only what each hyperparameter does but also how the parameters interact with each other is critical to their tuning. Furthermore, the tuning process is often iterative as we begin with wide ranges for each parameter and then begin to narrow our range until a specific parameter value is chosen.
GridSearchCV applies an exhaustive approach as it considers all combinations of supplied parameters. As balancing the target produced the best recall during classifier evaluation, we opted to include it into our pipeline along with the log regression classifier which was passed into gridsearchcv. This way each cross-validation training/test split was balanced only with its data. In other words, neither SMOTE nor the classifier suffered from data leakage during cross-validation.
Unfortunately, we did not increase our training recall but we were able to increase our test recall from 0.945 (log regression w/RF features and SMOTE) to 0.954. Furthermore, we decreased our false-negative count by 42. Our false positive count increased by 269 but once again it is better to been predicted to use opioids when in reality you’re not.
Understanding the full nature of a data science classification problem is key in your maturity as a data scientist. I hope you found this tutorial informative and easily understood. I welcome any feedback and suggestions as we are all just honing our craft.
All the code can be found on my GitHub. LINK