Feature Engineering for Election Result Prediction (in Python) | by Thamindu Dilshan Jayawickrama | Towards Data Science

I recently competed in a Kaggle competition where we have to predict the election results using machine learning. The dataset was from the 2019 India general election (see here). This article explains how to clean and prepare the dataset, create new features out of the existing ones and then predict the results using a popular machine learning algorithm. Most of the basic preprocessing, data visualization and machine learning steps haven’t clearly explained here; Rather I focused on feature engineering and how it affects the performance of the models. If you don’t have a clear understanding of what is feature engineering, please refer to my previous article on “Basic Feature Engineering to Reach More Efficient Machine Learning”.
First, we should load the dataset into a notebook. Most ML engineers and experts prefer to use notebooks for machine learning and data engineering tasks in the initial stage because it is really easy to do data visualization and all other steps with a notebook. I prefer to use pandas library to load the dataset because it makes all the data processing steps really easy without much effort. Below are the python libraries I used in the notebook.
import numpy as npimport pandas as pdimport osimport matplotlib.pyplot as pltimport seaborn as snsfrom sklearn.preprocessing import LabelEncoderfrom sklearn.preprocessing import MinMaxScalerfrom sklearn.model_selection import train_test_splitfrom sklearn.neighbors import KNeighborsClassifierfrom sklearn.feature_selection import SelectKBest, chi2
First, let’s run below commands to have a basic idea about the dataset and how the raw data look like.
dataset = pd.read_csv('/kaggle/input/indian-candidates-for-general-election-2019/LS_2.0.csv')dataset.head()
dataset.info()<class 'pandas.core.frame.DataFrame'>RangeIndex: 2263 entries, 0 to 2262Data columns (total 19 columns):STATE                                       2263 non-null objectCONSTITUENCY                                2263 non-null objectNAME                                        2263 non-null objectWINNER                                      2263 non-null int64PARTY                                       2263 non-null objectSYMBOL                                      2018 non-null objectGENDER                                      2018 non-null objectCRIMINALCASES                              2018 non-null objectAGE                                         2018 non-null float64CATEGORY                                    2018 non-null objectEDUCATION                                   2018 non-null objectASSETS                                      2018 non-null objectLIABILITIES                                 2018 non-null objectGENERALVOTES                               2263 non-null int64POSTALVOTES                                2263 non-null int64TOTALVOTES                                 2263 non-null int64OVER TOTAL ELECTORS IN CONSTITUENCY        2263 non-null float64OVER TOTAL VOTES POLLED IN CONSTITUENCY    2263 non-null float64TOTAL ELECTORS                              2263 non-null int64dtypes: float64(3), int64(5), object(11)memory usage: 336.0+ KB
We can see from the above analysis, the dataset contains few numerical columns and most of the columns are non-numeric. WINNER column contains the label indicating that either the candidate have won or lost the election. Also, note that the dataset contains some ‘NaN’ values which are basically missing values. Some column names contain ‘\n’ characters, which makes it annoying. Also, ASSETS and LIABILITIES column values contain ‘\n’ characters as well.
By considering only the available data, we can consider state, constituency, party, gender, category and education as categorical features. Additionally, you can run dataset.describe() command to have some statistical summary on numerical columns.
Next, we should clean the dataset, fix column names and treat the missing values. First I fixed the incorrect column names and replace all the spaces in column names with underscores (‘_’). This is not a mandatory step, but incorrect column names make it annoying to me.
# rename invalid column namesdataset = dataset.rename(columns={'CRIMINAL\nCASES': 'CRIMINAL_CASES', 'GENERAL\nVOTES': 'GENERAL_VOTES', 'POSTAL\nVOTES': 'POSTAL_VOTES', 'TOTAL\nVOTES': 'TOTAL_VOTES', 'OVER TOTAL ELECTORS \nIN CONSTITUENCY': 'OVER_TOTAL_ELECTORS_IN_CONSTITUENCY', 'OVER TOTAL VOTES POLLED \nIN CONSTITUENCY': 'OVER_TOTAL_VOTES_POLLED_IN_CONSTITUENCY', 'TOTAL ELECTORS': 'TOTAL_ELECTORS'})
Then let’s search for the missing values in each column.
dataset.isna().sum()STATE                                        0CONSTITUENCY                                 0NAME                                         0WINNER                                       0PARTY                                        0SYMBOL                                     245GENDER                                     245CRIMINAL_CASES                             245AGE                                        245CATEGORY                                   245EDUCATION                                  245ASSETS                                     245LIABILITIES                                245GENERAL_VOTES                                0POSTAL_VOTES                                 0TOTAL_VOTES                                  0OVER_TOTAL_ELECTORS_IN_CONSTITUENCY          0OVER_TOTAL_VOTES_POLLED_IN_CONSTITUENCY      0TOTAL_ELECTORS                               0dtype: int64
You can see that 10% of the row values are missing. There are multiple ways to treat missing values like deleting them, using back-fill or forward-fill, constant value imputation, mean/ median or mode imputation, etc. However, I just delete these rows here for the simplicity (only 10% is missing), but always remember that removing values will make the prediction model less accurate. You should try to impute missing values as much as possible.
# drop rows with NA valuesdataset = dataset[dataset['GENDER'].notna()]
Let’s convert ASSETS, LIABILITIES and CRIMINAL_CASES columns numeric because they represent money and count, and numeric will make sense to the models. In order to do that, we have to remove the ‘Rs’ sign, ‘\n’ character and commas in each value field. But also those columns contain ‘Nil’ and ‘Not Available’ as values. So before making them numeric, we have to replace those values with some meaningful value (for the moment I replaced them with 0).
# replace Nil values with 0dataset['ASSETS'] = dataset['ASSETS'].replace(['Nil', '`', 'Not Available'], '0')dataset['LIABILITIES'] = dataset['LIABILITIES'].replace(['NIL', '`', 'Not Available'], '0')dataset['CRIMINAL_CASES'] = dataset['CRIMINAL_CASES'].replace(['Not Available'], '0')# clean ASSETS and LIABILITIES column valuesdataset['ASSETS'] = dataset['ASSETS'].map(lambda x: x.lstrip('Rs ').split('\n')[0].replace(',', ''))dataset['LIABILITIES'] = dataset['LIABILITIES'].map(lambda x: x.lstrip('Rs ').split('\n')[0].replace(',', ''))# convert ASSETS, LIABILITIES and CRIMINAL_CASES column values into numericdataset['ASSETS'] = dataset['ASSETS'].astype(str).astype(float)dataset['LIABILITIES'] = dataset['LIABILITIES'].astype(str).astype(float)dataset['CRIMINAL_CASES'] = dataset['CRIMINAL_CASES'].astype(str).astype(int)
Now let’s make non-numeric columns numeric for better performance. Note that some model types cannot perform with non-numerical data. Here I’m focusing on a classification algorithm, which certainly cannot train on non-numerical data. So I label encoded those non-numerical columns using sklearn LabelEncoder.
# label encode categorical columnslblEncoder_state = LabelEncoder()lblEncoder_state.fit(dataset['STATE'])dataset['STATE'] = lblEncoder_state.transform(dataset['STATE'])lblEncoder_cons = LabelEncoder()lblEncoder_cons.fit(dataset['CONSTITUENCY'])dataset['CONSTITUENCY'] = lblEncoder_cons.transform(dataset['CONSTITUENCY'])lblEncoder_name = LabelEncoder()lblEncoder_name.fit(dataset['NAME'])dataset['NAME'] = lblEncoder_name.transform(dataset['NAME'])lblEncoder_party = LabelEncoder()lblEncoder_party.fit(dataset['PARTY'])dataset['PARTY'] = lblEncoder_party.transform(dataset['PARTY'])lblEncoder_symbol = LabelEncoder()lblEncoder_symbol.fit(dataset['SYMBOL'])dataset['SYMBOL'] = lblEncoder_symbol.transform(dataset['SYMBOL'])lblEncoder_gender = LabelEncoder()lblEncoder_gender.fit(dataset['GENDER'])dataset['GENDER'] = lblEncoder_gender.transform(dataset['GENDER'])lblEncoder_category = LabelEncoder()lblEncoder_category.fit(dataset['CATEGORY'])dataset['CATEGORY'] = lblEncoder_category.transform(dataset['CATEGORY'])lblEncoder_edu = LabelEncoder()lblEncoder_edu.fit(dataset['EDUCATION'])dataset['EDUCATION'] = lblEncoder_edu.transform(dataset['EDUCATION'])
Now let’s train a K-Nearest Neighbors model and see the accuracy. KNN is a supervised machine learning model which is categorized under classification algorithms. The algorithm works by taking a data point and finding out the k closest data points.
# separate train features and labely = dataset["WINNER"]X = dataset.drop(labels=["WINNER"], axis=1)# split dataset into train and test dataX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=1, stratify=y)# train and test knn modelknn = KNeighborsClassifier()knn.fit(X_train, y_train)knn.predict(X_test)print("Testing Accuracy is: ", knn.score(X_test, y_test)*100, "%")Testing Accuracy is:  70.79207920792079 %
The model has achieved 70% accuracy without much effect. Let’s normalize the dataset and see how the accuracy improves. I have used MinMaxScaler from the scikit-learn library to scale down all the values into the 0–1 range.
# scaling values into 0-1 rangescaler = MinMaxScaler(feature_range=(0, 1))features = [    'STATE', 'CONSTITUENCY', 'NAME', 'PARTY', 'SYMBOL', 'GENDER', 'CRIMINAL_CASES', 'AGE', 'CATEGORY', 'EDUCATION', 'ASSETS', 'LIABILITIES', 'GENERAL_VOTES', 'POSTAL_VOTES', 'TOTAL_VOTES', 'OVER_TOTAL_ELECTORS_IN_CONSTITUENCY', 'OVER_TOTAL_VOTES_POLLED_IN_CONSTITUENCY', 'TOTAL_ELECTORS']dataset[features] = scaler.fit_transform(dataset[features])
As you can see below, accuracy has improved well by just normalizing the column values. However, we could further improve accuracy by applying a few other feature engineering techniques.
Testing Accuracy is:  90.5940594059406 %
If we consider the EDUCATION column, it contains 11 categorical values which a particular candidate can have.
Illiterate, Literate, 5th Pass, 8th Pass, 10th Pass, 12th Pass, Graduate, Post Graduate, Graduate Professional, Doctorate, Other
If we think thoroughly, each of these value represents a particular level of education. By label encoding, we just assign some random integer for each of these values without thinking about their hierarchical level. However, if we assign that integer meaningfully according to the educational qualification, hopefully, the model will perform better. Note that there is a field value named as “Other”, which we don’t know the hierarchical position. I just assign a median value from the range for it.
dataset['EDUCATION'].value_counts()Post Graduate            502Graduate                 441Graduate Professional    33612th Pass                25610th Pass                1968th Pass                  78Doctorate                 73Others                    50Literate                  305th Pass                  28Not Available             22Illiterate                 5Post Graduate\n            1Name: EDUCATION, dtype: int64# encode education columnencoded_edu = []# iterate through each row in the datasetfor row in dataset.itertuples():    education = row.EDUCATIONif education == "Illiterate":        encoded_edu.append(0)    elif education == "Literate":        encoded_edu.append(1)    elif education == "5th Pass":        encoded_edu.append(2)    elif education == "8th Pass":        encoded_edu.append(3)    elif education == "10th Pass":        encoded_edu.append(4)    elif education == "12th Pass":        encoded_edu.append(7)    elif education == "Graduate":        encoded_edu.append(8)    elif education == "Post Graduate":        encoded_edu.append(9)    elif education == "Graduate Professional":        encoded_edu.append(10)    elif education == "Doctorate":        encoded_edu.append(11)    else:        encoded_edu.append(5)dataset['EDUCATION'] = encoded_edu
Let’s have some analysis of the PARTY column. If we list the number of candidates in front of each party, we can see that only a few parties have a significant number of candidates. The whole purpose of including the PARTY column is to have the impact of party of the candidate for the winning of the election. If the party doesn’t have a significant number of candidates, impact of that party for the winning of the candidate is low. So we can encode all of them into one common category (I encoded them as “other”).
dataset['PARTY'].value_counts()BJP       420INC       413IND       201BSP       163CPI(M)    100         ... AINRC       1SKM         1ANC         1YKP         1AJSUP       1Name: PARTY, Length: 132, dtype: int64# change party of the less frequent parties as Other# 'BJP','INC','IND','BSP', 'CPI(M)', 'AITC', 'MNM': high frequent# 'TDP', 'VSRCP', 'SP', 'DMK', 'BJD': medium frequentdataset.loc[~dataset["PARTY"].isin(['BJP','INC','IND','BSP', 'CPI(M)', 'AITC', 'MNM', 'TDP', 'VSRCP', 'SP', 'DMK', 'BJD']), "PARTY"] = "Other"dataset['PARTY'].value_counts()
Both of these steps should perform before label encoding columns. Let’s train our model again and see the accuracy.
Testing Accuracy is:  92.07920792079209 %
You can see that the accuracy has improved little more. Until now, we have only considered the existing features. However, we can make new features out of the existing ones.
In order to make new features, we should have a general idea on available raw features and how they affect the given scenario. Moreover, a good understanding of the problem domain is required. So here, we should think about what are the factors affecting the winning of a candidate in an election and what are the considerations for us to vote for a candidate or a party. For the given scenario, I’m not aware of the country factors they consider before voting for a candidate (what country-specific factors affect a candidate to win). For example, I have no idea how the CATEGORY affects election results. But I have come up with the following features, by assuming the general use case.
The dataset contains a lot of raw features, which will directly affect the winning chance of a candidate (For example, state, constituency, party, criminal records, educational qualifications, assets, etc). However, a particular candidate could also be able to win an election depending on the status of the nominated party. Below are some features that could represent the importance of the party for the winning probability of a candidate.
1. Total number of seats won by the party
We might consider the party of the candidate before voting him/ her. If a party has a high chance of winning, a candidate from that party might have the same chance to win (not true always). We can highlight that scenario by making a feature representing the number of seats won by the party (winning probability will be highlighted by the count).
2. Number of seats won in the constituency by the party
There can be some parties which will always win a particular state or a constituency. I have considered constituency here because it is the smallest area available.
3. Criminal cases against the party
We will often consider the number of criminal records against a party before voting. If a party has a higher number of criminal cases (higher number of corrupt politicians), that party might have a lesser chance to win the election (or vice versa ;-) ). We can highlight this scenario by making a feature representing the number of criminal records against a party.
4. Number of criminal cases for the party in each constituency
We can also divide the above scenario into subcases by considering criminal records for each constituency. That might represent the above feature well.
5. Education level of the candidates in the party
We will also consider the education qualifications of the candidates of the party, before voting them. If a party nominates more number of educated candidates, that party might win the election (higher probability). We have encoded educational qualifications meaningfully in an earlier step. So if we take the count of each candidate in a party, that feature will represent the educational level of the party.
6. Education level of the candidates in the party for each constituency
As we did with the criminal cases, we can also consider the educational level of a party for each constituency.
7. Number of candidates for the constituency by the party
As I have already mentioned, a particular party might have a higher chance of winning depending on the constituency. If a party nominates a higher number of candidates for a constituency, votes will be divided among candidates and the percentage of each candidate will be less. That will directly affect the winning probability.
Below are some more features I created that could affect the winning probability of a candidate.
1. Total number of voters per each state
2. Total number of voters in each constituency
3. Total number of constituencies per each state
So I have created a total of 10 new features based on the existing ones. Some of these features may not affect the performance of the model.
# Preparing feature valuescons_per_state = {}voters_per_state = {}party_winningSeats = {}party_criminal = {}party_education = {}party_totalCandidates_per_cons = {}party_winningSeats_per_cons = {}party_criminal_per_cons = {}party_education_per_cons = {}voters_per_cons = {}# group by statesubset = dataset[['STATE', 'CONSTITUENCY', 'TOTAL_ELECTORS']]gk = subset.groupby('STATE')# for each statefor name,group in gk:    # total constituencies per state    cons_per_state[name] = len(group)        # total voters per state    voters_per_state[name] = group['TOTAL_ELECTORS'].sum()# group by partysubset = dataset[['PARTY', 'CONSTITUENCY', 'CRIMINAL_CASES', 'EDUCATION', 'WINNER']]gk = subset.groupby('PARTY')# for each partyfor name,group in gk:    # winning seats by party    party_winningSeats[name] = group[group['WINNER'] == 1.0].shape[0]        # criminal cases by party    party_criminal[name] = group['CRIMINAL_CASES'].sum()        # education qualification by party (sum of candidates)    party_education[name] = group['EDUCATION'].sum()        # group by constituency    gk2 = group.groupby('CONSTITUENCY')        # for each constituency    for name2, group2 in gk2:        key = name2 + '_' + name    # cons_party                # total candidates by party in constituency        party_totalCandidates_per_cons[key] = len(group2)                # party winning seats in the constituency        party_winningSeats_per_cons[key] = group2[group2['WINNER'] == 1.0].shape[0]                # criminal cases by party in the constituency        party_criminal_per_cons[key] = group2['CRIMINAL_CASES'].sum()# education qualification by party in constituency (sum of candidates)        party_education_per_cons[key] = group2['EDUCATION'].sum()# Total voters per constituencysubset = dataset[['CONSTITUENCY', 'TOTAL_ELECTORS']]gk = subset.groupby('CONSTITUENCY')# for each constituencyfor name,group in gk:    voters_per_cons[name] = len(group)# Applying feature values# new feature columnstotal_cons_per_state = []total_voters_per_state = []total_voters_per_cons = []winning_seats_by_party = []criminal_by_party = []education_by_party = []total_candidates_by_party_per_cons = []winning_seats_by_party_per_cons = []criminal_by_party_per_cons = []education_by_party_per_cons = []# iterate through each row in the datasetfor row in dataset.itertuples():    subkey = row.CONSTITUENCY + '_' + row.PARTYtotal_cons_per_state.append(cons_per_state.get(row.STATE))    total_voters_per_state.append(voters_per_state.get(row.STATE))    total_voters_per_cons.append(voters_per_cons.get(row.CONSTITUENCY))    winning_seats_by_party.append(party_winningSeats.get(row.PARTY))    criminal_by_party.append(party_criminal.get(row.PARTY))    education_by_party.append(party_education.get(row.PARTY))    total_candidates_by_party_per_cons.append(party_totalCandidates_per_cons.get(subkey))    winning_seats_by_party_per_cons.append(party_winningSeats_per_cons.get(subkey))    criminal_by_party_per_cons.append(party_criminal_per_cons.get(subkey))    education_by_party_per_cons.append(party_education_per_cons.get(subkey))# append columns to datasetdataset['total_cons_per_state'] = total_cons_per_statedataset['total_voters_per_state'] = total_voters_per_statedataset['total_voters_per_cons'] = total_voters_per_consdataset['winning_seats_by_party'] = winning_seats_by_partydataset['criminal_by_party'] = criminal_by_partydataset['education_by_party'] = education_by_partydataset['total_candidates_by_party_per_cons'] = total_candidates_by_party_per_consdataset['winning_seats_by_party_per_cons'] = winning_seats_by_party_per_consdataset['criminal_by_party_per_cons'] = criminal_by_party_per_consdataset['education_by_party_per_cons'] = education_by_party_per_cons
Let’s train our model again and see how the accuracy improves. No need to label encode any of the created features as all of them are numeric. However, I normalize them into 0–1 range before training the model.
Testing Accuracy is:  96.78217821782178 %
Removing irrelevant features or less important features will improve the accuracy of a model. So far we have trained our latest model on 28 total features including the newly created 10. However, some of these features may not contribute a lot to derive conclusions based on the training data. Removing those features will not decrease the accuracy of the model, hopefully, will increase it, because irrelevant features may draw invalid inferences for the model.
First, we can remove some features by just analyzing their contribution. The NAME column won’t make any useful inferences for the model, because ideally, the name should be unique. However, the surname can be important in some cases, because some family names might have an impact on winning an election. Also the PARTY and SYMBOL both variables will represent the same feature, and we will be able to remove one of them without any impact on the accuracy. TOTAL_VOTES column contains the sum of GENERAL_VOTES column and POSTAL_VOTES column. So we can remove those two as well. If we plot the heat map representing the correlation matrix, we will see that those 3 features will be highly correlated.
# remove unnecessary columnsX.drop(labels=["NAME"], axis=1, inplace=True)X.drop(labels=["SYMBOL"], axis=1, inplace=True)X.drop(labels=["POSTAL_VOTES"], axis=1, inplace=True)X.drop(labels=["GENERAL_VOTES"], axis=1, inplace=True)
There are a few techniques available to select features based on their importance. I will use the univariate selection method to identify the most important features and remove others. I’ll be using SelectKBest from the Scikit learn library with chi-squared test to evaluate the feature importance.
# apply SelectKBest class to extract top most featuresbestfeatures = SelectKBest(score_func=chi2, k=10)fit = bestfeatures.fit(X, y)dfscores = pd.DataFrame(fit.scores_)dfcolumns = pd.DataFrame(X.columns)# concat two dataframes for better visualization featureScores = pd.concat([dfcolumns, dfscores], axis=1)featureScores.columns = ['Specs', 'Score']print(featureScores.nlargest(30, 'Score'))                                      Specs       Score25          winning_seats_by_party_per_cons  486.20778816  OVER_TOTAL_VOTES_POLLED_IN_CONSTITUENCY  285.34714115      OVER_TOTAL_ELECTORS_IN_CONSTITUENCY  262.17737314                              TOTAL_VOTES  216.93778812                            GENERAL_VOTES  216.13879921                   winning_seats_by_party  199.66252513                             POSTAL_VOTES   65.12686422                        criminal_by_party   36.5194373                                     PARTY   35.41643323                       education_by_party    6.57654811                              LIABILITIES    6.33033924       total_candidates_by_party_per_cons    5.75553820                    total_voters_per_cons    5.3026564                                    SYMBOL    4.1282838                                  CATEGORY    4.04703110                                   ASSETS    3.7555757                                       AGE    2.0777689                                 EDUCATION    0.88833027              education_by_party_per_cons    0.84018518                     total_cons_per_state    0.4816736                            CRIMINAL_CASES    0.43666719                   total_voters_per_state    0.29294826               criminal_by_party_per_cons    0.1787205                                    GENDER    0.1458701                              CONSTITUENCY    0.1432500                                     STATE    0.07683317                           TOTAL_ELECTORS    0.0544862                                      NAME    0.003039
Let’s drop all the columns with a score of less than 3. Note that some of the features I created earlier are also not important for the KNN model.
X.drop(labels=["TOTAL_ELECTORS"], axis=1, inplace=True)X.drop(labels=["STATE"], axis=1, inplace=True)X.drop(labels=["CONSTITUENCY"], axis=1, inplace=True)X.drop(labels=["GENDER"], axis=1, inplace=True)X.drop(labels=["criminal_by_party_per_cons"], axis=1, inplace=True)X.drop(labels=["total_voters_per_state"], axis=1, inplace=True)X.drop(labels=["CRIMINAL_CASES"], axis=1, inplace=True)X.drop(labels=["total_cons_per_state"], axis=1, inplace=True)X.drop(labels=["EDUCATION"], axis=1, inplace=True)X.drop(labels=["education_by_party_per_cons"], axis=1, inplace=True)X.drop(labels=["AGE"], axis=1, inplace=True)
Now let’s train our model again with the most important features and evaluate the accuracy.
Testing Accuracy is:  99.5049504950495 %
As you can see from the above results, feature engineering steps have increased the performance of the model drastically. We have reached an accuracy of 0.99 starting with an initial accuracy of 0.7. First, we clean the dataset and convert all the values into a numerical format. Then we create some new features using the existing ones and remove all less important features. Finally, we scale down the values into 0–1 range and train a K-Nearest Neighbors model. Note that we could further improve the accuracy by applying cross-validation. As a conclusion, I would like to summarize the accuracy after each major step as below.
Feature Engineering Step                   Testing AccuracyInitially without any data processing                0.7079Scale down values into 0-1 range                     0.9059Basic encoding of two columns                        0.9208Creating new features                                0.9678Removing unnecessary features                        0.9950