Analyzing Lending Club Loans with Python - A Tutorial | Towards Data Science

Marketplace lending is the site of possible discrimination or unfair lending practices. In this post series we study the spacial correlation between Redlining from 1935–40 and the peer-2-peer lending landscape of today.
As an Equal Housing Lender, a peer-2-peer lending marketplace should screen loan applications “without regard to race, color, religion, national origin, sex, handicap, or familial status.” Systematically refusing loans from specific zip codes can result in harming minority applicants . This is a practice called Redlining between 1935 -1940, when the Home Owners’ Loan Corporation or “HOLC” created maps that color-coded credit worthiness and risk on neighborhood levels in nearly 250 US cities. These maps were recently digitized and can be viewed on MAPPING INEQUALITY. Despite the federal Fair Housing Act, modern Redlining has been recently exposed in 61 US cities. Another investigation identified a couple of banks which apparently exclude some minorities from their clients’ portfolio.
We used Python, Pandas, Jupyter Notebook and the Geopandas library to visualize 24 million loan applications from Lending Club — the world’s largest peer-to-peer lending platform. The goal is to investigate whether applicants from zip codes which were “redlined” in the 1930s are more likely to be rejected by the marketplace today. If not, we will verify whether such loan applicants receive a higher interest rate per default. We also want to explore how these relationships evolve throughout the life span of Lending Club, from 2006-2007 till now.
The MAPPING INEQUALITY website offers an option to download shapfiles of all US cities which were “redlined” back in the 1930s. The shapefiles contain the geographical latitude and longitude coordinates of polygons describing the neighborhoods (zones) in each of the cities, as they were delimited by HOLC.
import geopandas as gpddf_redlines_map = gpd.read_file('data/HOLC_ALL.shp')df_redlines_map.head()
Each row represents a polygon (zone) with the following features:
state: US state,
city: US city,
looplat, looplng: polygon starting coordinates,
name: area name,
holc_id : area nr,
holc_grade: HOLC (security) grade,
area_descr: area description,
geometry: set of all coordinates building the polygon.
The HOLC grade describes the credit worthiness that was assigned to the zone back in the 1930s, as follows:
A — Best
B — Still Desirable
C — Definitely Declining
D — Hazardous
We clean out those polygons with invalid HOLC grade.
df_redlines_map = df_redlines_map[df_redlines_map.holc_grade!='E']
Before diving into our analysis, we look at the distribution of grades assigned by HOLC not only within each zip code but also at the state level. This requires an estimate of the population in each zone, back in the 1930s. A naive approach is to use the count of zones per HOLC grade. Because the size of the corresponding polygons varies a lot, we might fail to get an intuition about the population size. Therefore, we choose to use the zone area (in km2) as a benchmark unit.
We start by calculating the area of each zone in square kilometers, as well as its geographical center (“centroid”).
proj = partial(pyproj.transform, pyproj.Proj(init='epsg:4326'),               pyproj.Proj(init='epsg:3857'))df_redlines_map['area'] = [transform(proj, g).area/1e+6 for g in df_redlines_map['geometry'].values]df_redlines_map['centroid_lat'] = df_redlines_map.geometry.centroid.ydf_redlines_map['centroid_long'] = df_redlines_map.geometry.centroid.xdf_redlines_map.head()
We will now add the zip code of each zone using the coordinates of its centroid. For this purpose we rely on the free Python library uszipcode, which does geodecoding for each batch of 100 zones, using its own up-to-date database of geocoded data.
from uszipcode import SearchEnginei = 0while i < df_redlines_map.shape[0]:    df_redlines_map.loc[i:i+100,'zipcode'] = df_redlines_map[i:i+100].apply(lambda row: search.by_coordinates(row.centroid_lat,row.centroid_long, radius=30, returns=1)[0].zipcode, axis=1)        i = i + 100df_redlines_map.head()
Calculating the total redlined area per zip code is now straightforward.
df_redlines = df_redlines_map.groupby(['zipcode']).agg({'area': np.sum}).reset_index().rename(columns={'area': 'zip_area'})df_redlines.head()
We can now compute the total redlined area per state.
df_redlines_state = df_redlines_map.groupby(['state']).agg({'area': np.sum}).reset_index().rename(columns={'area': 'state_area'})df_redlines_state.head()
Using the previous calculations we can obtain the percentage distribution of HOLC grades within each state.
df_redlines_state_details = df_redlines_map.groupby(['state','holc_grade']).agg({'area': np.sum}).reset_index().rename(columns={'area': 'holc_area'})df_redlines_state_details = df_redlines_state_details.merge(df_redlines_state, on='state', how='left')df_redlines_state_details['holc_ratio'] = df_redlines_state_details.holc_area/df_redlines_state_details.state_areadf_redlines_state_details.head()
We can also compute similar values per zip code.
df_redlines_details = df_redlines_map.groupby(['zipcode','holc_grade']).agg({'area': np.sum}).reset_index().rename(columns={'area': 'holc_area'})df_redlines_details = df_redlines_details.merge(df_redlines[['zipcode','zip_area']], on='zipcode', how='left')df_redlines_details['holc_ratio'] = df_redlines_details.holc_area/df_redlines_details.zip_areadf_redlines_details.head()
As we will see later, some interesting visualizations require us to convert the grade values into features. This allows us to calculate the area ratio of A, B, C, and D subzones within each zone as shown below. We fill missing ratio values with a zero, to take into account the zip codes where there were no HOLC zones.
for grade in ['A', 'B', 'C', 'D']:    df1 = df_redlines_map[df_redlines_map.holc_grade==grade].groupby(['zipcode']).agg({'area': np.sum}).reset_index().rename(columns={'area': grade + '_area'})    df_redlines = df_redlines.merge(df1, on='zipcode', how='left')    df_redlines[grade + '_ratio'] = df_redlines[grade + '_area'] / df_redlines.zip_areadf_redlines.fillna(0, inplace=True)
These ratios help us estimating the so-called HOLC Reject Ratio, defined as the percentage of loan applications that were rejected within a zip code due to Redlining in the 1930s. The ratio is assumed to be 0% for applicants in subzone A, 100% for C and D, and 90% in subzone B. We choose 90% for B because it is close to the average reject ratio found in the literature [1][2].
df_redlines['holc_reject_ratio'] = (.9*df_redlines.B_ratio + df_redlines.C_ratio + df_redlines.D_ratio)df_redlines.head()
The bar chart below reveals that most zones considered by HOLC were ranked as Hazardeous or Definitely Declining in the 1930s.
redlines_labels = {'grade': ['A', 'B', 'C', 'D'],                   'desc':['A - Best', 'B - Still Desirable', 'C - Definitely Declining', 'D - Hazardous'],                   'color': ['g', 'b', 'y', 'r']         }fig, ax  = plt.subplots(1,1,figsize=(15,5))sns.countplot(y='holc_grade',data=df_redlines_map, palette=redlines_labels['color'], ax=ax)ax.set_title('Count of zones per grade');
The zones redlined by HOLC were usually not very large, less than 5 km2. We notice few outliers with surface reaching 180 km2 below.
fig, ax  = plt.subplots(1,1,figsize=(15,5))sns.boxplot(y="holc_grade", x="area", palette=redlines_labels['color'],data=df_redlines_map, ax=ax)sns.despine(offset=10, trim=True)ax.set_title('Distribution of zone areas (km^2) per HOLC grade');
States AL (Alabama) and KS (Kansas) had the largest Hazardous total area, as classified by HOLC in 1930s. Zones in the states of NH (New Hampshire) and IN (Indiana) were mostly classified as Definitely Declining.
fig, ax  = plt.subplots(1,1,figsize=(20,5))sns.barplot(x="state", y="holc_ratio", hue='holc_grade', data=df_redlines_state_details, ax=ax)ax.set_title('Holc Grade Ratio per state');
After exploring Redlining data, let’s now have a look at today’s lending. All loan applications received between 2007Q1 and 2018Q2 are downloaded from Lending Club website. The merged and cleansed data is made available to us by authors of the project cited here.
df_loan = pd.read_csv('data/df_reject_ratio_2007-2018.csv')df_loan.head()
Each loan application is described by the following features:
issue_d: the date when the application was received.
zip_code: 3 digits of the zip code of the applicant (borrower). Lending Club does not publish the full zip code for privacy concerns.
rejected: a flag with value 1 if the application was rejected by Lending Club or 0 otherwise.
grade: a category indicating the interest rate (applicable only for applications which were not rejected).
“Based on each loan application and credit report, every loan is assigned a grade ranging from A to G with a corresponding interest rate.” Each loan grade and its corresponding current interest rate is displayed on Lending Club website.
print('There have been {} loans requests received at Lending Club since 2007, of which {} have been rejected'.format(df_loan.shape[0], df_loan[df_loan.rejected==1].shape[0]))There have been 24473165 loans requests received at Lending Club since 2007, of which 22469074 have been rejected
One common approach to explore large time series is to aggregate features of interest according to bigger time units such as quarters. An issue with Lending Club data is the format of issue_d, which is YYYY-MM-DD for many rows but we can also find dates with the format b-YYYY. For example we see rows with 2007–05–26 and Dec-2011. It is desirable to handle each format differently when converting the date to quarter.
Specifically, we split the data into two sets, one for each date format.
df1 = df_loan[(df_loan.issue_d.str.len()==10)]df1['issue_q'] = pd.to_datetime(df1.issue_d, format='%Y-%m-%d').dt.to_period('Q')df1.head()
df2 = df_loan[(df_loan.issue_d.str.len()!=10)]df2['issue_q'] = pd.to_datetime(df2.issue_d, format='%b-%Y').dt.to_period('q')df2.head()
Now, we can merge both data sets.
df_loan = df1.append(df2)
We further use the data to calculate the Lending Club Loan Reject Ratio, defined as the percentage of loan applications which were rejected per zip code during each quarter.
df_loan_reject_ratio = df_loan[['issue_q','zip_code','rejected']].groupby(['issue_q','zip_code']).agg(['count', 'sum'])df_loan_reject_ratio.columns = df_loan_reject_ratio.columns.droplevel(level=0)df_loan_reject_ratio = df_loan_reject_ratio.rename(columns={'count':'lc_total_requested', 'sum':'lc_total_rejected'})df_loan_reject_ratio['lc_total_accepted'] = df_loan_reject_ratio.lc_total_requested - df_loan_reject_ratio.lc_total_rejecteddf_loan_reject_ratio['lc_reject_ratio'] = df_loan_reject_ratio.lc_total_rejected/df_loan_reject_ratio.lc_total_requesteddf_loan_reject_ratio = df_loan_reject_ratio.reset_index()df_loan_reject_ratio.head()
The summary table above gives the following quantities for each quarter and zip code:
lc_total_requested: number of loan applications received,
lc_total_rejected: number of loan applications rejected,
lc_total_accepted: number of loan applications accepted,
lc_reject_ratio: ratio of lc_total_rejected over lc_total_requested.
Similarly, we compute the Lending Club Loan Grade Ratio (the percentage of loans with a specific interest rate per zip code and quarter).
df_loan_grades = df_loan[df_loan.rejected==0][['issue_q','zip_code','issue_d']].groupby(['issue_q','zip_code']).count().reset_index().rename(columns={'issue_d':'total_accepted'})for grade in ['A', 'B', 'C', 'D', 'E', 'F', 'G']:    df1 = df_loan[(df_loan.rejected==0)&(df_loan.grade==grade)][['issue_q','zip_code','issue_d']].groupby(['issue_q','zip_code']).count().reset_index().rename(columns={'issue_d': 'LC_' + grade + '_accepted'})    df_loan_grades = df_loan_grades.merge(df1, on=['issue_q','zip_code'], how='left')    df_loan_grades['LC_' + grade + '_ratio'] = df_loan_grades['LC_' + grade + '_accepted'] / df_loan_grades.total_accepted
For example, LC_A_accepted is the number of application which were accepted and assigned the interest rate A. LC_A_ratio is the corresponding percentage ratio.
We are now ready to merge loans and Redlining data, allowing us to experiment with different visualizations, and to formulate interesting questions when looking at 2007–2018' loans vs Redlining.
Because Lending Club discloses only 3 out of 5 digits from the applicant’s zip code, we will have to summarize the data and merge it based on 123xx zip code patterns.
df_redlines[‘zip_code’] = df_redlines[‘zipcode’].astype(‘str’)df_redlines[‘zip_code’] = df_redlines[‘zip_code’].str.pad(5, ‘left’, ‘0’)df_redlines[‘zip_code’] = df_redlines[‘zip_code’].str.slice(0,3)df_redlines[‘zip_code’] = df_redlines[‘zip_code’].str.pad(5, ‘right’, ‘x’)df_redlines_aggr = df_redlines.fillna(0).groupby('zip_code').agg({'zip_area': np.sum, 'A_area': np.sum, 'B_area': np.sum, 'C_area': np.sum, 'D_area': np.sum, 'A_ratio': np.mean,'B_ratio': np.mean,'C_ratio': np.mean,'D_ratio': np.mean,'holc_reject_ratio': np.mean}).reset_index()df_redlines_aggr.head()
The table above shows the total size (km2) of areas ranked A, B, C or D by HOLC in the 1930s within the region represented by a 123xx zip code pattern, where 123 is a representative number for the zip code digits disclosed by Lending Club. The overall area ratio per HOLC grade as well as the overall HOLC reject ratio is also provided.
Continue with this aggregated Redlining dataset, we will now add the loan data. Empty ratios are replaced by 0 for 123xx regions where either loan data or Redlining data is missing.
df_redlines_loan = df_loan_reject_ratio.merge(df_loan_grades, on=['zip_code','issue_q'], how='left').merge(df_redlines_aggr, on='zip_code', how='left')df_redlines_loan.issue_q = df_redlines_loan.issue_q.astype('str')df_redlines_loan.fillna(0, inplace=True)df_redlines_loan.head()
Looking at the line plot below, we can see that Lending Club appears to reject more loans on average than HOLC did in 1930s. We can expect the rejects to increase further in the future.
fig, ax = plt.subplots(1,1,figsize=(15,5))sns.lineplot(x="issue_q", y="lc_reject_ratio",data=df_redlines_loan, ax=ax, label='LendingClub reject ratio')plt.axhline(df_redlines_loan[df_redlines_loan.holc_reject_ratio>0].holc_reject_ratio.mean(), color='r', label='HOLC reject ratio')plt.axhline(df_redlines_loan.lc_reject_ratio.mean(), color='black', label='LendingClub reject ratio Average')plt.xlabel('quater')plt.ylabel('ratio')plt.title('Average Loan Reject Ratio over time')plt.legend()plt.xticks(rotation=45);
The scatter plot below shows a positive correlation between the loan reject ratio of Lending Club and HOLC. This suggests the following hypothesis: zones where HOLC rejected the most or almost all loans in the 1930s are the same zones where Lending Club is likely to reject a lot of loans today.
fig, ax = plt.subplots(1,1,figsize=(15,5))sns.scatterplot(x=’lc_reject_ratio’, y=’holc_reject_ratio’, data=df_redlines_loan[df_redlines_loan.holc_reject_ratio>0], ax=ax)plt.title(‘Loan Reject Ratio per zipcode — LendingClub versus HOLC’);
As shown in the distplot below, the Lending Club reject ratio distribution appears to copy HOLC reject ratio distribution in regions with large areas which were classified to Hazardous or Definetely Declining for lending in the 1930s.
fig, ax = plt.subplots(1,1,figsize=(15,5))sns.distplot(df_redlines_loan[df_redlines_loan.holc_reject_ratio>0].holc_reject_ratio, color='r', hist = False, kde = True, kde_kws = {'shade': True, 'linewidth': 3}, label='HOLC', ax=ax)sns.distplot(df_redlines_loan[df_redlines_loan.holc_reject_ratio>0].lc_reject_ratio, color='g', hist = False, kde = True, kde_kws = {'shade': True, 'linewidth': 3}, label='LendingClub', ax=ax)plt.xlabel('ratio')plt.title('Loan Reject Ratio Distribution over zipcodes');
The correlation between HOLC and Lending Club features is very weak, looking at the heat-map below. This could imply that, in general, knowing the HOLC grade of a region does not help us confidently forecasting loan rejection or loan interest rate at Lending Club.
corr = df_redlines_loan.corr()mask = np.zeros_like(corr, dtype=np.bool)mask[np.triu_indices_from(mask)] = Truef, ax = plt.subplots(figsize=(11, 9))cmap = sns.diverging_palette(220, 10, as_cmap=True)sns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0,            square=True, linewidths=.5, cbar_kws={"shrink": .5})ax.set_title('Correlation between HOLC and LendingClub');
The following distribution plot suggests that there are almost no Lending Club applicants who are receiving the lowest interest rate (A) in redlined zones. The majority of applicants receive a medium interest rate (B, C, or D). Only few applicants have to deal with very high interest rates (E, F, or G). This distribution suggests that Lending Club may not be considering Redlining when accessing the risk of credit charge-off.
f = df_redlines_loan[(df_redlines_loan.holc_reject_ratio==1)&(df_redlines_loan.lc_reject_ratio!=1)].sort_values(by='lc_reject_ratio', ascending=False).reset_index()[['zip_code', 'holc_reject_ratio', 'lc_reject_ratio', 'LC_A_ratio', 'LC_A_ratio', 'LC_B_ratio', 'LC_C_ratio', 'LC_D_ratio', 'LC_E_ratio', 'LC_F_ratio', 'LC_G_ratio']]
Our study provides some evidence of modern Redlining by the Lending Club. The hypothesis of an intrinsic correlation between 1930’s redlining maps and today’s Lending Club policies for accepting loan requests and assessing risks of default remains valid.
There are some signs of unfair algorithms or historical bias in data used at Lending Club. This is clear from looking at our exploratory data analysis, especially the positive linear trend between HOLC reject ratio and Lending Club reject ratio.
Thank you for reading so far.
In Part 2 of this post series we will use Geopandas to visualize the Redlining areas, loan reject ratios and loan interest rates distribution on a geographical map of the United States. This will include a choropleth map animation for spatial time series visualization, showing how the geographical landscape of peer-2–peer lending evolved between 2007 and 2018 with regards to Redlining.
[1] Courtemanche, C., Snowden, K. (2011). Repairing a Mortgage Crisis: HOLC Lending and Its Impact on Local Housing Markets. The Journal of Economic History, 71(2), 307–337.
[2] Fishback, P., Flores-Lagunes, A., Horrace, W., Kantor, S., Treber. J. (2010). The Influence of the Home Owners’ Loan Corporation on Housing Markets During the 1930s. The Review of Financial Studies, 24(6), 1782–1813.