Relationships validated between population health chronic indicators
By: Daniel Wu
Tags: Digital Health, Public Health, Data Science, In Depth Analysis, Python

Relationships validated between population health chronic indicators

The ties among diabetes, chronic kidney disease, and cardiovascular disease

Photo by v2osk on Unsplash

In the last story, we started looking into a 15 year chronic disease dataset from the U.S. Center for Disease Control and Prevention, or CDC. The beginnings of the exploratory data analysis started with understanding the columns and rows of data and what was relevant for further analysis.

In this post, we are going to dig deeper to understand these 400K rows and 17 categories of topics, which requires a bit of data wrangling of the dataframe into a format for pivot table summary and visualization. After looking at the previous df_new.head(), I don’t think we will use the following columns, which results in a smaller set of 15 columns:

df_new = df_new.drop(columns=['YearEnd','LocationDesc','DataSource','DataValue','DataValueFootnoteSymbol','DatavalueFootnote','LowConfidenceLimit','HighConfidenceLimit','GeoLocation']) <class 'pandas.core.frame.DataFrame'>

RangeIndex: 403984 entries, 0 to 403983

Data columns (total 15 columns):

YearStart 403984 non-null int64

LocationAbbr 403984 non-null object

Topic 403984 non-null object

Question 403984 non-null object

DataValueUnit 374119 non-null object

DataValueType 403984 non-null object

DataValueAlt 273666 non-null float64

StratificationCategory1 403984 non-null object

Stratification1 403984 non-null object

LocationID 403984 non-null int64

TopicID 403984 non-null object

QuestionID 403984 non-null object

DataValueTypeID 403984 non-null object

StratificationCategoryID1 403984 non-null object

StratificationID1 403984 non-null object

dtypes: float64(1), int64(2), object(12)

Table 1: Summarizing by Topic, Question, and Value

Using a pivot table (much like Excel), let’s summarize each Question and its related values. To inform each of the questions and values, we’ll create multiple indexes from the category of each question (Topic), the QuestionID, Question, the unit of the data (DataValueUnit), and the type of the data relevant to the stratifications later (DataValueType). Pandas Pivot_table requires numeric values, and defaults to averages (numpy.mean). Further, I’ve dropped columns LocationID and YearStart since I won’t be using these later, and rounded to two decimals for readability:

df_QD = df_new.pivot_table(index=['Topic','QuestionID','Question','DataValueUnit','DataValueType'],columns=None,dropna=True) df_QD.drop(columns=['LocationID','YearStart']).round(2).head(25)

Table 1. Pivot Table showing the Question and Values

Table 2: Using Groupby with Stratification, Question, and Value

Alternative to the pivot_table, I can create a similar summary table using groupby(). I want to show a bit more information on the stratification in the resulting dataframe.

df_unittype_table1 = df_new.groupby(['Topic','QuestionID','Question','StratificationID1','Stratification1','DataValueUnit','DataValueType']).mean().round(2) df_unittype_table1.drop(columns=['YearStart','LocationID'])

Table 2. Using groupby() to summarize stratification

Table 3: Summarizing by Question and Location

An interesting approach would showing the data by each state (LocationAbbr) with the data values. The following pivot table includes additional indexes that provide context to the numbers such as the unit and type.

df_new_qloc = df_new.pivot_table(values='DataValueAlt',index=['Topic','QuestionID','Question','DataValueUnit','DataValueType'], columns='LocationAbbr',aggfunc='mean',dropna=True).round(2)

Table 3. Pivot Table of Question and Location

Visualizing Correlation Among All Indicators

Now we’ve seen the tables present the data values by question, by stratification, and by location. That’s still a lot of information! Let’s understand what the question indicators say about itself. What are the relationships between each pair of question indicators? This is where visualization of the data will paint a picture to understand the overall relationships.

Originally, I started out using the pivot_table from Table 3 but the axis labels were too long and difficult to read. Backing up slightly, we’re going to create a different pivot_table (df_new_qloc2) based on the new the column QuestionAbbr to take the first 37 characters of Question:

df_new['QuestionAbbr'] = df_new['Question'].str[:37] df_new_qloc2 = df_new.pivot_table(values='DataValueAlt',index=['Topic','QuestionID','QuestionAbbr'], columns='LocationAbbr',aggfunc='mean',dropna=True).round(2)

By transposing df_new_qloc2 pivot table, we’re ready to apply the correlation method .corr() and visualize the data. Using the code from the seaborn documentation, we can plot a correlation matrix heatmap. This visualizes each pair of indicators to understand where the positive correlation pairs reside. Since a correlation matrix yields duplicate views above and below the diagonal, we’ll mask the upper half of the heatmap for simplicity by creating an array of zeros using np.zeros_like() and returning the indices of the region above the diagonal using triu_indices_from(). Using matplotlib, we set a large figure size to allow zooming in on the jupyter notebook. The cmap sets the colormap for the figure (more options here). Lastly, sns.heatmap() renders the heatmap with the mask.