Analyzing Medicare Data in Python | by Sadrach Pierre, Ph.D. | Towards Data Science

Medicare is a single-payer national social health insurance program for Americans age 65 and older. It also includes some younger adults with disability status, people living with ALS, and people with end-stage renal disease. The Center for Medicare & Medicaid Services provides public access to Medicare data. You can access the data through Google BigQuery.
You can use the dataset to answer the following questions:
What is the total number of medications prescribed in each state?What is the most prescribed medication in each state?What is the average cost for inpatient and outpatient treatment in each city and state?Which are the most common inpatient diagnostic conditions in the United States?Which cities have the most number of cases for each diagnostic condition?What are the average payments for these conditions in these cities and how do they compare to the national average?
What is the total number of medications prescribed in each state?
What is the most prescribed medication in each state?
What is the average cost for inpatient and outpatient treatment in each city and state?
Which are the most common inpatient diagnostic conditions in the United States?
Which cities have the most number of cases for each diagnostic condition?
What are the average payments for these conditions in these cities and how do they compare to the national average?
In this post, we will perform some simple exploratory data analysis of data from Medicare database. While we will not answer all of the questions above, this post will provide a good start for interested readers.
First, to access the data, you need to create a Google Cloud account. Next, you need to install Google BigQuery and generate authentication credentials. You can find the instructions for doing so here.
Let’s get started!
First, let’s import the necessary google authentication and BigQuery packages:
from google.oauth2 import service_accountfrom google.cloud import bigquery
Next, we need to import pandas and define the column settings to be ‘None’. We do this so that we can display all columns since pandas truncate them by default.
import pandas as pdpd.set_option("display.max_columns", None)
Next, we define our key path, client credentials, and client object:
key_path = "key_path/key.json"credentials = service_account.Credentials.from_service_account_file(    key_path,    scopes=["https://www.googleapis.com/auth/cloud-platform"],)client = bigquery.Client(    credentials=credentials,    project=credentials.project_id,)
Next, we define the dataset reference where we pass in the name of the database, ‘medicare’, and the name of the project, ‘bigquery-public-data’:
dataset_ref = client.dataset("medicare", project="bigquery-public-data")
We can look at the 12 tables available:
columns = [x.table_id for x in client.list_tables(dataset_ref)]print(columns)
Let’s look at the first element in the list, ‘inpatient_charges_2011’:
table_ref = dataset_ref.table('inpatient_charges_2011')table = client.get_table(table_ref)
We can store the results in a dataframe and print the names of the columns:
df = client.list_rows(table).to_dataframe()print(df.columns)
We can also look at the length of the dataframe:
print(len(df))
We can also print the first five rows of the dataframe:
print(df.head())
We can import the counter method from the collections module in python to analyze the distribution in some of the categorical variables. Let’s take a look at the ‘provider_name’ column:
from collections import Counterprint(dict(Counter(df['provider_name'].values).most_common(20)))
We can also visualize some of this output. Let’s define a function that takes a category string as input and displays a bar chart of the four most common values:
import matplotlib.pyplot as pltimport seaborn as snsdef plot_most_common(category):    sns.set()    bar_plot = dict(Counter(df[category].values).most_common(4))    plt.bar(*zip(*bar_plot.items()))    plt.show()
Let’s now call the function with ‘provider_name’:
plot_most_common('provider_name')
We can also look at the four most common values for ‘drg_description’:
plot_most_common('drg_description')
We can also define a function to generate boxplots across categorical values. We use boxplots to visualize the distribution in numeric values based on the minimum, maximum, median, first quartile, and third quartile. If you are unfamiliar with them take a look at the article Understanding Boxplots.
The function takes a data frame, categorical column and numerical column and displays boxplots for the most common categories based on the limit:
def get_boxplot_of_categories(data_frame, categorical_column, numerical_column, limit):    keys = []    for i in dict(Counter(df[categorical_column].values).most_common(limit)):        keys.append(i)    print(keys)        df_new = df[df[categorical_column].isin(keys)]    sns.boxplot(x = df_new[categorical_column], y = df_new[numerical_column])
Let’s generate boxplots for average medicare payments in the 5 most commonly occurring provider names:
get_boxplot_of_categories(df, 'provider_name', 'average_medicare_payments', 5)
We can generate a similar plot for ‘total_discharges’:
get_boxplot_of_categories(df, 'provider_name', 'total_discharges', 5)
We can also build a scatterplot function. This function takes a data frame, categorical column, categorical value, and two numerical columns as input and displays a scatterplot:
def get_scatter_plot_category(data_frame, categorical_column, categorical_value, numerical_column_one, numerical_column_two):    import matplotlib.pyplot as plt    import seaborn as snsdf_new = data_frame[data_frame[categorical_column] == categorical_value]    sns.set()    plt.scatter(x= df_new[numerical_column_one], y = df_new[numerical_column_two])    plt.xlabel(numerical_column_one)    plt.ylabel(numerical_column_two)
Let’s generate a scatterplot of average_medicare_payments vs total_discharges for ‘GOOD SAMARITAN HOSPITAL’ :
get_scatter_plot_category(df, 'provider_name', 'GOOD SAMARITAN HOSPITAL', 'average_medicare_payments', 'total_discharges')
We can also look at the same plot for ‘NORTH SHORE MEDICAL CENTER’:
get_scatter_plot_category(df, 'provider_name', 'NORTH SHORE MEDICAL CENTER', 'average_medicare_payments', 'total_discharges')
To recap, in this post I performed simple exploratory data analysis on the Medicare dataset. I analyzed the distribution in categorical values of provider names. I also generated bar charts for visualizing the frequency of provider names in the dataset. Finally, I defined functions for generating boxplots across categorical values and visualizing scatterplots of numerical values. I hope this post was interesting.
The code from this post is available on GitHub. Thank you for reading!