Visualizing Capitol Riot Arrest Data
By: Paul Coogan
Tags: PHP, Python, Politics, Charts, Capitol Riot

Bringing in the data is incredibly simple using Pandas for reading in table data from a web site.



html = requests.get(url).content

df_list = pd.read_html(html) url = ‘ https://www.justice.gov/opa/investigations-regarding-violence-capitol' html = requests.get(url).contentdf_list = pd.read_html(html)

Once the information was stored in a data frame it could be cleaned up.

Translate the header to eliminate spaces.

Extract the state names and make them consistent.

Translate the state names to abbreviations using a dictionary.

df.rename(columns={‘CASE NUMBER’: ‘CASE_NUMBER’, ‘CHARGE(S)’: ‘CHARGES’, ‘LINKS TO PRESS RELEASES, CHARGING DOCUMENTS’: ‘LINKS’, ‘LOCATION OF ARREST’: ‘ARREST_LOCATION’, ‘CASE STATUS’: ‘STATUS’,’ENTRY LAST UPDATED’: ‘UPDATED’}, inplace=True)

df.STATE = df.STATE.replace(to_replace=r'.*Dist.* of ', value='', regex=True)

df.STATE = df.STATE.replace(to_replace=r'Washington,.*', value='Washington DC', regex=True)

df.STATE = df.STATE.replace(to_replace=r'Mass.*', value='Massachusetts', regex=True)

df.STATE = df.STATE.replace(to_replace=r'.*, ', value='', regex=True)

df.STATE = df.STATE.replace(to_replace=r'\)', value='', regex=True)

df_state = df.STATE.value_counts().rename_axis('STATE').reset_index(name='COUNTS')

us_state_abbrev = {

'Alabama': 'AL',

'Alaska': 'AK',

...

df_state['STATE'] = df_state['STATE'].map(us_state_abbrev)

My first thought was to render a map, because I love maps!

Not a good choice. I can zoom and mouse over for spot data but the sizes and populations of the states make the results very misleading — was this just Texas seceding from the Union again?

A bar chart was a much better solution with the non participating states falling out of the chart by default. Where no state was identified, the resulting NA was also removed.

Screen capture by the author

Again it looks like Texas is the big player, but what about NY and CA, those are “blue states”!

Normalizing the data by population is simply a matter of dividing the arrests in a state by the total population of that state, for Texas this is 13/28995881 (using 2010 census data). First the population data was pasted in as a dictionary and then converted to a data frame and concatenated on the end of the arrest data. Then the normalized data was calculated into an additional column. For Python gurus this probably looks excruciatingly procedural since the language can be extremely terse.

# The dictionary of populations

state_pop = {

‘California’: [‘39512223’],

‘Texas’: [‘28995881’],

‘Florida’: [‘21477737’],

... # Converted to a dataframe with state abbreviations as the index

df_pop = pd.DataFrame.from_dict(state_pop, orient=’index’)

df_pop = df_pop.reset_index()

df_pop = df_pop.rename(columns={‘index’: ‘STATE’, 0: ‘POP’})

df_pop[‘STATE’] = df_pop[‘STATE’].map(us_state_abbrev) # Add the new data to the master dataframe

df_normal = pd.concat([df_state.set_index('STATE'), df_pop.set_index('STATE')], axis=1, join="outer", sort=False) # Calculate the normalized figure into a new column

df_normal['NORMALIZED'] = df_normal['COUNTS'].astype('float') / df_normal['POP'].astype(float)

df_normal = df_normal.reset_index() # May as well dump the NA while we're here

df_normal = df_normal.rename(columns={'index': 'STATE'}).dropna()

The normalized data now has a huge outlier for Washington D.C. This make sense since that was the location of the event and it has a very low population (we are not going to discuss states versus districts and territories).

Screen capture by the author

Exclude DC is simply adding a filter to the data input into the chart.

fig = px.bar(df_normal[df_normal[‘STATE’] != ‘DC’], x=”STATE”, y=”NORMALIZED”, title=”Arrest Counts by Location Normalized for State Population (excluding DC)”)

fig.show()

Screen capture by the author

Now we can finally see that Texas is not so big and California is not so red. There is still some skew to the data as low population states can look worse when dealing with these small numbers. Even so, I find the last chart gives the best sense of which states participated.