From .txt file to d3.js visualization
By: Zach Alexander
Tags: Data Visualization, Regular Expressions, JavaScript, Data Science, D3js

From .txt file to d3.js visualization

Photo by Randy Fath on Unsplash

It was noted a few years ago by IBM that working as a Data Scientist means that you’ll likely have to abide by the “80/20 rule” quite often— 80 percent of your time will be spent simply finding, cleaning, and organizing data, leaving only 20 percent to actually performing analyses.

As a Data Science graduate student, one of my first courses focused solely on that initial 80 percent. One assignment in particular required cleaning the following text file that contained “elo-ratings” of fictional chess players, as well as scores and points from a fictional chess tournament:

Mock chess “elo” ratings and points

Truly atrocious! I clipped the bottom of this file since it was quite long, but in total, there were 64 players listed.

Now, you may be thinking, what can we possibly do with this? Is there any interesting information here? Well, we’ll use the following workflow to investigate:

Workflow

From text file to d3.js visualization

Reading the text file into R

The first thing we’ll have to do is grab the .txt file and pull it into R. To do this, we can use the base R functions paste() and readLines() :

library(stringr)

library(knitr) chess_data <- paste(readLines("https://raw.githubusercontent.com/zachalexander/data607_cunysps/master/Project1/chess_ratings.txt"), collapse = '

')

As you’ll see from the link, I saved the text file in a new GitHub repository and accessed the “Raw” option, in order to pull the correct file. Additionally, I’ve loaded in the stringr() and knitr() packages that we’ll use to process the text.

Using Regular Expressions to Process the Text File

Although I’ve pulled the text file into R, if I inspect my variable “chess_data” and print it to the console, you’ll see that the data is not formatted well at all:

After pulling the raw text file into R — before processing.

Fortunately, if you look closely, there are subtle patterns in this file that we can use to parse out key information with regular expressions. If you need to brush up on regular expressions, I’d recommend this resource.

First, it would be great to get player names. To do this, we can use the str_extract_all() function in combination with a custom regular expression to isolate this data. We can see that player names are listed in all capital letters and therefore, we can extract them all by running the following syntax:

player_names <- unlist(str_extract_all(chess_data, '([:upper:]+\\s[:upper:]+(\\s[:upper:]+)?(\\s[:upper:]+)?)')) # change to 'proper' case

player_names <- str_to_title(player_names)

head(player_names)

In the second portion of the syntax above, I simply converted the names from all capital letters to “proper” form. You can see the end product printed to the console above. We now have a vector of 64 player names! It was that easy!

Looking at the original text file, I think it would be interesting to also isolate the player’s state of origin, each player’s pre-tournament elo rating, and each player’s post-tournament elo rating. The following regular expressions took care of this:

# found that each player's state could be extracted through this regular expression combination

location <- unlist(str_extract_all(chess_data, '(

)\\s\\s\\s[:upper:]+'))

location <- unlist(str_replace_all(location, '^\\s+|\\s+$', "")) # found that each player's pre-rating could be extracted through this regular expression combination

pre_rating <- unlist(str_extract_all(chess_data, '(R:)\\s{1,}\\d+'))

pre_rating <- unlist(str_replace_all(pre_rating, '(R:)\\s', "")) # found that each player's post-rating could be extracted through this regular expression combination (used later in data visualization)

post_rating <- unlist(str_extract_all(chess_data, '(->)(\\s{1,})?\\d+'))

post_rating <- unlist(str_replace_all(post_rating, '(->)(\\s)?', ""))

Similar to isolating the player names, each one of these items above could be isolated due to certain patterns in the original text file.

Player Results — Other R Functions

With these pieces of information safely stored in organized vectors. Let’s take stock of where we are at, and what additional information we’d like to collect for analysis. All data highlighted in yellow below has been extracted, and all data in green are values that would be useful to grab from this text file — we’ll focus on these values in this next step.

Where we currently are in our workflow.

Initially, we can see that each player’s total points from the tournament contain numbers with decimal places. To isolate these values, we can extract all based on this sequence of characters (see below).

Then, since the rest of the data highlighted in green is formatted with “|” characters between each round, we can do some manipulation to grab these scores, continuing to take advantage of our regular expressions.

# found that each player's total points could be extracted through this regular expression combination

total_points <- unlist(str_extract_all(chess_data, '(\\d\\.\\d)')) results <- unlist(str_extract_all(chess_data, '(\\d\\.\\d\\s{2}(\\|[:upper:]\\s{2}(\\s)?(\\s)?(\\d+)?){1,})'))

results <- unlist(str_replace(results, '\\d\\.\\d\\s{2}\\|', ''))

In the end, I was able to convert these values for each player into a temporary dataframe by using a for loop to get rid of the extra symbols and making sure they matched with the players from my initial vector. You can see the for loop process below:

# used a for loop here to iterate over V1, using strsplit and the comma to separate out the opponent numbers into the appropriate columns. This also stored N/As in columns where a player didn't play an opponent.

for(i in 1:7){

tempdf[, paste0('game', i)] <- sapply(strsplit(as.character(tempdf$V1),','), "[", i)

}

After combining this data with my player data, we get the following dataframe:

Now we are getting somewhere!

As you can see, the data found in the game1, game2…gameN columns correspond to the index row number. Since we already have this information, I thought it would be interesting to replace the index number in these columns with the pre-tournament rating that corresponds to each player’s index number. If I do this correctly, I can then determine the “strength of schedule” for each player! To do this, I again utilized for loops:

# while iterating over the game columns, iterate over the player opponent number and match it with the opponent's pre-tournament elo rating. Then replace the opponent number with their elo rating.

for(i in 6:12) {

for(j in 1:64) {

value <- chess_ratings_df[,i][j]

chess_ratings_df[,i][j] <- chess_ratings_df$player_pre_rating[value]

}

}

And there we go! Here’s our final dataframe, cleaned and ready for analysis:

Final dataframe containing chess player information, the total points earned from the tournament, and the pre-elo rating of each opponent they faced.

Initial Analyses

Given this dataframe, I’m interested in determining which player(s) had the hardest tournament schedule based solely on the pre-tournament ratings of their opponents. Additionally, who had the biggest jump in their elo rating from pre-tournament to post-tournament?

To figure this out, I first used the rowMeans() function to take the average of the opponent pre-tournament elo ratings for the seven tournament games. I then created a new column called average_opp_rating to store this calculation for each player.

chess_ratings_df$average_opp_rating <- round(rowMeans(chess_ratings_df[,6:12], na.rm = TRUE), digits = 0)

I also decided to create a few more columns to determine the difference in player pre-tournament rating and player post-tournament rating (ultimately to see who had the highest elo rating gain/loss). Additionally, I created a column on the conditional of whether or not the rating difference between pre- and post- was negative or positive.

In the end, I developed a pretty comprehensive dataframe which could answer many of my questions. Many would find this satisfactory, and for most purposes it is. However, wouldn’t it be more appealing if we put this into a visualization?

Exporting Data into a .csv file

With my dataframe ready to go, I can export it to a .csv file with the following syntax in R:

filename <- './../.../filepath'

write.csv(final_df, file = filename)

For those that are familiar with d3.js, there is a built-in function that is capable of reading in a .csv file and iterate through its rows. However, I prefer to work directly with .json files (just my personal preference). You can quickly use a free online service such as csvjson.com to convert between the formats.

Once it’s ready, I have a workable .json file:

Example of my R dataframe that was converted to a .csv file and then a .json file

Making the d3.js visualization

For the sake of time, I’m going to brush over most of what it takes to make a d3.js visualization. However, if you are interested in learning, I’d recommend this book — Interactive Data Visualization — it really is a great guide to start getting your hands dirty with d3 and Javascript!

With my .json file ready to go from above, I can then do the following:

Create an Angular 2+ project using the angular-cli. Create a firebase project and deploy my Angular 2+ project using firebase. This is a great guide for this. I create a component, install the necessary modules, and begin building my visualization!

In the end, I generated a bar chart that displays each player’s pre-tournament elo rating, and then you can click a button to see their elo-rating change based on their post-tournament rating. You can also roll-over the initial bar and see the pre-tournament rating:

Final d3.js visualization — to see the final product go to https://zach-alexander.com/chess-elo

To see the full, interactive visualization, you can visit https://www.zach-alexander.com/chess-elo. Hopefully, you’ll find it appealing. You’ll notice too that I created a mock newspaper story related to this data — just for fun!

When all is said and done, although most people grumble about the “80/20 rule”, we can see how crucial it is for tasks like this one. There’s useful data everywhere, but only if you are willing to take the time to process it!