Explore Your Activity on Youtube With R: How to Analyze and Visualize Your Personal Data History
By: Saúl Buentello
Tags: Data Analysis, Programming, Data Visualization, Data Science, Technology

Explore Your Activity on Youtube With R: How to Analyze and Visualize Your Personal Data History

Find out how you consume Youtube using a copy of your personal data

Analysis and visualization of your data on YouTube — Plot based on YouTube categories

Let’s face it, dear reader, one of the websites that we visit frequently and became part of our life, whether for entertainment or looking for answers, and where we could spend hours, is YouTube. As with anything else we spend so much time on, I find it curious to see exactly how our consumption habits have changed. With Google Takeout and the YouTube API, analyzing your personal YouTube history is easy.

How can I get a copy of my data?

Your YouTube history is available through Google Takeout, a tool that Google provides to consult practically the history and stored data of any of the products that you have used. You have to enter the URL https://takeout.google.com/settings/takeout and log in with your personal account.

Screenshot: Google Takeout website

There you will find the possibility to request a complete copy, or by-product, or even select only some characteristics of a product. We are only interested in YouTube for now. Even if you have multiple accounts linked, you can switch between them by clicking on the profile picture in the upper right. If you have uploaded videos, I recommend that you make sure to uncheck the videos option of the file that it will generate, this will decrease its weight and delivery time.

Screenshot: Generating a copy of YouTube account data

Once the file is generated, you will be receiving a notification email, from where you can proceed to download it. For security, the file has an expiration date, so you have to do it before the date indicated in the email.

Screenshot: Data copy download notification email from Google Takeout

When you download the file you will get a .zip that when unzipped will create a structure of folders and files, depending on what you initially requested from Google Takeout.

Screenshot: Folder structure of the unzipped .zip

Currently, the generated file corresponding to your search and watching history on YouTube is only exported as HTML. You can use the “rvest” package to parse the generated HTML. Yes, I know, don’t judge me, “el canaca” and “dios eolo” will always be classics (viral memes in Mexico).

Screenshot: YouTube Search History

Working the exported data

Now you can proceed to create an R script. First, you have to include all the necessary packages and set the search history reading, contained in the “search-history.html” file located inside the “history” folder.

# REQUIRED PACKAGES

library(stringr)

library(rvest)

library(tidyverse)

library(jsonlite)

library(tidytext)

library(lubridate)

library(wordcloud)

library(httr)

library(ggplot2)

library(wordcloud2)

library(RCurl)

library(curl)

library(pbapply)

library(ggthemes) # READ SEARCH HISTORY

youtubeSearchHistory <- read_html("Takeout/YouTube and YouTube Music/history/search-history.html")

To parse text in HTML using “rvest” you can specify CSS classes that correspond to a particular section. For example, the line .header-cell + .content-cell> a finds the hyperlink in the content, corresponding to the searches you made.

# SCRAPING SEARCH HISTORY

youtubeSearch <- youtubeSearchHistory %>%

html_nodes(".header-cell + .content-cell > a") %>%

html_text()

Screenshot: YouTube Search History HTML Element Inspector

And of course, you can get the information you need, for example, the timestamp in the same way.

# SCRAPING TIMESTAMP

youtubeSearchContent <- youtubeSearchHistory %>%

html_nodes(".header-cell + .content-cell")

youtubeSearchTimeStr <- str_match(youtubeSearchContent, "<br>(.*?)</div>")[,2]

youtubeSearchTime <- mdy_hms(youtubeSearchTimeStr)

Now that you have the search and timestamp data, you can create a Data Frame with them.

# CREATING DATA FRAME SEARCH + TIMESTAMP

youtubeSearchDataFrame <- data.frame(search = youtubeSearch,

time = youtubeSearchTime,

stringsAsFactors = FALSE)

Watching history

The watching history you will find in another HTML with the name of “watch-history.html” is also contained within the “history” folder. First, you must read the file and obtain, in the same way as before by web scraping, each entry. You will have to analyze information with regular expressions.

# READ WATCH HISTORY

watchHistory <- read_html("Takeout/YouTube and YouTube Music/history/watch-history.html") watchedVideoContent <- watchHistory %>%

html_nodes(".header-cell + .content-cell") # POSSIBLE TIME CHARACTERS

watchVideoTimes <- str_match(watchedVideoContent,

"<br>([A-Z].*)</div>")[,2] # POSSIBLE ID VALUES

watchedVideoIDs <- str_match(watchedVideoContent,

"watch\\?v=([a-zA-Z0-9-_]*)")[,2] # VIDEO TITLE

watchedVideoTitles <- str_match(watchedVideoContent,

"watch\\?v=[a-zA-Z0-9-_]*\">(.*?)</a>")[,2]

Now you can put everything back together in a Data Frame.

# DATA FRAME WATCH HISTORY

watchedVideosDataFrame <- data.frame(id = watchedVideoIDs,

scrapedTitle = watchedVideoTitles,

scrapedTime = watchVideoTimes,

stringsAsFactors = FALSE) watchedVideosDataFrame$time <- mdy_hms(watchedVideosDataFrame$scrapedTime)

Get more data about videos with the YouTube API

I’m almost sure you’ll say up to this point… Well Saúl, it’s not a big deal, what else! This gets more interesting when you integrate with the YouTube API to get more data. Instead of just seeing basic data like titles and timestamps, you can get more information. In this way, you can see the popularity, descriptions, categories, and more about your video views.

If this is your first time obtaining credentials to use the YouTube API, follow the simple steps that the official YouTube documentation makes available to you: https://developers.google.com/youtube/v3/getting-started.

Once you establish a Google project, you will need to generate an API Key and enable YouTube Data API v3, to create your credentials in the sidebar menu.

Screenshot: Google API Console, setting the YouTube API

Screenshot: Google API Console, generating a new API Key

Once this is done, you can now use your API Key by assigning it to a new variable. You must also assign the connection to the youtube API to a new variable.



youtubeAPIKey <- "HERE_YOUR_API_KEY"

connectionURL <- ' # ESTABLISH API KEY AND CONNECTIONyoutubeAPIKey https://www.googleapis.com/youtube/v3/videos'

You can make a test, for example, I will take a video from my YouTube channel with ID “SG2pDkdu5kE”, to give you an overview.

# TRYIING QUERY RESPONSE

videoID <- "SG2pDkdu5kE"

queryParams <- list()

queryResponse <- GET(connectionURL,

query = list(

key = youtubeAPIKey,

id = videoID,

fields = "items(id,snippet(channelId,title,categoryId))",

part = "snippet"

))

parsedData <- content(queryResponse, "parsed")

str(parsedData)

You will find that the two important parameters are “fields” and “parts”. You must be careful in your queries because you could exceed the request quota or the response to the request may become very slow. You can find more information about these parameters in the official documentation: https://developers.google.com/youtube/v3/docs/videos

Screenshot: Testing Query Response

Get Video Category: Preparing Requests

To get more metadata like the video category, you can use the video IDs you got from your file and make requests to YouTube for more information about each video. You are about to make several thousand requests to the YouTube API, so you have to do additional preparation of the requests.

The most popular library for making web requests is “httr” (it only supports one request at a time). There exists also “RCurl” and “curl”. To make sure you get the data as quickly as possible, try all three, speed may differ based on established queries.



testConnection <- "

testCount <- 100 # REQUESTS OPTIONStestConnection https://www.google.com/ testCount # HTTR TEST

system.time(for(i in 1:testCount){

result <- GET(testConnection)

}) # RCURL Test

uris = rep(testConnection, testCount)

system.time(txt <- getURIAsynchronous(uris)) # CURL TEST

pool <- new_pool()

for(i in 1:testCount){curl_fetch_multi(testConnection)}

system.time(out <- multi_run(pool = pool))

At least in my case, the speed was much faster with “curl”, unlike the other two options.

Get Video Category: Formatting Requests

You will need to create a connection string for each request and eliminate duplicates to reduce the number of requests made. You will also need a function to parse the request-response data.

# CREATE REQUEST AND REMOVE DUPLICATES

createRequest <- function(id){

paste0(connectionURL,

"?key=",youtubeAPIKey,

"&id=",id,

"&fields=","items(id,snippet(channelId,title,description,categoryId))",

"&part=","snippet")

}

uniqueWatchedVideoIDs <- unique(watchedVideosDataFrame$id)

requests <- pblapply(uniqueWatchedVideoIDs, createRequest ) # PARSE OUT RESPONSE

getMetadataDataFrame <- function(response){

rawchar <- rawToChar(response$content)

parsedData <- fromJSON(rawchar)

data.frame <- cbind(id = parsedData$items$id, parsedData$items$snippet)

return(data.frame)

}

You can configure what to do in case the request is successful or in case it has failed.

videoMetadataDataFrame <- data.frame(id = c(),

channelId = c(),

title = c(),

description = c(),

categoryId = c()

) # SUCCESS

addToMetadataDataFrame <- function(response){

.GlobalEnv$videoMetadataDataFrame <- rbind(.GlobalEnv$videoMetadataDataFrame,getMetadataDataFrame(response))

} # FAIL

failFunction <- function(request){

print("fail")

}

A method, although slower but more reliable to obtain your data, is to take it from memory and configure our multiple requests.

# GRAB REQUEST RESPONSE FROM MEMORY

fetchMetadataFromMemory <- function(request){

return(getMetadataDataFrame(curl_fetch_memory(request)))

} system.time(out <- multi_run(pool = pool))

saveRDS(videoMetadataDataFrame, file = "videoMetadataDataframeAsync1.rds") length(requests)

nrow(videoMetadataDataFrame) listMetadata <- pblapply(requests, fetchMetadataFromMemory)

Once this is done, depending on the speed of your internet connection, and the size of your data frame, you can probably prepare a coffee, a tea, or open a beer, waiting as long as the indicator reaches 100%.

Screenshot: listMetadata console preview

You will also use the “bind_rows” function to combine the list into an ordered Data Frame.

# COMBINE LIST INTO A DATA FRAME

videoMetadataDataFrame <- bind_rows(listMetadata)

saveRDS(videoMetadataDataFrame, file = "videoMetadataDataFrame_memory.rds")

Get Video Category: Formatting Categories

Each category has a unique ID. You need to make another request to get them and add the data to a new column.



categoryListURL <- " # CATEGORY ID REQUESTcategoryListURL https://www.googleapis.com/youtube/v3/videoCategories categoryResponse <- GET(url = categoryListURL,

query = list(

key = youtubeAPIKey,

regionCode = "us",

part = "snippet"

))

parsedCategoryResponse <- content(categoryResponse, "parsed") categoryDataFrame <- data.frame(categoryId=c(), category=c())

for(item in parsedCategoryResponse$items){

categoryDataFrame <<-rbind(categoryDataFrame,

data.frame(categoryId = item$id, category=item$snippet$title))

} categoryDataFrame

videoMetadata <- merge(x = videoMetadataDataFrame, y = categoryDataFrame, by = "categoryId")

head(videoMetadata)

You can combine your new Data Frame with your watch history to get the video metadata along with when it was played.

# COMBINE WITH WATCH HISTORY

watchedVideos <- merge(watchedVideosDataFrame , videoMetadata, by="id")

str(watchedVideos)

Have your tastes of what you watch on YouTube changed over time?

With your search history, watch history, and category metadata, you can now answer this question. You can see which video categories you play the most and how this has been changing over time.

# VISUALIZE VIDEO CATEGORIES WATCHED

watchedVideos %>%

group_by(category) %>%

summarise(count = n()) %>%

arrange(desc(count)) watchedVideos %>%

ggplot(aes(x = time, fill = category)) +

labs(x= "Year", y= "Count") +

ggtitle("How much have your genre tastes changed over time?", "Most played categories")+

geom_area(stat = "bin") +

theme_economist_white()

For example, in my case, something that surely likes many of you is very present, are videos in the music category. And of course, the presence of the Film & Animation category makes sense, I follow many channels to find out about upcoming movies, trailers, and reviews.