Explore Your Activity on Youtube With R: How to Analyze and Visualize Your Personal Data History | by Saúl Buentello | Towards Data Science

Let’s face it, dear reader, one of the websites that we visit frequently and became part of our life, whether for entertainment or looking for answers, and where we could spend hours, is YouTube. As with anything else we spend so much time on, I find it curious to see exactly how our consumption habits have changed. With Google Takeout and the YouTube API, analyzing your personal YouTube history is easy.
Your YouTube history is available through Google Takeout, a tool that Google provides to consult practically the history and stored data of any of the products that you have used. You have to enter the URL https://takeout.google.com/settings/takeout and log in with your personal account.
There you will find the possibility to request a complete copy, or by-product, or even select only some characteristics of a product. We are only interested in YouTube for now. Even if you have multiple accounts linked, you can switch between them by clicking on the profile picture in the upper right. If you have uploaded videos, I recommend that you make sure to uncheck the videos option of the file that it will generate, this will decrease its weight and delivery time.
Once the file is generated, you will be receiving a notification email, from where you can proceed to download it. For security, the file has an expiration date, so you have to do it before the date indicated in the email.
When you download the file you will get a .zip that when unzipped will create a structure of folders and files, depending on what you initially requested from Google Takeout.
Currently, the generated file corresponding to your search and watching history on YouTube is only exported as HTML. You can use the “rvest” package to parse the generated HTML. Yes, I know, don’t judge me, “el canaca” and “dios eolo” will always be classics (viral memes in Mexico).
Now you can proceed to create an R script. First, you have to include all the necessary packages and set the search history reading, contained in the “search-history.html” file located inside the “history” folder.
# REQUIRED PACKAGESlibrary(stringr)library(rvest) library(tidyverse) library(jsonlite) library(tidytext)library(lubridate) library(wordcloud)library(httr)library(ggplot2)library(wordcloud2)library(RCurl)library(curl)library(pbapply)library(ggthemes)# READ SEARCH HISTORYyoutubeSearchHistory <- read_html("Takeout/YouTube and YouTube Music/history/search-history.html")
To parse text in HTML using “rvest” you can specify CSS classes that correspond to a particular section. For example, the line .header-cell + .content-cell> a finds the hyperlink in the content, corresponding to the searches you made.
# SCRAPING SEARCH HISTORYyoutubeSearch <- youtubeSearchHistory %>%  html_nodes(".header-cell + .content-cell > a") %>%  html_text()
And of course, you can get the information you need, for example, the timestamp in the same way.
# SCRAPING TIMESTAMPyoutubeSearchContent <- youtubeSearchHistory %>%  html_nodes(".header-cell + .content-cell")youtubeSearchTimeStr <- str_match(youtubeSearchContent, "<br>(.*?)</div>")[,2]youtubeSearchTime <- mdy_hms(youtubeSearchTimeStr)
Now that you have the search and timestamp data, you can create a Data Frame with them.
# CREATING DATA FRAME SEARCH + TIMESTAMPyoutubeSearchDataFrame <- data.frame(search = youtubeSearch,                                 time = youtubeSearchTime,                                stringsAsFactors = FALSE)
The watching history you will find in another HTML with the name of “watch-history.html” is also contained within the “history” folder. First, you must read the file and obtain, in the same way as before by web scraping, each entry. You will have to analyze information with regular expressions.
# READ WATCH HISTORY watchHistory <- read_html("Takeout/YouTube and YouTube Music/history/watch-history.html")watchedVideoContent <-  watchHistory %>%  html_nodes(".header-cell + .content-cell")# POSSIBLE TIME CHARACTERSwatchVideoTimes <- str_match(watchedVideoContent,                              "<br>([A-Z].*)</div>")[,2]# POSSIBLE ID VALUES watchedVideoIDs <- str_match(watchedVideoContent,                              "watch\\?v=([a-zA-Z0-9-_]*)")[,2]# VIDEO TITLEwatchedVideoTitles <- str_match(watchedVideoContent,                                 "watch\\?v=[a-zA-Z0-9-_]*\">(.*?)</a>")[,2]
Now you can put everything back together in a Data Frame.
# DATA FRAME WATCH HISTORYwatchedVideosDataFrame <- data.frame(id = watchedVideoIDs,                                      scrapedTitle = watchedVideoTitles,                                      scrapedTime = watchVideoTimes,                                      stringsAsFactors = FALSE)watchedVideosDataFrame$time <- mdy_hms(watchedVideosDataFrame$scrapedTime)
I’m almost sure you’ll say up to this point... Well Saúl, it’s not a big deal, what else! This gets more interesting when you integrate with the YouTube API to get more data. Instead of just seeing basic data like titles and timestamps, you can get more information. In this way, you can see the popularity, descriptions, categories, and more about your video views.
If this is your first time obtaining credentials to use the YouTube API, follow the simple steps that the official YouTube documentation makes available to you: https://developers.google.com/youtube/v3/getting-started.
Once you establish a Google project, you will need to generate an API Key and enable YouTube Data API v3, to create your credentials in the sidebar menu.
Once this is done, you can now use your API Key by assigning it to a new variable. You must also assign the connection to the youtube API to a new variable.
# ESTABLISH API KEY AND CONNECTIONyoutubeAPIKey <- "HERE_YOUR_API_KEY"connectionURL <- 'https://www.googleapis.com/youtube/v3/videos'
You can make a test, for example, I will take a video from my YouTube channel with ID “SG2pDkdu5kE”, to give you an overview.
# TRYIING QUERY RESPONSEvideoID <- "SG2pDkdu5kE"queryParams <- list()queryResponse <- GET(connectionURL,                     query = list(                       key = youtubeAPIKey,                       id = videoID,                       fields = "items(id,snippet(channelId,title,categoryId))",                       part = "snippet"                     ))parsedData <- content(queryResponse, "parsed")str(parsedData)
You will find that the two important parameters are “fields” and “parts”. You must be careful in your queries because you could exceed the request quota or the response to the request may become very slow. You can find more information about these parameters in the official documentation: https://developers.google.com/youtube/v3/docs/videos
To get more metadata like the video category, you can use the video IDs you got from your file and make requests to YouTube for more information about each video. You are about to make several thousand requests to the YouTube API, so you have to do additional preparation of the requests.
The most popular library for making web requests is “httr” (it only supports one request at a time). There exists also “RCurl” and “curl”. To make sure you get the data as quickly as possible, try all three, speed may differ based on established queries.
# REQUESTS OPTIONStestConnection <- "https://www.google.com/"testCount <- 100# HTTR TESTsystem.time(for(i in 1:testCount){   result <- GET(testConnection)})# RCURL Testuris = rep(testConnection, testCount)system.time(txt <-  getURIAsynchronous(uris))# CURL TESTpool <- new_pool()for(i in 1:testCount){curl_fetch_multi(testConnection)}system.time(out <- multi_run(pool = pool))
At least in my case, the speed was much faster with “curl”, unlike the other two options.
You will need to create a connection string for each request and eliminate duplicates to reduce the number of requests made. You will also need a function to parse the request-response data.
# CREATE REQUEST AND REMOVE DUPLICATEScreateRequest  <- function(id){  paste0(connectionURL,         "?key=",youtubeAPIKey,         "&id=",id,         "&fields=","items(id,snippet(channelId,title,description,categoryId))",         "&part=","snippet")}uniqueWatchedVideoIDs <- unique(watchedVideosDataFrame$id)requests <- pblapply(uniqueWatchedVideoIDs, createRequest )# PARSE OUT RESPONSEgetMetadataDataFrame <- function(response){  rawchar <- rawToChar(response$content)  parsedData <- fromJSON(rawchar)  data.frame <- cbind(id = parsedData$items$id, parsedData$items$snippet)  return(data.frame)}
You can configure what to do in case the request is successful or in case it has failed.
videoMetadataDataFrame <- data.frame(id = c(),                                        channelId = c(),                                        title = c(),                                        description = c(),                                        categoryId = c()                                       )# SUCCESSaddToMetadataDataFrame <- function(response){  .GlobalEnv$videoMetadataDataFrame <- rbind(.GlobalEnv$videoMetadataDataFrame,getMetadataDataFrame(response))}# FAILfailFunction <- function(request){  print("fail")}
A method, although slower but more reliable to obtain your data, is to take it from memory and configure our multiple requests.
# GRAB REQUEST RESPONSE FROM MEMORYfetchMetadataFromMemory <- function(request){  return(getMetadataDataFrame(curl_fetch_memory(request)))}system.time(out <- multi_run(pool = pool)) saveRDS(videoMetadataDataFrame, file = "videoMetadataDataframeAsync1.rds")length(requests)nrow(videoMetadataDataFrame)listMetadata <- pblapply(requests, fetchMetadataFromMemory)
Once this is done, depending on the speed of your internet connection, and the size of your data frame, you can probably prepare a coffee, a tea, or open a beer, waiting as long as the indicator reaches 100%.
You will also use the “bind_rows” function to combine the list into an ordered Data Frame.
# COMBINE LIST INTO A DATA FRAMEvideoMetadataDataFrame <- bind_rows(listMetadata)saveRDS(videoMetadataDataFrame, file = "videoMetadataDataFrame_memory.rds")
Each category has a unique ID. You need to make another request to get them and add the data to a new column.
# CATEGORY ID REQUESTcategoryListURL <- "https://www.googleapis.com/youtube/v3/videoCategories"categoryResponse <- GET(url = categoryListURL,                         query = list(                           key = youtubeAPIKey,                           regionCode = "us",                           part = "snippet"                         ))parsedCategoryResponse <- content(categoryResponse, "parsed")categoryDataFrame <- data.frame(categoryId=c(), category=c())for(item in parsedCategoryResponse$items){  categoryDataFrame <<-rbind(categoryDataFrame,                              data.frame(categoryId = item$id, category=item$snippet$title))}categoryDataFramevideoMetadata <- merge(x = videoMetadataDataFrame, y = categoryDataFrame, by = "categoryId")head(videoMetadata)
You can combine your new Data Frame with your watch history to get the video metadata along with when it was played.
# COMBINE WITH WATCH HISTORYwatchedVideos <- merge(watchedVideosDataFrame , videoMetadata, by="id")str(watchedVideos)
With your search history, watch history, and category metadata, you can now answer this question. You can see which video categories you play the most and how this has been changing over time.
# VISUALIZE VIDEO CATEGORIES WATCHED watchedVideos %>%   group_by(category) %>%   summarise(count = n()) %>%   arrange(desc(count))watchedVideos %>%   ggplot(aes(x = time, fill = category)) +   labs(x= "Year", y= "Count") +   ggtitle("How much have your genre tastes changed over time?", "Most played categories")+  geom_area(stat = "bin") +   theme_economist_white()
For example, in my case, something that surely likes many of you is very present, are videos in the music category. And of course, the presence of the Film & Animation category makes sense, I follow many channels to find out about upcoming movies, trailers, and reviews.
It would be interesting to see the times when you are most active by giving your valuable time to YouTube. You can look at this by creating a custom function that allows you to view it as a clock.
# VISUALIZE CLOCK WATCHES PER HOURclockPlot <- function (x, col = heat.colors(n), ...) {  if( min(x)<0 ) x <- x - min(x)  if( max(x)>1 ) x <- x/max(x)  n <- length(x)  if(is.null(names(x))) names(x) <- 0:(n-1)  m <- 1.05  plot(0, type = 'n', xlim = c(-m,m), ylim = c(-m,m),        axes = F, xlab = '', ylab = '', ...)  fig <- pi/2 - 2*pi/200*0:200  polygon( cos(fig), sin(fig) )  f2 <- .02  fig <- pi/2 - 2*pi/n*0:n  segments( (1+f2)*cos(fig), (1+f2)*sin(fig), (1-f2)*cos(fig), (1-f2)*sin(fig) )  segments( cos(fig), sin(fig),0, 0, col = 'light grey', lty = 3)  f1 <- -2*pi/n*(0:50)/50  for (i in 1:n) {    fig <- pi/2 - 2*pi/n*(i-1)    b <- pi/2 - 2*pi/n*i    polygon( c(0, x[i]*cos(fig+f1), 0), c(0, x[i]*sin(fig+f1), 0), col=col[i] )    f2 <- .1    text((1+f2)*cos(fig), (1+f2)*sin(fig), names(x)[i])  }}clockDataFrame <- watchedVideos %>%   mutate(hour = hour(time)) %>%   group_by(hour) %>%   summarise(count = n()) %>%   arrange(hour)clockPlot(clockDataFrame$count, main = "What hours do you spend the most time watching YouTube?")
Raise your hands, how many more here, like me, to be able to overcome insomnia sometimes spend hours at dawn watching YouTube? In my case, it is very noticeable that mostly from 2 am to 3 am I give the platform my valuable time.
Another interesting piece of information that you can obtain is the top 10 of the videos that you have played the most over and over again. You can visualize this with a simple table, with the package “kableExtra” and order it in descending order concerning the number of reproductions.
# TABLE MOST RE-WATCHED VIDEOSw1 <- watchedVideos %>%  mutate(year = year(time)) %>%   group_by(year, title) %>%   summarise(count = n()) %>%   arrange(year, desc(count)) %>%   top_n(5)mostReWatched <- knitr::kable(x = head(arrange(w1, desc(count)) %>%                                      select(year, title, ,count), 10),                                      col.names = c('Year', 'Video Title', 'Count'))kable_styling(mostReWatched, "striped", position = "left", font_size = 12)
You will get a table like the following. In my case, I find it curious to see that from 2016 to 2019 some video does not appear in my top 10. I suppose the reason is the arrival of Spotify in my life, and that before this I used mostly YouTube to play music. By the way, yes, I’m a fan of “Vicentico” and “Los Fabulosos Cadillacs” (former vocalist of, and one band of the most iconic rock/ska bands in Spanish of all time).
You can visualize this more attractively with the “wordcloud” package. It depends on how long your history is, of course. You can establish for example words with a minimum of 25 times repeating themselves.
# WORDCLOUD MOST SEARCHED WORDSmyWords <- youtubeSearchDataFrame %>%  unnest_tokens(word, search) %>%  anti_join(stop_words) %>%  count(word, sort = TRUE)myWordcloud <- myWords %>%  group_by(word) %>%  summarize(count = sum(n)) %>%  anti_join(stop_words)wordcloud(words = myWordcloud$word, freq = myWordcloud$count, min.freq = 25,           max.words = 100, random.order =FALSE, rot.per =.35,          colors=brewer.pal(9, "Set1"))
You will get a plot like the following one as a result, where you can discover the congruence with your tastes. If you are wondering, the answer is yes, I believe in the UFO phenomenon (OVNI in Spanish).
To answer this question, you can do the same as in the previous example, but this time with the descriptions of the videos, taking advantage of the fact that we have this information in our Data Frame. It is very common that by making advertising many videos in their descriptions they include URLs, we are going to filter this (or as much as possible), in addition to words that do not give us relevant information such as articles, pronouns, etc. This time we will do it with “wordcloud2” and words that are repeated at least more than 250 times.
# WORDCLOUD MOST FREQUENT WORDS IN VIDEO DESCRIPTIONSdescriptionsWordcloud <- watchedVideos %>%  unnest_tokens(word, description) %>%  anti_join(stop_words) %>%  count(word, sort = TRUE) %>%  filter(! word %in% c("www.instagram.com", "gmail.com", "www.twitter.com", "youtu.be", "como", "instagram", "instagram.com", "tú", "watch", "aquí", "pero", "su", "http", "al","se","si","goo.gl","smarturl.it","facebook","video","más", "twitter", "te","lo","este","tu", "para", "por", "con", "es", "del", "las", "una", "mi", "de", "en", "la", "el", "los", "https", "bit.ly" , "â", "www.youtube.com")) %>%  filter(n > 250)wordcloud2(descriptionsWordcloud)
As you can see, many of the words refer to subscribing to their channels or following their social networks, as expected
Thanks for your kind reading. Here you can find the complete code: https://github.com/cosmoduende/r-youtube-personal-history-analysis
I thank you for having come this far, I wish you have a happy analysis, that you can put it into practice, and be as surprised and amused as I am with the results!
Some rights reserved