Explore your activity on Google with R: How to Analyze and Visualize Your Search History | by Saúl Buentello | Towards Data Science

Can you imagine a world without the most popular search engine in the world? I bet, dear reader, that if you are between the ages of 18 and 25, touching a dictionary or encyclopedia is not a part of everyday life for you, as it was for many of us in our youth.
Finding answers, knowledge, information (and misinformation), is just a click away nowadays within the reach of anyone. That is why I find it interesting to be able to take a look at how our personal consumption habits have changed, especially if you have a long history since you opened the doors to Google to find out everything about you.
Thanks to Google Takeout, a tool that Google provides to consult your data stored in any of the products that you have used, you can consult your search history. You have to enter the URL https://takeout.google.com/settings/takeout to log in with your personal account.
You can request a complete copy of your data, or by-product, or select only some characteristics of a single product. For us, for purposes of this article, it is sufficient to select “My Activity”.
I also recommend that you make sure to uncheck the rest of the options it contains to only leave the “Search” option, to reduce the weight and delivery time of the file it will generate.
Once this is done, you will have to select “export once” and choose the .zip delivery format.
And voila, now you just have to be patient, waiting for the copy of the content you have selected to be generated.
When your file is generated, you will receive a notification email inviting you to download it, with an expiration date to be able to do so before the specified date, for security reasons.
When you unzip the .zip you downloaded, you will find a file named “MyActivity.html”.
This HTML file contains the complete data of your search history in Google, from when you started using the search engine for the first time, until the moment when you generated this data copy with Google Takeout.
Now you can proceed to work on the information obtained in a new R script. First, you must include all the packages that you will use, and establish the reading of your search history, contained in the file “MyActivity.html” that is inside the “Search” folder, contained in the “My Activity” folder.
# REQUIRED LIBRARIESlibrary(wordcloud)library(lubridate)library(rvest)library(tm)library(tidyverse)# READ DATAfileHTML <- "Takeout/My Activity/Search/MyActivity.html"mySearchFile <- read_html(fileHTML, encoding = "UTF-8")
You will extract the data that interests you and you will do web scrapping of the HTML, using the “rvest” package. In this way, using regular expressions, you can extract the date and time of your searches, the searched text, and the type of search you did on Google, to create a data frame with all this information.
# SCRAPPING SEARCH DATE AND TIMEdateSearch <- mySearchFile %>%   html_nodes(xpath = '//div[@class="mdl-grid"]/div/div') %>%   str_extract(pattern = "(?<=<br>)(.*)(?<=PM|AM)") %>%  mdy_hms()dateSearch[1:5]# SCRAPING SEARCH TEXTtextSearch <- mySearchFile %>%   html_nodes(xpath = '//div[@class="mdl-grid"]/div/div') %>%  str_extract(pattern = '(?<=<a)(.*)(?=</a>)') %>%   str_extract(pattern = '(?<=\">)(.*)')textSearch[1:5]# SCRAPING SEARCH TYPEsearchType <- mySearchFile %>%   html_nodes(xpath = '//div[@class="mdl-grid"]/div/div') %>%   str_extract(pattern = "(?<=mdl-typography--body-1\">)(.*)(?=<a)") %>%   str_extract(pattern = "(\\w+)(?=\\s)")searchType[1:5]# CREATE DATA FRAME USING SCRAPED DATAsearchedData <- tibble(timestamp = dateSearch,                      date = as_date(dateSearch),                      year = year(dateSearch),                      month = month(dateSearch, label = TRUE),                      day = weekdays(dateSearch),                      hour = hour(dateSearch),                      type = searchType,                      search = textSearch)searchedData$day <- factor(searchedData$day, levels = c("Sunday", "Monday", "Tuesday", "Wednesday","Thursday", "Friday", "Saturday"))searchedData <- na.omit(searchedData)head(searchedData)
As a result, you will get a new data frame with 8 variables.
This could be the first question that you can answer. With the information obtained, you can view the number of searches carried out concerning the years of your history.
# PLOT SEARCHED BY YEARsearchByYear <- ggplot(searchedData, aes(year, fill=..count..)) +  scale_fill_gradient(low = "yellow", high = "red")+  geom_bar(width=0.7)+  labs(x= "Year", y= "Count") +   ggtitle("How much your search frequency has changed over time", "Search activity by year")searchByYearggplotly()
In my case, for example, my history goes from 2007 to today, in 2020. That is, 13 years of searches are represented in volume as follows. There is a very notable increase, mainly in comparison with years before 2014, in which I continued to use the Yahoo search engine mostly, also it seems to me that it must be taken into consideration that access to increasingly faster and faster Internet connections also facilitated the possibility of making more and more queries to Google.
You can dig a little deeper into the results previously obtained, now visualizing how much the frequency with which you have searched on Google has changed, detailed by months over the years of age that your history has.
# PLOT SEARCH BY MONTHsearchByMonth <- searchedData[(searchedData$year > 2007 & searchedData$year< 2021), ]ggplot(searchByMonth, aes(year, fill=..count..)) +   scale_fill_gradient(low = "yellow", high = "red")+  geom_bar(aes(x = month, group = year)) +  theme(axis.text.x = element_text(angle=90)) +  facet_grid(.~year, scales="free") +   labs(x= "Year / Month", y= "Count") +   ggtitle("How much your search frequency has changed over time", "Month activity on detail")
I am almost sure, dear reader, that like me, if you look carefully, a lot had to do with the increase in the number of searches carried out concerning the worst months in which the pandemic covid-19 in your country forced you to stay more at home, and therefore with your devices such as phones, tablets, and computers with internet access. For example in my case, I do not think it is a coincidence that the maximum records plotted in 2020 coincide with the red traffic light (closure of all non-essential establishments and remote work from home) established in the country where I live (Mexico).
You can get this information as well since you have a record of the exact time you searched with the popular Google engine.
# PLOT SEARCH BY HOURsearchByHour <- ggplot(searchedData, aes(hour, fill=..count..)) +  scale_fill_gradient(low = "yellow", high = "red") +  geom_bar() +   labs(x= "Hour", y= "Count") +   ggtitle("What time of day do you have the highest frequency of searches?", "Hour activity on detail")searchByHour
You will get the following plot as a result, where, for example, in my case, it is notable that I am mostly not a Google user at dawn.
Following the previous example, you can also visualize which days of the week register the highest number of searches performed by you.
# PLOT SEARCH BY WEEKDAYseearchByWeekD <- ggplot(searchedData, aes(day, fill=..count..)) +   scale_fill_gradient(low = "yellow", high = "red") +  geom_bar() +  labs(x= "Day", y= "Count") +   ggtitle("What day of the week do you have the highest frequency of searches?", "Weekday activity on detail")seearchByWeekD
It is to be expected that if, like me, you occupy your weekends to get away a little from staying online, to go see your friends, for example, the decrease in search volume will be noticeably visible.
Taking up the last two plots obtained, where you can see the time and day of the week in which you register the most searches in Google, you can create a new plot to see in detail the relationship between the two.
# PLOT SEARCH BY WEEKDAY AND TIME searchWdayTime <- ggplot(searchedData) +   scale_fill_gradient(low = "yellow", high = "red")+  geom_bar(aes(x = hour, group = day, fill=..count..) ) +  labs(x= "Hour / Day", y= "Count") +   ggtitle("Relationship between day / time you have a higher frequency of searches", "Weekday/Time activity on detail") +  facet_grid(.~day, scales = "free")searchWdayTime
In this way, you will get a plot like the following. You could also cross data with other variables that you already used such as months and years.
This is another interesting question that you can answer by viewing the data using the “wordcloud” package. First, you will have to extract the terms to later clean them and thus be able to create a Text Corpus. You should also remove words that do not provide relevant information such as articles and pronouns.
# CLEAN AND EXTRACT TEXT TO CREATE A TEXT CORPUSlastTwoYears <- searchedData[(searchedData$year > 2007 & searchedData$year< 2010), ]search <- tolower(lastTwoYears$search)search <- gsub('(http|https)\\S+\\s*|(#|@)\\S+\\s*|\\n|\\"', " ", search)search <- gsub("(.*.)\\.com(.*.)\\S+\\s|[^[:alnum:]]", " ", search)search <- trimws(search)textCorpus <-  Corpus(VectorSource(search))textCorpus <- tm_map(textCorpus, content_transformer(removePunctuation))textCorpus <- tm_map(textCorpus, content_transformer(removeNumbers))stopwords <- c(stopwords("english"), "que", "com", "cómo", "como", "para", "con", "qué", "las", "los", "del", "can")textCorpus <- tm_map(textCorpus, removeWords, stopwords)searchTDM <- TermDocumentMatrix(textCorpus)searchMatrix <- as.matrix(searchTDM)
Finally, you can create a new Data Frame to be able to visualize your cloud of most searched terms in Google for the last two years.
# CREATE DATA FRAME WITH WORDSarrange <- sort(rowSums(searchMatrix), decreasing = TRUE)twNames <- names(arrange)dataCloud <- data.frame(word = twNames, freq = arrange)wordcloud(dataCloud$word, dataCloud$freq, min.freq = 40, scale = c(2 , 0.5), max.words = 100, colors=brewer.pal(9, "Paired"))
You should then get your wordcloud with the terms you searched for at least 40 times or more. This is my cloud of most searched terms from 2018 to 2020. Do not judge me for searching so much “torrent” please hehe.
As a bonus, you can repeat the exercise to find out what the most searched terms were during the first two years that you allowed Google to enter your life to know everything about you. You only need to modify the range of years covered in the lastTwoYears variable. In my case, for example, from 2008 to 2010 (years of a lot of Flash and forums where to find answers about web development), this is my wordcloud. Obviously, the search volume was much lower compared to the previously generated wordcloud.
Thank you very much for your kind reading. As with most of my articles, I share the plots generated with plotly in a flexdashboard that I put together a little more aesthetic: https://rpubs.com/cosmoduende/google-search-history-analysis
Here you can find the complete code: https://github.com/cosmoduende/r-google-search-history-analysis
I thank you for making it to the end, I wish you a very happy analysis, that you can put everything into practice, and that you are surprised and have as much fun as I am with the results!
Some rights reserved