Scraping Reviews from Google Maps
Tags: Python, Web Scraping, Google Maps, Easy, Data Science

One of the common factors that we consider before visiting a place now a day is what the others say about it. At least, once in your life, you have seen the reviews of places like restaurants, ice cream stores, pizzas stores, etc…

So, let’s talk about how to scrap the reviews from multiple places in a results list of a certain search.

To extract reviews from Gmaps, we must need BeautifulSoup (bs4), Selenium, intuition, and a bit of HTML knowledge, because we are going to navigate with python through Gmaps as well as we usually do it manually within the site.

This purpose is merely educational and not for profit.

Google Maps is an asynchronous web page, i.e., the web application process any request to the server in the background.

This is the actual view when you search any place in Google Maps, on the right side is the map and on the left side are all the search results.

As I mentioned before, we will use various modules for our code.

>from selenium import webdriver

>from webdriver_manager.chrome import ChromeDriverManager

>from bs4 import BeautifulSoup

>from tqdm import tqdm

>import time, re, os

>import pandas as pd

Notice that if you don’t have these modules, you can easily install them by pip vias.

Then we open our web driver that will be manipulated by Python. and we add 5 seconds of delay before exploring the web content.

>browser = webdriver.Chrome(ChromeDriverManager().install())

>browser.implicitly_wait(5)

Now, let’s input the URL of any search, in my case I will use the link of the previous image.

>browser.get(‘https://www.google.com.mx/maps/search/restaurants/@40.7727474,-73.9739719,15z/data=!3m1!4b1')

Perfect, We are in the same view with my search results on the left.

To collect all the head names of the search results, we need to know where do we can find them, I will show you how to look them up. just press F12 or right-click and Inspect. you will see the page source, then click on ‘Select an element in the page’ and choose one element of the results. I will select the title of an element of the results list.

We can look that the title is inside the class name “section-result-header-container”, we write the next lines to get all the titles from all the elements in the search results and we save them in a list variable.

>review_titles = browser.find_elements_by_class_name(“section-result-header-container”)

>title_elements=[a.text[:-len(a.text)+a.text.find(‘

’)] for a in review_titles]

The variable title_elements will have a list with all the names of the places of our results list like this: [‘Robert’,’Daniel’,’Asiate’,…]

In the same way, we can extract the details of each element, the location, and also the ranking price by finding the class names of the elements that contain this information on the web page.

>loc_rest = browser.find_elements_by_class_name(“section-result-location”)

>locrest_all=[a.text for a in loc_rest] >det_rest = browser.find_elements_by_class_name(“section-result-details”)

>detrest_all=[a.text for a in det_rest]

After we get all the relative information from each element of the results list, we will navigate inside the elements, We need to use our logic in this part because is the part that changes frequently. but we can solve it if we follow the pattern.

If we explore the Xpaths of each element, we will have the next.

/html/body/jsl/div[3]/div[9]/div[7]/div/div[1]/div/div/div[4]/div[1]/a[1] /html/body/jsl/div[3]/div[9]/div[7]/div/div[1]/div/div/div[4]/div[1]/a[2] /html/body/jsl/div[3]/div[9]/div[7]/div/div[1]/div/div/div[4]/div[1]/a[3] /html/body/jsl/div[3]/div[9]/div[7]/div/div[1]/div/div/div[4]/div[1]/a[4]

We can see that only change the last part, the ‘a’ group. so we can get inside each element by changing the last part.

>for x in tqdm(range(len(restaurante)),unit= ‘ Elements‘, desc=’Scrapping Page [1]…’, position=0):

> inside_element = browser.find_element_by_xpath(‘/html/body/jsl/div[3]/div[9]/div[7]/div/div[1]/div/div/div[4]/div[1]/a[‘+str(x+1)+’]’)

> inside_element.click()

The code above will click each element for here you have to have care because when you click one element, you will be inside and for the next element of the loop will find the next path but an error will occur because the page view has changed. This is a view inside each element.

To avoid the error you have to turn back to search results by finding ‘Back to results’ is located above the panel info and below the search bar. Then I will have other two new lines in my loop.

>for x in tqdm(range(len(restaurante)),unit= ‘ Elements‘, desc=’Scrapping Page [1]…’, position=0):

> inside_element = browser.find_element_by_xpath(‘/html/body/jsl/div[3]/div[9]/div[7]/div/div[1]/div/div/div[4]/div[1]/a[‘+str(x+1)+’]’)

> inside_element.click()

> return_results= browser.find_element_by_xpath(‘/html[1]/body[1]/jsl[1]/div[3]/div[9]/div[7]/div[1]/div[1]/div[1]/div[1]/button[1]/span[1]’)

> return_results.click()

At this point, we can get inside and then outside each element.

The next step is to find the button review, the easy way is to find the CSS selector button review, A simple tool that will help you is ChroPath. It is an extension that lets you see more details when you are searching for an element on the page, you just expand more the inspector and then the second inspector and in the end, you will see ChroPath.

To use this extension, just do a simple element search like the picture below.

In the CroPath tab, you will see the real Xpath, CssSelector, Absolute Xpath, and Tag name of the review button. We are more interested in CssSelector, if we copy it, we will have something like this:

body.keynav-mode-off.screen-mode:nth-child(2) div.vasquette.pane-open-mode div.widget-pane.widget-pane-visible div.widget-pane-content.scrollable-y:nth-child(1) div.widget-pane-content-holder div.section-layout.section-layout-root div.jqnFjrOWMVU__root:nth-child(33) div.jqnFjrOWMVU__row div.jqnFjrOWMVU__right > button.gm2-button-alt.jqnFjrOWMVU__button-blue

The bold part is the CssSelector we need to click. Unfortunately, Google Maps website is mainly implemented using AJAX, not all elements are loaded if Selenium looks for them immediately after loading a page, that is why we will use a sentence to execute JavaScript to click on the CssSelector.

The code below will find de CssSelector and then will click it.

>button_review_altern=browser.find_element_by_css_selector(‘button.gm2-button-alt.jqnFjrOWMVU__button-blue’)

>browser.execute_script(“arguments[0].click();”, button_review_altern)

And now we are in the reviews part, and to navigate from one element to another is necessary to find the turn back button that is at the bottom, beside ‘All reviews’.

Notice that we will have more reviews if we scroll down more, so, the next step is to find the CssSelector for the scrollbar as same as we did for the review button using ChroPath.

body.keynav-mode-off.screen-mode:nth-child(2) div.vasquette.pane-open-mode div.widget-pane.widget-pane-visible div.widget-pane-content.scrollable-y:nth-child(1) div.widget-pane-content-holder div.section-layout.section-layout-root > div.section-layout.section-scrollbox.scrollable-y.scrollable-show

Once we get the CssSelector, we proceed to execute a JavaScript sentence again.

>scrollable_div=browser.find_element_by_css_selector(‘div.section-layout.section-scrollbox.scrollable-y.scrollable-show’)

>browser.execute_script(‘arguments[0].scrollTop = arguments[0].scrollHeight’,scrollable_div)

Now you can scroll as many times you want, you can do an iterable task to scroll down for n-times.

After all n-times scrolls down, It’s time to collect the usernames, the rates, and the reviews.

We apply the same technique that I showed you before, We have to find the class names that contain the respective data information.

Here, the class name for the usernames is ”section-review-title”.

The class name for the ratings is “section-review-title”.

For the review text, we will find it with the class name “section-review-text”.

We finally have the data we want to extract, let's save the whole data in different lists.

> names=browser.find_elements_by_class_name(“section-review-title”)

> all_names=[a.text for a in names]

> date_rate=browser.find_elements_by_class_name(‘section-review-publish-date’)

> daterate_all=[a.text for a in date_rate]

> revtext=browser.find_elements_by_class_name(‘section-review-text’)

> revtext_all=[a.text for a in revtext]

> content= browser.page_source

> soup=BeautifulSoup(content)

> rating = soup.find_all(‘span’, class_=’section-review-stars’)

> ratings_all=[]

> for y in range(len(rating)):

> ratings_all.append(re.search(‘\d’,rating[y][‘aria-label’]).group())

Now, our code is completed to scrap any search in Google Maps, do not forget to save the scraped data into a Data Frame.

> title_elements2=[]

> lloc=[]

> ddet=[]

> for y in range(len(all_names)):

> title_elements2.append(title_elements[x])

> lloc.append(locrest_all[x])

> ddet.append(detrest_all[x]) >all_data.append(pd.DataFrame({‘Search’:title_elements2,’Location’:lloc,’Details’:ddet,’Name’:all_names,’Rating’:ratings_all,’Date Rate’:daterate_all,’Review’:rev_text_all2}))

Here is the output data after we scraped the first element.

Thanks for reading! :)

BR.

Dante Rosey Salazar