Scrape Google Search Results using Python BeautifulSoup

29 Dec, 2020
In this article, we are going to see how to Scrape Google Search Results using Python BeautifulSoup.
bs4: Beautiful Soup(bs4) is a Python library for pulling data out of HTML and XML files. This module does not come built-in with Python. To install this type the below command in the terminal.
pip install bs4

requests:  Requests allows you to send HTTP/1.1 requests extremely easily. This module also does not come built-in with Python. To install this type the below command in the terminal.
pip install requests

Import the beautifulsoup and request libraries.
Make two strings with the default Google search URL, ‘https://google.com/search?q=’ and our customized search keyword.
Concatenate these two strings to get our search URL.
Fetch the URL data using requests.get(url), store it in a variable, request_result.
Create a string and store the result of our fetched request, using request_result.text.
Now we use BeautifulSoup to analyze the extracted page. We can simply create an object to perform those operations but beautifulsoup comes with a lot of in-built features to scrape the web. We have created a soup object first using beautifulsoup from the request-response
 We can do soup.find.all(h3) to grab all major headings of our search result, Iterate through the object and print it as a string.
Example 1: Below is the implementation of the above approach.
Python3
# Import the beautifulsoup # and request libraries of python.import requestsimport bs4  # Make two strings with default google search URL# 'https://google.com/search?q=' and# our customized search keyword.# Concatenate themtext= "geeksforgeeks"url = 'https://google.com/search?q=' + text  # Fetch the URL data using requests.get(url),# store it in a variable, request_result.request_result=requests.get( url )  # Creating soup from the fetched requestsoup = bs4.BeautifulSoup(request_result.text,                         "html.parser")print(soup)
Output:
Let’s We can do soup.find.all(h3) to grab all major headings of our search result, Iterate through the object and print it as a string.
Python3
# soup.find.all( h3 ) to grab # all major headings of our search result,heading_object=soup.find_all( 'h3' )  # Iterate through the object # and print it as a string.for info in heading_object:    print(info.getText())    print("------")
Output:
Example 2: Below is the implementation. In the form of extracting the city temperature using Google search:
Python
# import moduleimport requests import bs4   # Taking thecity name as an input from the usercity = "Imphal"  # Generating the url  url = "https://google.com/search?q=weather+in+" + city  # Sending HTTP request request_result = requests.get( url )  # Pulling HTTP data from internet soup = bs4.BeautifulSoup( request_result.text                          , "html.parser" )  # Finding temperature in Celsius.# The temperature is stored inside the class "BNeawe". temp = soup.find( "div" , class_='BNeawe' ).text     print( temp ) 
Output:
Python BeautifulSoup
Python web-scraping-exercises
Technical Scripter 2020
Web-scraping
Python
Technical Scripter
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
How to iterate through Excel rows in Python?
Deque in Python
Defaultdict in Python
Queue in Python
Rotate axis tick labels in Seaborn and Matplotlib
Check if element exists in list in Python
Python Classes and Objects
Bar Plot in Matplotlib
Python OOPs Concepts
How To Convert Python Dictionary To JSON?