Scraping Amazon Product Information using Beautiful Soup

07 Oct, 2021
Web scraping is a data extraction method used to exclusively gather data from websites. It is widely used for Data mining or collecting valuable insights from large websites. Web scraping comes in handy for personal use as well. Python contains an amazing library called BeautifulSoup to allow web scraping. We will be using it to scrape product information and save the details in a CSV file.
In this article, Needed the following are prerequisites.
url.txt: A text file with few urls of amazon product pages to scrape
Element Id: We need Id’s of objects we wish to web scrape, Will cover it soon...
Here, our text file looks like.
Module needed and installation:
BeautifulSoup: Our primary module contains a method to access a webpage over HTTP.
pip install bs4
lxml: Helper library to process webpages in python language.
pip install lxml
requests: Makes the process of sending HTTP requests flawless.the output of the function
pip install requests
Approach:
First, we are going to import our required libraries.
Then we will take the URL stored in our text file.
We will feed the URL to our soup object which will then extract relevant information from the given URLbased on the element id we provide it and save it to our CSV file.
Let’s have a look at the code, We will see what’s happening at each significant step.
Step 1: Initializing our program.
We import our beautifulsoup and requests, Create/Open a CSV file to save our gathered data. We declared Header and added a user agent. This ensures that the target website we are going to web scrape doesn’t consider traffic from our program as spam and finally get blocked by them. There are plenty of user agents available here.
Python3
from bs4 import BeautifulSoupimport requests File = open("out.csv", "a") HEADERS = ({'User-Agent':           'Mozilla/5.0 (X11; Linux x86_64)                AppleWebKit/537.36 (KHTML, like Gecko)                    Chrome/44.0.2403.157 Safari/537.36',                           'Accept-Language': 'en-US, en;q=0.5'}) webpage = requests.get(URL, headers=HEADERS)soup = BeautifulSoup(webpage.content, "lxml")
Step 2: Retrieving element Ids.
We identify elements by seeing the rendered web pages, but one can’t say the same for our script.  To pinpoint our target element, we will grab its element id and feed it to the script. 
Getting the id of an element is pretty simple. Suppose I need the element id of the products name, All I have to do 
Get to the URL and inspect the textIn the console, we grab the text next to id=
Get to the URL and inspect the text
In the console, we grab the text next to id=
copy the element id
We feed it to soup.find and convert the function’s output into a string. We remove commas from the string so that it won’t interfere with the CSV try-except writing format. 
Python3
try:        title = soup.find("span",                          attrs={"id": 'productTitle'})       title_value = title.string         title_string = title_value                    .strip().replace(',', '')           except AttributeError:         title_string = "NA"         print("product Title = ", title_string)
Step 3: Saving current information to a text file
We use our file object and write the string we just captured, and end the string with a comma “,” to separate its column when it’s interpreted in a CSV format.
Python3
File.write(f"{title_string},")
Doing the above 2 steps with all of the attributes we wish to capture from weblike Item price, availability etc.
Step 4: Closing the file.
Python3
File.write(f"{available},\n") # closing the fileFile.close()
While writing the last bit of information, notice how we add “\n” to change the line. Not doing so will give us all the required information in one very long row. We close the file using File.close(). This is necessary, if we do not do this we might get an error next time we open the file.
Step 5: Calling the function we just created.
Python3
if __name__ == '__main__':  # opening our url file to access URLs    file = open("url.txt", "r")     # iterating over the urls    for links in file.readlines():        main(links)
We open the url.txt in reading mode and iterate over each of its lines until we reach the last one. Calling the main function on each line.
This is how our entire code looks like:
Python
# importing librariesfrom bs4 import BeautifulSoupimport requests def main(URL):    # opening our output file in append mode    File = open("out.csv", "a")     # specifying user agent, You can use other user agents    # available on the internet    HEADERS = ({'User-Agent':                'Mozilla/5.0 (X11; Linux x86_64)                    AppleWebKit/537.36 (KHTML, like Gecko)                            Chrome/44.0.2403.157 Safari/537.36',                                'Accept-Language': 'en-US, en;q=0.5'})     # Making the HTTP Request    webpage = requests.get(URL, headers=HEADERS)     # Creating the Soup Object containing all data    soup = BeautifulSoup(webpage.content, "lxml")     # retrieving product title    try:        # Outer Tag Object        title = soup.find("span",                          attrs={"id": 'productTitle'})         # Inner NavigableString Object        title_value = title.string         # Title as a string value        title_string = title_value.strip().replace(',', '')     except AttributeError:        title_string = "NA"    print("product Title = ", title_string)     # saving the title in the file    File.write(f"{title_string},")     # retrieving price    try:        price = soup.find(            "span", attrs={'id': 'priceblock_ourprice'})                                .string.strip().replace(',', '')        # we are omitting unnecessary spaces        # and commas form our string    except AttributeError:        price = "NA"    print("Products price = ", price)     # saving    File.write(f"{price},")     # retrieving product rating    try:        rating = soup.find("i", attrs={                           'class': 'a-icon a-icon-star a-star-4-5'})                                    .string.strip().replace(',', '')     except AttributeError:         try:            rating = soup.find(                "span", attrs={'class': 'a-icon-alt'})                                .string.strip().replace(',', '')        except:            rating = "NA"    print("Overall rating = ", rating)     File.write(f"{rating},")     try:        review_count = soup.find(            "span", attrs={'id': 'acrCustomerReviewText'})                                .string.strip().replace(',', '')     except AttributeError:        review_count = "NA"    print("Total reviews = ", review_count)    File.write(f"{review_count},")     # print availablility status    try:        available = soup.find("div", attrs={'id': 'availability'})        available = available.find("span")                    .string.strip().replace(',', '')     except AttributeError:        available = "NA"    print("Availability = ", available)     # saving the availability and closing the line    File.write(f"{available},\n")     # closing the file    File.close()  if __name__ == '__main__':  # opening our url file to access URLs    file = open("url.txt", "r")     # iterating over the urls    for links in file.readlines():        main(links)
Output:
product Title = Dremel DigiLab 3D40 Flex 3D Printer w/Extra Supplies 30 Lesson Plans Professional Development Course Flexible Build Plate Automated 9-Point Leveling PC & MAC OS Chromebook iPad Compatible Products price = $1699.00 Overall rating = 4.1 out of 5 stars Total reviews = 40 ratings Availability = In Stock. product Title = Comgrow Creality Ender 3 Pro 3D Printer with Removable Build Surface Plate and UL Certified Power Supply 220x220x250mm Products price = NA Overall rating = 4.6 out of 5 stars Total reviews = 2509 ratings Availability = NA product Title = Dremel Digilab 3D20 3D Printer Idea Builder for Brand New Hobbyists and Tinkerers Products price = $679.00 Overall rating = 4.5 out of 5 stars Total reviews = 584 ratings Availability = In Stock. product Title = Dremel DigiLab 3D45 Award Winning 3D Printer w/Filament PC & MAC OS Chromebook iPad Compatible Network-Friendly Built-in HD Camera Heated Build Plate Nylon ECO ABS PETG PLA Print Capability Products price = $1710.81 Overall rating = 4.5 out of 5 stars Total reviews = 351 ratings Availability = In Stock. 
Here’s how our out.csv looks like.
adnanirshad158
simmytarika5
arorakashish0911
Python web-scraping-exercises
Web-scraping
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.