A Short Guide to Web Scraping with Python
By: Haider Imtiaz
Tags: Python, Web Scraping, Scraping, Data Mining, Web Development

Web scraping is a tool which is used to Scrap the important data from the Website, Take a simple example You want to Scrap the Amazon.com for analyzing the cheapest price of Laptops. You need to Surf every page of amazon to collect the data which consume a lot of time right, Why not do this job with the help of Web scraper.

In this is an article we gonna Scrap a website with the help of Python.

Install the Required Libraries:

1:BeautifulSoup

2:Request

Our Target is to Scrap the Names of President of America from wikipedia.com.

Import the Both libraries in the Python Script. To import the Library we gonna use import keyword, Take a Look on the Below Code!

Now its time for Real Coding, Here is the Link of Our Target Site Wikipedia, I'm gonna Create a Link Variable and store the site Url in it. After that, we gonna get the HTML data of the target website with help of the requests library. Take a look at the Code below.

We send a request to Wikipedia that we need HTML data, and by using request.get() we get HTML raw data as a response.

Beautiful Soup in Action:

We get Html Data from Wikipedia and we store this data in the ‘r’ variable. If you try to print the ‘r’, you get output in the Raw order, to fix it we gonna use BeautifulSoup library, Take a look at the Code below.

We use Beautifulsoup() function and in this parenthesis, I include ‘r’ which store our Html Raw data, and ‘html5lib’ this is keyword used to reformat the HTML raw format into HTML5 format, similarly, you can also use ‘lxml’ for converting the format of raw data into HTML format.

Lets Scraping Begin:

BeautifulSoup provide us so many tools to scrape the data from the internet, so our target is to find all the name of the President of American, So Goto to Wikipedia Link, Press F12 from your keyboard to access the inspection tool on Google Chrome.

Scarping the data require a basic understanding of the Html tags, So we analyze that the name of the President is written in Bold and big Font by using <big> tag, and if you search the big in the find field you will notice that only President names are written in this tag, Lets target this tag in our Script. Take a Look at the Code Below.

Now in the code, I create an empty LIST to store Scrap data in it, Next, I create iteration with Loop, the basic purpose of creating a loop so we find the tag and scrap the data from it one by one, and next, I used find_all function which is Beautifulsoup amazing tool, it is used to find a string or tag in the HTML data. Well in the find_all() I write ‘big’ so when our Script run the find_all function will find the matching string or tag in our HTML data, and in the Indentation of the loop we gonna add the Scrap name in to the LIST by using append(), But why I use get_text(), well this is the Beautifulsoup function which is used to find the Text in the Scrap data, When we scrap the <big> we get data in the form like this

<a href=”/wiki/George_Washington” title=”George Washington”>George Washington</a>

Check out the Google inspection to know about it.

Well, we don't need to get this whole link data we need only the name so in this case, I use the get_text() function.

Finally, We get the Scrap data you can Print it out on the console Screen by typing print(names).

Conclusion

This is a simple Guide to Scrap Data from the Website by using Python Programming and BeautifulSoup Library. Website Scraping Depends on your Logic how you choose the right tag and classes to scrap the data.

If you like this Article helpful Please help the Arthur by Clicking on the 👏 Clap emoji