How to scrape data using Python?
By: Siva Santosh S
Tags: Python Web Scraper, Python, Web Scraping, Beautifulsoup, Requests Library

See the above image, when you click the URL mentioned above you will get redirected to this page. We have to go to the developer tools.In my case, I’m using the chrome, (CTRL + SHIFT + I ) is sufficient to go to the developer tools. Once you are in it, just keep the cursor on the text or numerical data you want to extract. Here while scraping, I noted few things which I want to extract that are :

About us Title Description of About us Management Team Title Names of the Management Team Description of Management Team

All the HTML content which is extracted is stored in the variable named Soup, in that Soup object, we want to extract the data which is more relative to us. In our case, the first and the foremost thing I want is About us (Title) which is present in the ‘class’ = ‘Red XCN’.

RedbusAbout = Soup.findAll(attrs= {‘class’: ‘Red XCN’})[0].text

In the above code snippet, I’m finding all the attributes which have the class name as Red XCN and extracting the most related data I want from the Soup object. For instance, if I run this code.

Soup.findAll(attrs= {‘class’: ‘Red XCN’}) #the output will be [<h3 class="Red XCN">About us</h3>,

<h3 class="Red XCN" id="BirdsEye">Management Team</h3>]

In our case, we only want the first line of the output, because it has ‘About us’ in it. In order to extract the first line from the snippet, we are using the indexing concept, where the first line represents the 0th index here.

# here the [0], represents the Oth index, which will help us the extract only the fisrst line of the code. Soup.findAll(attrs= {‘class’: ‘Red XCN’})[0] #the output for this code is: <h3 class="Red XCN">About us</h3>

If you had observed it clearly, the above output is in the HTML form, in order to extract only the text from the above output, we are using the .text extension.

Soup.findAll(attrs= {‘class’: ‘Red XCN’})[0].text #output 'About us'

Now we had saved the whole output in the variable name RedbusAbout, I believe now you got the idea of this line.

RedbusAbout = Soup.findAll(attrs= {‘class’: ‘Red XCN’})[0].text

Similarly, we had extracted all the data, we want

AboutRedBus = Soup.findAll(attrs= {‘class’: ‘western’})[0].text ManagementTitle = Soup.findAll(attrs= {'class': 'Red XCN'})[1].text Name1 = Soup.findAll(attrs= {'class': 'Red TextBold XCN'})[0].text[0:14] Name2 = Soup.findAll(attrs= {'class': 'Red TextBold XCN'})[1].text[0:11] Name1Desc = Soup.findAll(attrs= {'class' : 'western'})[1].text Name2Desc = Soup.findAll(attrs= {'class' : 'western'})[2].text lastline = Soup.findAll('div',attrs= {'class': False, 'id' : False})[3].text.replace('

', '')

Now, try to do it on your own, if you feel you got stuck, then follow my youtube video, which demonstrates the same.

Now, if you observe the variable lastline, it doesn’t have any class or id, in that case just keep the attributes as False, like class is False (that means the class is not present) and id is False (this means id is not present), and ‘div’ means to find all the patterns with word ‘div’. In short, the whole line says that from Soup find all the lines which have ‘div’ in it and which doesn’t have the class and id attributes in them. Once we found it we had indexed it [3] and replace the ‘

’ words with the ‘ ’ empty words.