BeautifulSoup Python Library
By: Jitendra Singh Balla
Tags: Machine Learning, Python, Beautifulsoup, Data Science, Web Scraping

Let’s start scraping on real websites rather than the save html file.

To grab the source code of any website we’ve to use requests and pass the url of website as an argument. In this article we use this website ‘http://coreyms.com' just for example purpose.

from bs4 import BeautifulSoup

import requests

import csv source = requests.get('http://coreyms.com').text

soup = BeautifulSoup(source, 'lxml') print(soup.prettify()) #output will be source code of the website in HTML form.

If you want to see the structure of website page than just right click and select Inspect option you’ll be able to see the HTML code of that page.

Now for getting first article from website :

article = soup.find('article')

print(article.prettify()) #the output will be HTML structured code of the article section. #to grab the headline from the first article

headline = article.h2.a.text

print(headline) #'a' stands for anchor tag, 'h2' stands for the href tag which contains the link & the headline of the article. The output will be headline of the article in plain text.

use print(article.prettify()) again to see where the summary tag is , after that make it comment using ‘#’.

#to grab the summary from the article

summary = article.find('div', class_ = 'entry-content').p.text

print(summary) #'p' stands for paragraph and as an output we'll have whole paragraph of section 'div' tag.

Again uncomment the print(article.prettify()) and see where the video source , if you’ve any idea of YouTube videos every video have an ID. Now make comment print(article.prettify()).

#to grab the source of video

vid_src = article.find('iframe', class_ = 'youtube-player')['src']

print(vid_src) #output will be the link to the embedded video.

Now to grab the video ID:

# make comment print(vid_src). vid_id = vid_src.split('/').[4]

vid_id = vid_id.split('?').[0]

print(vid_id) #'split' will split the vid_src in '/' seprated , [4] is the index of the video id, split('?') will seprated id from further info, [0] will give the 0th index. The output will be video id.

Now we can create YouTube video link using that video id :

# make comment print(vid_id).

yt_link = f'https://youtube.com/watch?v={vid_id}'

print(yt_link) # in above code '?' stands for query, 'v' stands for video where we've given video id. The output will be link to YouTube, just copy that and paste in your browser you'll be directed to the video on YouTube.

Above was just for one article from website. Now for scraping all the articles from the website we have to use for loop to extract the video links from the website.

#to save all info into a csv file, 'cms_scrape.csv' is name and opening it to write all info in it

csv_file = open('cms_scrape.csv', 'w') #'w' stands for write csv_writer = csv.writer(csv_file)

csv_writer = csv_writerow(['headline', 'summary', 'video_link']) for article in soup.find_all('article'): headline = article.h2.a.text

print(headline) summary = article.find('div', class_ = 'entry-content').p.text

print(summary) try: #use for dealing with some missing links

vid_src = article.find('iframe', class_ = 'youtube-player')['src'] vid_id = vid_src.split('/').[4]

vid_id = vid_id.split('?').[0]



yt_link = f'https://youtube.com/watch?v={vid_id}'

except Exception as e: #use for dealing with some missing links

yt_link = None

print(yt_link) print()

csv_writer.writerow([headline, summary, yt_link

csv_file.close() #In output first thing will be headline. 2nd will be summary and 3rd will be YouTube link of videos.

BeautifulSoup’s Alternatives :

Scrapy

It is the most popular web scraping framework in Python. An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

Selenium

Selenium automates browsers. That’s it! What you do with that power is entirely up to you. Primarily, it is for automating web applications for testing purposes, but is certainly not limited to just that. Boring web-based administration tasks can (and should!) also be automated as well.

import.io

import.io is a free web-based platform that puts the power of the machine readable web in your hands. Using our tools you can create an API or crawl an entire website in a fraction of the time of traditional methods, no coding required.

ParseHub

You can extract data from anywhere. ParseHub works with single-page apps, multi-page apps and just about any other modern web technology. ParseHub can handle Javascript, AJAX, cookies, sessions and redirects. You can easily fill in forms, loop through dropdowns, login to websites, click on interactive maps and even deal with infinite scrolling.

Portia

Portia is an open source tool that lets you get data from websites. It facilitates and automates the process of data extraction. This visual web scraper works straight from your browser, so you don’t need to download or install anything.