Super Simple Way to Scrape BBC News Articles in Python | by Petur Einarsson | Towards Data Science

I work in London as a Data Scientist for a consultancy. This effectively means I work for multiple companies, doing different Data Science stuff.
Mostly I build Machine Learning solutions in the Cloud. Sometimes general Data Solutions, Dashboards or recommending Data Science best practises. Recently though, it's been mostly NLP or Computer Vision. But enough about me.
If you are into Data Science, Machine Learning or Big Data, there’s a good chance you’ve had to collect data from a website.
For small projects or Proof-of-Concepts (PoC), doing this by hand is fine. Other times, this is best done in code. This is where Python shines, with many open-source libraries to interact with various websites.
Knowing how to wield any scraping library, is a very useful skill. Well worth the investment 💰
The secret is that it’s easy to scrape websites
Together, we’ll build a simple Python class that scrapes the BBC.
Ps: If you only opened the article for the final code, feel free to skip to the end where it’s all laid out.
BeautifulSoup is my all-time favourite way to scrape websites. It’s simple to learn and easy to use.
Plus there’s something comical about imagining fishing the good bits out of soup when web scraping!
To get started, install the library into your environment using the command line. As a Data Scientist, I usually use Anaconda.
We will also need the requests module to get the HTML code from the BBC. Run these in your terminal:
conda install beautifulsoup4 requests -y
You can also use pip if you prefer.
pip install beautifulsoup4 requests
Once installed, create a python file (.py) and import the libraries at the top of your code.
I recommend using PyCharm, but if you are just starting, Jupyter Notebooks are a great way to get your feet wet.
import requestsfrom bs4 import BeautifulSoup as bs
We are mostly concerned with the two find functions. They allow us to find any HTML element in our soup, just by specifying its name.
find(element_tag, attribute) #return first matching itemfind_all(element_tag, attribute) #return list of matching items
Always use a python environment to manage your projects
Let’s pick a BBC article to work with. I’ve chosen one where Iceland is commemorating the loss of its first glacier to climate change.
Access the raw article via Python:
url = 'https://www.bbc.co.uk/news/world-europe-49345912'article = requests.get(url)
Then pass the article content over to BeautifulSoup and specify the HTML parser.
soup = bs(article.content, 'html.parser')
Simple as that and the soup is ready! Let’s fish out the good bits 🎣
The next step is manually inspecting the HTML of the page to find the element we are interested in. I’m using Chrome for this bit, but you can use any browser.
Open the webpage in your browser, right-click and ‘inspect’.
Hover the mouse over any of the text, and the browser will tell you in which div that paragraph is in.
If it’s not working, press Ctrl+ Shift+ C or CMD + shift + C and try again.
In this case, “story-body__introduction” contains the highlighted paragraph.
The div class “story-body__inner” with the property “articleBody” contains the entire article, and that is what we are after.
Find the body in the soup:
body = soup.find(property='articleBody')
This gives us the content of the page as a list of paragraphs (p) in raw HTML format.
Access the text of each paragraph and assign it to a new list using list comprehension:
text = [p.text for p in body.find_all('p')
We can combine the above in a single function, that takes in the URL of a BBC article:
If we wanted to access the title, we again inspect the site, find the div that the title is in, and fish it out of the soup.
This is how that code would look like:
More experienced coders probably already see the issue with these functions...
They are the same for the most part. That’s not very DRY.
One way is to refactor them into 3 functions instead and put get_soup in its own function.
However, that would mean if we wanted to access both the title and the body, we’d have to download the BBC article twice!
This feels like it should be a class instead.
Plus, classes look and feel more professional instead of calling functions within functions. They also make it easier to maintain, when the scraper grows in complexity.
As the great coders we are, let’s turn it into a class!
Start by defining the class.
Every class should have an __init__ function, where you pass it any variables to initialise the functions.
The __init__ function is called automatically when you call the class.
Each function in the class also needs to have self as the first argument. This is the actual instance of that particular class.
class BBC:    def __init__(self):        pass
Next, add URL as an argument in __init__, and turn it into a soup. Changes are highlighted in bold.
class BBC:    def __init__(self, url):        article = requests.get(url)        self.soup = bs(article.content, "html.parser")
Writing the soup variable as self.soup makes it an attribute of that class. Class attributes can be accessed anywhere within a class, within any function.
To access the body of the article, we use the body function from before. body is now also an attribute of the class
class BBC:    def __init__(self, url):        article = requests.get(url)        self.soup = bs(article.content, "html.parser")        self.body = self.get_body()    def get_body(self) -> list:        body = self.soup.find(property="articleBody")        return [p.text for p in body.find_all("p")]
Same with the title:
class BBC:    def __init__(self, url):        article = requests.get(url)        self.soup = bs(article.content, "html.parser")        self.body = self.get_body()        self.title = self.get_title()    def get_body(self) -> list:        body = self.soup.find(property="articleBody")        return [p.text for p in body.find_all("p")]    def get_title(self) -> str:        return self.soup.find(class_="story-body__h1").text
The entire code looks like this then:
That’s it for now. Here’s how it works in action:
In my next post, I apply NLP to the BBC article and build a Streamlit app to interact with it!