Tutorial: Scrape 100 Headlines in Seconds With 23 Lines of Python | by Jack Bandy | Towards Data Science

If you need to do any kind of web scraping, Scrapy is almost impossible to pass up. With built-in features such as parallel requests, user agent spoofing, robots.txt policies, and more, you can build a powerful web scraper with just a few lines of code.
In this tutorial, I will show how to create a basic Scrapy Spider that collects headlines from news articles. On most websites, you can do this without worrying about paywalls, bot detection, or weird parsing tricks, so I will leave those worries for another tutorial.
Here is a file with 100 random urls, from a dataset I have been collecting from NewsAPI. The first few lines look like this:
We will use three open-source libraries (and their dependencies): Pandas, Scrapy, and Readability. Assuming you already have pip installed, you can ensure your computer has these by running the following command in your terminal:
pip install scrapy pandas readability-lxml
Then, make the folder where you want to organize the project and navigate to it:
mkdir headline_scrapercd headline_scraper
Now, we can create the spider. Scrapy has a startproject command that sets up a whole project, but I have found that can be bloated. We’ll keep it simple by making our own spider which, as promised, comes in at just 23 lines (including comments and whitespace 🙂). These are the contents of a file titled headline_scraper.py :
Unlike a normal python script, we need to use scrapy’s runspider command to run the file. Choose where to save the output using the -o flag:
scrapy runspider headline_scraper.py -o scraped_headlines.csv
That’s everything! Below is an explanation of what’s going on in the code.
Besides importing libraries and declaring PATH_TO_DATA(a link to the gist where the 100 urls are stored), here is what the initial lines are doing:
class HeadlineSpider(scrapy.Spider) makes a new kind of class, “HeadlineSpider,” that inherits the base functionality of scrapy Spiders
name="headline_spider" names the new Spider “headline_spider,” for Scrapy’s reference
start_urls=read_csv(PATH_TO_DATA).url.tolist() uses the csv reader from pandas to download the file with 100 urls, then gets the “url” column and turns it into a python list. start_urls are needed for any spider, but in this case, they are the only urls that will be visited.
(If you want to use a different file with your own urls, simply replace PATH_TO_DATA with the location of your file. Then, either make sure the column of urls is labeled “url,” or replace .url with .name_of_your_column It can be on your machine, i.e., PATH_TO_DATA=/Users/Jack/Desktop/file.csv )
Now that we have initialized the spider, we need to tell it what to actually do when it gets to a web page.
To actually extract the headline, we must implement the parse method. Scrapy automatically sends the http response to this method whenever it retrieves a url from the list of start_urls , then it is totally up to you what to do with it. Here’s what this version does:
doc=Document(response.text)this line uses readability’s functionality to parse through the html. You could also use alternative libraries, such as BeautifulSoup or Newspaper, for this part.
yield is essentially a return method that creates the output, which you can think of as a row in the resulting spreadsheet.
'short_title':doc.short_title() and the lines below it simply set different attributes of the return object. You can think of 'short_title' as a column in the resulting spreadsheet, and the contents will be the output of doc.short_title() (for example, “How the coronavirus outbreak is affecting the global economy”)
Again, this part is very customizable. If you wanted to try getting the full text of the article, just change the yield lines to include doc.summary() as follows.
yield {    'full_text': doc.summary(),    'short_title': doc.short_title(),    'full_title': doc.title(),    'url': response.url}
This will make an additional column calledfull_text in the resulting .csv file, with readability’s best attempt at extracting the article text.
Here is the output file from runningscrapy runspider headline_scraper.py -o scraped_headlines.csv:
Boom! Well, at least for the most part. Scrapy is very error tolerant out of the box, so only 95/100 urls came back. The 5 that didn’t come back encountered some kind of http error, like the 404 error you sometimes see browsing the web. Additionally, some rows (7 by my count) say “warning” and “Are you a robot?” rather than providing actual headline. (Bloomberg seems to have very effective bot detection.)
There are many strategies to deal with bot detection, some of which are already implemented in Scrapy. Let me know if you are interested, and I can cover those strategies in a future tutorial.
If you have any questions, comments, or issues with the code, feel free to reach out by responding to this post or messaging me on Twitter. Thanks!