How to Collect Data From The New York Times Over Any Period of Time | by Brienna Herold | Towards Data Science

Suppose we want to find out how discourse in The New York Times has evolved over a period of time. These days it would be neat to analyze this, since whenever we visit the front page we get firehosed with news about the coronavirus. How has the pandemic shaped the headlines of one of the most popular newspapers in the United States?
To answer a question like this, we would need to collect article metadata from The New York Times. Here I describe how to do this in Python.
Table of Contents:
Where to find The New York Times article metadataHow to request article metadata from The New York Times
Where to find The New York Times article metadata
How to request article metadata from The New York Times
The New York Times offers a developer network consisting of several APIs to let non-employees request articles and metadata from them, which is super convenient for this analysis.
Not many newspapers do this, actually. Maintaining an API takes a lot of work. For newspapers that do not offer an API, we would need a different approach to get their headlines. It’s still possible, just not as easy.
To begin using The New York Times APIs:
Create a free developer’s account here. Sign in after you open the link in the verification email. Make your way to the developer’s home page.
Register a new app in your developer’s account.
Click on your e-mail address in the upper-right corner.Select Apps in the drop-down menu that appears.Select +NEW APP on your Apps page.Enter any name and description.Activate the Archive API.Select Create.
Click on your e-mail address in the upper-right corner.
Select Apps in the drop-down menu that appears.
Select +NEW APP on your Apps page.
Enter any name and description.
Activate the Archive API.
Select Create.
Take note of your just registered app’s API key. This is the only piece of information we need to get data from The New York Times, and we will need it soon.
We will use the Archive API to get all article metadata from the past year. This code can be modified for any period of time between 1851 and today.
Load dependencies.
import osimport jsonimport timeimport requestsimport datetimeimport dateutilimport pandas as pdfrom dateutil.relativedelta import relativedelta
Specify the date range.
end = datetime.date.today()start = end - relativedelta(years=1)
Make a list of the months that fall within this range, even if partially. We need this information for making calls to the Archive API, since it works with only one month at a time.
months_in_range = [x.split(' ') for x in pd.date_range(start, end, freq='MS').strftime("%Y %-m").tolist()]
I wrote some code to request and process article metadata from the Archive API. This code works with only one month at a time for optimal memory management. We send a request to the API for a given month, receive and parse the response, and populate a data frame with some details about each article, including its publication date, main headline, section, subject keywords, document type, and material type. Finally, we save the data frame as a CSV file and move on to the next month, until we have reached the end of the desired time range.
Note that there are two rate limits per API: 4,000 requests per day and 10 requests per minute. We sleep for 6 seconds between calls to avoid hitting the per minute rate limit.
Don’t forget to switch out YOUR_API_KEY on line 4 with your API key.
Run the code to get and process articles from months_in_range.
We have collected metadata for 80,460 articles from the past year! Each month has been saved to a CSV file in the headlines directory.
Let’s see what the response looks like for the last month that we processed, which is still in memory.
response
There’s a lot more. The documentation tells us what is all there.
See what the CSV file for this month looks like.
df
As seen in the data frame, there is the occasional consistency quirk to watch out for. Sometimes the 1st of a month is considered part of the previous month. Sometimes there is missing data, like with September and October 1978 due to a multi-union strike.
There is a lot that we can do with all of the article metadata we now have in our headlines directory. One of the things we could do is explore how The New York Times has evolved in its reporting over the past year, with an emphasis on the coronavirus pandemic. This would be interesting as we consider not just what happened, but how the media discussed it.
The complete notebook is here.
If you’d like to read more of my articles or explore millions of other articles, you can sign up for a Medium membership:
brienna.medium.com
You can also subscribe to my email list to get notified whenever I publish a new article:
brienna.medium.com
Some other stories from me that might interest you: