How News Agencies Can Leverage Data For Editorial Decision Making
By: Zihui Zhou
Tags: Dashboard, News, Machine Learning, Statistics, Data Analysis

Have you ever wondered how news agencies decide what stories their journalists cover? To produce in-demand content, it is crucial for any news agency to allocate journalists to stories and topics that are of most interest to its readers. However, it’s not an easy task to grasp where readers’ interests lie now, let alone predict where readers’ interests will lie in the future.

In Columbia Business School’s “Analytics in Action” class, we (a team of three MBA students and two engineering students) collaborated with The Associated Press (“AP”) to tackle this immense and fundamental challenge for news agencies by putting the data analytics tools we’ve learned at CBS into action (as the course name may suggest). We investigated patterns in readers’ activities (a proxy for reader interest) on AP’s news platforms to create a tool that can help AP make smarter decisions about where to allocate their journalists. Here is how we approached this challenge and our findings after three months of “analytics in action”.

The Problem

AP is a not-for-profit news agency that offers fact-based news on (1) its end-user facing mobile app and website for free (the “B2C platform”) and (2) its business-customer facing AP Newsroom website via a paid subscription (the “B2B platform”). Examples of B2B customers are international and local news networks that distribute AP’s content on their own platforms. While the breadth of news coverage expands with globalization and digitalization, AP’s B2B customers (i.e. the source of AP’s revenue) are increasingly demanding in-depth story coverage. In other words, AP faces a resource allocation problem: where should AP focus its journalistic resources in order to deliver maximum value to its B2B customers?

The Data

To tackle this problem, we received 3 months (from June to August 2020) of user activity data for AP’s B2B and B2C platforms. The data included the number of page views and screen views for each article on the B2C platform, as well as the number of previews and downloads for each article on the B2B platform. Both B2C and B2B data included a timestamp for each page view, screen view, preview, and download. In addition, for each article we received data on attributes such as subject type (e.g. sports) and content type (e.g. investigations).

NOTE: We acknowledge that 2020 has been an outlier year as COVID-19 has received extensive coverage from news agencies across the world, including AP. Although insights drawn from our analysis of this given dataset may not entirely be representative of broader trends in general reader interest, we believe that the analytic tools developed through this project will continue to be valuable for AP in making data-driven decisions in the post-pandemic world.

Our Analysis

Although B2B customers are a primary focus for AP’s business model, richer data is available for the B2C platform. Thus, our initial thought was to see if we can use activity on the B2C platform as a proxy or predictor of activity on the B2B platform. If there is a strong correlation or leading/lagging relationship between B2B and B2C activity, then we could potentially use the larger B2C activity data to draw insights for B2B activity, specifically regarding what stories AP should focus on in the near future.

We first set out to clean up and merge the datasets we received to analyze the correlation between B2B and B2C user activity across different subjects, content types, and time of activity. We found that B2C activity is positively correlated with B2B activity for certain content types and subjects. However, the magnitude of correlation varied across topics, and the correlation, though positive, was generally not very strong. For example, the correlation between B2B activity and B2C activity is stronger for “Business” news than for “Sports” news, but correlation coefficients for each case did not exceed 0.7, which is generally considered as a threshold for a “strong” correlation.