Hadoop statistics collection and applications
By: Pinterest Engineering
Tags: Pinball, Pinterest, Engineering, Hadoop, Data

Xinding Sun | Pinterest engineer, Discovery

The massive volume of discovery data that powers Pinterest and enables people to save Pins, create boards and follow other users, is generated through daily Hadoop jobs. Managed by Pinball, these jobs are organized into indexing workflows that outputs dozens of terabytes of data daily.

Previously, the only statistics collected were each job’s native Hadoop counters, which presented a few problems. For one, they’re expensive, because each counter change has to synchronize with the Hadoop job tracker. Additionally, counter history isn’t preserved, so it’s impossible to trace statistics trends and strange changes, which are often strong indicators of data corruption or code bugs.

To address these issues, we designed a new statistics collection engine to power the indexing workflows, which has allowed us to develop real-time alerts to protect our data, and can be used to collect useful statistics for ongoing data analysis.

Architecting the statistics collection engine

The statistics collection engine stores statistics for each job at run time, and then are merged with native Hadoop counters and stored to S3 immediately after job completion. We then do alerts on job completion, copy the data to our database, and alerts and data analysis are further developed on database. The statistics data are structural and small compared to that of a Hadoop job output. The MySQL DB serves our applications very well in practice.

Designing the engine

We kept the following design values in mind when building the statistics collection engine.

Lightweight: To avoid expensive communication between the Hadoop job tracker and the tasks, we implemented the statistics collection engine at per shard layer. In our initial effort, the engine was implemented inside Hadoop reducer. Since each reducer has its own shard, the collection of statistics is local to the reducer task. Note the reducer doesn’t have to communicate with the job tracker.

Our job output is stored on AWS S3, where each reducer takes one partition. On completion, a reducer uses its partition number to store its own statistics. Once all the reducers finish, we aggregate the total statistics by scanning through the per shard statistics. At the end, we merge the statistics with the Hadoop native counters to get the final result. We store both intermediate and final statistics in thrift format.

Simple: We implemented the statistics collection engine as a singleton class. As the reducer runs within one process, it simply grabs the singleton at preparation phase. During the merging phase, the reducer calls the exposed APIs to update the statistics.

We support multiple types of statistics including counters, sums, top/bottom items, histograms, each of which were chosen because the final results can be derived by merging their per-shard intermediate results. The APIs corresponding to the statistics are listed in the following. Note that the “name” is unique within a reducer process, and used to identify a specific statistics metric. For example, we use “PageRank” in a pagerank reducer to identify all the pagerank related information.

1. void incrCount(String name) and void incrCount(String name, long count) are used to increase a metric count, where the first one is a special case of the second, with count = 1L.

2. void addSum(String name, double value) is used to add up the values of a specific metric.

3. void addToHistogram(String name, double value) uses a histogram buckerizer and adds the value to the histogram counts of the metric.

4. void addItem(String name, Object object, double value) adds an item into two priority queues for the metric. The queues keep fixed numbers of top and bottom items ranked by their values. The object’s name string is used to identify the item in the queue.

To make it simple to use, we also set the defaults for the statistics APIs so one can use the above APIs without customizing any parameters. By default, a log bucketizer is used for histograms (the size of top/bottom items are set to 100). However, a programmer can implement a new bucketizer and expose it for research purpose, and also set the size of top/bottom queue with a few extra lines.

Alert and data analysis applications

We mainly use the collected statistics information for two types of applications — alert and data analysis.

Alerts: We’ve implemented two types of alerts based on absolute value and on temporal estimation.

1) Alert based on absolute value: Any number that’s below a preset threshold will trigger an alert. One example is USER_WITH_INTERESTS metric in UserJoins calculation. In the figure, we can clearly see the dip due to a bug on 12/23.

2) Alert based on temporal estimation: In this case, we use recent data to predict current value. Any number that’s out of track will trigger an alert. One example is the the S3 output size check of a job. In the figure, the output size of the job changed significantly around 07/22, which was due to corrupted data input.

We also have two degrees of alert: warning and fatal. In the former case, we only send warning email to the job owner. In the latter, we stop the job immediately and alert the owner to do the investigation. Since the alert is useful in detecting sudden changes, programmers tend to tolerate false positives in practice. This is purely based on the fact that post debugging on corrupted data and bugs are way more expensive.

Data analysis: Almost all the statistics information we collect are useful for data analysis. For example, after the UserJoins calculation job is done, we can collect the user related board and pin counters. In another example on top/bottom items, the pagerank job collects top ranked users, boards from the joins and makes them available for post analysis at each iteration. As a third example, we collect the histograms of social ranks in the PinJoins calculation job for research and experiment. At each iteration, we tune the social rank parameters and get the result on the fly.

Looking ahead

The success of the statistics collection framework enables us to manage the indexing workflow in a total new way. There are still many things we want to explore in the future.

We may use an extra Hadoop job to do the final aggregation to improve performance if the per shard statistics data become too big.

We may create more sophisticated statistical model for better alert and analysis.

To reduce false positives, we may allow the programmer to initialize the workflow with some parameters on data or code changes.

We may also use the statistics to tune the scheduling of the jobs in the workflows to achieve optimum cluster usage.

If you’re interested in tackling these possibilities with us, join our team!

Xinding Sun is a software engineer at Pinterest.

Acknowledgements: The framework presented here is a collaboration with Tao Tao. We want to thank the whole Discovery team for actively applying it to their Hadoop jobs/workflows and giving invaluable feedbacks. Thanks to Hui Xu for the suggestions and technical insights.