Building a scalable and available home feed
By: Pinterest Engineering
Tags: Engineering, Pinterest, Hbase, Data, Qps

Dan Feng | Pinterest engineer, Discovery

We pride ourselves on being a company focused first and foremost on the user experience. In order to deliver a great experience, including showing related content in the home feed, we’re building a service that’s fast and highly available. From a Pinner’s point of view, availability means how often they’ll get errors. For service owners, availability means how many minutes the service can be down without violating SLA (service level agreement). We use number of nines to measure the availability of our site and each service.

The Pinterest home feed is a personalized collection of Pins for each person. One third of Pinterest’s traffic lands on home feed, which makes it one of our most critical pages. When building our new home feed, achieving four nines or higher was one of the metrics used for measuring the success of the project. The full discussion for the new home feed architecture can be found at ‘Building a smarter home feed’. Here, I’ll focus on the design decisions from behind the scenes.

Isolating challenges

The home feed system can be simplified to support three use cases: writing Pinners’ feed to a storage, serving feed from the storage and removing feed when it’s required. Writing feed can have a huge QPS (query per second). Fortunately it’s not user-facing and certain delay (e.g. seconds or even minutes) is tolerable. Serving has relatively small QPS when comparing the writing operation, but it’s user-facing and has a tight performance requirement.

A simple design can include writing all feed to a storage and serving and deleting from it. At our current scale, we keep hundreds of terabyte data and support millions of operations per second. We’ve had success with HBase in our past iterations of the home feed system. After evaluating all the options, we chose HBase as our backend storage. The problem with the design is it’s very challenging to tune the same storage to meet the requirements for both a high volume of writing and a high performance of reading and updating. For example, when a person creates a new Pin, we’ll fan out the Pin to all his or her followers. Followers are sharded across all HBase regions. When we fan out the same Pin to hundreds of Pinners, the write operation will hit multiple regions, lock the WAL (write ahead log) on each region server, update it and unlock it after use. Locking the WAL for each write/update/delete operation isn’t efficient and quickly becomes a bottleneck. A better approach is to batch operations and push the changes to HBase once in a while, which increases the throughput of the HBase cluster dramatically. But the latency of each operation can be as high as the flush interval. For user-facing operations, our latency requirement is at millisecond level and the approach will fail us miserably.

To satisfy the different requirements, we designed a system with two HBase clusters and save data to different HBase clusters at different stages (see the component diagram below).

Zen is a service that provides a graph data model on top of HBase and abstracts the details of HBase operations from data producer and consumer.

SmartFeed worker is pushing feed from all sources (we also reference sources as pools) to HBase through Zen, and called by PinLater, an asynchronous job execution system that can tolerate certain delays and failures.

HBase for materialized content saves the Pins that have potentially been shown in the home feed before, and its content is accessed through Zen.

SmartFeed content generator is in charge of selecting new Pins from the pools, scoring and ordering them.

SmartFeed service is indirectly retrieving feed (content) from both of the HBase clusters, and only talks to the pools cluster through SmartFeed content generator.

When a Pinner hits their home feed:

SmartFeed service calls content generator to get new Pins

Content generator decides how many Pins should be returned and how they should be ordered in the returned result

Simultaneously SmartFeed service retrieves saved Pins from HBase for materialized content

SmartFeed service will wait for the results from the above two steps, mix and return them. (If the calls to content generator fails or timeouts, the result from step 2 will still be returned.)

Offline, SmartFeed service will save the new result to HBase for materialized content and delete them from HBase for pools

With this design, we separate user-facing components from non user-facing components. Since different HBase clusters have different volumes of data and usage patterns, we can scale and configure them individually to meet their needs. In reality, we have far less Pins in materialized content cluster than the cluster for pools. We can make it more reliable and faster without too much cost.

Speculative execution

With the design above, the availability is as good as the HBase for materialized content since we’re serving content only when it’s available. From time to time, HBase cluster can have JVM (Java virtual machine) garbage collection, node failure, region movement, etc. With a single HBase cluster, the availability can occasionally drop below four nines.

To improve the availability over four nines, we implement something called speculative execution. We always keep a hot standby HBase cluster in a different EC2 availability zone to avoid losing Pinners’ data. Any changes made to the primary HBase cluster will be synced to the standby cluster within a few hundred milliseconds. In the event of a partial failure of the primary cluster we’ll serve the data from a standby cluster. This technique helps make the whole system four nines of read availability (not write) and provides a much better Pinner experience than failing the requests. The way that the speculative execution works is:

Make a call to the primary cluster to retrieve data

If the call fails or doesn’t return within certain time, make another call to the standby cluster

Return the data from the cluster which returns first

With this approach, SmartFeed service will be able to return data if either of the clusters is available and the overall availability is close to the combined availability of the two clusters. The tricky part is to pick a proper waiting time. Since syncing data from the primary cluster to the standby cluster has some delay, the data returned from the standby cluster can be stale. If the waiting time is too small, Pinners will have a higher chance of getting stale data. If the waiting time is too long, Pinners have to wait unnecessarily long even if we can return results from the standby cluster much earlier. For us, we find if a call doesn’t return within time x, it’ll eventually time out in most cases. The time x is also larger than the 99.9 percentile of the call’s latency. We decided to use this as the cutoff time, which means results may be returned from the standby cluster for one out of 1,000 calls.

Another interesting finding is that the latency of the standby cluster is higher than the primary cluster because so few calls fall back to the standby cluster, and it’s in a ‘cold’ state for most of the time. To warm up the pipeline and get it ready for use, we randomly forward x percent of calls to the standby cluster and drop the result.

One time the primary HBase was down for almost one hour because of some hardware issue. Thanks to speculative execution, all home feed requests automatically failover to the standby cluster. The performance and success rate of home feed was not impacted at all during the whole HBase incident.

Outcomes

Since the launch of SmartFeed project, we’ve been handling hundreds of millions of calls per day and haven’t had a major incident with the availability dropping below 95 percent for more than five minutes. Overall, our availability is better than four nines.

If you’re interested in tackling challenges and making improvements like this, join our team!

Dan Feng is a software engineer at Pinterest.

Acknowledgements: This technology was built in collaboration with Chris Pinchak, Xun Liu, Raghavendra Prabhu, Jeremy Carroll, Dmitry Chechik, Varun Sharma and Tian-Ying Chang. This team, as well as people from across the company, helped make this project a reality with their technical insights and invaluable feedback.

For Pinterest engineering news and updates, follow our engineering Pinterest, Facebook and Twitter. Interested in joining the team? Check out our Careers site.