Taming data inconsistencies
By: Pinterest Engineering
Tags: Pinterest, Infrastructure, Engineering, Data

Vamsi Ponnekanti | Pinterest engineer, Infrastructure

On every Pinner’s profile there’s a count for the number of Pins they’ve both saved and liked. Similarly, each board shows the number of Pins saved to it. At times, Pinners would report that counts were incorrect, and so we built an internal tool to correct the issue by recomputing the count. The Pinner Operations team used the tool to fix inconsistencies when they were reported, but, over time, such reports were growing. It wasn’t only a burden on that team, but it could have caused Pinners to perceive that Pinterest wasn’t reliable. We needed to determine the root of the problem and substantially reduce such reports.



Here I’ll detail why some of those counts were wrong, the possible solutions, the approach we took and the results we’ve seen.

Addressing the problem

After digging into the issue, we found a number of reasons counts appeared wrong, including:

Pinner deactivations: If user A liked a Pin saved by user B, and user B deactivated their account, the Pin was not shown to user A, but the Pin was still included in the count. (Note: there’s a very small number of Pinners who deactivate and reactivate their accounts multiple times.)

Spam/Porn filtering: If a Pin was suspected to have spam or porn content, it wasn’t shown, but was still reflected in the count. The scheme to identify spam/porn content is continually evolving, and domains are added or removed from the suspect list almost every day.

Non-transactional updates: Some counts were updated asynchronously in a separate transaction to optimize the latency of operations such as Pin creation. In some rare failure scenarios, it was possible the count wasn’t updated.

Possible solutions

Option 1: Fix the root causes

We first looked at ideas for fixing the root causes. Whenever a Pinner deactivated/reactivated their account, we could update the counts in the profiles of all Pinners who may have liked their Pins.

Likewise, whenever there’s a change in porn/spam filtering schemes, we could update the counts in the profiles of the owners of those impacted Pins, and in the profiles and boards of the Pinners who’ve liked and saved such Pins.

To address the problem caused by non-transactional updates, we would need to update the count in same transaction, which may increase latency of operations such as Pin creation.

This solution wouldn’t only be expensive, but it also wouldn’t fix the inconsistencies that already exist.

Option 2: Offline analysis and repair

Another approach commonly used in similar situations is offline analysis and repair, where we could dump data from our online databases to Hadoop daily, and they’d be queryable from Hive. We could have an offline job(s) that finds the inconsistencies, and another background job(s) fixing those inconsistencies in the online databases. Since online data could’ve changed after offline analysis was performed, it would need to be revalidated before updating the count in the online database.

We found this to be a good solution and used a similar method to fix inconsistencies in our HBase store. However, based on that past experience, we knew the effort to build it was no small task.

Option 3: Online detection and quick repair

We also thought about online detection and quick repair. If a Pinner detects an inconsistency in the data on their profile, our system should also be able to detect it. For example, if a Pinner scrolls through all of the Pins they’ve liked, the system could check if the count of liked Pins shown matched the displayed count in their profile. This way of detection is simpler than the previous solutions, with small overhead.

Once an incorrect count is detected, we could queue a job to fix the count so we could display the correct count the next time. We already had a framework, PinLater, that could queue arbitrary jobs for later execution. Typically, such jobs run within seconds of queuing.

The job to fix the count would have information about which counter to fix (such as user ID or board ID), as well as information about the stored count (i.e. the displayed count) and the actual count. This job could use the same logic as our internal tool to recompute and fix the counts.

The count fixing job would check if both the stored count and actual count are still same as at the time of queueing. If so, it would update the stored count to the actual count. If the stored count or the actual count are different from what they were at the time of queuing, the count isn’t updated as it indicates some change of state, such as new Pins that may have come in.

However, this solution doesn’t detect all inconsistencies. For example, if a Pinner doesn’t click on ‘likes’ or ‘Pins’ in their profile, or if they don’t scroll through their entire list of Pins, it can’t detect inconsistencies. This solution also isn’t appropriate for inconsistencies that are expensive to detect during online reads.

The chosen solution

In the short term, the third solution (online detection and quick repair) was preferred since it fixes existing inconsistencies in the count in near real-time, and is simple to implement. In the longer term, we may still build the second solution.

Results

After deploying and gradually ramping up the solution to repair counts for one percent of users to 100 percent, nearly all Pinner reports about count inconsistencies vanished. The table below shows the number of reports over a period of about 12 weeks, including a few weeks before the launch, the ramp-up period and the weeks following launch.