Building nearline system for scale and performance: Part I
By: Rolland He
Tags: Software Architecture, Search, Data Intelligence, Software Design

The (Offline) Tortoise and the (Nearline) Hare

To launch our design process, we determined at a high level which approach would achieve the results we wanted. We didn’t want to spend time building a complex system if our requirements didn’t call for it. Conversely, we didn’t want to under-engineer a solution, only to learn that what we built wouldn’t achieve our goals.

Starting with the Basics

A straightforward implementation would be an offline system that filters out sent jobs:

Log the jobs that we show for each user during email generation Schedule a batch process job to read from those logs Write the data we need to some database

The next time we generated an email for a user, we would just read from that database and filter out all the jobs the user had already seen.

This type of offline processing is common for many data systems because it can handle large amounts of data at once (i.e., high throughput), is straightforward to implement, and is relatively fault-tolerant.

This offline system allows us to increase our lookback window for jobs, since we now have a mechanism for filtering out sent jobs — challenge 1 solved. Before our project began, there was already such an offline system to help filter out repeated jobs for users.

Need for Speed

Why would we need to do additional work if we already had an offline system that functions reasonably well?

The problem is that users often are subscribed to multiple different email campaigns, and may receive more than one email in a day. Our offline pipeline has a relatively high data latency — sometimes as long as a day — since it runs infrequently and is computationally intensive. This means that if recent data hasn’t been fully processed yet and the user receives multiple emails in a day, then we might end up showing the same job listing across these different emails. Hence, challenge 2 from above is not actually addressed.

To remedy this data latency issue, we needed a process that incurs much lower latency. We needed a real-time (online) or close-to-real-time (nearline) system.

All Systems Online (Nearly)

First things first: our system wouldn’t have to be fully realtime. After all, we probably wouldn’t be sending two emails to the same user within seconds of each other. When designing this part of the system, it was useful to know that the latency requirement was less stringent. We didn’t want to spend unnecessary time or system resources trying to optimize the data flow. This is why we ultimately ended up with a nearline system instead of a fully online one.

The nearline system needed to function similarly to the offline system: it should be processing some raw data and writing the needed information in some database table that we could easily query from when we needed to show jobs to that user. But instead of reading from logged data on a periodic basis, the nearline system would need to be continually processing event data, and writing that data to the table whenever there was new event information.

The focus of this project, and this blog series, is on the nearline portion of this system. I will also discuss how I’ve connected these two different parts of the system.

The Power of 2

With a nearline system in place, we would ideally be able to retire the offline pipeline in order to reduce system complexity and avoid redundant data processing. However, one important reason prevented us from doing away with the offline system: determining whether a user has opened an email.

Glassdoor uses open tracking to track the emails users have opened. We base this determination on data we receive from Sendgrid. If an email hasn’t been opened for some period of time, then the user has not actually seen any of the job listings in the email. It therefore doesn’t make much sense to filter those jobs out in future emails. To give users ample time to open the emails, the offline process handles open tracking. Currently, we don’t have any near-realtime support for such information, and therefore cannot tackle it solely with our nearline system.

Consequently, we needed to utilize the power of both approaches. The offline system provides ground truth data for open tracking, whereas the online system allows us to have quasi-realtime for the job listings we have sent a user. If we could use the nearline system as a way to prevent showing the same jobs in the short-term, while leveraging the offline system to prevent showing the same jobs ONLY if they have been seen, then our system would address both of our initial problems.