System Design — How to query status of 10,000 machines
By: Omar Faroque
Tags: 1m Machine Health Status, Million Query Job Status, System Design

The general answer is to 1) maintain a queue of 10,000 jobs, each one is to query a machine’s status 2) have a bunch of worker machines run in parallel, retrieving and executing one status querying job at a time.

Following are the steps of getting deeper in this question and ultimately reach a satisfying state.

Step 1 — Naive Approach

A pitfall when the interviewer presents this question is that, he won’t tell you how many machines you can use to do such an operation. Thus, the first impression an inteviewee got is it can be a dead simple a loop.

List<Status> getStatus(List<Host> hosts) {

List<Status> status = Lists.newArrayList();

hosts.forEach(h -> {

status.add(h.getStatus());

})



return status;

}

Step 2 — this is not an algorithm question, THIS IS A SYSTEM DESIGN QUESTION!

The inteviewer will attack you by asking the complexity of the above solution — it’s O(N) . You’ll doubt if there’s any better way. It seems not, because you have to visit each machine and O(N) is the best one you can think of. Can it be O(logN) ??!!

But wait, the only feasible answer is that… this is not an algorithm question, THIS IS A SYSTEM DESIGN QUESTION!

Then it makes sense, since you can make it faster by enabling Parallel Computing !

The naive parallel solution that comes right to your mind is, having multiple worker machines, say 100, each one querying 100 host machines (100 * 100 = 10,000) and forward the query results to UI.

Step 3 — Naive Parallelization

But what if there are huge delays when querying a single machine? It’s not uncommon due to:

huge network latency

request timeout

several host machines are actually dead!

One of the worst case is that, 9000 host machines are healthy and the other 100 are dead, and it happens to be that one of the workers is responsible for querying the 1000 dead machines. Your UI would take forever to show the results!

Step 4 — Determine a timeout

We should have a timeout for each status query!

Him: Well, how long is the timeout?

You: If we have a collected history of how long each request takes, we can make the timeout such that, we can not only ensure most healthy calls are not missed, but also fail on dead machines fast enough

Him: How can you do that? Let’s say we have a distribution graph for you, can you elaborate more?

Step 5 — Cold Start and Load Balancing? -Worker and Task Queue Paramdigm

There are two main problems with this above solution:

1) what about cold start ? What if there’s no historical data? When you are building this from scratch, not optimizing it, then no data is ready for you to reference.

? What if there’s no historical data? When you are building this from scratch, not optimizing it, then no data is ready for you to reference. 2) what about load balancing ? What if some workers finished all their jobs very fast, and don’t have things to do, but some workers still have a ton of work remaining? This can happen easily when a worker gets assigned 100 healthy host machines, and another worker happens to get assigned 100 dead machines

The only feasible way to go further now seems require changing the naive parallel computing paradigm.

And the promising one is the Worker and Task Queue Paradigm , looks like this:

Each worker pulls an uncompleted task from the task queue, and execute it. When finishing it, the worker pulls a new one.

This parallel paradigm greatly reduces the chance of a worker got assigned all dead machines, and therefore greatly reduces the overall expected time for the whole job to finish.

Let’s compare this with the former approach in Step 3 in the case of consecutive dead machines. We are counting how much time it cost to finish the whole job in both ways, based on the following assumption:

Say the first 100 machines are dead, and the other 9900 are good.

The time to process a health machine (time to send requests + double way network communication time + time to process response) is 1s, and that for a dead machine is 5s (because of the standard timeout).

Ignore the time to actually retrieve a task from task queue

In first approach, the worst scenario is that one worker gets all the 100 dead machines. That worker will take 100 * 5 sec to finish, and all the other 99 workers will take 100 * 1 to finish. So the total time to finish the job is max(100 * 5, 100 * 1) = 500 sec.

In the second approach: all 100 workers will pulls each of query job for a dead machine (100 in total), so it takes 5 sec to finish all the dead machine querying jobs in parallel. Then it takes 9900 / 100 = 99 sec to finish all the remaing jobs for healthy machines. The overall time is 5 + 99 = 104 sec, which is far less than 500 sec.

Step 6 — Priority Queue

This is a kind of bonus answer to this specific problem, but is quite common and important to a general task queue. The incentive is about having tasks of different priorities.

You can set the priority by considering weight of different elements.

Summary

Finally, we came up with a quite sound solution for this question.

Have a worker and task queue paradigm

Maintain the queue as a priority queue

Remember this paradigm in the future when you run into similar problems!

References: https://bowenli86.github.io/2016/06/17/system%20design/task%20queue/System-Design-How-to-query-status-of-10-000-machines/