System Design Interview: Globally Unique ID Generation
By: Eileen Pangu
Tags: Tech, Distributed Systems, System Design Interview, Software Development, Backend Development

It’s been a while since my last system design interview post. I figure it’s about time to pick up the series again. Today, I want to talk about globally unique ID generation. It sounds like an interesting topic, through which interviewers can see the depth of knowledge candidates have.

Requirements

First, let’s decipher the topic word by word.

Globally: it implies that there are multiple systems, components, or machines that are working together. We should consider them as a whole. At the same time we should not forget that they may be somewhat decoupled, making coordination challenging.

Unique: it needs to be exclusive, one and only. Duplicates are disallowed. Repetition is forbidden. Collision is catastrophic.

ID: it may be an integer. If it’s an integer, we need to ask the range — signed vs unsigned, 32 bits vs 64 bits. It may be a string. If it’s a string, we need to ask about the character set, the encoding — UTF-8 vs ASCII, and the min-max length.

Generation: the system needs to create those IDs. It may provide it as a service to other systems.

The next step in a system design interview is usually asking clarification questions. But a trick is that for those open ended questions, interviewers are often amenable to us stating our assumption and asking for confirmation. So we have a chance to steer the conversation towards the direction that we feel comfortable about. In this case, I’m going to assume that the system provides an ID generation service. Clients can inquire about the system through RPC, and the system responds with a string ID that it has never issued before. I’m going to further assume that the ID is encoded in UTF-8 and we don’t worry about the character set or the string length (as long as we don’t go crazy).

UUID

Before our mind wanders into those complicated distributed system design patterns, pay attention to the requirement. It didn’t say the ID should be generated in any order. That’s great. Because generating IDs in a certain order leads to a lot more complexity. Without an order requirement, can we just hand out UUIDs? Sounds doable. The chance of collision is so low that in practice we don’t worry about them. It’s far more likely to have software bugs or even hardware bugs than to have UUID collisions. So that’s our solution 1.0. The system just randomly generates an UUID in response.

Using System Clock?

The interviewer may say “Good, now I want those IDs to be monotonically increasing”. You saw that coming, didn’t you. The most natural thing associated with ascending order is time. So how about just using timestamp as the ID? Each time the system gets a request for a new ID, it Sprintf the current timestamp. Well, in general, using clock time is an anti-pattern due to clock skew. A machine’s clock may drift. It needs to be recalibrated via network time protocol periodically. So, in theory, the machine’s time could go backward if it goes too fast and gets reset by clock synchronization. Not to mention that in a distributed setting, multiple machines’ clocks are almost guaranteed to go out of sync. So relying on time alone is dangerous. But it doesn’t hurt asking the interviewer if that’s acceptable. It may be OK depending on the use cases. If clients request IDs concurrently, the IDs they get back have non-deterministic order anyway. Client A may get back “456” while client B may get back “123”. But that’s fine. If they’re really sending concurrent requests — meaning that they have no dependency on each other or communication channel between each other — we can’t really tell who’s first and who’s second. The problem only arises when a single client requests IDs in quick succession or multiple clients that have a way to know what each other gets. It will then be really confusing to see “456” first, and then the next returned ID is “123”. In academic terms, this is called violation of linearizability.

Using a Database

The interview may say “I care about clock skew. And I want the next ID I get to be always strictly larger than the ones I got before”. Fine. We’ll hook up a database with the server, store a counter in the database, and do a transactional increment whenever we serve a request. That’ll work as long as the system throughput can sustain the incoming requests. We could horizontally scale the server instances to increase availability, but they’re still bottlenecked by the underlying database. Not the mention that this creates a single point of failure. But again, the use cases may not care — they may have a backup plan if our ID service is temporarily unavailable.

Distributed Consensus

The interviewer may then say “I want high availability and high throughput”. OK, you really want me to get into distributed consensus, don’t you. There are many distributed consensus algorithms and implementations out there. Paxos is too complicated. People seem to like Raft. I also think Raft is a better fit here. Raft achieves distributed log replication. A cluster of nodes can tolerate minority failure. Committed log sequences are guaranteed to be the same across nodes. I’ve written a more elaborate series about Raft [link] that you may want to check out. With Raft, we can use the next log entry position as the ID. A request will get back the ID if that log entry is committed. Once that log entry is committed, the next request will have to use the next position. Raft is also relatively high throughput because appending log entry is fast. The interviewer may continue to hector that he wants many many IDs and our log can’t expand indefinitely. In that case, we can store a number inside the log entry. The number can be composite if a large range is required. Every time a new ID is needed, we write the number_from_last_entry + 1 to the new entry. Raft is a leader-based protocol. So there is only one machine at a time doing the increment (no coordination needed for that). The machine can cache the latest number in memory — in a thread-safe way of course — so that it doesn’t need to read from disk. If a new leader is elected, it just needs to load the number from its last entry to memory. Now we have the number saved, we can clean up ancient logs.

Google Spanner’s TrueTime

A bonus point: we can mention how Google’s Spanner handles this with its TrueTime API. I’ve written another blog post that explains in details how Googler Spanner uses TrueTime [link]. Google deploys highly accurate GPS clocks in each data center. They manage to control the clock skew so well that they can provide time with a very small (a few milliseconds) [earliest, latest] interval . The likelihood of the true time falling outside the interval is so low that we don’t have to worry about that (the same reasoning as for UUID above). If we have access to this TrueTime API, what we can do is that for every ID request, we inquire the current true time to retrieve a [earliest, latest] interval. Then instead of responding immediately, we wait until the next true time response [earliest’, latest’] ’s earliest’ is larger than that previous latest . Now we can respond with that previous latest as the ID since subsequent ID is guaranteed to be larger than that. Essentially we wait out the clock uncertainty until the previous time is guaranteed to be in the past. This does have the constraint of serialising the ID requests through one server but it’s likely to be faster than Raft since no data is written to disk. In addition, the server is stateless so we can easily fall back to different one if it fails.

Closing

I think this should be the end of the interview discussion. Hopefully throughout the interview, you can demonstrate that you understand the system design constraints and tradeoffs. I wish you have a wonderful interview experience. Keep an eye out for more system design interview posts to come.