For the Layman—What Is a Distributed System? (Part 1)
By: Kislay Verma
Tags: Learning Programming, Programming, Software Engineering, Beginners Guide, Distributed Systems

What Is a Distributed System?

Server: A computer connected to the internet or some other network.

Distributed systems are made up of “independent” components which are not necessarily located next to each other.

This seemingly simple definition of distributed systems has huge ramifications and gives these systems their unique strengths and weaknesses. Let’s cover some of these in detail.

Location transparency

Components in a distributed system communicate with each other via methods/protocols that don’t require the calling component to know where the called component is located. So the engine can potentially be located at home even when you drive your car. Somehow, when the accelerator is pressed, the engine generates more power, which is somehow transferred to the wheels. Another analogy is the remote working style we now see everywhere. Team members are not physically located together but still cooperate by performing their respective jobs in benefit of a shared objective.

This is called location transparency, which is a form of loose coupling. (All developers get a dreamy-eyed look when they hear this word — try it!)

The advantages of location transparency are obvious. If software components need not be co-located, then we can move them to different physical machines, each of which can then be vertically scaled. That is, we can buy powerful machines for each component separately instead of fitting all components on one machine. This directly leads to a more powerful system. Note that we are not mandating that components must be located on separate servers, just that it shouldn’t matter where they are located as long as there is a way to locate them.

How do these distributed components find each other? There are many ways to do this. One of the most popular mechanisms is DNS (Domain Name System) which maps names to IP addresses (unique identities of servers all over the internet). The backbone of all networking is the ability to locate one specific machine given its IP address or domain name. This seemingly small (but actually extremely complicated) technique has allowed software to eat the world.

Partial failure mode

A fallout of the distributed nature is that our failure mode is not all-or-nothing anymore. The card reader component may have failed but the account management component may be running. This means that some functionality related to account management may still be accessible even though our card swiping users are frustrated.

Horizontal scalability

Yet another corollary of location transparency is that there need not be only one more instance of a component. If it doesn’t matter where the engine is located, we can now add ten or more independent engines to add that much more power.

It is even possible to add and remove engines from the car as needed. This ability to add more instances of a component is called horizontal scaling and is the currently preferred mechanism for scaling software systems since it bypasses the physical limitations of how powerful a single server can be — we just add more low-power servers to compensate.

So we have a bunch of components living on different servers (machines) and communicating over a network (internet/LAN). Is that it?

Eventual consistency

There is one more interesting thing to understand here: The network is unreliable and slow.

If your Zoom call has ever hung mid-sentence or YouTube has buffered, you know what I’m talking about. Data is sent from one component to the other, but sometimes doesn’t reach it or reaches it after some noticeable time. Maybe the wire is cut, maybe the other component reads the data but crashes before it could do anything with it, maybe a lot of data was flowing over the wire and hence everything is stuck.

Any which way this occurs, this results in different parts of the system not having complete information about each other. This happens all the time in the systems we encounter on the web — payment was taken but the order could not be placed (payment component is not able to talk to order component), money transfer is triggered but will be reflected in 24 hours (transfer component knows that there is to be a transfer, but the account component has not been told yet).

Another way of putting this is to say that the independent components of a distributed system understand their own state (e.g., position, orientation, amount of load) absolutely but may be out of sync (to a greater or lesser extent) with other components. The out-of-sync-ness is a result of the network acting as a queue of unshared knowledge between them.

This is called inconsistency and is the other side of the global state we encountered in non-distributed systems. The remedy for this in distributed systems is eventual consistency, meaning we must implement mechanisms which will ensure that all components eventually agree with each other on what the overall state of the system is. Note that this is a catchup game, and components are out of sync by design rather than by mistake. This gives us some buffer time in which we can perform knowledge transfer instead of making every component aware of everything instantaneously (a physical impossibility in the distributed world, given that nothing can travel faster than the speed of light).

The bad parts

While distributed systems can be extremely resilient to failures and very responsive under high loads, building well-designed distributed systems is an extremely complicated undertaking.

The first problem is user experience. There is no way to hide the eventually consistent nature of the system from the users. With instant gratification being the increasingly accepted norm, it can sometimes take a lot of clever UX to keep the system distributed and the user happy.

A distributed system is necessarily much more complicated than a monolithic (everything in one place) design to compensate for the fallacies of distributed computing. A lot of new tools have evolved to help developers build reliable distributed systems, but this is still far from an easy or solved problem.