Backend-to-Backend Communication
By: Gregory Pabian
Tags: Distributed Systems, Asynchronous Programming, Backend Development, Distributed Transaction, Event Based Architecture

Any distributed system resolves around sharing information between its distinct components, frontend or backend alike. In this article, I would like to focus on establishing communication between different backends. I have outlined reasons, goals, and methods for communication for the reader to understand the concept from all angles.

Photo by Pavan Trikutam on Unsplash

Reasons for Communication

Backends have to reach other backends within a distributed system because of:

an explicit user action (sometimes acting as their proxy),

an implicit system operation (e.g., periodical tasks).

A rule of thumb dictates that a user action should require fewer permission than a system operation. If feasible, a backend should execute a command on a different service only with the privileges of the user in question.

Firstly, the following diagram shows an exemplary user action in an online game, where the User can send a command to the Game Backend in order to leave a particular game. After accepting the command, the Session Backend has to ask the Notification Backend to inform other game participants that the user has left, as the former has no capabilities for sending notifications. An avid reader might point out that the Session Backend should have no knowledge about notifications whatsoever; on the other hand, such separation of concerns requires switching to an event-based or command-based architecture.

An example of an (explicit) user action. Created using PlantUML.

Secondly, the diagram underneath displays an example of a proxied user action. The User may request information about the particular game from the Game Backend, which needs to collect data from the User Backend first to complete the request. The Game Backend can obtain the user information with the privileges limited to the ones of the User in question.

An example of a proxied user action. Created using PlantUML.

Lastly, the diagram underneath illustrates a typical scheduled system operation. The Scheduler periodically requests the Session Backend to log out all inactive users, which forces the aforementioned backend to ask the Notification Backend to notify the participants of all games about terminated sessions. To clarify, the Session Backend does not act as a proxy because no physical user executed the Scheduler, the very system did.

An example of an (implicit) system operation. Created using PlantUML.

Goals of Communication

Backends need to send messages to other backends in the following scenarios:

they require a piece of information from them,

they want to change a state of another system component.

Even though one could consider immutable operations safer than mutable ones, the callee should verify the permissions of the caller anyway. Again, the system in question might not allow certain users to access particular information.

For instance, the Session Backend might ask the Authentication Backend about token validity, as shown in the diagram below. Verifying a token serves as the most basic example of an immutable request, as such an action either confirms or negates validity and nothing more. Immutable requests do not produce issues with distributed transactions, as, even if they fail, no changes require a rollback.

An example of an immutable operation. Created using PlantUML.

On the other hand, designing systems with mutable commands requires rudimentary knowledge about distributed transactions. Such a transaction might span over multiple microservices, triggering many fallible operations. Should even a single operation fail, the system needs to either:

cancel the changes performed by the remaining operations, or

commit the changes undertaken by the remaining operations and execute a compensating operation.

A software developer can use the two-phase commit protocol to roll back the changes over multiple services. The protocol assumes stable communication between the services, their runtime stability (no crashes) and atomicity of the related databases used by the said services. The algorithm requires one service to act as a distributed transaction coordinator, which manages the two phases of committing:

the voting phase — when the coordinator tells each service what to commit, and they respond with either success or failure,

the commit phase — when the coordinator collects the votes and asks all the services to either commit or roll back.

The time necessary to complete some transactions might span over minutes because of external systems. Such long-running transactions, as they cannot rely on the two-phase commit protocol, use compensating transactions when failures happen. For instance, if a user had booked a concert ticket and had paid, and the concert host did not confirm the ticket within a specified time frame, the system needs to inform the user about the problem, cancel the ticket and return the money, as shown underneath.

An example of a long-running transaction. Created using PlantUML.

Methods for Communication

Backends can communicate with other backends using:

explicit and synchronous messages (e.g., over an HTTP request),

implicit and asynchronous messages (e.g., over an event-based system).

Regardless of the chosen method of communication, each message should contain a piece of information to enable identification of the sender. Even if backends use isolated subnets for data exchange, knowing the source of a message helps to enforce access control throughout a system.

System that prioritise speed over e.g., separation of concerns, might prefer to use synchronous messages, especially for immutable operations. For HTTP, services might call idempotent endpoints using a CDN (Content Delivery Network) or other cache layers, enabling faster results over time. Using synchronous messages means that a caller remains blocked until a receiver responds; blocking too many threads simultaneously in a threaded backend might prevent an entire instance from accepting new connections.

Command-based or event-based systems use asynchronous messages, which means that a backend might send a command (or an event) to a message pipeline for further processing by other backends, as shown down below. The system will eventually process such a message, but the original sender never waits explicitly for any result. Using a message-driven system allows for service decoupling, as backends with specific functionality do not need to know about the existence of each other.

An example of a command-based system. Created using PlantUML.

Summary

As modern software development discarded monolithic backends in favour of microservices, and consequently, distributed systems, software professionals needed to establish backend-to-backend communication. There exist diverse ways of such communication, which depend on the exact reasons for bridging two services together and goals to achieve. I find it significant to understand that once developers set up communication between two backends, migrating to a different method of exchanging information might turn the whole system design upside down.