Next Generation Distributed Systems On The BEAM
By: Gordon Guthrie
Tags: Software Development, Technology, Elixir, Erlang, Distributed Systems

The correct way to do this is a proper structured survey of BEAM shops with a well designed questionnaire — but it is possible to sketch out a vision of what that world might look like.

The key elements are mainstays of distributed systems as they are built:

causality

consensus

some sort of hashing/routing/distribution algo

distributed Finite State Machines

separation of the control plane from the data plane

streaming/back-pressure and overload protection

shared config

cluster management/cluster join/leave for various cluster types

So a NewTP would aim to give your unwritten software the ability to:

have pre-defined causal properties

have pre-defined consensus

use predefined hashing/routing/distribution mechanisms

be composed of understandable, testable and documented distributed FSMs

have plane separation between data/control

be overload resistant

be part of a well-behaved and operationally-manageable cluster

etc, etc

The challenge, as always, is how to eat the elephant. Notwithstanding that the right place to start is a with a properly structured survey, we can have a stab at that as well.

An Architecture Of Distributed Systems

Finite State Machines are simple and well understood things:

this FSM has three states A , B and C

, and A is ready for work, the start and end state

is ready for work, the start and end state B is attempting to do work

is attempting to do work C is cleaning up after work

is cleaning up after work the following transitions are allowed A -> B, B -> B, B -> C, C -> B, C -> A

Simple Finite State Machine

Traditionally we think of an FSM as being uni-polar — but that is just a degenerate case, more often we build multi-polar FSMs:

Multi-polar Finite State machines

In this world the state B of the first FSM actual contains the remote FSM — B wishes that some thing remote happens and I is the clean initial state on the remote node that it contacts to make it so.

J tries to do it, and K and L report back success or failure. (This is a happy-path diagram only).

There is on top of that a set of failure states associated with partition — did the remote machine get the message? Did the ack just time out? and all the other normal dist-sys problems.

In this world we start seeing a separation of concerns:

Separation Of Concerns

Many of the things we want out NewTP to have can be implemented as standard libraries — we can cleanly separate the plane of data from control at this point as well.

The plane of control needs additional things tho:

Configuration And Manageability

There are additional problems that we need to systematise the handling of. For reliability you need multi-polar FSMs, not bi-polar ones, and multi-polarity means you must handle consensus and causality:

Consensus And Causality

This functionality needs to be baked into our FSMs.

In addition the clusters so-built must be upgradeable incrementally — by moving individual nodes from version N to version N+1 including full and partial roll-back.

Upgrade And Roll-Back

A Possible Way Forward

Needless to say, it must be re-iterated that the correct way to go about this is to do a survey of major BEAM shops and get confirmation about exactly what design patterns are in use and represent major blocks of commonality that could be systematised.

That notwithstanding, here are my thoughts.

The big shift from plain Erlang to Erlang/OTP came via the addition of two things to the language:

a new set of standard libraries

major software components that implemented new functionality and that were implemented as behaviours and customisable via callbacks.

I think there is a room for an auxiliary language on the BEAM for writing distributed FSMs. It is an auxiliary language only because the core business logic would be implemented in callbacks which would be written in Erlang, Elixir or LFE. You might consider it a specification language (supported by libraries) that helps the end user design and specify distributed, updatedable, robust multi-polar FSMs with standard routing/hashing and plane of data implementations available at write time.

The task is pretty large:

Components Of A Distributed FSM auxiliary language for the BEAM

Lots of the necessary library parts already exist — as other libraries — but systematizing them is not a small task. The issue of how to implement causality and eventual consistency mechanisms across dFSMs alone is a big task.

It is worth stepping through the toolchain.

Clearly the core output of the compiler is the behaviours that implement the distributed FSMs.

However, the dFSMs have plumbing built in — so there is an opportunity to make them sort of self-mocking where the plumbing is replaced by Erlang messages — and can be made to iterate over the failure modes (partition, remote node down, remote going down during comms, remote node coming back, etc, etc) — in otherwords you should be able to compile the code into testing dFSMs and run detailed extensive tests of many types (common test, fuzz testing, model testing, failure testing, upgrade/downgrade testing, etc, etc) on a single machine.

The toolchain/language will also need to be able to have versioned interfaces with upgrade/downgrade built into the contracts.

The language should be typed using the dialyzer typing — and this means that the compiler should spit out generators and (possibly) models for things like Quick Check model testing or Fuzz Testing.

There will be a need for config output to, to be able to bring the cluster up — things that both ends of a dFSM pair can talk to each other.

One of the key outputs of the toolchain should be state transition documents that reflect the actual state of the FSM — that is the diagrams should be first class objects that are in synch with the code and then enable detailed design review *prior* to the actual detailed implementation of the code.

Implementation Notes

There is an anti-pattern of YASLs — Yet Another Shitty Lisp — and there are a range of suggested compiler targets for BEAM languages — like Core Erlang.

My proposal would be to use a hypothetical LISP called Liffey as a target — it would be a LISP composed of Erlang terms that would have a 1-to-1 correspondence with LFE so that the AST generated by the compiler could just be to_string()ed to LFE source code and then would be pushed though the LFE compiler.

The goal, obviously, would be do partial, iterative implementations — the old design-big-build-small way.