akkadotnet/akka.net — pull request #400 by Aaronontheweb (2014-09-17 04:10:47+00:00)
Title: Akka.Cluster development - usable beta

After a couple month's of work, @smalldave and I have an initial beta of Akka.Cluster available to use.

We're in the process of porting the multi-node unit tests and doing bugfixes on the `ClusterDaemon`, but the core of Akka.Cluster is running smoothly and includes a couple of examples.

I'm going to leave this PR open for a bit as it explains some of the conceptual underpinnings of how clustering works, but also because it includes many new additions. Most of this work has been going into the `akka-cluster` branch but I think now is the right time to begin making this work available to other developers for review, testing, and having fun.

## How Clustering Works
Akka.Cluster is easy to use, but it's important to understand conceptually how it works as this will have an impact on how you design your clusters when using it in production. It's a straight port of Akka's clustering module, which itself borrows concepts from verifiably successful clustering implementations such as Cassandra, Riak, and others.

I'll explain some of the terminology as we go.

First, here's what the initial state of a new Akka cluster looks like:

![image](https://cloud.githubusercontent.com/assets/326939/4298098/426d9b44-3e1a-11e4-99b0-482917c20005.png)

There are two types of nodes in an Akka.Cluster:
* __Seed nodes__, which have well-known addresses that are described inside your `Config`
* __Non-seed nodes__, which are dynamically deployed nodes that join the cluster by establishing contact with a seed nodes.

The majority of your nodes will typically be __non-seed nodes__ for production clusters - this is because they're easier to manage from a configuration standpoint and it also requires fewer static IP addresses and deployment overhead.

So here's why seed nodes are important:

![image](https://cloud.githubusercontent.com/assets/326939/4298128/e2886046-3e1a-11e4-9188-46ef2dcc4318.png)

Seed nodes are used to start the conversation (known as _gossip_) between all nodes - every node has to contact them in order to join the cluster. You can have a cluster with a single seed node - if it goes down then you can't deploy any new nodes without changing your configuration, which is why you typically see 2-3 seed nodes in live deployments.

For this example, I gave all of the nodes different deployment configurations just to show how establishing a cluster works - in real-life these nodes would all be deployed with the same configuration usually.

![image](https://cloud.githubusercontent.com/assets/326939/4298147/58cbe584-3e1b-11e4-8984-dd15c30770d2.png)

Once nodes establish connectivity to each other, they begin exchanging information using what's known as a [Gossip protocol](http://en.wikipedia.org/wiki/Gossip_protocol) - regularly delivered peer-to-peer messages that contain time-versioned information about the status of all known peers in the network. Each of these gossip messages includes a version number expressed as a [Vector clock](http://en.wikipedia.org/wiki/Vector_clock) which gives each node the ability to determine which information arriving from the network is actually more recent than what the node's most recent gossip state already contains.

After a period of initial gossip, the nodes have begun to establish connectivity with each other and know now about at least 1 neighbor each. In addition, a `leader` node has been elected for every role in the cluster (more on that later.) The leader's job is to dictate which nodes are up and which nodes are down, based on the consensus of gossip information from the other peers.

![image](https://cloud.githubusercontent.com/assets/326939/4298217/e73f701e-3e1c-11e4-9dd5-1f0619daa5ad.png)

Once the gossip has had a chance to propagate across the cluster, all nodes are now aware of each other and the ring of nodes are all considered `Up` - therefore you can start doing things such as cluster-aware routers.

![image](https://cloud.githubusercontent.com/assets/326939/4298247/759479b8-3e1d-11e4-86b9-a8f3a902d237.png)

Whoops! Looks like a node just died! At least two of the other nodes in the cluster (usually nodes that _are not_ adjacent to it) noticed either because (1) C started missing cluster heartbeats or (2) the TCP connection to C died. The remaining nodes will gossip this information and mark the node as unreachable for now and will eventually prune it from the set of known nodes if the node doesn't return after a period of time.

If the node that died was the leader, a new leader would be elected.

![image](https://cloud.githubusercontent.com/assets/326939/4298297/5e9c0d7e-3e1e-11e4-89f4-52d62951dc6e.png)

And thus the cluster's network topology changes. New nodes may join later and other nodes may still leave, but the gossip protocol and interconnectivity between peers is what keeps the wheels turning no matter how things change.

## Using Akka.Cluster
I haven't added build support for Akka.Cluster NuGet packages just yet, but for the time being you can manually reference Akka.Cluster and its dependencies (Akka.Remote and Microsoft.Bcl.Immutable.)

To run Akka.Cluster inside your application, use this as a base configuration:

````
akka {
actor {
provider = "Akka.Cluster.ClusterActorRefProvider, Akka.Cluster"
}

remote {
log-remote-lifecycle-events = DEBUG
helios.tcp {
hostname = "127.0.0.1"
port = 0
}
}

cluster {
seed-nodes = [
"akka.tcp://ClusterSystem@127.0.0.1:2551",
"akka.tcp://ClusterSystem@127.0.0.1:2552"]

auto-down-unreachable-after = 10s
}
}
````

This gives you access to the `Cluster` actor system extension, specifies a couple of seed nodes, and the `ClusterActorRefProvider` gives you access to cluster routers, which are the primary way you integrate clusters into your existing applications.

Here's how you can configure a cluster router:
````
/myAppRouter {
router = consistent-hashing-pool
nr-of-instances = 100
cluster {
enabled = on
max-nr-of-instances-per-node = 3
allow-local-routees = off
}
}
````

So this defines a consistent hashing pool router that automatically deploys actors on active nodes participating in the cluster, and it explicitly forbids running any actors locally on the calling machine (`allow-local-routees = off`).

I'll be including some more examples and better documentation on all of the cluster options available for each type of router.

## Roles and Clusters
From one of the new samples added to `examples/Cluster` in this pull request:
`akka.cluster.roles = [backend]`

Clustering would not be very useful if you could only cluster identical pieces of software together - so that's why you can define __roles__ for each node in your cluster. That way if you can have multiple different pieces of software built on Akka.NET still form a cluster together, but only assign work to nodes that fulfill specific roles - and this is something you can specify in your routing configuration:

````
/myAppRouter {
router = consistent-hashing-pool
nr-of-instances = 100
cluster {
enabled = on
max-nr-of-instances-per-node = 3
allow-local-routees = off
use-role = backend
}
}
````

Now this router will only assign work to nodes that have the `backend` role enabled in their configuration.

## Wrap Up
@smalldave and I still have more work to do yet, and we'll be periodically submitting PRs to dev. In the meantime though, we would love it if you could run the samples, document bugs, or help us port the MultiNode testkit (located in Akka.Remote.TestKit.).

If you have any questions about this PR, put them here - if you have any questions about the design or usage please post on the [Akka.NET users group](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!forum/akkadotnet-user-list) or create an issue here.