erlang/otp — pull request #558 by dumbbell (2014-12-11 18:35:36+00:00)
Title: mnesia: Double-check the nodes list returned after protocol negotiation

During Mnesia startup, after protocol negotiation, the list of connected
nodes is written to `recover_nodes`. This list is later used to merge
the schema.

If Mnesia was stopped on a remote node between the protocol negotiation
and the moment the list is stored in `recover_nodes`, the remote node
is still considered running: the value of `recover_nodes` stored during
`mnesia_down/1` is overwritten. Therefore, this node may be used to
acquire a write lock on the schema in order to perform the merge. In
this case, the remote node never answers to the lock request and Mnesia
hang forever (`application:start(mnesia)` never returns).

To fix the problem, we check the list one last time and remove from it
all nodes where Mnesia is stopped.

We first discovered the problem with RabbitMQ. Here is a trace of the messages that shows the issue:

The cluster is made of three nodes:
* a@magellan
* b@magellan
* c@magellan

Mnesia is currently running on all of them, but all three nodes are partitioned. To recover, Mnesia is restarted on both a@magellan and b@magellan.

The PIDs are the following processes on b@magellan:
* <0.1030.0> mnesia_monitor
* <0.1032.0> mnesia_locker
* <0.1033.0> mnesia_recover
* <0.1034.0> mnesia_tm

This is the trace of messages sent on b@magellan:

1. b@magellan wants to connect to the two other nodes. This starts the protocol negotiation.

```
{1418,50172,633940} <0.1034.0> ----> <0.1033.0> {'$gen_call',{<0.1034.0>,#Ref<0.0.0.5135>},{connect_nodes,[c@magellan,a@magellan]}}
{1418,50172,633950} <0.1033.0> ----> <0.1030.0> {'$gen_call',{<0.1033.0>,#Ref<0.0.0.5136>},{negotiate_protocol,[a@magellan,c@magellan]}}
{1418,50172,633982} <0.1042.0> ----> {rex,a@magellan} {'$gen_call',{<0.1042.0>,{#Ref<0.0.0.5137>,a@magellan}},{call,mnesia_monitor,call,[{negotiate_protocol,<0.1030.0>,"4.12.3",[{8,1},{8,0},{7,6}]}],<0.1025.0>}}
{1418,50172,633994} <0.1042.0> ----> {rex,c@magellan} {'$gen_call',{<0.1042.0>,{#Ref<0.0.0.5137>,c@magellan}},{call,mnesia_monitor,call,[{negotiate_protocol,<0.1030.0>,"4.12.3",[{8,1},{8,0},{7,6}]}],<0.1025.0>}}
```
2. The connection are established with both nodes.

```
{1418,50172,733991} <0.1042.0> ----> mnesia_monitor {protocol_negotiated,{<0.1033.0>,#Ref<0.0.0.5136>},[a@magellan,c@magellan]}
{1418,50172,734046} <0.1030.0> ----> <0.1033.0> {#Ref<0.0.0.5136>,[a@magellan,c@magellan]}
```
3. Before the list of connected nodes is written to `recover_nodes`, Mnesia on a@magellan is stopped.

```
{1418,50172,734063} <0.1030.0> ----> <0.1033.0> {'$gen_cast',{mnesia_down,a@magellan}}
{1418,50172,734068} <0.1030.0> ----> <0.1030.0> {'$gen_cast',{mnesia_down,mnesia_controller,a@magellan}}
{1418,50172,734074} <0.1030.0> ----> <0.1034.0> {mnesia_down,a@magellan}
```
4. `mnesia_recover` overwrites the value of `recover_nodes` updated by `mnesia_down` and continue.

```
{1418,50172,734089} <0.1033.0> ----> <0.1033.0> {'$gen_cast',{announce_all,[a@magellan,c@magellan]}}
```
5. Later, to perform the merge of the schema, the value of `recover_nodes` is used to acquire a lock. Here, a@magellan is picked.

```
{1418,50172,812223} <0.1058.0> ----> {mnesia_locker,a@magellan} {<0.1058.0>,{write,{tid,122,<0.1058.0>},{schema,'______WHOLETABLE_____'}}}
```
6. a@magellan doesn't run Mnesia anymore and `mnesia_locker` on b@magellan never gets an answer.

I used the following test program to help trigger and fix the problem:
https://gist.github.com/dumbbell/7f398e3781e6b5e52a57