xing/beetle — Issue #18: How the RCC-server copes with the fall of the rabbit-server (cluster)?
Author: clarkdevis
Created: 2012-02-01 12:17:32+00:00
State: open

* Software:
* rabbitmq-server: 2.6.1-1
* redis-server: 2:2.4.2-1~bpo60+1
* beetle: 0.3.0.rc.11

I have three servers with ip-addresses: 192.168.111.170 (www1), 192.168.111.171 (www2), 192.168.111.172 (www3).

__On the server WWW1:__
service redis-server start
service rabbitmq-server start
rabbitmqctl cluster_status
Cluster status of node rabbit@www1 ...
[{nodes,[{disc,[rabbit@www1,rabbit@www3,rabbit@www2]}]},
{running_nodes,[rabbit@www3,rabbit@www2,rabbit@www1]}]
...done.

beetle configuration_server start -- --verbose --redis-servers 192.168.111.170:6379,192.168.111.171:6379,192.168.111.172:6379 --redis-master-file /root/beetle-redis-master-file --amqp-servers 192.168.111.170:5672,192.168.111.171:5672,192.168.111.172:5672 --pid-dir /var/log/beetle

beetle configuration_client start -- --verbose --client-id rcc-1 --redis-master-file /root/beetle-redis-master-file --amqp-servers 192.168.111.170:5672,192.168.111.171:5672,192.168.111.172:5672 --pid-dir /var/log/beetle

Log:
I, [2012-02-01 14:59:31#10500] INFO -- : Beetle: connecting to rabbit 192.168.111.170:5672
I, [2012-02-01 14:59:31#10500] INFO -- : Beetle: connecting to rabbit 192.168.111.171:5672
I, [2012-02-01 14:59:31#10500] INFO -- : Beetle: connecting to rabbit 192.168.111.172:5672

__On the server WWW2:__
service redis-server start
service rabbitmq-server start

beetle configuration_client start -- --verbose --client-id rcc-2 --redis-master-file /root/beetle-redis-master-file --amqp-servers 192.168.111.170:5672,192.168.111.171:5672,192.168.111.172:5672 --pid-dir /var/log/beetle

__On the server WWW3:__
service redis-server start
service rabbitmq-server start

beetle configuration_client start -- --verbose --client-id rcc-3 --redis-master-file /root/beetle-redis-master-file --amqp-servers 192.168.111.170:5672,192.168.111.171:5672,192.168.111.172:5672 --pid-dir /var/log/beetle

Everything works, including all of the RCC-processes. They receive messages from the RCS-process.

Strange scenario:

WWW1: service rabbitmq-server stop
RCC1 Log:
W, [2012-02-01 15:04:28#10500] WARN -- : Beetle: lost connection: 192.168.111.170:5672. reconnecting.
W, [2012-02-01 15:04:28#10500] WARN -- : Beetle: lost connection: 192.168.111.170:5672. reconnecting.

RSS1 Log:
D, [2012-02-01 15:05:41#10483] DEBUG -- : Beetle: sending reconfigure
D, [2012-02-01 15:05:41#10483] DEBUG -- : Beetle: trying to send message reconfigure:ae545770-4cc4-11e1-a7bb-97834f7db210 to 192.168.111.170:5672
W, [2012-02-01 15:05:41#10483] WARN -- : Beetle: error closing down bunny Connection refused - connect(2)
D, [2012-02-01 15:05:41#10483] DEBUG -- : Beetle: trying to send message reconfigure:ae545770-4cc4-11e1-a7bb-97834f7db210 to 192.168.111.170:5672
W, [2012-02-01 15:05:41#10483] WARN -- : Beetle: error closing down bunny Connection refused - connect(2)
I, [2012-02-01 15:05:41#10483] INFO -- : Beetle: server 192.168.111.170:5672 down: Connection refused - connect(2)
D, [2012-02-01 15:05:41#10483] DEBUG -- : Beetle: trying to send message reconfigure:ae545770-4cc4-11e1-a7bb-97834f7db210 to 192.168.111.171:5672
D, [2012-02-01 15:05:41#10483] DEBUG -- : Beetle: message sent!
....
I, [2012-02-01 15:06:52#10483] INFO -- : Publishing reconfigure message with server '192.168.111.171:6379'
D, [2012-02-01 15:06:52#10483] DEBUG -- : Beetle: sending reconfigure
D, [2012-02-01 15:06:52#10483] DEBUG -- : Beetle: trying to send message reconfigure:d82c46b6-4cc4-11e1-a58d-1b3141ff5712 to 192.168.111.172:5672
D, [2012-02-01 15:06:52#10483] DEBUG -- : Beetle: message sent!

RCC2:
W, [2012-02-01 15:07:39#9306] WARN -- : Beetle: lost connection: 192.168.111.170:5672. reconnecting.
W, [2012-02-01 15:07:49#9306] WARN -- : Beetle: lost connection: 192.168.111.170:5672. reconnecting.

RCC3:
W, [2012-02-01 15:08:19#5026] WARN -- : Beetle: lost connection: 192.168.111.170:5672. reconnecting.
D, [2012-02-01 15:08:22#5026] DEBUG -- : Beetle: processing message msgid:system_reconfigure_rcc-3:0df9069e-4cc5-11e1-b599-c3fcbcee9bd7
D, [2012-02-01 15:08:22#5026] DEBUG -- : Beetle: ack! for message msgid:system_reconfigure_rcc-3:0df9069e-4cc5-11e1-b599-c3fcbcee9bd7
I, [2012-02-01 15:08:22#5026] INFO -- : Received reconfigure message with server '192.168.111.171:6379' and token '1328093960543'
D, [2012-02-01 15:08:22#5026] DEBUG -- : Beetle: message processing completed

So, what am I doing wrong?

The documentation was written:
beetle configuration_client start -- --help
--amqp-servers LIST AMQP server list (e.g. 192.168.0.1:5672,192.168.0.2:5672)

Do I understand that if stop the first AMQP-server, then the RCC / RCS should start receiving messages from the other?