openstack/oslo.messaging — pull request #1 by roopakparikh (2014-07-03 23:51:20+00:00)
Title: Add heartbeat support to kombu

This change adds heartbeat support for kombu by passing in heartbeat parameter
of default value 60 second and adding a thread(eventlet) that spawns after heartbeat/2
seconds to check for the connection.

This is a critical bug fix for cases where network connections are not reliable.
One of my test scenario had a NAT between nova-compute and the nova-conductor/nova-scheduler.
NAT devices are known to drop idle connections without telling corresponding parties of
the state of the connection. This behavior caused interesting 'hung' operation on nova side.

Here are the details:
Setup:

Nova-Conduct/Scheduler are running on a machine 'P' along with Broker
Nova-compute C is running on a virtual machine local at my workplace behind a NAT.
Both P, C are using Kombu 3.0.16 (https://pypi.python.org/pypi/kombu)

P creates a 'topic' exchange and create a Queue (nova_compute_) which needs acknowledgements.
C waits for the message upon arrival sends an ack back to B.

Scenarios:

Positive:
P and C are connected to B, P publishes messages, C receives it and everything works fine.

Negative:
While C is connected to B,

step-1) I disconnect the 'virtual network' which disconnects the virtual machine C is running on.
So effectively C is not reachable anymore. (This should be similar to unplugging network cable to a machine)

At this point of time, I observed the following on broker machine:

rabbitmqctl list_connections name state timeout : it shows the connection between the B and C as running, timeout value=0
rabbitmqctl list_queues name messages_ready messages_unacknowledged: shows name and both ready and unacked messages =0

step-2) I then publish a message from P on the Q.

Now

rabbitmqctl list_connections name state timeout : it shows the connection between the B and C as running, timeout value=0
rabbitmqctl list_queues name messages_ready messages_unacknowledged: shows name and both ready =0 and unacked messages =1

step-3) I then publish another message from P on the Q.

Now

rabbitmqctl list_connections name state timeout : it shows the connection between the B and C as running, timeout value=0
rabbitmqctl list_queues name messages_ready messages_unacknowledged: shows name and both ready =0 and unacked messages =2

step-4) I plug 'connect' the network for C again but I never see the messages on the C they remain in the unacked state > 8 hrs (I didn't check after that).

Solution:
After the fix, the timeout value in Step-1,2,3 = 60 and step-4 succeeds and the messages are delivered.

Some references:

http://kombu.readthedocs.org/en/latest/reference/kombu.connection.html
https://www.rabbitmq.com/amqp-0-9-1-reference.html
https://github.com/celery/py-amqp/