hornetq/hornetq — pull request #1304 by gaohoward (2013-09-29 13:58:23+00:00)
Title: Fixing random failure of test org.hornetq.tests.integration.cluster.failover.BackupSyncLargeMessageTest.testDeleteLargeMessages()

Cause of the failure:

This test sets up a live-backup, sends a few large messages, consumes half of them before crash the server and failover, and check the
large messages dir of the backup that the half consumed large message files are deleted.

The deletion of each large message file from the backup is driven by a ReplicationLargeMessageEndMessage sent from live via the
replication channel asynchronously. When server stops and notifies the backup to failover

(see HornetQServerImpl.stop(...))

replicationManager.sendLiveIsStopping(LiveStopping.STOP_CALLED);

it is possible that some of those ReplicationLargeMessageEndMessages are still cached at the live server side even after the backup
has received the 'STOP_CALLED' message and initiates failover process. This may leave some of those messages un-processed and so those
large messages are not deleted from backup's large message dir, resulting in assertion failure at the end of test.

How to Fix:

Before sending out the STOP_CALLED message to backup, the live should flush the replication connections of any pending messages.

//this make sure all pending replication messages are flushed before failover
localReplicationManager.getBackupTransportConnection().getTransportConnection().checkFlushBatchBuffer();
replicationManager.sendLiveIsStopping(LiveStopping.STOP_CALLED);