Project: springxd
Issue XD-3100: module.*.count > 1 duplicates messages on taps
Using module.name.count > 1 when deploying taps causes duplication of messages in those modules. This impacts balancing of the containers and modules in a cluster as messages should not be duplicated across modules if the same module is deployed twice to two containers in order to spread the load. We use taps quite heavily in our project mainly for analytics of the life feed in real time but due to issue we have discovered and described in this bug we are currently facing a limitation where heavily processing modules can not be load balanced across the cluster as they are causing duplication of the messages and therefore the same module deployed to two containers would still process the same message twice. To demonstrate the problem please see test case scenario set up below: h4. 1. Environment - Spring-XD version 1.1.1-RELEASE - Running two spring-xd containers and one spring-xd admin h4. 2. Set up Stream definition is as follows: {quote}stream create --name test-module-count --definition "syslog-udp --port=5140 | transform | log" stream deploy --name test-module-count --properties "module.*.count=2" stream create --name tap-test-module-count --definition "tap:stream:test-module-count.syslog-udp > transform --expression='payload.toString() + \"TAPPED\"' | log" stream deploy --name tap-test-module-count --properties "module.*.count=2"{quote} Please refer to the screen shots attached to see that after deploying those two streams we have: - streams successfully deployed ( module-count-spring-xd-streams.png ) - streams successfully deployed with count=2 to both containers ( module-count-spring-xd-containers.png ) - 5 queues created in Rabbit ( module-count-rabbit.png ) where two were created for the syslog-udp collector as a result of using module.syslog-udp.count=2 - this is causing messages to be duplicated. Normally the expectation would be to have only one queue for the tap h4. 3. Test input data I have sent a very simple UDP message to the listening udp collector running on second container: {quote}echo test-module-count >> /dev/udp/host02/5140{quote} h4. 4. Test output data in the logs ( module-count-container01.log and module-count-container02.log ) h5. Expected result: Below messages logged only on 1 container (it does not matter which one) {quote}2015-05-26 09:52:21,630 1.1.1.RELEASE INFO xdbus.test-module-count.1-1 sink.test-module-count - {UNDECODED=test-module-count}{quote} Below message logged only on one container (it does not matter which one) {quote}2015-05-26 09:52:21,843 1.1.1.RELEASE INFO xdbus.tap-test-module-count.0-1 sink.tap-test-module-count - {UNDECODED=test-module-count }TAPPED{quote} h5. Actual result: Stream that has been create as a tap has duplicated the same message and as a result the same message was proccessed twice on both containers by the same module ( transformer ) and logged twice to the console on both containers Container01: {quote}2015-05-26 14:52:21,143 1.1.1.RELEASE INFO xdbus.tap-test-module-count.0-1 sink.tap-test-module-count - {UNDECODED=test-module-count }TAPPED{quote} Container02: {quote}2015-05-26 09:52:21,630 1.1.1.RELEASE INFO xdbus.test-module-count.1-1 sink.test-module-count - {UNDECODED=test-module-count } 2015-05-26 09:52:21,843 1.1.1.RELEASE INFO xdbus.tap-test-module-count.0-1 sink.tap-test-module-count - {UNDECODED=test-module-count }TAPPED{quote}
Story points: 5