spring-projects/spring-xd — pull request #690 by markfisher (2014-04-02 12:26:33+00:00)
Title: XD-1397 thru XD-1400: New ZooKeeper Runtime

this commit includes XD-1397, XD-1398, XD-1399, XD-1400 and much much more

supporting module deployment via ZooKeeper

added support for stream module parameters

added support for stream module classloader

added support for composite modules

added named channel support in stream definitions

added ZooKeeperStreamDefinitionRepository

the /streams endpoints in the REST API are now backed by this implementation.

added support for Module undeployment

removed RedisStreamDefinitionRepository and InMemoryStreamDefinitionRepository

Stream delete functionality

Introduced the ability to undeploy and delete a stream.
The Stream object has a new property in its map to
determine if the stream's modules should be deployed
upon creation. This property is exposed through
a "getter" in the Stream object and is examined
when a stream is created or updated in ZK.

Stream module graceful undeployment/destroy

When a stream is undeployed (by marking the 'deployed' flag to
'false' in the stream data in ZK) the admin listener will
remove the source, processors, and sink paths in the order
in which the stream is processed.

Furthermore, when a stream is destroyed via the REST controller,
the stream will be undeployed before it is deleted from ZK.
See StreamDeployer.beforeDelete() for details.

support for job deployment and undeployment

support for job redeployment when container departs cluster

support for job launching from shell/REST via the job queue

removed 'deployChannel' and 'undeployChannel'

When a stream is created (but not deployed) a znode will be created in the following format:

/xd/streams/stream-name

When the stream is deployed, /source, /sink/, and
(optionally) /processors znodes will be created, with
their children being the labels for the modules of those
types that belong to the stream.

When the stream is undeployed, the admin will delete
the /source, /sink, and /processors nodes to bring
the znode back to its original undeployed state:

/xd/streams/stream-name

cleanup after control bus removal

Uniquely identify deployed modules

ContainerRegistrar maintains a map of deployed modules via
ModuleDescriptor.Key. The key uses module type and label
to identify a module. This prevents the tracking of multiple
modules of the same type and label for different streams.
One of the problems this causes is the inability to undeploy
a module for a stream if a previous stream had the same module
label and type undeployed before it.

This fix introduces the stream name to the key in order
to correctly identify the module runtime instance.

Also, now using job label in key used for deploy and undeploy.

removed launch booleans from ModuleDeploymentRequests

The ZooKeeper repository implementations were attempting to
delete children in a znode (kind of like a truncate option)
but this isn't supported by ZK. Therefore the deleteAll
implementation will iterate all of the children and delete
them one by one. This isn't efficient but this is the best
that can be done via the ZK API.

splitting on colon plus space for label

Fixed AbstractSingleNodeInitializationTests

Fixed invalid map references

Fixed AbstractSingleNodeInitializationTests

Added embeddedZooKeeper.start() in setup

Modified Curator path cache listener

Modified the Curator path cache implementation to
correctly add and remove listeners for a path's
children.

Fixed ZooKeeperClientConnectionTests. Make RetryPolicy Pluggable

spring-xd-dirt/src/test/java/org/springframework/xd/dirt/stream/AbstractSingleNodeStreamDeploymentIntegrationTests.java

fixed module indices and test's undeploy order

reversing order of parsed modules for StreamFactory

no longer reverse order in Stream build

avoiding test crosstalk

support named channels on both ends of a stream

ZooKeeperStreamDefinitionRepositoryTests pass; minor cleanup

parser needs StreamDefinitionRepository for taps

refactoring StreamTestSupport

ensuring the /xd/streams path exists

increased test sink polling timeouts

Added stream name to getModule in SingleNodeIntegrationTestSupport

initialize StreamDefinitionRepository before tests

creating /xd/streams with create instead of ensurePath

JobsContollerIntegrationTests passing

StreamsControllerIntegrationWithRepositoryTests passing

fixed check for composite module

removed StreamPluginModuleDeploymentTests and created a similar test in AbstractSingleNodeStreamDeploymentIntegrationTests

using creatingParentsIfNeeded instead of ensurePath

properly deleting module dependencies for streams

tap can be deleted after its stream

Implemented paging in ZKSDR

xd-shell-test work in progress

ZooKeeperConnection autoStartup=true

Fix shell tests - work in progress

- Added DeploymentsListener in shell tests that would check deploy/undeploy event before any assertions

Added DefinitionsListener (wip)

Fix hanging FileSink tests

- Throw RuntimeException instead of returning false after waitFor times out

Implemented ZooKeeperStreamRepository

Added listeners for Stream and Job commands

corrected path for JobCommandListener

increased test file sink timeout

implemented ZooKeeper backed RuntimeContainerInfoRepository

Fix ModuleCommandTests

improving reliability of command listeners

using JobsPath

improved reliability of JobCommandTests

implemented ZooKeeperJobRepository

Fix AggregatorProcessorModuleOptionsMetadata's release prop

launch error is now async

Fix XdConfigLoggingInitializer

- Handle Embedded zookeeper connect

Fix SingleNode non-embedded ZooKeeperServer

- Also moved the properties to be referred from SharedServerContextConfiguration

Fixed ZooKeeperClientConnectionTests

Fixed compile warnings

Improved stream-command-tests

Misc cleanup, Javadoc, refactoring

Removed PathChildrenCache for jobs and streams as members
of AdminServer and placed them as local variables in
takeLeadership since they are only used in that method.

Added JavaDoc to AdminServer.

Minor formatting.

XD-1427 upgrade zookeeper to 3.4.6

Misc cleanup

Mostly JavaDoc, whitespace, and minor refactoring.