Adapt to latest Spring shell
XD-1458 Upgrade to Spring for Apache Hadoop 2.0 RC2
XD-1397 thru XD-1400: New ZooKeeper Runtime

this commit includes XD-1397, XD-1398, XD-1399, XD-1400 and much much more

supporting module deployment via ZooKeeper

added support for stream module parameters

added support for stream module classloader

added support for composite modules

added named channel support in stream definitions

added ZooKeeperStreamDefinitionRepository

the /streams endpoints in the REST API are now backed by this implementation.

added support for Module undeployment

removed RedisStreamDefinitionRepository and InMemoryStreamDefinitionRepository

Stream delete functionality

Introduced the ability to undeploy and delete a stream.
The Stream object has a new property in its map to
determine if the stream's modules should be deployed
upon creation. This property is exposed through
a "getter" in the Stream object and is examined
when a stream is created or updated in ZK.

Stream module graceful undeployment/destroy

When a stream is undeployed (by marking the 'deployed' flag to
'false' in the stream data in ZK) the admin listener will
remove the source, processors, and sink paths in the order
in which the stream is processed.

Furthermore, when a stream is destroyed via the REST controller,
the stream will be undeployed before it is deleted from ZK.
See StreamDeployer.beforeDelete() for details.

support for job deployment and undeployment

support for job redeployment when container departs cluster

support for job launching from shell/REST via the job queue

removed 'deployChannel' and 'undeployChannel'

When a stream is created (but not deployed) a znode will be created in the following format:

/xd/streams/stream-name

When the stream is deployed, /source, /sink/, and
(optionally) /processors znodes will be created, with
their children being the labels for the modules of those
types that belong to the stream.

When the stream is undeployed, the admin will delete
the /source, /sink, and /processors nodes to bring
the znode back to its original undeployed state:

/xd/streams/stream-name

cleanup after control bus removal

Uniquely identify deployed modules

ContainerRegistrar maintains a map of deployed modules via
ModuleDescriptor.Key. The key uses module type and label
to identify a module. This prevents the tracking of multiple
modules of the same type and label for different streams.
One of the problems this causes is the inability to undeploy
a module for a stream if a previous stream had the same module
label and type undeployed before it.

This fix introduces the stream name to the key in order
to correctly identify the module runtime instance.

Also, now using job label in key used for deploy and undeploy.

removed launch booleans from ModuleDeploymentRequests

The ZooKeeper repository implementations were attempting to
delete children in a znode (kind of like a truncate option)
but this isn't supported by ZK. Therefore the deleteAll
implementation will iterate all of the children and delete
them one by one. This isn't efficient but this is the best
that can be done via the ZK API.

splitting on colon plus space for label

Fixed AbstractSingleNodeInitializationTests

Fixed invalid map references

Fixed AbstractSingleNodeInitializationTests

Added embeddedZooKeeper.start() in setup

Modified Curator path cache listener

Modified the Curator path cache implementation to
correctly add and remove listeners for a path's
children.

Fixed ZooKeeperClientConnectionTests. Make RetryPolicy Pluggable

spring-xd-dirt/src/test/java/org/springframework/xd/dirt/stream/AbstractSingleNodeStreamDeploymentIntegrationTests.java

fixed module indices and test's undeploy order

reversing order of parsed modules for StreamFactory

no longer reverse order in Stream build

avoiding test crosstalk

support named channels on both ends of a stream

ZooKeeperStreamDefinitionRepositoryTests pass; minor cleanup

parser needs StreamDefinitionRepository for taps

refactoring StreamTestSupport

ensuring the /xd/streams path exists

increased test sink polling timeouts

Added stream name to getModule in SingleNodeIntegrationTestSupport

initialize StreamDefinitionRepository before tests

creating /xd/streams with create instead of ensurePath

JobsContollerIntegrationTests passing

StreamsControllerIntegrationWithRepositoryTests passing

fixed check for composite module

removed StreamPluginModuleDeploymentTests and created a similar test in AbstractSingleNodeStreamDeploymentIntegrationTests

using creatingParentsIfNeeded instead of ensurePath

properly deleting module dependencies for streams

tap can be deleted after its stream

Implemented paging in ZKSDR

xd-shell-test work in progress

ZooKeeperConnection autoStartup=true

Fix shell tests - work in progress

- Added DeploymentsListener in shell tests that would check deploy/undeploy event before any assertions

 Added DefinitionsListener (wip)

Fix hanging FileSink tests

- Throw RuntimeException instead of returning false after waitFor times out

Implemented ZooKeeperStreamRepository

Added listeners for Stream and Job commands

corrected path for JobCommandListener

increased test file sink timeout

implemented ZooKeeper backed RuntimeContainerInfoRepository

Fix ModuleCommandTests

improving reliability of command listeners

using JobsPath

improved reliability of JobCommandTests

implemented ZooKeeperJobRepository

Fix AggregatorProcessorModuleOptionsMetadata's release prop

launch error is now async

Fix XdConfigLoggingInitializer

- Handle Embedded zookeeper connect

Fix SingleNode non-embedded ZooKeeperServer

- Also moved the properties to be referred from SharedServerContextConfiguration

Fixed ZooKeeperClientConnectionTests

Fixed compile warnings

Improved stream-command-tests

Misc cleanup, Javadoc, refactoring

Removed PathChildrenCache for jobs and streams as members
of AdminServer and placed them as local variables in
takeLeadership since they are only used in that method.

Added JavaDoc to AdminServer.

Minor formatting.

XD-1427 upgrade zookeeper to 3.4.6

Misc cleanup

Mostly JavaDoc, whitespace, and minor refactoring.

XD-1471 Migrate repositories to ZooKeeper

Removed repository implementations that are no longer used

ContainerMetadataRepository

Removed container events and listeners

container now writes ephemeral metadata node per deployed module

added ZooKeeperModuleDependencyRepository

added ZooKeeperModuleDefinitionRepository

Add ZooKeeperModuleMetadataRepository

- Cleanup RuntimeModulesInfoRepository, resource assemblers, module event listeners
- Add ZooKeeperModuleMetadataRepository and update store
- Update Controller for runtime modules command

Fix RuntimeCommandTests

Fix ModulesControllerIntegrationTests

- Check existence of data in case of composed module definitions in the /xd/modules path

Fix Job undeploy

- Add deletingChildrenIfNeeded() so that the job node gets deleted along with child node (metadata)

Fix StreamTestSupport

- Use ModuleDeployer's deployedModules to get the Module info by streamName and index

Removed the --store option

when deleting a stream, we are now explicitly traversing child nodes so we can avoid NoNodeExceptions
those might occur if the stream is in the process of being undeployed when destroy is called

Introduced RepositoryConnectionListener

RepositoryConnectionListener creates the required paths
for the repository implementations that use ZooKeeper.

This class replaces the static inner classes previously
used by ZooKeeperJobDefinitionRepository and
ZooKeeperStreamDefinitionRepository.

Fix script tests

- Add explicit sleep to wait for the stream/job to get deployed before acting on that stream/job
- Handle adding tap on filejdbc module after the module is created but before it is deployed.

Suppress NodeExists log message

When invoking invoking Curator's creatingParentsIfNeeded node builder,
if the parent does not exist the following is logged at INFO level:

14:28:40,189  INFO ProcessThread(sid:0 cport:-1): server.PrepRequestProcessor:645
Got user-level KeeperException when processing sessionid:0x14523b328600000
type:create cxid:0x5 zxid:0x5 txntype:-1 reqpath:n/a Error Path:/xd/streams
Error:KeeperErrorCode = NodeExists for /xd/streams

This is harmless but very confusing for people reading the logs.
The modifications to the log4j.properties in this commit will
suppress INFO level messages from this ZooKeeper class.

Provided names for PathChildrenCache threads

When reading a thread dump or stack trace generated
by a PathChildrenCache thread it is helpful to
know which cache that thread is serving. This
change provides a meaningful name to the thread
Curator creates for these caches.

Also introduced an AtomicReference for the containers
cache. The idea is that the reference isn't set
until the cache is started and the reference is
removed before the cache is closed.

Fix NoNodeException in tests

- Suppress org.apache.zookeeper.KeeperException when repositories reset
- Fix zk Stream/Job definition repositories deleteAll()
  - continue on NoNodeException here
- Make delete(), deleteAll() operations in ZKModuleMetadata repos no-op
- Fix streamName in AbstractSingleNodeStreamDeploymentIntegrationTests

Speedup build process

- Support tests to run in parallel
  - Fix some of the tests so that they can run in parallel.
  - Configure maxParallelForks to use available core
  - Increase forkEvery value for spring-xd-dirt tests and for the rest of the tests.
- Modify jvm args for compile and test

polished deleteAll() methods on merge
XD-1278 Renaming avro sink to hdfs-dataset

- Adding support for parquet in addition to avro
  - use new option --format=parquet
  - for this option to work the payload currently MUST be a JavaBean with properties for all fields to be stored

- Adding support for not allowing nulls
  - this avoids generating an Avro UNION for every field in the schema
  - use new option --allowNullValues=false
Make parallel tests configurable

- Default maxParallelForks to 1 if not provided as a property.
  - To override, -PmaxParallelForks=<?>
- Fix jdbc_tests to add sleep in between creation/deployment of job and tap on job execution listener output
Update refs to spring.io
XD-1444: Update to boot 1.0 release
XD-1449 Upgrade Spring Shell to 1.1 RC1
Fix gemfire & twitter script tests

- Added explicit sleep to make sure there are no timeout issues.
shutdown leader Admin on context stop
XD-1483, XD-1484, and XD-1485

XD-1485: Modify module deployments path

https://jira.spring.io/browse/XD-1485

Modified deployments path from /xd/deployments/[container]
to /xd/deployments/modules/[container]. This will pave the
way to allowing for the use of /xd/deployments/streams
to handle stream deployment.

XD-1483: StreamListener watches /xd/deployments/streams

https://jira.spring.io/browse/XD-1483

XD-1484: JobListener watches /xd/deployments/jobs

https://jira.spring.io/browse/XD-1484

Removed "deploy" flag from Streams and Jobs. Deployments are
now handled via additions/removals from /xd/deployments/streams
and /xd/deployments/jobs.

merged after addressing review comments
fix race condition for stream properties
XD-1441: Implement XD_MODULE_CONFIG_LOCATION and XD_MODULE_CONFIG_NAME

* Supports leaf propety lookup with _<profile> suffix
* Supports root module config file with _<profile> suffix
Fix dependencies of PMML module
XD-1459: Insulate module from Environment
XD-1208 Add Module options for OOTB jobs

- Add HdfsMongodbJobOptionsMetadata for hdfsmongodb batch job module
  - Update hdfsmongodb module
- Add module option Mixins that are common to batch job modules
  - Add mixins for BatchJobDeleteFilesOptionMixin, BatchJobFieldNamesOptionMixin, BatchJobResourcesOptionMixin,
BatchJobRestartableOptionMixin
  - Update existing OptionsMetadata pojos
- Fix jdbc_test to have more sleep between job_create and job_deploy

XD-1208 Remove idField default

- Specify "null" string as default in hdfsmongodb.xml

XD-1208 Better handling of idField

- Update hdfsmongodb.xml
  - If idField is empty, then the converter would use Tuple's id
- Fix Assertions in converter

Various fixes