XD-1428 Add more Container logging info

- Display Hadoop Distro, Zookeeper client connect string info
- Fix  javadoc link in MessageBusAwareChannelResolver

Improve logging options

- Specify the hadoopDistro config for the hadoop version being used

Fix review comments

- Specify default hadoopDistro option in ContainerOptions
- Update comments on xd scripts

XD-1428 Update env for singlenode

- Instead of setting zkConnectString, update the environment with
zkClientConnect string (use Embedded zookeeper server's port once available)

XD-1428 Fix code review comments

- Change logging text based on the suggestions
- In case of version mismatch, log error instead of info
- Fix typo in var name

XD-1428 Refactor ZookeeperConnectionConfiguration

- Make both the ZooKeeperConnectionConfiguration classes environment aware so that zk-properties is
always available in the environment.

XD-1428 Make cmdLineArgs propertysource available

- Make cmdLineArgs propertysource available even if there are no arguments being passed.
In this case, the cmdLineArgs will have the default options set.
- Update SingleNodeOptions with default hadoopDistro option
Workaround for intermittent HSQLDB start failure

When running the build there is an occasional HSQLDB startup failure.
The root cause for this failure is:

  java.net.SocketException: Invalid argument

This exception is thrown on JDK 7 OS X when there are > 1024
open file descriptors. See this Stack Overflow thread:

http://stackoverflow.com/questions/16191236/tomcat-startup-fails-due-to-java-net-socketexception-invalid-argument-on-mac-o

This is fixed in Java 7u60:

https://bugs.openjdk.java.net/browse/JDK-8021820

In the meantime, this workaround will trap that exception and
make several attempts at starting up HSQLDB before giving up.

Previously it was assumed that a startup failure was due to
a port conflict. This is unlikely because an open port is
selected by the test prior to configuring HSQLDB. If this does
happen the log file will indicate so:

org.springframework.beans.factory.BeanCreationException:
  Error creating bean with name 'hsqldb' defined in class...
...
Caused by: java.lang.IllegalStateException: HSQLDB could not be started on 0.0.0.0:6379, state: SHUTDOWN
...
Caused by: java.net.BindException: Address already in use
	at java.net.PlainSocketImpl.socketBind(Native Method)
	at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:376)
...

Rework of the while loop

Modified the while loop to use a boolean instead of
always forcing a break. A server start failure
still causes a break. This avoids the need for
another boolean to track whether an error occurred.
Added command line option to generate heap dumps

If a test generates an OutOfMemoryError, a heap
dump will be generated that can be used to determine
the root cause.

See http://docs.oracle.com/javase/7/docs/webnotes/tsg/TSG-VM/html/clopts.html
XD-1445: Rethrow unexpected exceptions in completion logic

- Refine caught throwable to Exception
- Add Javadocs
XD-1464 - Upgrade to Spring Integration 4.0.0.M4
XD-1460 Enable JMX by default

- Removed jmxEnabled commandLine option
- Set XD_JMX_ENABLED by default in application.yml
- Added configs in xd-config.yml
- Disable JMX by default in RandomConfigurationSupport
  - This will allow the tests to run without JMX enabled.
- Fixed xd.container JMX MbeanExporter bean configuration
  - Moved the bean configuration from ContainerServerApplication to ContainerConfiguration
XD-1465 - Update management context path

- Boot endpoints are server management.contextPath
  - Update this path to use '/management'
XD-1453 Update Spring-AMQP

Spring-AMQP : 1.3.1.RELEASE
Spring-Integration : 4.0.0.M4
XD-1324 Changing default Hadoop version to Hadoop 2.2.0 (hadoop22)
XD-1467: Defer access to repo in completion logic

Address review comments
XD-1468 Updates Reactor to version 1.1.0.M3
Add sleep to see if it fixes CI script tests
XD-1470: Exclude jackson jars from module lib dirs
XD-1827: Reject taps on non-existent streams.

When calling ChannelNode.resolve on a stream tap, a lookup on the stream
name is performed before continuing and an exception raised if no matching
stream is found.
Adapt to latest Spring shell
XD-1458 Upgrade to Spring for Apache Hadoop 2.0 RC2
XD-1397 thru XD-1400: New ZooKeeper Runtime

this commit includes XD-1397, XD-1398, XD-1399, XD-1400 and much much more

supporting module deployment via ZooKeeper

added support for stream module parameters

added support for stream module classloader

added support for composite modules

added named channel support in stream definitions

added ZooKeeperStreamDefinitionRepository

the /streams endpoints in the REST API are now backed by this implementation.

added support for Module undeployment

removed RedisStreamDefinitionRepository and InMemoryStreamDefinitionRepository

Stream delete functionality

Introduced the ability to undeploy and delete a stream.
The Stream object has a new property in its map to
determine if the stream's modules should be deployed
upon creation. This property is exposed through
a "getter" in the Stream object and is examined
when a stream is created or updated in ZK.

Stream module graceful undeployment/destroy

When a stream is undeployed (by marking the 'deployed' flag to
'false' in the stream data in ZK) the admin listener will
remove the source, processors, and sink paths in the order
in which the stream is processed.

Furthermore, when a stream is destroyed via the REST controller,
the stream will be undeployed before it is deleted from ZK.
See StreamDeployer.beforeDelete() for details.

support for job deployment and undeployment

support for job redeployment when container departs cluster

support for job launching from shell/REST via the job queue

removed 'deployChannel' and 'undeployChannel'

When a stream is created (but not deployed) a znode will be created in the following format:

/xd/streams/stream-name

When the stream is deployed, /source, /sink/, and
(optionally) /processors znodes will be created, with
their children being the labels for the modules of those
types that belong to the stream.

When the stream is undeployed, the admin will delete
the /source, /sink, and /processors nodes to bring
the znode back to its original undeployed state:

/xd/streams/stream-name

cleanup after control bus removal

Uniquely identify deployed modules

ContainerRegistrar maintains a map of deployed modules via
ModuleDescriptor.Key. The key uses module type and label
to identify a module. This prevents the tracking of multiple
modules of the same type and label for different streams.
One of the problems this causes is the inability to undeploy
a module for a stream if a previous stream had the same module
label and type undeployed before it.

This fix introduces the stream name to the key in order
to correctly identify the module runtime instance.

Also, now using job label in key used for deploy and undeploy.

removed launch booleans from ModuleDeploymentRequests

The ZooKeeper repository implementations were attempting to
delete children in a znode (kind of like a truncate option)
but this isn't supported by ZK. Therefore the deleteAll
implementation will iterate all of the children and delete
them one by one. This isn't efficient but this is the best
that can be done via the ZK API.

splitting on colon plus space for label

Fixed AbstractSingleNodeInitializationTests

Fixed invalid map references

Fixed AbstractSingleNodeInitializationTests

Added embeddedZooKeeper.start() in setup

Modified Curator path cache listener

Modified the Curator path cache implementation to
correctly add and remove listeners for a path's
children.

Fixed ZooKeeperClientConnectionTests. Make RetryPolicy Pluggable

spring-xd-dirt/src/test/java/org/springframework/xd/dirt/stream/AbstractSingleNodeStreamDeploymentIntegrationTests.java

fixed module indices and test's undeploy order

reversing order of parsed modules for StreamFactory

no longer reverse order in Stream build

avoiding test crosstalk

support named channels on both ends of a stream

ZooKeeperStreamDefinitionRepositoryTests pass; minor cleanup

parser needs StreamDefinitionRepository for taps

refactoring StreamTestSupport

ensuring the /xd/streams path exists

increased test sink polling timeouts

Added stream name to getModule in SingleNodeIntegrationTestSupport

initialize StreamDefinitionRepository before tests

creating /xd/streams with create instead of ensurePath

JobsContollerIntegrationTests passing

StreamsControllerIntegrationWithRepositoryTests passing

fixed check for composite module

removed StreamPluginModuleDeploymentTests and created a similar test in AbstractSingleNodeStreamDeploymentIntegrationTests

using creatingParentsIfNeeded instead of ensurePath

properly deleting module dependencies for streams

tap can be deleted after its stream

Implemented paging in ZKSDR

xd-shell-test work in progress

ZooKeeperConnection autoStartup=true

Fix shell tests - work in progress

- Added DeploymentsListener in shell tests that would check deploy/undeploy event before any assertions

 Added DefinitionsListener (wip)

Fix hanging FileSink tests

- Throw RuntimeException instead of returning false after waitFor times out

Implemented ZooKeeperStreamRepository

Added listeners for Stream and Job commands

corrected path for JobCommandListener

increased test file sink timeout

implemented ZooKeeper backed RuntimeContainerInfoRepository

Fix ModuleCommandTests

improving reliability of command listeners

using JobsPath

improved reliability of JobCommandTests

implemented ZooKeeperJobRepository

Fix AggregatorProcessorModuleOptionsMetadata's release prop

launch error is now async

Fix XdConfigLoggingInitializer

- Handle Embedded zookeeper connect

Fix SingleNode non-embedded ZooKeeperServer

- Also moved the properties to be referred from SharedServerContextConfiguration

Fixed ZooKeeperClientConnectionTests

Fixed compile warnings

Improved stream-command-tests

Misc cleanup, Javadoc, refactoring

Removed PathChildrenCache for jobs and streams as members
of AdminServer and placed them as local variables in
takeLeadership since they are only used in that method.

Added JavaDoc to AdminServer.

Minor formatting.

XD-1427 upgrade zookeeper to 3.4.6

Misc cleanup

Mostly JavaDoc, whitespace, and minor refactoring.

XD-1471 Migrate repositories to ZooKeeper

Removed repository implementations that are no longer used

ContainerMetadataRepository

Removed container events and listeners

container now writes ephemeral metadata node per deployed module

added ZooKeeperModuleDependencyRepository

added ZooKeeperModuleDefinitionRepository

Add ZooKeeperModuleMetadataRepository

- Cleanup RuntimeModulesInfoRepository, resource assemblers, module event listeners
- Add ZooKeeperModuleMetadataRepository and update store
- Update Controller for runtime modules command

Fix RuntimeCommandTests

Fix ModulesControllerIntegrationTests

- Check existence of data in case of composed module definitions in the /xd/modules path

Fix Job undeploy

- Add deletingChildrenIfNeeded() so that the job node gets deleted along with child node (metadata)

Fix StreamTestSupport

- Use ModuleDeployer's deployedModules to get the Module info by streamName and index

Removed the --store option

when deleting a stream, we are now explicitly traversing child nodes so we can avoid NoNodeExceptions
those might occur if the stream is in the process of being undeployed when destroy is called

Introduced RepositoryConnectionListener

RepositoryConnectionListener creates the required paths
for the repository implementations that use ZooKeeper.

This class replaces the static inner classes previously
used by ZooKeeperJobDefinitionRepository and
ZooKeeperStreamDefinitionRepository.

Fix script tests

- Add explicit sleep to wait for the stream/job to get deployed before acting on that stream/job
- Handle adding tap on filejdbc module after the module is created but before it is deployed.

Suppress NodeExists log message

When invoking invoking Curator's creatingParentsIfNeeded node builder,
if the parent does not exist the following is logged at INFO level:

14:28:40,189  INFO ProcessThread(sid:0 cport:-1): server.PrepRequestProcessor:645
Got user-level KeeperException when processing sessionid:0x14523b328600000
type:create cxid:0x5 zxid:0x5 txntype:-1 reqpath:n/a Error Path:/xd/streams
Error:KeeperErrorCode = NodeExists for /xd/streams

This is harmless but very confusing for people reading the logs.
The modifications to the log4j.properties in this commit will
suppress INFO level messages from this ZooKeeper class.

Provided names for PathChildrenCache threads

When reading a thread dump or stack trace generated
by a PathChildrenCache thread it is helpful to
know which cache that thread is serving. This
change provides a meaningful name to the thread
Curator creates for these caches.

Also introduced an AtomicReference for the containers
cache. The idea is that the reference isn't set
until the cache is started and the reference is
removed before the cache is closed.

Fix NoNodeException in tests

- Suppress org.apache.zookeeper.KeeperException when repositories reset
- Fix zk Stream/Job definition repositories deleteAll()
  - continue on NoNodeException here
- Make delete(), deleteAll() operations in ZKModuleMetadata repos no-op
- Fix streamName in AbstractSingleNodeStreamDeploymentIntegrationTests

Speedup build process

- Support tests to run in parallel
  - Fix some of the tests so that they can run in parallel.
  - Configure maxParallelForks to use available core
  - Increase forkEvery value for spring-xd-dirt tests and for the rest of the tests.
- Modify jvm args for compile and test

polished deleteAll() methods on merge
XD-1278 Renaming avro sink to hdfs-dataset

- Adding support for parquet in addition to avro
  - use new option --format=parquet
  - for this option to work the payload currently MUST be a JavaBean with properties for all fields to be stored

- Adding support for not allowing nulls
  - this avoids generating an Avro UNION for every field in the schema
  - use new option --allowNullValues=false
Make parallel tests configurable

- Default maxParallelForks to 1 if not provided as a property.
  - To override, -PmaxParallelForks=<?>
- Fix jdbc_tests to add sleep in between creation/deployment of job and tap on job execution listener output
Update refs to spring.io