modularize build.gradle
downgraded some dirt dependencies
Restore 'compile' dep to spring-xd-hadoop
XD-2041: Check asciidoc cross ref links
XD-2067 update asciidoctor toolchain
XD-2058: Renamed DeploymentVerifier

Renamed DeploymentVerifier to ResourceStateVerifier.

Added methods to SingleNodeIntegrationTestSupport to allow blocking
until incomplete status.
reverted to compile scope on 'spring-integration-groovy' for java 8 compiles
XD-2004: Containers stopped responding to Admin

Fixed race condition between the container creating the
/xd/deployments/modules/allocated/container-id path and the
supervisor removing the path when the container exits.
The container will now wait for the supervisor to remove
the path before proceeding with startup.

Added onSuspend and onResume to ZooKeeperConnectionListener.
This allows us to distinguish between events where container
ephemeral nodes need to be created after a network disconnect.

Temporary network disconnects will raise SUSPENDED events
followed by RECONNECTED events. Containers will now avoid
undeploying modules upon SUSPENDED events. LOST events still
cause modules to be undeployed. RECONNECTED events will
force a shutdown and recreation of the ZooKeeper client
if the prior event was LOST. If the prior event was
SUSPENDED, the container will re-register if the existing
ephemeral node for that container was created by the previous
ZooKeeper client session.

Testing was done on Linux via iptables DROP rules. Additionally,
testing was performed with a ZooKeeper ensemble, including
the shut down of 2 out of 3 servers to disrupt quorum.
XD-2047 Expose Redis Bus Headers Customization

JIRA: https://jira.spring.io/browse/XD-2047

The redis bus supports passing custom (string-valued) headers but
there was no easy way to configure them.

Add a property to `application.yml`, `servers.yml` to allow
customization.
XD-2015 Fix UI end-to-end (E2E) tests

* Made the documentation (README.md) clearer regarding issue `Could not find chromedriver...`
* Due to the recently introduced addition of polling the E2E tests were partially broken also.
  Add `browser.ignoreSynchronization=true` to the tests as described at https://github.com/angular/protractor/issues/49

XD-2015 Reference `protractor.conf.js` explicitly

* Bump versions in `package.json`
* Have NPM execute `webdriver-manager update`

XD-2015 Polishing, updating `README.md`
XD-1329 Add Kafka source module

- Add kafka source module with consumer configurations
- Add `spring-xd-extension-kafka` subproject
  - Add Kafka module options mixin and  source module options metadata

Fix code review comments
  - Add kakfa zkconnect chroot path for kafka zk namespace
     - Allow the `zkconnect` property be overridden via modules.yml/kafka.properties
  - Update javadoc for StringDecoder
  - Update kafka inbound channel adapter's poller interval to `100ms` and
    max-messages per poll to `-1` which is unlimited.

Add `encoding` module option for string ValueDecoder

- Use `util:properties` for consumer-properties
  - Add config/kafka-consumer.properties to externalize the properites

Add Kafka Message transformer
  - Message transformer that transforms the kafka messages into
Map<Integer,Object> where key is partition index and value is payload.
XD-2097 Shutdown containers from admin

- Enable `shutdown` endpoint for containers
- Keep the `mangementContextPath` the same for both admin/container
- ContainerRegistrar stores the container's port as container attribute
- If the container uses local port `0` then the port is known only when
  the `EmbeddedServletContainer` is initialized
- Add `update` method in `ContainerRepository` to update the container data
- Update container data (attributes) along with container port info
- Add REST endpoint that shuts down container by `containerId`
- Get container host and port from container attributes
- Use `RestTemplate` to send `POST` request to shutdown the container
- Throw `NoSuchContainerException` if container is not found
- Add test

Handle error scenarios
- During `containerRepository.update` if the container path doesn't exist,
  throw `NoSuchContainerException`
- Add tests for `containerRepository.update` scenarios
- Check `zkConnection` before calling `containerRepository.update`
- Throw `ContainerShutdownException` when the response to shutdown container
  results in exception

Use the management port
- Use container server's `management` server port to access shutdown
  endpoint.
- Handle the `EmbeddedServletContainerInitializedEvent` when the event
  comes out of management server's servlet container. The namespace for this
  context is set to `management`. Hence using that to distinguish the event
- Make sure the management port is always updated when the above
  event is processed.
- Update the container attributes with mangement port (if used)
  before registering with ZK on the `ZKContainerConnectionListener`s
  onResume() and onConnect events
Added synchronization for container attributes

Changed ContainerAttributes internal map to ConcurrentHashMap
to take into account the possibility of multiple threads
reading & writing attributes.

When the admin port for a container is extracted from
an application context event, the port number will
be updated in the container's ephemeral node if the
node already exists. If the node does not exist, it
is assumed that the container is in the middle of
the registration process on another thread and therefore
the container attributes (including the management port)
will eventually be written to the container node.

The update to container attributes, check for node, and
update of node is now an atomic operation. Additionally,
the serialization of container attributes and subsequent
creation of the container node is also an atomic operation.
This guarantees that operations requiring read/write of
container attributes followed by a write to ZooKeeper
will happen on one thread at a time.

Improved assert message in functional test.
XD-2093,XD-2092,XD-2097 - Expose container/modules info from XD cluster

- XD-2093 Add deployment status to module metadata
  - ModuleMetadata will now have:
     - Unit (Stream/Job) name
     - Module name (module label + index)
     - Module type
     - Unit deployment status
- Update `ModuleMetadataResource` with the changes
- Add a method in `ModuleMetadataController` to update the
  deployment status for each stream/job module by referring
  to stream/job repositories
- Fix `runtime modules` command and add the above fields in the result
- Change `runtime modules` to `cluster modules`
- Fix the tests
- Add `deploymentStatus` to ModuleMetadata
  - Previously it was updated with `ModuleMetadataResource`
- XD-2092 Enhance container domain object
- Add `RuntimeContainer` that extends `Container` and also has:
  - DeploymentSize
  - DeployedModules
- Change `/runtime/containers` to `cluster/containers` and change the
- REST endpoint to return the `DetailedContainer` objects
  - Update resource assembler and domain object to use `DetailedContainer`
- Update tests

- XD-2097 Shutdown containers from admin
  - Enable `shutdown` endpoint for containers
    - Keep the `mangementContextPath` the same for both admin/container
  - ContainerRegistrar stores the container's port as container attribute
    - If the container uses local port `0` then the port is known only when the `EmbeddedServletContainer` is initialized
    - Add `update` method in `ContainerRepository` to update the container data
    - Update container data (attributes) along with container port info
  - Add REST endpoint that shuts down container by `containerId`
    - Get container host and port from container attributes
    - Use `RestTemplate` to send `POST` request to shutdown the container
    - Throw `NoSuchContainerException` if container is not found
    - Add test
- Handle error scenarios
  - During `containerRepository.update` if the container path doesn't exist, throw `NoSuchContainerException`
  - Add tests for `containerRepository.update` scenarios
  - Check `zkConnection` before calling `containerRepository.update`
  - Throw `ContainerShutdownException` when the response to shutdown container results in exception
- Use the management port
  - Use container server's `management` server port to access shutdown endpoint.
    - Handle the `EmbeddedServletContainerInitializedEvent` when the event comes out of management server's servlet container. The namespace for this context is set to `management`. Hence using that to distinguish the event
    - Make sure the management port is always updated when the above event is processed.
    - Update the container attributes with mangement port (if used) before registering with ZK on the `ZKContainerConnectionListener`s onResume() and onConnect events
- Message rates from admin server
  - Access jolokia endpoints for deployed modules' message rates from admin controller
    - This will allow us the client don't make direct connection to the container
- Fix review comments
  - use `ModuleType` and `DeploymentUnitStatus.State` in ModuleMetadata model
  - fix javadoc
  - Fix hasLength -> hasText
  - fine tune updateDeploymentStatus method
XD-721 Test Job Module addition after XD Start
XD-788 Job Command Integration Tests

* Extracted job methods from AbstractIntegrationTest and moved them to AbstractJobTest subclass
* Jobs Inherit from AbstractJobTest
* Moved methods from JobUtil to AbstractJobTest
* Removed JobUtil
* Using waitForTablePopulation instead of waitForXD in JDBC Source/Job/JobCommand tests.
* Removed un-necessary waitForXD's
* Implemented the following JobCommandTests
** testJobLifeCycle
** testJobCreateDuplicate
** testStreamDestroyMissing
** testJobCreateDuplicateWithDeployFalse
** testJobDeployUndeployFlow
** testInvalidJobDescriptor
** testMissingJobDescriptor
** testListStepExecutions
** testListJobExecutions
Refreshed code based on code review requests

* Removed super.setup and replaced with initializer
* Removed outdated docs
* Removed overloaded job method(wait-time)
* Removed overloaded stream method (wait-time)
* Renamed jobLaunch to launchJob
* Removed jobDeploy that accepts a boolean where true will deploy the job, false will not.
* Rebased and resolved conflicts