Refactor small helper out of get_work_dir_outputs for reuse.

Leave comment about not understanding the complexity :).
Rename variable for clarity post output collections.

After tools can explicitly create collections the term created_collections becomes ambigious - these are history collection instances (HDCA) implicitly created by mapping over collections so renaming this field implicit_collections.
Rename variable for clarity.

This method does not just return datasets.
Add file missed with pull request 505.
Refactor dataset collection creation into reusable pieces.

In downstream output collection work this allows creating elements after initial construction - which is useful for output collections composed of dynamically discovered datasets after a job has completed.
Rearrange verify method to make duplication more obvious.
Break up a couple test methods for reuse downstream.

Remove duplication related to verifying a single dataset in tests - allows reuse in downstream when verifying the content of explicitly defined tool output collections.

Refactor out waiting for jobs - seems wrong place to put that logic this should allow easier moving.
Script to build workflows from simpler YAML description.
Split all workflow extraction testing out of test/api/test_workflows.py.

test_workflows.py was pretty unwieldy and the workflow extraction tests were very different than the other tests in that file (e.g. very different helper functions).
Simplify new test_workflow_extraction.py.

Now makes sense to assume all tests consume a history.
Simplify interface to test helper _extract_and_download_workflow.

Makes tests in this file marginally more readable.
Introduce another helper to simplify test_workflow_extraction.py.
Fix for running test_extract_workflow.py stand-alone.
Remove dated TODO from workflow extraction code.
More unit tests for workflow extraction summarize method.
Refactor lib/galaxy/workflow/extract.py to breakup big loop.
Add helper for waiting on a job instead of a history.

Used downstream - should try to use this more in API tests.
Allow tools to output collections with static or determinable structure.

By "static" I mean tools such as a FASTQ de-interlacer that would produce a "paired" collection with two datasets everytime. By "determinable" I mean tools that perform N->N operations within the same job - such as a tool that needs to normalize a bunch of datasets all at once and not in separate jobs. (For N->N collection operations that should or can be done in N separate jobs tool authors should just write tools that operate over a dataset and produce a dataset and let the end-user 'map over' that operation.)

There are still large classes of operations where the structure of the output collection cannot be pre-determined - such as splitting files (e.g. bam files by read group) - that are not implemented in this commit.

Model:

The models have been updated to do a more thorough job of tracking collection outputs. Jobs just producing HistoryDatasetCollectionAssociations works fine for simple jobs producing collections - but you don't want to map a list over a tool that produces a pair and produce a bunch of pairs HDCAs and a list:pair HDCA- you just want a bunch of pieces and the one list:pair at that the top.

Workflow:

Workflows containing such operations can be executed - but the workflow editor has not been updated to handle this complexity (and it will require a significant overhaul) so such tools are not available in the workflow editor.

Tool Testing:

This commit also introduces a new tool XML syntax for describing tests on output collections. See files test/functional/tools/collection_creates_list.xml and test/functional/tools/collection_creates_pair.xml for examples.

Tests:

Includes two tools to test this - one that uses explicit pair output names and one that iterates over the structure of input list to produce an output list.

Includes several new tools API tests that test the tools described above via the API and implicit mapping over such tools. Includes two new workflow API tests - one that verifies a simple workflow with output collections works and one that verifies mapping over workflow steps in collections works.
Allow tools to output collections with a dynamic number of datasets.

Models:

Track whether dataset collections have been populated yet.

Dataset collections are still effectively immutable once populated - but dynamic output collections require them to be sort of likely `final` fields in Java (analogy courtesy of JJ) - allowing them to be declared before they are initialized or populated. This is tracked by the `populated_state` field.

Tools:

Output collections can now describe `discover_datasets` elements just like datasets - except in this case instead of dynamically populating new datasets in the history - they will comprise the collection. `designation` has been reused to serve as the element_identifier for the collection element corresponding to the dataset.

See Pull Request 356 for more information on the discover_datasets tag https://bitbucket.org/galaxy/galaxy-central/pull-request/356/enhancements-for-runtime-discovered.

Workflows:

Update workflow execution and recovery for dynamic output collections.

Galaxy workflow data flow before collections

* - * - * - * - * - *

Galaxy worfklow data flow after collections (iteration 1)

* - * - * \
           * - * - *
* - * - * /         \
                     * - * - *
* - * - * \         /
           * - * - *
* - * - * /

Galaxy worfklow data flow after this commit

              / * - * \
         * - *         * - *
        /     \ * - * /     \
       /                     \
      /                       \
     /        / * - * \        \
* - * -- * - *         * - * -- * - *
     \        \ * - * /        /
      \                       /
       \                     /
        \     / * - * \     /
         * - *         * - *
              \ * - * /
More configurable format and metadata handling for output collections.

Imporvements to testing code.