CDK-139: Updating FS impl to use Descr locations.

Now that DatasetDescriptors have getLocation and getConfiguration, the
FileSystem Dataset/Repo/Provider implementations can use the Descriptor
to pass Dataset paths. Now, the FileSystemMetadataProvider assigns a
location when creating a Dataset. FileSystemDatasetRepository uses the
paths that are set on the DatasetDescriptor.

Because location and configuration are being passed by the
DatasetDescriptor, all of the classes necessary to get a FileSystem and
Path for a FS-based Dataset are available. So this commit also removes
the FileSystem and root Path from the FS repo and the FileSystem option
from the FileSystemDataset.Builder. This makes the FS classes more
flexible and handles Configurations more cleanly: only the
MetadataProvider is instantiated with a Configuration, which is passed
via Immutable objects where it needs to go.

Several helper methods that were previously protected because they were
shared between the FS MetadataProvider and DatasetRepository are now
private and owned by the MetadataProvider.

Tests are up-to-date and passing for core, but this breaks the Hive
integration. This update is primarily to fix problems in the Hive
integration anyway.
CDK-139: Separating common tests out of FS.

This extracts tests that should pass on all DatasetRepository
implementations into the TestDatasetRepositories set. The FS tests now
inherit from this add add other tests, like making sure that the data
directory exists after a dataset is created. Minor updates to the FS
repository are included where tests found bugs.

A major update to the FS repository and common tests is the new
MemoryMetadataProvider, which is a simple provider with no durability
intended for testing. Both the common and FS tests use this provider
rather than the FS provider. MemoryMetadataProvider is tested by
TestMemoryMetadataProvider, which inherits from the new common set,
TestMetadataProviders.

The common tests include checks that verify IllegalArgumentException is
thrown, which has been added to the API documentation in the interface.
Similar additions haven't been made to the FS MetadataProvider yet.
CDK-139: Cleaning up DatasetReader tests.

Clean up to match the recent repo and provider tests
* Renamed TestDatasetReaderCommon to TestDatasetReaders
* Merged Test...DatasetReaderCommon and Test...DatasetReader classes
CDK-139: Updating FS MetadataProvider tests.

The FS MetadataProvider now uses the common test methods in
TestMetadataProviders. Tests are passing.
CDK-139: Updating MP tests to run on local and DFS.

This fixes MiniDFSTest so that it returns a Configuration that defaults
to HDFS, and parameterizes the TestMetadataProviders superclass to run
with both HDFS and the local FS. This caught a bug in
FileSystemMetadataProvider, where FileNotFoundException is thrown for an
empty directory in HDFS.

Test classes that depend on TestMetadataProviders need to have a
Constructor added to pass along the local/distributed mode string.
CDK-139: Updating repo tests to run on local, DFS.
CDK-139: Adding changable providers to repo tests.

Adding test of FS provider and FS repo. This is in preparation for
testing the FS repo with the Hive external provider.

Caught and fixed problems:
* When validating the DatasetDescriptor, equals was not implemented
* FS repo delete should return true if the provider delete returns true
CDK-123: Clarifying assumptions in Hive Provider.

The FileSystem is now final and set in the constructors. Before, a call to save
assumed that the FileSystem was set if a dataDirectory was set, which happens
when an unmanaged HCatalogDatasetRepository creates (external) data sets.
Changes to the HCatalogDatasetRepository could easily cause a NPE by violating
these undocumented assumptions, so always setting up a FileSystem prevents
that. Also, nothing set the dataDirectory to null, so table locations weren't
read on subsequent saves.

In addition, this commit makes save more defensive. Rather than relying on
dataDirectory being null to signal that save should read the managed table's
location, it now handles both modes explicitly. In managed mode, the table is
created, read, and then dataDirectory is set (it is an outgoing field). In
unmanaged mode, the dataDirectory (an incoming field) is checked with a
Precondition at the start of the function, used to create the table metadata,
then set to null so that it doesn't leak into other save calls.
CDK-123: Refactoring Hive repository for clarity.

Previously, both create and get would lazily create a special inline
FileSystemDatasetRepository for the purpose of opening Hive-managed Datasets
because the location of the those sets is controlled by Hive. This commit
consolidates those inline classes into a private inner class called
ExternalFileSystemDatasetRepository with docs to explain what's going on: it is
only used immediately after the MetadataProvider creates or loads because it
will only load the data set that the MP points to.

The special data set repo was previously stored in the same field as the normal
repo, used when Hive does _not_ manage the data set, which was confusing so it
is now stored in a new field: externalDatasetRepository. Both fields (the other
is fileSystemDatasetRepository) are final and initialized in the constructors.

Next, the Hive repository used to assume that all tables were in the
filesystem-backed repository that is opened in unmanaged mode (now null in
managed mode). This isn't necessarily the case, so the special external
repository is now used when loading and deleting for both modes because its
location is set from the Hive metadata when the set is loaded. An extra
consequence is that the DatsetDescriptor must be successfully loaded before it
can be deleted. With this change, the regular FS repository is only used for
creation.

This commit also removes the call to set the MetadataProvider's file system in
create. The FileSystem for the HCatalogMetadataProvider is now set in the
constructors for unmanaged mode, from the created FileSystemDatasetRepository
or from the FileSystem passed in. For managed mode, the handling has not
changed.

Known bug: in create, the FS repository determines the location of the data set
in create and calls metadataProvider.save. But the path where the data will be
stored must be set before calling save (or else the Precondition is not met).
This currently gets around the problem by guessing (accurately) where the FS
repository will put the data set, violating encapsulation.
CDK-123: Merging Hive branch, updating for CDK-139.

CDK-139 adds a location to DatasetDescriptor, which is what CDK-123
needed in order to stop passing Dataset locations through a field on the
MetadataProvider instance, which created a brittle dependence between
the Hive MetadataProvider and DatasetRepository. Also, because the
FileSystemDatasetRepository no longer determines dataset location, and
instead uses whatever is set in the DatasetDescriptor, it can be used
without modification for external Datasets. For managed Datasets, the
HCatalogDatasetRepository is now a thin wrapper around the FS
DatasetRepository that avoids creating and deleting---handled by
Hive---but delegates to FS to load and do other tasks.

The other major change in this commit is to split the
HCatalogMetadataProvider into two implementations, one for external
(HCatalogExternalMetadataProvider) and one for managed datasets
(HCatalogManagedMetadataProvider). The shared methods still exist in
HCatalogMetadataProvider, from which both inherit. This is easier to
reason about than selecting behavior with a boolean field because they
need to do certain operations in different orders (create) and should
handle certain situations differently. Utility methods for CDK-Hive
translation are now in HiveUtils and have been extended for incoming
Parquet support.

The new MetadataProviders are working and passing standard tests with FS
repositories. This caught a bug where just setting the TableType to
EXTERNAL does not create an external table; a table parameter, EXTERNAL,
must be set to TRUE.

To do:
* Translate directly between Hive and CDK partitioning and add tests
* Add round-trip tests, via core's TestDatasetRepositories
CDK-139: Updating Crunch tests for FS repo changes.
Adding a String-URI method to Hive repo builder.
Adding Parquet support for Hive/Impala
CDK-142: Use schema literal for Hive unless hdfs:.
[squash me] Trying another parquet format.

parquet.hive.DeprecatedParquetInputFormat
squash me (and find where I go)
squash me: fixing test setup to use boolean
squash me: formatting in HCMP header
CDK-139: Removing Configuration from descriptors.

This commit removes the optional Configuration from the
DatasetDescriptor interface. Instead, optional properties can be set on
the descriptor. To avoid assuming the FileSystem for a Dataset, the
metadata for FileSystem-based data sets now includes a property,
cdk.filesystem.uri, which is used to open the correct FileSystem for
each DatasetDescriptor.

Removing the Configuration from the descriptor interface requires adding
it back to FileSystemDatasetRepository and FileSystemDataset.Builder, so
that those classes can open the correct FileSystem from the descriptor's
FS URI.

This also exposed a bug in Parquet, where new Configuration() is called
in a constructor, which causes parquet to get the wrong file system and
fail tests. There isn't a constructor that takes a FS or a
Configuration, so I'm not sure how to fix this. It doesn't happen when
the default FS from the environment is used.

To do:
* Add general property storage to the hive tables.
* Also need to remember to remove deprecated methods before 0.8.0