enhance performance of removeFields command
CDK-484: Clean up Registration and Datasets.

This cleans up the separation between the Registration and Datasets
classes. Registration handles URI matching and returns DatasetRepository
objects. Datasets handles any other operations that use those
repositories and the data. Load and view are now one method that returns
the appropriate object based on the URI.

This also expands Datasets, adding create, update, and delete methods
that call the appropriate method on the repository responsible for a
given URI. This is to enable the CLI to use Dataset URIs, solving the
problem of how to use more than one repository for CLI operations.
CDK-484: Add Datasets URI tests and Datasets#exists.

This also fixes inconsistencies in how view URIs are handled. Create or
load operations will return View objects for view URIs and create will
create the underlying dataset correctly. Delete, update, and exists will
reject view URIs with an IllegalArgumentException. Update applies to an
entire dataset, so it doesn't make sense to allow it to be applied to a
view. Delete and exists could be applied to views in the future, but
this is not implemented (and may not be)
CDK-481 Remove SchemaTool references to StorageKey

SchemaTool createOrMigrateSchemaDirectory required that all schemas come with a StorageKey object,
which is a reference to code that is no longer necessary. Remove that requirement and add a unit test
for SchemaTool.
Improved tests by removing static block and some duplicated code.
CDK-474: CrunchDatasets takes Dataset or View URI
CDK-474: delay loading Dataset from URI
CDK-353: Move location choice for FS datasets into the repo.

This is to enable temporary datasets, where the MetadataProvider is not
used. The FS MetadataProvider no longer sets the location, this is done
by the FS DatasetRepository. The repo will still use a different
location if the provider overrides the incoming location, which is the
case with Hive.
CDK-353: Allow dataset locations to be passed to the FS repo.

Imports need to store metadata using the FS repository that can be read
as a Dataset in MR, but pass in the location of existing data. This
removes the check that ensured location was not set. This used to throw
an exception, but is now allowed.
CDK-353: Add TemporaryDatasetRepository for MR.

This updates MR to use datasets from a temporary repository rather than
datasets in the target repository for temporary MR records and files.
Otherwise, jobs writing to Hive will create a dataset for each reduce
task attempt, which all send partition updates.

This introduces a TemporaryDatasetRepository and a
TemporaryDatasetRepositoryAccessor so that repository implementations
can supply a more appropriate repository. FS datasets return
repositories rooted in .temp/ underneath the repository's root
directory, and Hive's temporary repository does not interact with Hive.
Fix TestCrunchDatasets to use Datasets#load.
Update dataset script, remove HCatalog and add MR 0.20.

Jobs fail on CDH 4.x unless the hadoop-0.20-mapreduce libs are in the
classpath when using the old JobTracker. This also works on CDH 5.x,
so it is the current default. The script will fall back to
hadoop-mapreduce libs if hadoop-0.20-mapreduce is not found.
CDK-192: Update CLI to use Dataset URIs.

This enables the copy command to work between repositories.
CDK-192: Add view copy command.

Crunch's MemPipeline won't write to MapReduceTargets, so the CopyTask
has to write to the output View manually. Crunch is still used to
materialize the incoming collection so that DoFn transforms can be used
later.
CDK-192: Fix csv-import to run in parallel.

This updates the csv-import command to use the CopyTask and a temporary
repository to set up a Dataset for the incoming CSV data, stored in the
same FileSystem as the data. This is compatible with the
kite-data-mapreduce formats.
CDK-192: Use MR framework for record count.
CDK-192: Fix csv-import test by avoiding DistCache.

The distributed csv-import tests are failing because the jar files added
by TaskUtil are not found. This works on the 4.4 VM and is requried for
teh job to complete. The fix here is to not use the distributed cache if
testing.
CDK-192: Dynamcially resolve MR input record counter.

This fixes hadoop-1 tests.
Fix Hive MetaStore URI handling.

The hive.metastore.uris property is set in hive-site.xml, but the Loader
and HCatalog class were checking the property before creating a HiveConf
that reads the hive-site.xml file. This prevented the environment's
MetaStore URI from being loaded.

HiveConf is now required in the Loader to initialize URIs correctly, but
this caused test failures kite-data-mapreduce because the MR tasks did
not have Hive jars in the classpath. This commit adds dynamic resolution
of HiveConf so that the hive URIs will not be registered if it can't be
loaded without failing registration for all implementations.