Merge pull request #134 from fiserro/master

kite-morphlines-protobuf: hasValue check - added corresponding user doc
CDK-332: Deprecate DatasetReader#open and DatasetWriter#open.

This deprecates the reader and writer open methods so that users know
not to use them the design of the readers and writers remains the same,
with an initialization step. But, the initialization is done in an SPI
method, initialize, that is called after the reader or writer is
constructed, but before it is returned from newReader or newWriter.

The code has been moved to the new method by adding open to the
AbstractDatasetReader and a new AbstractDatasetWriter that logs a
warning that the call is no longer required. Tests that called log have
been migrated by either replacing the open call with a call to
initialize, or by removing the call to open. The first case is used
whenever the test instantiates the reader (or writer) itself, as in
TestCSVFileReader, and the latter is used when the reader comes from a
dataset or view.

Some helper methods expected an unopened reader either from newReader or
instantiated directly. These methods now check if the reader is open
and will call initialize if needed.
Add some missing javadoc.
CDK-511: Integrate kite-docs into release process
Merge pull request #137 from joey/cdk-511

CDK-511: Integrate kite-docs into release process
Skip site generation for kite-app-parent cdh4 pom
CDK-510. Add PartitionedDataset.
CDK-496: Added the type parameter to the Datasets and DatasetRepository APIs

 * When creating new Datasets or Views, types are also resolved with respect
to the Schema. This preserves the ability to use the reflect data model by
passing a type of Object.class.

 * The patch also fixes CDK-517 by using Avro serialization to convert
constraints specified in the View URI into the type expected by the data
model in use.

 * The type parameters is also maintained in the MR APIs so the right
data model can be used in MR and Crunch jobs.

 * A number of util methods were added to a new DataModelUtil class to make
it easier to deal with data models.

 * The patch also has the benefit of enabling projection over a Dataset by
loading the Dataset with a type that contains a subset of the fields of the
Dataset schema. There are some minimal tests for this, but more comprehensive
testing should be done before documenting this as a feature.
Merge pull request #122 from joey/cdk-496

CDK-496: Added the type parameter to the Datasets API
CDK-238. Move PartitionKey to SPI.
Add back org.kitesdk.data.PartitionKey
CDK-456: Added PartitionStrategy to Constraints.

This adds a PartitionStrategy to Constraints and updates its
EntityPredicate and KeyPredicate classes. The entity and key predicates
can now use the partition strategy to prepare the constraints in the
constructor so they can be applied quickly, rather than looking up
constraints for schema or partition fields each call to apply.

This also updates utility methods in SchemaUtil to work with a partition
strategy. It now has code to get a schema for fields in a partiiton
strategy and to build a key schema, which is needed by the CLI and the
HBase implementation.
CDK-456: Update AvroKeySchema to use SchemaUtil#keySchema.
CDK-456: Fix assumption in HBase avro util and tests.

The tests previously assumed that the key schema field names matched the
original field names. Now that the key fields are created based on the
partition strategy, this isn't possible because the source fields can be
used multiple times. The fix is to look up the correct source field name
using the partition strategy.

This also fixes tests that assumed the names matched by adding the
matching names explicitly in the partition strategy. This is allowed for
HBase, but was broken because FieldPartitioner itself enforced that
source and partition names had to be different. This removes that
restriction and adds a test to make sure it is properly enforced for
Hive tables (non-Hive HDFS produces a warning). The enforcement was
already implemented in the Compatibility class, it just wasn't used
because FieldPartitioner was doing it.
CDK-456: Fix review items.

* Removed partition strategy arg from toKeyRanges, alignedWithBoundaries
* Added @throws to javadoc for missing partition strategy
* Made predicate combination more thorough
* Added notes about source and partition name collisions
Add DurableParquetAppender.

This appender wraps an AvroAppender and a ParquetAppender to provide
durability by writing to Avro in tandem with writing parquet files
directly. After the file is closed, the avro copy is removed in a new
cleanup phase in the appender. Any failure requires recovery using the
Avro files.

The default is to use the regular ParquetAppender and the durable
appender can be enabled by setting the descriptor property
"kite.parquet.durable-writes" to true.
CDK-492: Make durable writes default and clean up.

Durable writes should be the default when writing to parquet, and not
using them should be an optimization. This switches the
kite.parquet.durable-writes to kite.parquet.non-durable-writes and adds
the setting to the DatasetKeyOutputFormat because durable writes are not
required when using an OutputCommitter in MR where failed task attempts
are run again.

This also refactors the FS descriptor properties into a common
FileSystemProperties class and moves the isTrue method into a
DescriptorUtil class in the SPI.
CDK-511: Replace guide markdown with redirect to new guide.

This replaces all of the guide markdown files with html pages that
redirect to the new locations, except for langingpage.md, which was
present but wasn't linked from the other pages.
CDK-530: Deprecate CrunchDatasets#asSource with type.

Now that Views carry a type, this is no longer needed. Because the type
parameter matched, it was forced to be the view's type already.
Remove type parameter in CrunchDatasets.asSource in HBase test.