NGDATA/hbase-indexer — pull request #4 by whoschek (2013-07-29 18:57:35+00:00)
Title: CDH-13042: add morphline support to HBase indexer

/**
* Pipes a given HBase Result into a morphline and extracts and transforms the specified HBase cells
* to a SolrInputDocument. Note that for proper functioning the morphline should not contain a
* loadSolr command because loading documents into Solr is the responsibility of the enclosing Indexer.
*
* Example config file:
*
*

*
*
*
* table="record"
*
*
* mapper="com.ngdata.hbaseindexer.morphline.MorphlineResultToSolrMapper">
*
*
*
*
* <--
* Parameter morphlineId (optional): Name used to identify a morphline if there are multiple morphlines in a morphline config file
* -->
*
*
*
*

*/

morphlines : [
{
id : morphline1
importCommands : ["com.cloudera.**", "com.ngdata.**"]

commands : [
{
# The extractHBaseCells morphline command extracts cells from an HBase Result
# and transforms the resulting values into a SolrInputDocument.
# It consists of an array of zero or more mapping specifications.
extractHBaseCells {
mappings : [
{
# The inputColumn specifies the data to be used from HBase for populating the field in Solr.
# It takes the form of a column family name and qualifier, separated by a colon.
# The qualifier portion can end in an asterisk, which is interpreted as a wildcard.
# In this case, all matching column-family and qualifier expressions will be used.
#
# The following are examples of valid inputColumn values:
#
# mycolumnfamily:myqualifier
# mycolumnfamily:my*
# mycolumnfamily:*
inputColumn : "cfA:qualifierA"

# The outputField specifies the morphline record field (aka Solr document field) to add output values to
outputField : fieldA

# The type parameter defines the datatype of the content in HBase.
# Because all data is stored in HBase as byte arrays, but all content in Solr is indexed as text,
# a method for converting from byte arrays to the actual datatype is needed.
# The type parameter can be the name of a type that is supported by org.apache.hadoop.hbase.util.Bytes.toXXX
# (currently: "byte[]", "int", "long", "string", "boolean", "float", "double", "short", bigdecimal"),
# or it can be the name of a class that implements the ByteArrayValueMapper interface.
# Use type "byte[]" to pass the byte array through to the morphline without any conversion.
type : int

# The source parameter determines what portion of an HBase KeyValue will be used as indexing content.
# It has two possible values: "value" or "qualifier".
# When "value" is specified (which is the case by default), then the cell value is used as input for indexing.
# When "qualifier" is specified, then the column qualifier is used as input for indexing.
source : value
},

{
inputColumn : "cfB:qual*"
outputField : fieldB
type : string
}
]
}
}

{ logDebug { format : "output record: {}", args : ["@{}"] } }
]
}
]

For those new to morphlines here is some background:

- Introduction: http://blog.cloudera.com/blog/2013/07/morphlines-the-easy-way-to-build-and-integrate-etl-apps-for-apache-hadoop/

- Reference Guide: http://cloudera.github.io/cdk/docs/current/cdk-morphlines/morphlinesReferenceGuide.html

- Source code: https://github.com/cloudera/cdk/tree/master/cdk-morphlines

BTW, On August 1, CDK 0.5.0 will be released with a new version of morphlines. For example, this will allow to get HTML cells from HBase via the Lily Morphline HBase Indexer, convert the HTML and XML and use XPath/XQuery to extract links, and other metadata from the HTML and index it in Solr - https://github.com/cloudera/cdk/blob/master/src/site/markdown/release_notes.md

Wolfgang.