Package: etllib
**********etllib***************.. contents::ETLlib provides functionality for munging through and repackagingJSON, TSV and other data for preparation and submission (ETL) to Apache Solr. Thelibrary takes advantage of Apache Tika, and is callable from Apache OODT.Using ETLlib****************Installing the ETLlib package makes available four things on yourcomputer:``repackage`` commandThe ``repackage`` command takes an original paged JSON file, stripsoff the paging information, isolates the identified object type (e.g., "teams"or "journal_entries", etc.), and may perform some basic cleaning and metadataaddition using Apache Tika on the fields within the JSON document.``poster`` commandThe ``poster`` command lets post individual reformulated  JSON documentsto Apache Solr.``repackageandpost`` commandThe repackageandpost command combines repackage, and poster, andobviates the need to store the repackaged JSON doc as an intermediate file, andthen repackages (keeps docs in memory) and then posts directly to Apache Solr.``tsvtojson`` commandTakes an input TSV file and parses it with a set of column headers and outputs aJSON file.``translatejson`` commandTakes an input JSON file and a column header file and cred file and translates fromsource lang to dest lang using Bing's API and Apache Tika.``imagesimilarity`` commandComputes the similarity between a directory full of image files using a feature-basedapproach based on Jaccard's algorithm. Clusters scores. Uses Tika.ETLlib LibraryThe ETLlib Library is a Python-based API for munging  data anddoing ETL. The library was originally developed as a set of Python scriptsto be integrated into an Apache OODT ETL process through parsing/Apache Tikacleanup and then on to Apache Solr for analytics.This document describes how to use the above three items, with specialattention to the ETLlib library.Commands========After installing the ETLlib package,  new commands are made available onyour system, ``repackage`` and ``poster`` and ``repackageandpost`` and``tsvtojson`` and ``translatejson`` and ``imagesimiliarity``..These commands enable you to reformulateaggregate  JSON documents, cleanse their fields (which may contain UTF-8 or otherweird encodings), convert from different formats (e.g., TSV to JSON), translatefields within the documents using Apache Tika, and then to post those documents to Apache Solr.These were developed initially independently as python scripts that are wrapped usingApache OODT ETL workflows, but later Chris Mattmann <mattmann@apache.org>decided they would be useful as a installable python library.To use these commands from your interactive prompt, you just need to make sureyour shell's PATH environment variable includes the directory where thecommands are installed.  On most systems, these two commands are installed in::/usr/binHowever, on Mac OS X, the installation location may be::/Library/Frameworks/Python.framework/Versions/Current/binAnd on Windows, it may be::c:\Program Files\PythonNote also that some interactive shells create a cache of commands in order toexecute your requests more quickly.  You may need to force your shell tore-build that cache.  The csh and tcsh shells are two such examples; you canmake these shells rebuild their caches by running the ``rehash`` command.Use from Shell Scripts----------------------The ``repackage`` and ``poster`` and ``repackageandpost`` and ``tsvtojson`` and``translatejson`` and ``imagesimilarity`` commands  may be usedfrom shell scripts as well.  The onlyrequirement for making these  commands available to shell scripts is the same asfor interactive sessions:  the shell's PATH environment variable must include thedirectory that contains the commands.Here is a sample shell script that repackages a  Teams JSON file of 20 aggregaterecords, and outputs 20 individual Teams JSON files ::#!/bin/bashfor ag in $(ls /data/xdata/Kiva/raw/RAW__json_teams_9Feb2013); dorepackage -j $ag -o teamsdoneThe above shell script assumes that ``repackage`` will be found in``/usr/local/bin``, ``/usr/bin``, or ``/bin``.  It then loops throughthe aggregate teams JSON files from the Kiva raw dataset and then handseach aggregate JSON file to the repackage script, which unravels those1234 teams JSON data files into 1234 * 20 = 24680 individual team JSON files.The rest of the commands may also be used from a shell script.Some example working commands are:Pipe a single  JSON journal entries file into the repackageandpost script::echo "/data/xdata/Kiva/raw/RAW__json_journalEntries_04Mar2013/365/191677_journalentries_pg1_retreived-2013_03_04_21_56.json" \| repackageandpost -u "http://localhost:8080/solr/journalentries/update/json?commit=true" -o journal_entries -vTake in an input TSV file named computrabajo-ve-20121108.tsv and turn it into a JSON file with a root object named employmentjobs using the provided colheaders.txt file::tsvtojson -t data/staging/computrabajo-ve-20121108.tsv -j data/jobs/tsvtojson/1/output/computrabajo-ve-20121108.json -c conf/colheaders.txt -o employmentjobsExtract out the ~9000 or so jobs present in computrabajo-ve-20121108.json under the "employmentjobs" key:repackage -j ../../../../../data/jobs/tsvtojson/1/output/computrabajo-ve-20121108.json -o employmentjobsTranslate the fields defined in translate.cols in the JSON file named 648c3a4a-22d1-4a43-b0da-9c8e45716e40.json from spanish ("es") to english ("en") and output the translated JSON named 648c3a4a-22d1-4a43-b0da-9c8e45716e40-t.json, using Bing and Apache Tika and the provided credentials::translatejson -i data/jobs/repackager/1/output/648c3a4a-22d1-4a43-b0da-9c8e45716e40.json -j data/jobs/translate/1/output/648c3a4a-22d1-4a43-b0da-9c8e45716e40-t.json -c src/tika-python/lib/translate.cols -p src/tika-python/lib/translator.creds -f es -t en -vCompute the similarity of images in your $HOME/Pictures directory on Mac:cd $HOME/Pictures && imagesimilarity -m -f . > similarity-scores.txtETLlib Library===========The ETLlib Library is a Python-based application programming interface (API) formunging and processing  JSON data for ETL and analytics. In fact, the commands``poster`` and ``repackage`` and ``repackageandpost`` and ``tsvtojson`` and ``translatejson``and ``imagesimilarity`` are implemented using the ETLlib Library.  Ifshell-script programming is not to your taste, and youknow Python, then using the ETLlib Library may be right for you.Changelog*********0.0.2 - Refactor to use new Python-Tika lib===========================================Current Development.0.0.1 - Updated to include ClusterScores and Image Similarity=============================================================Includes tools to handle image similarity.0.0.0 - Initial===============This is an initial release of etllib supporting capability forreformulating JSON data using Tika_ and JSON read/write in prepfor ETL using OODT_ into Solr_... References:.. _OODT: http://oodt.apache.org/.. _Tika: http://tika.apache.org/.. _Solr: http://lucene.apache.org/solr/