ARGOeu/ar-compute-engine — pull request #45 by kaggis (2014-11-30 22:52:17+00:00)
Title: Nov sprint

The following commits add support for handling avro encoded sync files during calculation init.

lib/calculate.sh script is responsible for initializing pig computations.

calculate.sh prepares the following sync files and passes them as parameters to pig latin calculation script:
- poem_sync
- sites_sync
- downtimes_sync
- hepspec_sync

An additional configuration parameter has been added in /etc/ar-compute-engine.conf : serialization
if serialization=avro then the avro serialization framework has been enabled
if serialization=none then flat delimited text files are being used

calculate.sh now checks for the status of serialization config parameter.
If serialization=none then calculate.sh operates as it used to by handling the flat text files
If serialization=avro then calculate.sh looks for the avro encoded sync files (.avro extension) and calls the corresponding decoders (in /usr/libexec/lib/avro)

The decoders include:
../lib/avro/poem_decode.py -- decoder for poem_sync avro files
../lib/avro/downtimes_decode.py -- decoder for downtimes_sync avro files
../lib/avro/sites_decode.py -- decoder for downtimes_sync avro files
../lib/avro/hepspec_decode.py -- decoder for hepspec_sync avro files

The decoders currently use the minimal fields of the schema files + some additional fields (egi specific) when necessary:

More specifically:

hepspec_decode retrieves the following schema fields:
[ 'site', 'weight' ]

downtimes_decode retrieves the following schema fields:
['hostname', 'service', 'start_time', 'end_time']

poem_decode retrieves the following schema fields:
['name', 'service', 'metric', 'vo' ] -- vo is additional (egi specific) field

poem_decode retrieves the following schema fields:
[hostname','service','production','monitored','scope','site','roc','infrastructure','certification','site_scope']
-- production,monitored,scope,roc,infrastructure,certification,site_scope are additional (egi specific) fields

The avro encoded data of the consumer files doesn't concern calculate.sh since it passes through the prefilter script and the prefilter output (non-encoded) is placed directly to hive database (when in cluster mode) or to prefilter.out text file (when in local mode)