Shopify/spark — pull request #18 by dklassen (2014-11-12 21:05:04+00:00)
Title: Add log reporter

This PR's adds a LogSink which is rolling log file for spark metrics events. The purpose is to ingest these log files into splunk for dashboard metrics at the application/job level. Other options like DataDog won't work because of the requirement for Application/Job level tagging. Once this is up and running into splunk we can extract more information from the fields to make it easier to query with splunk. For now I just wanted to see if we can do anything interesting with a splunk dashboard

if this works and we go ahead would be worth pulling this out into its own library to keep the spark fork clean.

@orenmazor @airhorns