azkaban/azkaban2 — pull request #110 by davidzchen (2014-01-17 01:08:03+00:00)
Title: Store Job Stats to DB in execution_jobs attachments field

Originally, when a job type wrote stats, it would store them to a file on disk, which is later read by a web app servlet such as the Pig visualizer or the flow summary. Configuring this disk location is error-prone, and the better approach is to write it to the attachments field of the execution_jobs table in the database, which was unused and was added for this exact purpose.

Write side:

1. When preparing the job to be run, the JobRunner creates an attachment filename, which would be a file for the current attempt of the job stored in the execution directory of the flow.
2. JobRunner saves a property, azkaban.job.attachment.file, which is then passed to the job by the JobTypesManager.
3. While the job is running, the job can write stats to the attachment file.
4. When the job has finished running, JobRunner uploads the attachment file to the database.

Read side:

1. If a web server servlet wants to read the job stats for a job, it calls getExecutionJobStats method of ExecutorManager.
2a. If the ExecutorManager finds the flow in the list of currently executing flows, then it calls the executor server.
a. The request is received by the executor server ExecutorServlet.
b. ExecutorServlet calls FlowRunnerManager to read the job attachment file.
c. FlowRunnerManager gets the filename of the job attachment file from the FlowRunner for the flow and reads the file.
2b. If the flow is not in the list of currently running flows, then ExecutorManager retrieves the job attachment from the database.