atlasapi/atlas — pull request #159 by fredvdd (2012-08-02 10:13:27+00:00)
Title: Make Magpie ingest job restartable, only ingest changed files

The Magpie ingest job keeps persistent track of the lastModifiedTime of
the last results file successfully ingested. File are ingested by order
of lastModifiedTime. The job will ingest all files that have a
lastModifiedTime greater (after) the persisted time. This means the job
is restartable though it will re-ingest in its entirety the job that was
in progress when the previous run was stopped.