harvard-library/librarycloud_ingest — pull request #14 by jlicht (2014-10-19 21:54:44+00:00)
Title: Refactor splitter; add support for data file URL as well as local file

@michael-lts I've refactored the 'splitter' logic to separate the logic for getting hold of an input stream from a file, from the file-type-specific logic to split it. (As a result, I've also added the ability to download from a URL rather than a file, which opens the door to using S3 for the files).

So, for example, to implement the EAD splitter you just need to write a class that inherits from ```ISplitter``` and implements ```getIterator()```

This also means that:

* The splitter gets invoked once per file - it should not be responsible for iterating over all files in a directory
* We will need to generate one command XML file for __each__ EAD file to be imported. This will be implemented in a script, similar to [the one for aleph files](https://github.com/harvard-library/librarycloud_ingest/blob/master/util/ingest_aleph.sh).

Advantages of this approach:

* Keep all infrastructure logic (downloading files, looking through files in a directory) out of file-format-specific code (aleph, oasis, via, etc.)
* Enable using a central data store (such as S3) for the data files, which means that we can avoid copying files to ingestion servers. (The problem with the current approach is that, to distribute the load evenly, we need to look at the total number of ingestions servers, and copy a subset of the files to be processed to each server. Which is cumbersome, especially when the servers are transient and only being spun up for a few hours. Better to put the command files on a queue, put the data files in a central location, and allow servers to pull command files and associated data files as needed)