biometrics/openbr — pull request #117 by caotto (2013-10-23 16:22:00+00:00)
Title: Leverage stream to do enrollment more efficiently

One outcome of investigating issue #115

When dealing with large enrollment jobs, file I/O can take a significant amount of time, and we can mitigate this by pipe-lining the enrollment and output steps.
So instead of enrolling images 1 block at a time, then doing file I/O for the block, this branch builds a stream with the base algorithm as the first stage, followed by a (single-threaded) stage that outputs to galleries and does performance counting.

Currently this branch introduces several issues:
breaks support for read, cache gallery flags
breaks support for [distribute=false] (this is a little complicated, this branch actually removes the need for this flag in some cases, but it should still be supported for some other cases)
probably damages training (because the stream transform is created in enroll, something similar has to be done in train to maintain consistency)

and not really a correctness issue, but this branch has exposed stream's data source hierarchy as ill-conceived to some extent, and it should be re-designed at some point.

So not ready to merge just yet, but comments are welcome.