spacetelescope/finf-standard — pull request #9 by mdboom (2014-05-15 14:44:25+00:00)
Title: Streaming indeterminately sized data sets via streaming

FINF currently shares this shortcoming with FITS -- you can't just keep writing out rows in a table if you don't know how many there are at the beginning. It would require updating both the block header (not so hard) and the YAML tree (potentially expensive if it requires an insertion in the file).

One possible solution is to stream to a separate BFF file (see "exploded form"), with a special mode that says "the size of the block == the size of the file", and then implode the block after the fact (or leave it external, depending on the use case).