openmicroscopy/bioformats — pull request #1232 by csachs (2014-07-24 00:14:21+00:00)
Title: Use chunkmap for Nikon ND2 files to speed up processing

Hello,

we're regularly working with large time lapse Nikon .nd2 files in our lab, and unfortunately, the initial reading step makes it almost unusable for larger files.
This is due to the bioformats reader sequentially reading (at least the header) of all chunks. The NIS software has almost no delay in opening files. This is possible, because modern ND2 files contain a chunk map (table of contents) at the end of the file. By reading just this map, and skipping the individual processing of image chunks at the initial read phase, a dramatic speed up is possible.

I've tried to keep the impact on the other reader code minimal ... the chunk map is read, and as the first image block is observed, the data is added to the particular data structures, and all the image blocks are skipped. Furthermore, if the chunk map reader function does not find the magic chunk map identifier, it will deactivate chunk map processing altogether. That way, broken files should still be as readable as before.

The speed increase is dramatic, some tests with showinf on files:

13 GiB file (local hard drive):
before: Initialization took 60.735s
after: Initialization took 2.894s

115 GiB file (over 100 MBit network):
before: Initialization took 610.772s
after: Initialization took 16.348s

I realize that other nd2 optimizations are currently ongoing on the dev_5_0 branch (melissalinkert/nd2-final), if you would prefer to have this patch for that branch, please just drop me a note and I'll submit a pull request for that branch (I've tested my changes with both branches).

Remark: This work has been partly created at the Research Centre Jülich. Use this patch as you see fit (i.e. CC0/GPL).

Regards,
Christian Sachs