dstuebe/pyon — pull request #3 by swarbhanu (2012-02-28 17:19:57+00:00)
Title: Hdf array iterator

Added hdf_array_iterator. It has a function acquire_data, which has a yield statement in it and so when it is used, it provides a generator.

The function uses the ArrayIterator class to make slices of size, buffer_size, from arrays stored in multiple hdf files with multiple datasets in them (each dataset having a name such as 'conductivity' and holding a one dimensional array of data).

A dict is also created (called arrays_out). The dict contains the variable names as keys and concatenated one dimensional arrays as items. This dict is always kept within a chunk size that is to be provided by the user as the input parameter, concatenated_block_size, and it is refreshed when more data is read from the hdf files through the ArrayIterator class.

Using next() for the generator provided, yields the above mentioned arrays_out dict as one of the outputs. As mentioned, the arrays_out dict holds the concatenated scientific data.