The advantage of being sample oriented is that it will be much easier to
tie this back to observation maps, and subsequently demultiplexed sequence.
This is a critical design consideration for QiiTa -- the flat COO approach
requires interrogating the full data structure in order to fish out sample
specific data. It supports minimal parsing in order to filter samples or
"take" samples from the table.

This structure is effectively COO, and does map directly into the
coo_matrix constructor however the table_factory does not allow for this
yet (you can pass (the coo_matrix constructor accepts (data, (rows, cols))
or whatever the nesting is).

The number of groups is N to the number of samples, which is fine for HDF5.
This has a dramatic reduction in diskspace required as we don't need to
store the row values explicitly.

Normalization is nice, and I agree it is redundant, but a small amount of
denormalization can make life a lot easier. It would be possible to just
use the sample index, for instance, instead of the sample name as the group
but it is a nice amount of verbosity that makes fetching sets of samples
from a table more intuitive. The other approach would be to abolish the
"samples/" group at the cost of mixing data and metadata. The current
distinction is sample oriented (which I'd argue is a good thing), and while
we've previously made strides to make samples and observations effectively
the same thing, we get performance and usability benefits by treating these
axes differently.

Please remember that the structure we defined during the sprint was prior
to the consideration of being able to easily fetch data from observation
maps and demultiplexed sequence. These considerations change up the needs
on the representation of the data. I believe strongly that the flat
structure we laid out previously is a bad idea given the revised goals.


On Mon, Jan 6, 2014 at 11:30 AM, Jai Ram Rideout
<notifications@github.com>wrote:

> In biom/table.py:
>
> > +        # ./samples/ids
> > +        samp_ids = samp_grp.create_dataset('ids', (n_samp,), dtype=vlen_str)
> > +        samp_ids[...] = self.SampleIds
> > +
> > +        # ./samples/metadata
> > +        if self.SampleMetadata is not None:
> > +            samp_md = samp_grp.create_dataset('metadata', (1,), dtype=vlen_str)
> > +            samp_md_serial = dumps(self.SampleMetadata)
> > +            samp_md[...] = samp_md_serial
> > +
> > +        # ./data
> > +        data_grp = h5grp.create_group('data')
> > +
> > +        # A new group will be created for each sample of the form:
> > +        #
> > +        # ./data/<sample_name>
>
> I'm not sure about this new data group structure. It seems overly
> complicated and it isn't as clear or as simple as we had it defined in the
> code sprint google doc:
>
> data/
>   rows # 1D ndarray of unsigned integers
>   cols # 1D ndarray of unsigned integers
>   vals # 1D ndarray of floats
>
> There are advantages to the above approach:
>
>    - conceptually simple
>       - keeps the file format simple and easy to describe to others
>       - uses a common sparse matrix storage technique
>       - simpler design means less room for others to create
>       invalid/nonsensical BIOM files
>     - fewer groups in the HDF5 file (though I'm not sure where this
>    starts to break down)
>    - directly maps to scipy coo_matrix constructor (see here<https://github.com/jrrideout/protobiom/blob/master/protobiom/hdf5.py#L27>for how simple the protobiom parser is)
>       - there may also be performance issues with the new data design
>       because you're having to manually iterate over things within Python to fish
>       pieces out and build the matrix. In general, we should try to avoid this
>       sort of thing wherever possible.
>     - less redundancy: the new data design stores sample IDs twice
>
> We could stay with the simple row/col/val structure and keep an array of
> external links, similar to how we're storing samples/ids. What do you
> think?
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/biom-format/biom-format/pull/268/files#r8672644>
> .
>