In the original structure, you have to assume the data are unsorted. From a
practical standpoint, this eliminates any possible data locality benefit.
While it is of course possible to pass an array of index positions into a
the data structure, the amount of fetches to disk required will always be
high. If you repeat this for multiple samples, the expense for a large
table is likely to be large. Grouping the data by sample allows immediately
for more efficient use of on-disk (and even in-memory) data accesses.

Is this actually hard to parse? The parsing code is light, and the
structure is more verbose than a flat COO structure.

In the current table, as was the case in protobiom and the flat COO
structure discussed during the code sprint, all data needed to reside in
memory -- table operations are currently independent of on-disk
representation. However, this structure better allows for performing table
operations in the future without loading the full table into memory
(assuming sample-oriented operations are being performed) as it is trivial
to walk samples and only examine the relevant ones.

What are the performance concerns with this structure beyond the current
acknowledged limitation with table_factory? Please keep in mind that the
acknowledged performance bottleneck would still exist with the flat COO
structure, and also keep in mind that this structure is parsed directly
into COO anyway.

I disagree that the existing design offers any benefit over this structure.
It requires more space on disk, does not allow random access to samples
efficiently, ignores natural structure in the data, and limits future
on-disk use to essentially:

```python
idx = where(cols == some_sample_index, True, False)
data = take(cols, idx)
```

And I suspect the above will actually require loading everything into
memory anyway...


On Tue, Jan 7, 2014 at 10:29 AM, Jai Ram Rideout
<notifications@github.com>wrote:

> In biom/table.py:
>
> > +        # ./samples/ids
> > +        samp_ids = samp_grp.create_dataset('ids', (n_samp,), dtype=vlen_str)
> > +        samp_ids[...] = self.SampleIds
> > +
> > +        # ./samples/metadata
> > +        if self.SampleMetadata is not None:
> > +            samp_md = samp_grp.create_dataset('metadata', (1,), dtype=vlen_str)
> > +            samp_md_serial = dumps(self.SampleMetadata)
> > +            samp_md[...] = samp_md_serial
> > +
> > +        # ./data
> > +        data_grp = h5grp.create_group('data')
> > +
> > +        # A new group will be created for each sample of the form:
> > +        #
> > +        # ./data/<sample_name>
>
> If you want to tie samples to observation maps and demultiplexed
> sequences, why couple these links so tightly with the observation counts
> data? What about having samples/external_links, similar to samples/ids?
> You can still take slices of these arrays if you only want to retrieve
> certain samples.
>
> Also note that the on-disk size of a BIOM table is not a primary concern.
> IMO the primary concerns are 1) be able to parse the files and 2) be able
> to perform table operations efficiently.
>
> I don't know what the revised goals are (I only learned what QiiTa was a
> week ago), so it may make sense to take your approach. However, I want to
> be sure that whatever design we decide on doesn't bite us in the ass
> performance-wise and stays reasonably simple to enable adoption by other
> developers. If we are aiming to keep interacting with BIOM tables as we
> currently are, then the design we came up with during the code sprint is
> the better option.
>
> —
> Reply to this email directly or view it on GitHub<https://github.com/biom-format/biom-format/pull/268/files#r8699749>
> .
>