astropy/astropy — pull request #2790 by taldcroft (2014-09-30 14:33:58+00:00)
Title: Remove numpy structured array as data container in Table

This issue is to explore the idea of entirely removing the `np.ndarray` or `np.ma.MaskedArray` structured array object that currently serves as the primary data storage. This object is referenced as `Table._data` in the code.

The way `Table._data` currently works is that it is essentially the owner of all the table data and the `Column` objects then hold references to the columns in `Table._data` (hereafter just `_data`). A fair bit of the trickier table code is maintaining that duality when mutable table operations are performed. When you start trying to integrate other data types as `Columns` it gets very tricky / untenable.

The proposal is to make a `Table` be a collection of `Columns` where the `Column` objects are the primary and independent data owners. I am hoping that implementation of this will actually result in a net reduction of code and overall simplification of logic. Some advantages:

- Issues #1833, #1835, and #1839 should all become much easier. This is a big driver.
- Relatedly, potential for more flexibility to match needs of VOtable and FITS tables (#1964 and #199). Nested tables and columns with different properties should be much easier.
- Cheap addition / removal of columns. An idiom that people (@eteq!) apparently like is to create an empty table and then progressively build the table by adding one column at a time. This is currently very inefficient since the `_data` object needs to be re-built from scratch for each column.
- Columnar access (probably the dominant mode) may be faster because the data values in memory won't be strided.

Issues:

- The `Row` class will need work to replicate the functionality currently handled by `np.void` and `np.mvoid` classes. There is something weird and magical about those classes I have never entirely understood, so fully replicating them represents a real uncertainty right now.
- It's likely that iterating through a table by rows will be even slower. Even now there is a lot of Python code running for each row so it's starting out slow.
- Users that are defying convention and using `_data` directly will be out of luck, though we could put in a compatibility property that basically creates the equivalent `ndarray` on request (and issues a warning saying don't use this!).
- The accepted way to create a pure `np.array` from a `Table` will always generate a copy. Currently it can be a reference.
- The high level table operations are implemented at a low level using pure `ndarray`s. This will need some work.

Comments / discussion please!

@astrofrog @eteq @embray @mhvk @mdboom