ContinuumIO/blaze — pull request #255 by mrocklin (2014-05-08 17:43:26+00:00)
Title: [WIP] - Indexing on data layer

This adds a new, relatively heavyweight abstraction onto the data layer. Indexing with `__getitem__`.

```Python
In [1]: from blaze.data import *

In [2]: schema = '{date: date, sum: real, mean: real, std: real, open: real, high: real, low: real, close: real, volume_btc: real, volume_currency:real, price: real}'

In [3]: dd = CSV('btc_us_dollar_table.csv', schema=schema)
In [6]: dd[::100, 'date']
Out[6]:
(datetime.date(2010, 7, 17),
datetime.date(2010, 10, 25),
datetime.date(2011, 2, 2),
datetime.date(2011, 5, 13),
datetime.date(2011, 8, 27),
datetime.date(2011, 12, 5),
datetime.date(2012, 3, 14),
datetime.date(2012, 6, 22),
datetime.date(2012, 9, 30),
datetime.date(2013, 1, 8))

In [10]: dd[::100, 0] == dd[::100, 'date'] # can also index by column number
Out[10]: True

In [7]: timeit dd[::100, 'date'] # we only parse what's necessary
1000 loops, best of 3: 1.22 ms per loop

In [8]: timeit dd.as_py()
10 loops, best of 3: 101 ms per loop
```

Honestly this PR feels pretty hacky, there is a huge amount of micro-logic everywhere. This may be necessary because we're trying to wrap all of the complexity of fancy-ish indexing along with the datashape record-row similarity. I've tried to pull out common functionality into `utils.py` but I expect that refactoring this code would be helpful long term. It's sorta well tested but that's the only thing that gives me confidence in its robustness.

# Other changes

## Tuples Everywhere

Previously data descriptors would often emit and consume some mix of tuples and dicts. Covering this ambiguity along with all of the other dimensions of complexity (fancy indexing, records) was too much. I now implement a tuples everywhere policy. Data descriptors emit ordered immutable collections of data (i.e. nested tuples). All information lost in this transformation is still in the datashape. Datashape and data are now more orthogonal.

I've been building utility functions to swap formats for usability though.

## `Files` data descriptor becomes `Concat` and `Stack`

We also swap out the old `Files` data descriptor with `Stack` and `Concat`. These are nice because they can be composed with any data descriptor (not just file based ones) but they're less clever about searching for new files. They're also much simpler and more explicit.

## Performance

Just a final note, I'm not thinking much about constant factor performance. I'm supporting laziness when it makes sense and I'm not doing anything terribly dumb, but for the most part I expect that there is an order of magnitude slowdown in the code that I'm writing. I hope that this is easy to benchmark and correct once a cohesive plan is in place.