Repository: blaze/blaze

Issue:
Bug: columns attribute of TableSymbol is None when creating a schema with discover(tables.Table)
Example:

``` python
from blaze import discover, TableSymbol
import tables as tb
f = tb.open_file('/data/bitcoin/btc.h5')
d = f.root.btc
schema = discover(d)
t = TableSymbol('t', schema=schema)
assert t.columns is not None
```



Discussion:
`discover` gives you the full datashape

``` Python
In [7]: discover(d)
Out[7]: dshape("37450461 * { transaction : int64, sender : int64, recipient : int64, timestamp : int64, value : float64 }")
```

We use the term `schema` to mean the datashape of a row, i.e.

``` Python
In [12]: discover(d).subshape[0]
Out[12]: dshape("{ transaction : int64, sender : int64, recipient : int64, timestamp : int64, value : float64 }")
```

TableSymbol is willing to accept either argument

``` Python
In [13]: TableSymbol('t', discover(d)).schema
Out[13]: dshape("{ transaction : int64, sender : int64, recipient : int64, timestamp : int64, value : float64 }")

In [14]: TableSymbol('t', discover(d).subshape[0]).schema
Out[14]: dshape("{ transaction : int64, sender : int64, recipient : int64, timestamp : int64, value : float64 }")
```

But apparently it fails silently when improper keyword arguments are used.

> We use the term `schema` to mean the datashape of a row

thanks for clarifying, didn't know this.

should this only accept schema as you've defined above and not a full data shape?

We used to do this. Full datashapes were allowed in order to enable reasoning on the lengths of tables.  This change happened recently in https://github.com/ContinuumIO/blaze/pull/473 

Note that if you just do `Table(d)` it will always do the right thing.

i wanted to write down an expression that was independent of the data and not have `Table` do things automatically for me. i wanted to do 

``` python
gb = by(t, t.date, t.value.mean())
result = compute(gb, into(pd.DataFrame, into(np.ndarray, data)))
```

am i doing this in a non blaze-y way? (`data` is an instance of `tables.Table`)

Seems fine.  Thoughts that arise are. 

Why not `compute(gb, data)`?  Is it stored in an inefficient container?

Why not `into(pd.DataFrame, data)`.  Was this not implemented?

The first thing I would have tried is:

```
gb = by(t, t.date, t.value.mean())
result = compute(gb, data)
```

as long as `t` has the same shape as `data`

> Why not `compute(gb, data)`?

it dispatches to the python streaming groupby which is much slower than pandas in this case

> Why not `into(pd.DataFrame, data)`?

Didn't try that yet.

`into(pd.DataFrame, data)` works great, but the issue is not that.

> `gb = by(t, t.date, t.value.mean())`
> `result = compute(gb, data)`

in the above lines i can't access the columns if the way i constructed `t` was like this:

``` python
t = TableSymbol('t', schema=discover(data))
```

Replace

```
t = TableSymbol('t', schema=discover(data))
```

with 

```
t = TableSymbol('t', dshape=discover(data))
```

or even just 

```
t = TableSymbol('t', discover(data))
```

oh gosh feeling slow now ... didn't realize that one could specify schema and or dshape

should this raise an error if `schema` is not a `Record`?

likewise if `dshape` is not a `DataShape` instance?

They'll both be DataShape instances.  Probably it should err based on the result of `isdimension(ds[0])`

this issue revealed that the schema/datashape distinction was also ambiguous for me. Thanks for bringing this up! I think the error should be if `dshape` is not a `DataShape` or not convertible to one via `datashape.dshape`

pr in the works

ok cool. I would think we could help people out with a try/except block around this in Table **init**:

```
       if dshape and isinstance(dshape, _strtypes):
            dshape = datashape.dshape(dshape)
        if dshape and not isdimension(dshape[0]):
            dshape = var * dshape
```

why not just have `dshape` and get rid of `schema`? the `isdimension` check essentially turns `schema`-like things into a `dshape`-like thing

`dshape` is stored internally.  `schema` is used only at the user interface.  We could require dshapes but then we'd have to explain `var` to people.  I'm open to it, not particularly worried about having schema around though.  

ok. i'm not sure why you'd have to explain `var`, since what's currently a valid schema would is also be a valid datashape. people can just continue doing what they're doing, except that now they have to pass in `dshape=a_datashape` instead of `schema=a_datashape`. i think having both is more confusing than having to explain `var`. i ended up needing an explanation of `schema` vs `dshape` anyway.

also i find the constructor is hard to reason about (mostly because i don't know `datashape` very well) whereas if it was just _one_ kind of datashape-y thing i think it would be easier to understand

just my 2c and not a huge priority, but maybe a point for potential user confusion.

Ah, you're saying "just remove the schema=" keyword argument, not "remove the input of just a schema"

yep

Yeah, that sounds super-reasonable


Resolving patch:
diff --git a/blaze/compute/chunks.py b/blaze/compute/chunks.py
--- a/blaze/compute/chunks.py
+++ b/blaze/compute/chunks.py
@@ -59,7 +59,7 @@ def __iter__(self):
 
 @dispatch(tuple(reductions), ChunkIter)
 def compute_one(expr, c, **kwargs):
-    t = TableSymbol('_', schema=expr.child)
+    t = TableSymbol('_', dshape=expr.child)
     a, b = reductions[type(expr)]
 
     return compute_one(b(t), [compute_one(a(t), chunk) for chunk in c])
diff --git a/blaze/expr/table.py b/blaze/expr/table.py
--- a/blaze/expr/table.py
+++ b/blaze/expr/table.py
@@ -168,16 +168,14 @@ class TableSymbol(TableExpr):
     __slots__ = '_name', 'dshape', 'iscolumn'
     __inputs__ = ()
 
-    def __init__(self, name, dshape=None, iscolumn=False, schema=None):
+    def __init__(self, name, dshape=None, iscolumn=False):
         self._name = name
-        if schema and dshape:
-            raise ValueError("Please specify one of schema= or dshape= keyword"
-                    " arguments")
-        if schema and not dshape:
-            dshape = datashape.var * schema
         if isinstance(dshape, _strtypes):
             dshape = datashape.dshape(dshape)
-        if not isdimension(dshape[0]):
+        try:
+            if not isdimension(dshape[0]):
+                dshape = datashape.var * dshape
+        except TypeError:
             dshape = datashape.var * dshape
         self.dshape = dshape
         self.iscolumn = iscolumn


Test patch:
diff --git a/blaze/compute/tests/test_chunks_compute.py b/blaze/compute/tests/test_chunks_compute.py
--- a/blaze/compute/tests/test_chunks_compute.py
+++ b/blaze/compute/tests/test_chunks_compute.py
@@ -76,7 +76,7 @@ def test_head():
 
 
 def test_join():
-    cities = TableSymbol('cities', schema='{id: int, city: string}')
+    cities = TableSymbol('cities', dshape='{id: int, city: string}')
     j = join(t, cities, 'id')
 
     city_data = [[1, 'NYC'], [1, 'Chicago'], [5, 'Paris']]
diff --git a/blaze/compute/tests/test_pandas_compute.py b/blaze/compute/tests/test_pandas_compute.py
--- a/blaze/compute/tests/test_pandas_compute.py
+++ b/blaze/compute/tests/test_pandas_compute.py
@@ -406,7 +406,7 @@ def test_outer_join():
 
 def test_by_on_same_column():
     df = pd.DataFrame([[1,2],[1,4],[2,9]], columns=['id', 'value'])
-    t = TableSymbol('data', schema='{id:int, value:int}')
+    t = TableSymbol('data', dshape='{id:int, value:int}')
 
     gby = by(t, t['id'], t['id'].count())
 
diff --git a/blaze/compute/tests/test_pytables_compute.py b/blaze/compute/tests/test_pytables_compute.py
--- a/blaze/compute/tests/test_pytables_compute.py
+++ b/blaze/compute/tests/test_pytables_compute.py
@@ -10,6 +10,8 @@
 
 from blaze.compute.core import compute
 from blaze.expr import TableSymbol
+from blaze import discover
+import tables as tb
 
 
 t = TableSymbol('t', '{id: int, name: string, amount: int}')
@@ -60,8 +62,16 @@ def eq(a, b):
     return (a == b).all()
 
 
+def test_discover_datashape(data):
+    ds = discover(data)
+    t = TableSymbol('t', dshape=ds)
+    columns = t.columns
+    assert columns is not None
+
+
 def test_table(data):
     assert compute(t, data) == data
+    assert isinstance(data, tb.Table)
 
 
 def test_single_column(data):
diff --git a/blaze/expr/tests/test_table.py b/blaze/expr/tests/test_table.py
--- a/blaze/expr/tests/test_table.py
+++ b/blaze/expr/tests/test_table.py
@@ -297,6 +297,27 @@ def symsum():
     return t, t.amount.sum()
 
 
+@pytest.fixture
+def ds():
+    return dshape("var * { "
+            "transaction_key : int64, "
+            "user_from_key : int64, "
+            "user_to_key : int64, "
+            "date : int64, "
+            "value : float64 "
+            "}")
+
+
+def test_discover_dshape_symbol(ds):
+    t_ds = TableSymbol('t', dshape=ds)
+    assert t_ds.columns is not None
+
+    t_sch = TableSymbol('t', dshape=ds.subshape[0])
+    assert t_sch.columns is not None
+
+    assert t_ds.isidentical(t_sch)
+
+
 class TestScalarArithmetic(object):
     ops = {'+': add, '-': sub, '*': mul, '/': truediv, '//': floordiv, '%': mod,
            '**': pow, '==': eq, '!=': ne, '<': lt, '>': gt, '<=': le, '>=': ge}