Repository: blaze/blaze

Issue:
Slicing on Interactive Data with SQL does not work
Hi all,

Big fan of the project. I've started working on a 10 minutes to blaze tutorial (#423) and noticed that Slicing doesn't work for interactive objects. See [here](http://nbviewer.ipython.org/github/Will-So/Blaze-Workbook/blob/Slicing.ipynb) for an example.

I'd be happy to work on this but I still don't have a good grasp how exactly the `Data` function works. Any tips would be appreciated. 



Discussion:
Hi Will,
Thanks for the kind words. Slicing doesn't work for the SQL backend because
the ordering of rows isn't predictable. Relational algebra deals with sets
of tuples which don't have a deterministic ordering by default. Of course
order by clauses can induce an order but there currently isn't a
generic way to tell whether an expression is ordered. However maybe having
generic metadata on the expression regarding its order isn't necessary. It
might be possible to inspect the current sqlalchemy expression and see if
there's an order by clause present and then slice and otherwise raise an
error.

On Friday, May 1, 2015, Will So notifications@github.com wrote:

> Hi all,
> 
> Big fan of the project. I've started working on a 10 minutes to blaze
> tutorial (#423 https://github.com/ContinuumIO/blaze/issues/423) and
> noticed that Slicing doesn't work for interactive objects. See here
> http://nbviewer.ipython.org/github/Will-So/Blaze-Workbook/blob/master/Slicing.ipynb
> for an example.
> 
> I'd be happy to work on this but I still don't have a good grasp how
> exactly the Data function works. Any tips would be appreciated.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ContinuumIO/blaze/issues/1072.

## 

## 

Best,
Phillip Cloud

Checking to see if an SQLAlchemy object has an `order_by` statement would make sense.

In the case of there being no `order_by` clause, a good fallback might be to treat the slice as `df[offset:limit]` and print a warning that the order is not guaranteed to be the same each time. Indeed, this is how SQLAlchemy [handles](http://docs.sqlalchemy.org/en/improve_toc/orm/query.html) its `slice` function (without the warning). This would satisfy most of the use cases for slicing. 

Without functionality like this it will be difficult to explore rows outside of the first 10 using Blaze. This is also a problem with functions like `distinct()`. You have to iterate through the results in order to view more than 10 distinct items. e.g., to get a list of 50 distinct items:

``` python
items = []
for item in df.H.distinct():
    items.append(item)
    if len(items) == 50:
        break
items
```

Rather than Pandas/Numpy which is just `df.H.unique()[:50]`. Operations like these are often one of the first things we do when examining a new dataset.  Although this particular problem can be solved by allowing `head()` to return more than 10 items. 

Head can return more than 10 items.  What you're running into here is that
the default repr function of interactive symbols slaps a head(10) on to the
end of your expression to avoid dumping everything to the screen.  If you
actually want the full result you need to specify how you would like to
represent it in memory.  This might be done, for example, by

```
odo(expr.head(50), pd.DataFrame)
```

or

```
list(expr.head(50))
```

One could reinterpret your request as "If the user has already specified a
head(...) length on their expression then please respect that when
printing", which seems pretty reasonable.

On Fri, May 1, 2015 at 5:23 AM, Will So notifications@github.com wrote:

> Checking to see if an SQLAlchemy object has an order_by statement would
> make sense.
> 
> In the case of there being no order_by clause, a good fallback might be
> to treat the slice as df[offset:limit] and print a warning that the order
> is not guaranteed to be the same each time. Indeed, this is how SQLAlchemy
> handles http://docs.sqlalchemy.org/en/improve_toc/orm/query.html its
> slice function (without the warning). This would satisfy most of the use
> cases for slicing.
> 
> Without functionality like this it will be difficult to explore rows
> outside of the first 10 using Blaze. This is also a problem with functions
> like distinct(). You have to iterate through the results in order to view
> more than 10 distinct items. e.g., to get a list of 50 distinct items:
> 
> items = []for item in df.H.distinct():
>     items.append(item)
>     if len(items) == 50:
>         break
> items
> 
> Rather than Pandas/Numpy which is just df.H.unique()[:50]. Operations
> like these are often one of the first things we do when examining a new
> dataset. Although this particular problem can be solved by allowing head()
> to return more than 10 items.
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/ContinuumIO/blaze/issues/1072#issuecomment-98156115.

@Will-So The way to do this would be something like this:

``` python
@dispatch(Slice, Selectable)
def compute_up(expr, data, **kwargs):
    index = expr.index
    if isinstance(index, slice):
        start = index.start
        if start < 0:
            raise ValueError('start value of slice cannot be negative')
        stop = index.stop
        if stop < 0:
            raise ValueError('stop value of slice cannot be negative')
        if index.step is not None:
            raise ValueError('step parameter in slice objects not supported')
    elif isinstance(index, (np.integer, numbers.Integral)):
        if index < 0:
            raise ValueError('integer slice cannot be negative for the SQL backend')
        start = index
        stop = start + 1
    else:
        raise TypeError('type %r not supported for slicing' % type(index).__name__)
    return data.offset(start).limit(stop)
```

Thanks for the detailed responses Phillip and Matthew. I now understand Blaze a better. I'll start working on a PR. 


Resolving patch:
diff --git a/blaze/compute/sql.py b/blaze/compute/sql.py
--- a/blaze/compute/sql.py
+++ b/blaze/compute/sql.py
@@ -35,6 +35,11 @@
 from toolz import unique, concat, pipe
 from toolz.curried import map
 
+import numpy as np
+import numbers
+
+import warnings
+
 from multipledispatch import MDNotImplementedError
 
 from odo.backends.sql import metadata_of_engine
@@ -43,7 +48,7 @@
 
 from .core import compute_up, compute, base
 
-from ..expr import Projection, Selection, Field, Broadcast, Expr, IsIn
+from ..expr import Projection, Selection, Field, Broadcast, Expr, IsIn, Slice
 from ..expr import (BinOp, UnaryOp, USub, Join, mean, var, std, Reduction,
                     count, FloorDiv, UnaryStringFunction, strlen, DateTime)
 from ..expr import nunique, Distinct, By, Sort, Head, Label, ReLabel, Merge
@@ -538,11 +543,19 @@ def compute_up(t, s, **kwargs):
                      order_by=get_clause(getattr(s, 'element', s), 'order_by'))
 
 
-@dispatch(Sort, ClauseElement)
+@dispatch(Sort, (Selectable, Select))
+def compute_up(t, s, **kwargs):
+    s = select(s.cte())
+    direction = sa.asc if t.ascending else sa.desc
+    cols = [direction(lower_column(s.c[c])) for c in listpack(t.key)]
+    return s.order_by(*cols)
+
+
+@dispatch(Sort, (sa.Table, ColumnElement))
 def compute_up(t, s, **kwargs):
     s = select(s)
     direction = sa.asc if t.ascending else sa.desc
-    cols = [direction(lower_column(getattr(s.c, c))) for c in listpack(t.key)]
+    cols = [direction(lower_column(s.c[c])) for c in listpack(t.key)]
     return s.order_by(*cols)
 
 
@@ -793,3 +806,40 @@ def post_compute(_, s, **kwargs):
 @dispatch(IsIn, ColumnElement)
 def compute_up(expr, data, **kwargs):
     return data.in_(expr._keys)
+
+
+@dispatch(Slice, (Select, Selectable, ColumnElement))
+def compute_up(expr, data, **kwargs):
+    index = expr.index[0]  # [0] replace_slices returns tuple ((start, stop), )
+    if isinstance(index, slice):
+        start = index.start or 0
+        if start < 0:
+            raise ValueError('start value of slice cannot be negative'
+                             ' with a SQL backend')
+
+        stop = index.stop
+        if stop is not None and stop < 0:
+            raise ValueError('stop value of slice cannot be negative with a '
+                             'SQL backend.')
+
+        if index.step is not None and index.step != 1:
+            raise ValueError('step parameter in slice objects not supported '
+                             'with SQL backend')
+
+    elif isinstance(index, (np.integer, numbers.Integral)):
+        if index < 0:
+            raise ValueError('integer slice cannot be negative for the'
+                             ' SQL backend')
+        start = index
+        stop = start + 1
+    else:
+        raise TypeError('type %r not supported for slicing wih SQL backend'
+                         % type(index).__name__)
+
+    warnings.warn('The order of the result set from a Slice expression '
+                  'computed against the SQL backend is not deterministic.')
+
+    if stop is None:  # Represents open-ended slice. e.g. [3:]
+        return select(data).offset(start)
+    else:
+        return select(data).offset(start).limit(stop - start)
diff --git a/blaze/expr/expressions.py b/blaze/expr/expressions.py
--- a/blaze/expr/expressions.py
+++ b/blaze/expr/expressions.py
@@ -415,6 +415,20 @@ def sliceit(child, index):
 
 
 class Slice(Expr):
+    """
+    Elements `start` until `stop`. On many backends, a `step` parameter
+    is also allowed.
+
+    Examples
+    -------
+    >>> from blaze import symbol
+    >>> accounts = symbol('accounts', 'var * {name: string, amount: int}')
+    >>> accounts[2:7].dshape
+    dshape("5 * {name: string, amount: int32}")
+    >>> accounts[2:7:2].dshape
+    dshape("3 * {name: string, amount: int32}")
+
+    """
     __slots__ = '_hash', '_child', '_index'
 
     @property
diff --git a/blaze/expr/utils.py b/blaze/expr/utils.py
--- a/blaze/expr/utils.py
+++ b/blaze/expr/utils.py
@@ -58,8 +58,21 @@ def hashable_index(index):
     return index
 
 
-
 def replace_slices(index):
+    """
+    Takes input from Slice expression and returns either a list,
+    slice object, or tuple.
+
+    Examples
+    -------
+    >>> replace_slices([1, 2, 345, 12])
+    [1, 2, 345, 12]
+    >>> type(replace_slices(_slice(1, 5, None))) is slice
+    True
+    >>> type(replace_slices((2, 5))) is tuple
+    True
+
+    """
     if isinstance(index, hashable_list):
         return list(index)
     elif isinstance(index, _slice):
@@ -67,5 +80,3 @@ def replace_slices(index):
     elif isinstance(index, tuple):
         return tuple(map(replace_slices, index))
     return index
-
-


Test patch:
diff --git a/blaze/compute/tests/test_sql_compute.py b/blaze/compute/tests/test_sql_compute.py
--- a/blaze/compute/tests/test_sql_compute.py
+++ b/blaze/compute/tests/test_sql_compute.py
@@ -202,9 +202,21 @@ def test_join():
 
     result = compute(joined.sort('amount'), {L: lhs, R: rhs})
     assert normalize(str(result)) == normalize("""
-    SELECT amounts.name, amounts.amount, ids.id
-    FROM amounts JOIN ids ON amounts.name = ids.name
-    ORDER BY amounts.amount ASC""")
+    with anon_1 as (select
+            amounts.name as name,
+            amounts.amount as amount,
+            ids.id as id from amounts
+        join
+            ids
+        on
+            amounts.name = ids.name) select
+        anon_1.name,
+        anon_1.amount,
+        anon_1.id
+    from
+        anon_1
+    order by
+        anon_1.amount asc""")
 
 
 def test_clean_complex_join():
@@ -1540,6 +1552,31 @@ def test_not():
     assert normalize(result) == normalize(expected)
 
 
+def test_slice():
+    start, stop, step = 50, 100, 1
+    result = str(compute(t[start:stop], s))
+
+    # Verifies that compute is translating the query correctly
+    assert result == str(select(s).offset(start).limit(stop))
+
+    # Verifies the query against expected SQL query
+    expected = """
+    SELECT accounts.name, accounts.amount, accounts.id FROM accounts
+    LIMIT :param_1 OFFSET :param_2
+    """
+
+    assert normalize(str(result)) == normalize(str(expected))
+
+    # Step size of 1 should be alright
+    compute(t[start:stop:step], s)
+
+
+@pytest.mark.xfail(raises=ValueError)
+def test_slice_step():
+    start, stop, step = 50, 100, 2
+    compute(t[start:stop:step], s)
+
+
 def test_datetime_to_date():
     expr = tdate.occurred_on.date
     result = str(compute(expr, sdate))
@@ -1577,6 +1614,23 @@ def test_timedelta_arith(sql_with_dts):
     ).all()
 
 
+def test_sort_compose():
+    expr = t.name[:5].sort()
+    result = compute(expr, s)
+    expected = """with anon_1 as (select
+            accounts.name as name
+        from
+            accounts
+        limit :param_1
+        offset :param_2) select
+            anon_1.name
+        from
+            anon_1
+        order by
+            anon_1.name asc"""
+    assert normalize(str(result)) == normalize(expected)
+    assert normalize(str(compute(t.sort('name').name[:5], s))) != normalize(expected)
+
 
 @pytest.yield_fixture
 def sql_with_float(url):