Repository: blaze/blaze

Issue:
Combining `join` and indexing generates far too many records
I join two tables. Before taking the join I select only two columns from one of the tables. Instead of getting 2433 records (entry 10) I get 19524825 (entry 11).

``` py

In [1]: from blaze import Data, join

In [2]: d = Data("sqlite:///sonair.sqlite3")

In [3]: d.event.head()
Out[3]: 
          name operation  datetime_nearest_receiver aircraft  temperature_2m  \
0  24_002_A320       S16 2014-05-15 07:46:15.100000     A320             7.3   
1  24_005_A321       S16 2014-05-15 08:08:23.200000     A321             7.5   
2  24_006_A321       S16 2014-05-15 08:11:50.250000     A321             7.5   
3  24_007_A333       S16 2014-05-15 08:15:52.850000     A333             8.0   
4  24_008_F100       S16 2014-05-15 08:21:51.650000     F100             8.0   
5  24_011_B762       S16 2014-05-15 08:55:40.500000     B762             9.0   
6  24_012_A320       S16 2014-05-15 09:22:46.050000     A320             9.5   
7  24_016_A320       S16 2014-05-15 10:27:42.100000     A320            10.0   
8  24_017_B764       S16 2014-05-15 10:35:20.050000     B764            10.1   
9  24_020_A333       S16 2014-05-15 11:03:01.650000     A333             9.8   

   temperature_5cm  humidity  windspeed  pressure  include  
0              8.5      91.7        1.6     978.9        1  
1              9.4      84.8        1.2     978.9        1  
2              9.4      84.8        2.3     978.9        1  
3             11.1      84.7        2.3     979.0        1  
4             11.1      84.7        1.8     979.0        1  
5             14.9      77.6        2.2     978.7        1  
6             14.1      73.5        8.3     978.7        1  
7             16.8      72.9        2.4     978.5        1  
8             13.6      75.9        2.8     978.5        1  
9             11.3      68.4        1.8     978.4        1  

In [4]: d.event.count()
Out[4]: 2675

In [5]: d.operation.head()
Out[5]: 
  name  runway takeoff datetime_nearest_close
0  S28      28    True                      A
1  S16      16    True                      Q
2  L14      14   False                      I

In [6]: d.operation.count()
Out[6]: 3

In [7]: d.event[d.event.include==True]
Out[7]: 
           name operation  datetime_nearest_receiver aircraft  temperature_2m  \
0   24_002_A320       S16 2014-05-15 07:46:15.100000     A320             7.3   
1   24_005_A321       S16 2014-05-15 08:08:23.200000     A321             7.5   
2   24_006_A321       S16 2014-05-15 08:11:50.250000     A321             7.5   
3   24_007_A333       S16 2014-05-15 08:15:52.850000     A333             8.0   
4   24_008_F100       S16 2014-05-15 08:21:51.650000     F100             8.0   
5   24_011_B762       S16 2014-05-15 08:55:40.500000     B762             9.0   
6   24_012_A320       S16 2014-05-15 09:22:46.050000     A320             9.5   
7   24_016_A320       S16 2014-05-15 10:27:42.100000     A320            10.0   
8   24_017_B764       S16 2014-05-15 10:35:20.050000     B764            10.1   
9   24_020_A333       S16 2014-05-15 11:03:01.650000     A333             9.8   
10  24_023_A333       S16 2014-05-15 11:21:32.650000     A333             9.7   

    temperature_5cm  humidity  windspeed  pressure  include  
0               8.5      91.7        1.6     978.9        1  
1               9.4      84.8        1.2     978.9        1  
2               9.4      84.8        2.3     978.9        1  
3              11.1      84.7        2.3     979.0        1  
4              11.1      84.7        1.8     979.0        1  
5              14.9      77.6        2.2     978.7        1  
6              14.1      73.5        8.3     978.7        1  
7              16.8      72.9        2.4     978.5        1  
8              13.6      75.9        2.8     978.5        1  
9              11.3      68.4        1.8     978.4        1  
...

In [8]: d.event[d.event.include==True].count()
Out[8]: 2433

In [9]: join( d.event, d.operation, 'operation', 'name').count()
Out[9]: 2675

In [10]: join( d.event[d.event.include==True], d.operation, 'operation', 'name').count()
Out[10]: 2433

In [11]: join( d.event[d.event.include==True], d.operation[['name', 'datetime_nearest_close']], 'operation', 'name').count()
Out[11]: 19524825

In [12]: join( d.event, d.operation[['name', 'datetime_nearest_close']], 'operation', 'name').count()
Out[12]: 2675

In [13]: join( d.operation[['name', 'datetime_nearest_close']], d.event[d.event.include==True], 'name', 'operation').count()
/home/rifr/anaconda3/envs/sonair/lib/python3.4/site-packages/sqlalchemy/sql/base.py:508: SAWarning: Column 'name' on table <sqlalchemy.sql.selectable.Select at 0x7ff5d3104898; Select object> being replaced by Column('name', TEXT(), table=<Select object>), which has the same key.  Consider use_labels for select() statements.
  self[column.key] = column
/home/rifr/anaconda3/envs/sonair/lib/python3.4/site-packages/sqlalchemy/sql/base.py:508: SAWarning: Column 'name' on table <sqlalchemy.sql.selectable.Alias at 0x7ff5d3104630; alias_6> being replaced by Column('name', TEXT(), table=<alias_6>), which has the same key.  Consider use_labels for select() statements.
  self[column.key] = column
/home/rifr/anaconda3/envs/sonair/lib/python3.4/site-packages/sqlalchemy/sql/base.py:508: SAWarning: Column 'name' on table <sqlalchemy.sql.selectable.Alias at 0x7ff5d311b080; alias_6> being replaced by Column('name', TEXT(), table=<alias_6>), which has the same key.  Consider use_labels for select() statements.
  self[column.key] = column

Out[13]: 19524825


```



Discussion:
@FRidh, Can you provide a link to this dataset, or a small subset of it? I'd like to debug this and see what's going on. Thanks!

Hrm, that is strange.  I'll echo @cpcloud 's request for the dataset (are the RDatasets all available in sqlite form somewhre?)

I'd also be interested in seeing the generated SQL for line 11.

``` python
>>> from blaze import compute
>>> print compute(join( d.event[d.event.include==True], d.operation[['name', 'datetime_nearest_close']], 'operation', 'name').count())
```

Here is the generated SQL

``` py

In [1]: from blaze import *

In [2]: d = Data('sqlite:///sonair.sqlite3')

In [3]: print( compute(join( d.event[d.event.include==True], d.operation[['name', 'datetime_nearest_close']], 'operation', 'name').count()) )
SELECT count(alias_1.operation) AS __count 
FROM (SELECT event.operation AS operation, event.name AS name, event.datetime_nearest_receiver AS datetime_nearest_receiver, event.aircraft AS aircraft, event.temperature_2m AS temperature_2m, event.temperature_5cm AS temperature_5cm, event.humidity AS humidity, event.windspeed AS windspeed, event.pressure AS pressure, event.include AS include, operation.datetime_nearest_close AS datetime_nearest_close 
FROM event, operation, (SELECT event.name AS name, event.operation AS operation, event.datetime_nearest_receiver AS datetime_nearest_receiver, event.aircraft AS aircraft, event.temperature_2m AS temperature_2m, event.temperature_5cm AS temperature_5cm, event.humidity AS humidity, event.windspeed AS windspeed, event.pressure AS pressure, event.include AS include 
FROM event 
WHERE event.include = 1) JOIN (SELECT operation.name, operation.datetime_nearest_close 
FROM operation) ON event.operation = operation.name) AS alias_1

```

The dataset is not publicly available. I could create a file containing a small subset. What is the recommended way to select a certain amount of rows? Should I go the `sqlite -> DataFrame.loc -> sqlite` route for this?

I'm using the following versions:

```
blaze                     0.7.2                np19py34_0  
blaze-core                0.7.2                np19py34_0  
into                      0.2.2                np19py34_0 
```

It seems some kind of product is done somewhere since `19524825 = 2433 * 2675 * 3`

Here is the [link](https://de.owncube.com/public.php?service=files&t=415d715e3c25234f8f67e3eed86ffc37&download) to a subset. It shows the same behaviour.

``` py
In [10]: d = Data('sqlite:///subset.sqlite3')

In [11]: join( d.event[d.event.include==True], d.operation[['name', 'datetime_nearest_close']], 'operation', 'name').count()
Out[11]: 210
```

7 out of 10 with `include=True`, and 3 operations.

Thanks for the subset.  Looking at the SQL below gives a clue as to what's going wrong

``` sql
Good version

SELECT count(alias_8.operation) AS __count
FROM (SELECT event.operation AS operation, event.name AS name, 
             event.datetime_nearest_receiver AS datetime_nearest_receiver, 
             event.aircraft AS aircraft, 
             event.temperature_2m AS temperature_2m, 
             event.temperature_5cm AS temperature_5cm, 
             event.humidity AS humidity, 
             event.windspeed AS windspeed, 
             event.pressure AS pressure, 
             event.include AS include, 
             operation.runway AS runway, 
             operation.takeoff AS takeoff, 
             operation.datetime_nearest_close AS datetime_nearest_close
       FROM event JOIN operation ON event.operation = operation.name
       WHERE event.include = 1) AS alias_8

Bad version

SELECT count(alias_9.operation) AS __count
FROM (SELECT event.operation AS operation, 
             event.name AS name, 
             event.datetime_nearest_receiver AS datetime_nearest_receiver, 
             event.aircraft AS aircraft, 
             event.temperature_2m AS temperature_2m, 
             event.temperature_5cm AS temperature_5cm, 
             event.humidity AS humidity, 
             event.windspeed AS windspeed, 
             event.pressure AS pressure, 
             event.include AS include, 
             operation.datetime_nearest_close AS datetime_nearest_close
      FROM event, operation, (SELECT event.name AS name, 
                                     event.operation AS operation, 
                                     event.datetime_nearest_receiver AS datetime_nearest_receiver,                                           event.aircraft AS aircraft, 
                                     event.temperature_2m AS temperature_2m, 
                                     event.temperature_5cm AS temperature_5cm, 
                                     event.humidity AS humidity, 
                                     event.windspeed AS windspeed, 
                                     event.pressure AS pressure, 
                                     event.include AS include
                                 FROM event           
                                 WHERE event.include = 1) JOIN (SELECT operation.name, 
                                                                       operation.datetime_nearest_close
                                                                 FROM operation) 
                                                            ON event.operation = operation.name) 
                                                            AS alias_9
```

It's joining on the event table twice, hence the blowup.  I'll try to track this down today.


Resolving patch:
diff --git a/blaze/compute/sql.py b/blaze/compute/sql.py
--- a/blaze/compute/sql.py
+++ b/blaze/compute/sql.py
@@ -23,7 +23,7 @@
 from sqlalchemy.sql import Selectable, Select
 from sqlalchemy.sql.elements import ClauseElement, ColumnElement
 from sqlalchemy.engine import Engine
-from operator import and_
+from operator import and_, eq
 import itertools
 from copy import copy
 import toolz
@@ -207,26 +207,28 @@ def compute_up(t, lhs, rhs, **kwargs):
     rhs = alias_it(rhs)
 
     if isinstance(lhs, Select):
-        ldict = dict((c.name, c) for c in lhs.inner_columns)
+        lhs = lhs.alias(next(aliases))
+        left_conds = [lhs.c.get(c) for c in listpack(t.on_left)]
     else:
-        ldict = lhs.c
+        ldict = dict((c.name, c) for c in inner_columns(lhs))
+        left_conds = [ldict.get(c) for c in listpack(t.on_left)]
 
     if isinstance(rhs, Select):
-        rdict = dict((c.name, c) for c in rhs.inner_columns)
+        rhs = rhs.alias(next(aliases))
+        right_conds = [rhs.c.get(c) for c in listpack(t.on_right)]
     else:
-        rdict = rhs.c
+        rdict = dict((c.name, c) for c in inner_columns(rhs))
+        right_conds = [rdict.get(c) for c in listpack(t.on_right)]
 
+    condition = reduce(and_, map(eq, left_conds, right_conds))
 
-    condition = reduce(and_,
-            [lower_column(ldict.get(l)) == lower_column(rdict.get(r))
-        for l, r in zip(listpack(t.on_left), listpack(t.on_right))])
-
+    # Perform join
     if t.how == 'inner':
         join = _join_selectables(lhs, rhs, condition=condition)
         main, other = lhs, rhs
     elif t.how == 'left':
-        join = _join_selectables(lhs, rhs, condition=condition, isouter=True)
         main, other = lhs, rhs
+        join = _join_selectables(lhs, rhs, condition=condition, isouter=True)
     elif t.how == 'right':
         join = _join_selectables(rhs, lhs, condition=condition, isouter=True)
         main, other = rhs, lhs
@@ -234,11 +236,19 @@ def compute_up(t, lhs, rhs, **kwargs):
         # http://stackoverflow.com/questions/20361017/sqlalchemy-full-outer-join
         raise ValueError("SQLAlchemy doesn't support full outer Join")
 
-    def cols(x):
-        if isinstance(x, Select):
-            return list(x.inner_columns)
-        else:
-            return list(x.columns)
+    """
+    We now need to arrange the columns in the join to match the columns in
+    the expression.  We care about order and don't want repeats
+    """
+    if isinstance(join, Select):
+        def cols(x):
+            if isinstance(x, Select):
+                return list(x.inner_columns)
+            else:
+                return list(x.columns)
+    else:
+        cols = lambda x: list(x.columns)
+
     main_cols = cols(main)
     other_cols = cols(other)
     left_cols = cols(lhs)


Test patch:
diff --git a/blaze/compute/tests/test_sql_compute.py b/blaze/compute/tests/test_sql_compute.py
--- a/blaze/compute/tests/test_sql_compute.py
+++ b/blaze/compute/tests/test_sql_compute.py
@@ -150,21 +150,21 @@ def test_clean_complex_join():
 
     result = compute(joined, {L: lhs, R: rhs})
 
+    expected1 = """
+        SELECT amounts.name, amounts.amount, ids.id
+        FROM amounts JOIN ids ON amounts.name = ids.name
+        WHERE amounts.amount > :amount_1"""
 
-    assert (normalize(str(result)) == normalize("""
-    SELECT amounts.name, amounts.amount, ids.id
-    FROM amounts JOIN ids ON amounts.name = ids.name
-    WHERE amounts.amount > :amount_1""")
-
-    or
-
-    normalize(str(result)) == normalize("""
-    SELECT amounts.name, amounts.amount, ids.id
-    FROM amounts, (SELECT amounts.name AS name, amounts.amount AS amount
-    FROM amounts
-    WHERE amounts.amount > :amount_1) JOIN ids ON amounts.name = ids.name"""))
+    expected2 = """
+        SELECT alias.name, alias.amount, ids.id
+        FROM (SELECT amounts.name AS name, amounts.amount AS amount
+              FROM amounts
+              WHERE amounts.amount > :amount_1) AS alias
+        JOIN ids ON alias.name = ids.name"""
 
 
+    assert (normalize(str(result)) == normalize(expected1) or
+            normalize(str(result)) == normalize(expected2))
 
 
 def test_multi_column_join():
@@ -582,13 +582,25 @@ def test_clean_join():
 
 
     expr = join(join(tfriends, tname, 'a', 'id'), tcity, 'a', 'id')
-    assert normalize(str(compute(expr, ns))) == normalize("""
+
+    result = compute(expr, ns)
+
+    expected1 = """
     SELECT friends.a, friends.b, name.name, place.city, place.country
     FROM friends
         JOIN name ON friends.a = name.id
         JOIN place ON friends.a = place.id
-        """)
+        """
+
+    expected2 = """
+    SELECT alias.a, alias.b, alias.name, place.city, place.country
+    FROM (SELECT friends.a AS a, friends.b AS b, name.name AS name
+          FROM friends JOIN name ON friends.a = name.id) AS alias
+    JOIN place ON alias.a = place.id
+    """
 
+    assert (normalize(str(result)) == normalize(expected1) or
+            normalize(str(result)) == normalize(expected2))
 
 
 def test_like():
@@ -661,10 +673,20 @@ def test_join_complex_clean():
     expr = join(tname[tname.id > 0], tcity, 'id')
     result = compute(expr, ns)
 
-    assert normalize(str(result)) == normalize("""
-    SELECT name.id, name.name, place.city, place.country
-    FROM name JOIN place ON name.id = place.id
-    WHERE name.id > :id_1""")
+    expected1 = """
+        SELECT name.id, name.name, place.city, place.country
+        FROM name JOIN place ON name.id = place.id
+        WHERE name.id > :id_1"""
+
+    expected2 = """
+        SELECT alias.id, alias.name, place.city, place.country
+        FROM (SELECT name.id as id, name.name AS name
+              FROM name
+              WHERE name.id > :id_1) AS alias
+        JOIN place ON alias.id = place.id"""
+    assert (normalize(str(result)) == normalize(expected1) or
+            normalize(str(result)) == normalize(expected2))
+
 
 
 def test_projection_of_join():
@@ -686,11 +708,24 @@ def test_projection_of_join():
 
     ns = {tname: name, tcity: city}
 
-    assert normalize(str(compute(expr, ns))) == normalize("""
-    SELECT place.country, name.name
-    FROM name JOIN place ON name.id = place.id
-    WHERE place.city = :city_1""")
+    result = compute(expr, ns)
+
+    expected1 = """
+        SELECT place.country, name.name
+        FROM name JOIN place ON name.id = place.id
+        WHERE place.city = :city_1"""
+
+    expected2 = """
+        SELECT alias.country, name.name
+        FROM name
+        JOIN (SELECT place.id AS id, place.city AS city, place.country AS country
+              FROM place
+              WHERE place.city = :city_1) AS alias
+        ON name.id = alias_6.id"""
+
 
+    assert (normalize(str(result)) == normalize(expected1) or
+            normalize(str(result)) == normalize(expected2))
 
 def test_lower_column():
     metadata = sa.MetaData()
@@ -1049,12 +1084,23 @@ def test_join_count():
 
     result = compute(expr, {db: engine}, post_compute=False)
 
-    assert normalize(str(result)) == normalize("""
+    expected1 = """
     SELECT count(alias.x) as count
     FROM (SELECT t1.x AS x, t1.y AS y, t2.b AS b
           FROM t1 JOIN t2 ON t1.x = t2.a
           WHERE t1.x > ?) as alias
-          """)
+          """
+    expected2 = """
+    SELECT count(alias2.x) AS __count
+    FROM (SELECT alias1.x AS x, alias1.y AS y, t2.b AS b
+          FROM (SELECT t1.x AS x, t1.y AS y
+                FROM t1
+                WHERE t1.x > ?) AS alias1
+          JOIN t2 ON alias1.x = t2.a) AS alias2"""
+
+
+    assert (normalize(str(result)) == normalize(expected1) or
+            normalize(str(result)) == normalize(expected2))
 
 
 def test_merge_compute():
@@ -1080,3 +1126,52 @@ def test_head_limit():
     assert compute(t.head(5).head(10), s)._limit == 5
     assert compute(t.head(10).head(5), s)._limit == 5
     assert compute(t.head(10).head(10), s)._limit == 10
+
+
+def test_no_extraneous_join():
+    ds = """ {event: var * {name: ?string,
+                            operation: ?string,
+                            datetime_nearest_receiver: ?datetime,
+                            aircraft: ?string,
+                            temperature_2m: ?float64,
+                            temperature_5cm: ?float64,
+                            humidity: ?float64,
+                            windspeed: ?float64,
+                            pressure: ?float64,
+                            include: int64},
+             operation: var * {name: ?string,
+                               runway: int64,
+                               takeoff: bool,
+                               datetime_nearest_close: ?string}}
+        """
+    db = resource('sqlite:///:memory:', dshape=ds)
+
+    d = symbol('db', dshape=ds)
+
+    expr = join(d.event[d.event.include==True],
+                d.operation[['name', 'datetime_nearest_close']],
+                'operation', 'name')
+
+    result = compute(expr, db)
+
+    assert normalize(str(result)) == normalize("""
+    SELECT alias.operation, alias.name, alias.datetime_nearest_receiver,
+           alias.aircraft, alias.temperature_2m, alias.temperature_5cm,
+           alias.humidity, alias.windspeed, alias.pressure,
+           alias.include, alias.datetime_nearest_close
+          FROM (SELECT event.name AS name,
+                       event.operation AS operation,
+                       event.datetime_nearest_receiver AS datetime_nearest_receiver,
+                       event.aircraft AS aircraft,
+                       event.temperature_2m AS temperature_2m,
+                       event.temperature_5cm AS temperature_5cm,
+                       event.humidity AS humidity,
+                       event.windspeed AS windspeed,
+                       event.pressure AS pressure,
+                       event.include AS include
+                FROM event WHERE event.include = 1) AS alias1
+                JOIN (SELECT  operation.name AS name,
+                              operation.datetime_nearest_close as datetime_nearest_close
+                      FROM operation) AS alias2
+                ON alias1.operation = alias2.name
+    """)