Repository: dask/dask

Issue:
read_sql_table gives AttributeError when reading a database
When I try to read from a sql database:
`df_database = create_engine('sqlite:///' + database_path)`
`df = dd.read_sql_table('table', df_database, 'id_driver')`

I get this error: `AttributeError: 'Engine' object has no attribute '_instantiate_plugins'`

If I use the pandas `read_sql_query` function on the same db, it works just fine.

Here's a full trace:
> AttributeError Traceback (most recent call last)
> ipython-input-10-2da58b7e1773 in <module>()
>       3 t0 = time.time()
> ----> 4 df = dd.read_sql_table('table', df_database, 'id_driver')
>       5 t1 = time.time()
>       6 
> 
> ~/lib/python3.6/site-packages/dask/dataframe/io/sql.py in read_sql_table(table, uri, index_col, divisions, npartitions, limits, columns, bytes_per_chunk, **kwargs)
>      66     if index_col is None:
>      67         raise ValueError("Must specify index column to partition on")
> ---> 68     engine = sa.create_engine(uri)
>      69     meta = sa.MetaData()
>      70     if isinstance(table, six.string_types):
> 
> ~/lib/python3.6/site-packages/sqlalchemy/engine/__init__.py in create_engine(*args, **kwargs)
>     389     strategy = kwargs.pop('strategy', default_strategy)
>     390     strategy = strategies.strategies[strategy]
> --> 391     return strategy.create(*args, **kwargs)
>     392 
>     393 
> 
> ~/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py in create(self, name_or_url, **kwargs)
>      50         u = url.make_url(name_or_url)
>      51 
> ---> 52         plugins = u._instantiate_plugins(kwargs)
>      53 
>      54         u.query.pop('plugin', None)
> 
> AttributeError: 'Engine' object has no attribute '_instantiate_plugins'


Discussion:
reprex: 

```python
In [3]: import dask.dataframe as dd
imp
In [4]: import pandas as pd

In [5]: from sqlalchemy import create_engine

In [6]: engine = create_engine("sqlite:///tmp.db")

In [7]: df = pd.DataFrame({"A": [1] * 100})

In [8]: df.index.name = 'idx'

In [9]: df.to_sql("table", engine, index=True)

In [10]: r = dd.read_sql_table("table", engine, "idx")
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-10-63977ffca8c7> in <module>()
----> 1 r = dd.read_sql_table("table", engine, "idx")

~/Envs/dask-dev/lib/python3.6/site-packages/dask/dask/dataframe/io/sql.py in read_sql_table(table, uri, index_col, divisions, npartitions, limits, columns, bytes_per_chunk, **kwargs)
     66     if index_col is None:
     67         raise ValueError("Must specify index column to partition on")
---> 68     engine = sa.create_engine(uri)
     69     meta = sa.MetaData()
     70     if isinstance(table, six.string_types):

~/Envs/dask-dev/lib/python3.6/site-packages/sqlalchemy/engine/__init__.py in create_engine(*args, **kwargs)
    389     strategy = kwargs.pop('strategy', default_strategy)
    390     strategy = strategies.strategies[strategy]
--> 391     return strategy.create(*args, **kwargs)
    392
    393

~/Envs/dask-dev/lib/python3.6/site-packages/sqlalchemy/engine/strategies.py in create(self, name_or_url, **kwargs)
     50         u = url.make_url(name_or_url)
     51
---> 52         plugins = u._instantiate_plugins(kwargs)
     53
     54         u.query.pop('plugin', None)

AttributeError: 'Engine' object has no attribute '_instantiate_plugins'

```
The second argument to `read_sql_table` should not be an engine, but a URI from which an engine can be made.
Ah, thanks @martindurant. I suppose it would be nice to allow `uri` to be an engine, in case the user has some special configuration on it. Though perhaps I'm missing a good reason to require a string.

@gsavaia could you submit a PR allowing `uri` to be an existing `Engine`, along with some tests and docs?
I would not be surprised if it was difficult to serialize an engine with
active database connections.

On Fri, Dec 8, 2017 at 11:49 AM, Tom Augspurger <notifications@github.com>
wrote:

> Ah, thanks @martindurant <https://github.com/martindurant>. I suppose it
> would be nice to allow uri to be an engine, in case the user has some
> special configuration on it. Though perhaps I'm missing a good reason to
> require a string.
>
> @gsavaia <https://github.com/gsavaia> could you submit a PR allowing uri
> to be an existing Engine, along with some tests and docs?
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/2975#issuecomment-350312256>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AASszEoJprUzFHS5WIBDju21q7ewIfjAks5s-WiwgaJpZM4Q6-Z1>
> .
>

That's an excellent reason :) We could perhaps extract `engine.url` then.

On Fri, Dec 8, 2017 at 10:51 AM, Matthew Rocklin <notifications@github.com>
wrote:

> I would not be surprised if it was difficult to serialize an engine with
> active database connections.
>
> On Fri, Dec 8, 2017 at 11:49 AM, Tom Augspurger <notifications@github.com>
> wrote:
>
> > Ah, thanks @martindurant <https://github.com/martindurant>. I suppose it
> > would be nice to allow uri to be an engine, in case the user has some
> > special configuration on it. Though perhaps I'm missing a good reason to
> > require a string.
> >
> > @gsavaia <https://github.com/gsavaia> could you submit a PR allowing uri
> > to be an existing Engine, along with some tests and docs?
> >
> > —
> > You are receiving this because you are subscribed to this thread.
> > Reply to this email directly, view it on GitHub
> > <https://github.com/dask/dask/issues/2975#issuecomment-350312256>, or
> mute
> > the thread
> > <https://github.com/notifications/unsubscribe-auth/
> AASszEoJprUzFHS5WIBDju21q7ewIfjAks5s-WiwgaJpZM4Q6-Z1>
>
> > .
> >
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/2975#issuecomment-350312736>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIjM3aiTHMq8GMsA8W_M4dB5SMwGDks5s-WkhgaJpZM4Q6-Z1>
> .
>

Running into the same issue and was hoping for some guidance.
Understanding that read_sql_table expects a URI, how could one supply a configuration such as:
connect_args={'sslmode': 'prefer'})

I am unfortunately unable to connect to Redshift without supplying this configuration.

Thanks!
@ChrisMcPherson your suggestion to use `connect_args` seems good. They could be passed through to https://github.com/dask/dask/blob/d80957501680522291de0653f3d9e2eda010295e/dask/dataframe/io/sql.py#L68.
@TomAugspurger Thanks for pointing me in the right place, Tom! I'll take a stab at adding the connect_args pass-through feature.
@TomAugspurger was this resolved?
I don't believe so.

IIUC, this would be fixed by adding a `conncect_args=None` parameter to `DataFrame.to_sql` and `read_sql` that's a dict of args passed through to `create_engine`'s `conncect_args`.

I notice that `create_enginge` takes a lot of keyword arguments, not just `connect_args`. We may want to have a more general `engine_kwargs` that is passed through to `create_engine` as `**kwargs`. Then it would be something like

```python
dd.read_sql_table(..., engine_kwargs={'connect_kwargs': {'sslmode': 'prefer'}, ...})
```
Passing through an `engine_kwargs` parameter from the `read_sql_table` method to `create_engine` seems easy to fix.  Marking this as a good first issue.
In the pandas API for `read/to_sql`, you pass an instance of SQLAlchemy engine. What do you think if you do:

```python
dd.read_sql_table(query, eng, ...)
```

instead of `uri` or a dict?
IIRC, sqlalchemy engines tend to not pickle well.

On Thu, Nov 8, 2018 at 5:00 AM Damien Garaud <notifications@github.com>
wrote:

> In the pandas API for read/to_sql, you pass an instance of SQLAlchemy
> engine. What do you think if you do:
>
> dd.read_sql_table(query, eng, ...)
>
> instead of uri or a dict?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/2975#issuecomment-436956109>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIgUwE0cLyMLYfBji32td8GNQDQJnks5utA7agaJpZM4Q6-Z1>
> .
>

OK. I didn't know. Thanks. I'll try to push something soon about this "good first" issue.

Resolving patch:
diff --git a/dask/dataframe/io/sql.py b/dask/dataframe/io/sql.py
--- a/dask/dataframe/io/sql.py
+++ b/dask/dataframe/io/sql.py
@@ -8,7 +8,7 @@
 
 def read_sql_table(table, uri, index_col, divisions=None, npartitions=None,
                    limits=None, columns=None, bytes_per_chunk=256 * 2**20,
-                   head_rows=5, schema=None, meta=None, **kwargs):
+                   head_rows=5, schema=None, meta=None, engine_kwargs=None, **kwargs):
     """
     Create dataframe from an SQL table.
 
@@ -53,17 +53,19 @@ def read_sql_table(table, uri, index_col, divisions=None, npartitions=None,
         ``sql.func.abs(sql.column('value')).label('abs(value)')``.
         Labeling columns created by functions or arithmetic operations is
         recommended.
-    bytes_per_chunk: int
+    bytes_per_chunk : int
         If both divisions and npartitions is None, this is the target size of
         each partition, in bytes
-    head_rows: int
+    head_rows : int
         How many rows to load for inferring the data-types, unless passing meta
-    meta: empty DataFrame or None
+    meta : empty DataFrame or None
         If provided, do not attempt to infer dtypes, but use these, coercing
         all chunks on load
-    schema: str or None
+    schema : str or None
         If using a table name, pass this to sqlalchemy to select which DB
         schema to use within the URI connection
+    engine_kwargs : dict or None
+        Specific db engine parameters for sqlalchemy
     kwargs : dict
         Additional parameters to pass to `pd.read_sql()`
 
@@ -81,7 +83,8 @@ def read_sql_table(table, uri, index_col, divisions=None, npartitions=None,
     from sqlalchemy.sql import elements
     if index_col is None:
         raise ValueError("Must specify index column to partition on")
-    engine = sa.create_engine(uri)
+    engine_kwargs = {} if engine_kwargs is None else engine_kwargs
+    engine = sa.create_engine(uri, **engine_kwargs)
     m = sa.MetaData()
     if isinstance(table, string_types):
         table = sa.Table(table, m, autoload=True, autoload_with=engine,


Test patch:
diff --git a/dask/dataframe/io/tests/test_sql.py b/dask/dataframe/io/tests/test_sql.py
--- a/dask/dataframe/io/tests/test_sql.py
+++ b/dask/dataframe/io/tests/test_sql.py
@@ -206,3 +206,18 @@ def test_select_from_select(db):
                     ).select_from(sql.table('test'))
     out = read_sql_table(s1, db, npartitions=2, index_col='number')
     assert_eq(out, df[['name']])
+
+
+def test_extra_connection_engine_keywords(capsys, db):
+    data = read_sql_table('test', db, npartitions=2, index_col='number', engine_kwargs={'echo': False}
+                          ).compute()
+    # no captured message from the stdout with the echo=False parameter (this is the default)
+    out, err = capsys.readouterr()
+    assert "SELECT" not in out
+    assert_eq(data, df)
+    # with the echo=True sqlalchemy parameter, you should get all SQL queries in the stdout
+    data = read_sql_table('test', db, npartitions=2, index_col='number', engine_kwargs={'echo': True}
+                          ).compute()
+    out, err = capsys.readouterr()
+    assert "SELECT" in out
+    assert_eq(data, df)