Repository: pandas-dev/pandas

Issue:
ENH: sql support with SQLAlchemy - follow-up
#5950 is now merged! (Thanks to @mangecoeur)

But there were still some issue to flesh out/fix:
- [x] release notes / 0.14.0 whatsnew (Here is the section in whatsnew for when you are ready): http://pandas-docs.github.io/pandas-docs-travis/whatsnew.html#whatsnew-0140-sql -> #7120
- [ ] figure out user API (#6300) and update api.rst/docs accordingly
  - ~~functional api (read_table/read_sql)~~ (fixed in #6867)
  - ~~other public functions? (execute, has_table)`~~ (see #6300 and #6987)
  - OO API (PandasSQLAlchemy?)
- [x] get travis working (not skipping the tests)
  MySQL was taken out of the 2.7 build, for some reason was failing the build process: https://github.com/pydata/pandas/commit/043d165290180c6d9e3a480c18e220548af56c53, and failure here (see install.sh): https://travis-ci.org/jreback/pandas/jobs/18477137 (note Joris: I think this is OK now with pymysql)
- [x] get MySQL working (failing tests, see https://github.com/pydata/pandas/pull/6316#issuecomment-34960181) -> fixed in #6613
- [x] handle `OperationsError` (e.g. have PyMySQL installed but server is not running) -> #6651 
- lower bar on sql alchemy version -> moved to #6340, #7022
- [x] add postgresql tests (#6316) (and other flavors?)
- [x] move DataFrame.to_sql to generic and update it with new functionality -> fixed in #6598 
- timezone issues (see also https://github.com/pydata/pandas/pull/5950#issuecomment-33630742) -> moved to #7139
- [x] check if relevant parts of old legacy tests pass with master -> #6987
- [x] check coverage of the test suite
- [ ] add vbench (esp. vs old code)
- [x] `columns` argument in `read_table` not working
- [x] use `itertuples` rather than `iterrows` #6509
- [x] error when having integer column names (fixed in #6902)
- [x] read and write a `MultiIndex` (so `index_col` should take a list as well as a scalar) (fixed in #6735)
- [x] Allow mapping as parameters for SQL DBAPI2 #6708 (fixed in #6709)
- [x] Conversion of invalid column names is broken (you get column with None's) #6796, fixed in #6902 
- [x] Data written with to_sql legacy mode (sqlite/mysql) is not persistent (not written, only in memory) #6846, fixed in #6875

There is also a todo list in https://github.com/pydata/pandas/issues/4163, but I thought it is cleaner to start a new one (but maybe we should copy some over). 
Complement the list if you think of others!



Discussion:
@jorisvandenbossche: Is there a minimum sqlalchemy version this is targeting?

Edit: I see v0.8.1 (https://github.com/pydata/pandas/pull/5950/files#diff-f2202167afc438a107cf9f73ac4f9fb1R98).

It may be helpful to know if it's a requirement / recommendation. If it's a larger project and the sqlalchemy version is locked and Pandas can be upgraded, they may still want to know if it's possible for them to utilize this at an older version.

@tony this should work for pretty much all versions of SQLAlchemy; it uses mainly the connection features

Hello,

With the new sql interface, will the 'classic' DB2API still works with params of a sql being a dictionnary ?

typical example =   "execute('SELECT \* FROM ATABLE WHERE AFIELD=%(x)s', {'x':x})"

I added 'read and write a `MultiIndex`' to the list.

I (finally) have a need to use SQL so I'm going to be using sqlalchemy and the new pandas sql stuff pretty extensively in the next few weeks. I'll report what I find. Doing a `df.to_sql()` with a MultiIndex was the first thing I tried.

@TomAugspurger Super! Certainly report (and fix? :-) everything, bugs, annoyances, suggestions. There are still a lot of issue to report and fix.

@stonebig I think so. Would you be able to test it with?

@jorisvandenbossche 

if you can move residual issues that are not likely to be closed in 0.14 to 0.14.1 (or later)

release notes / v0.14.0.txt needed as well

not pushing, but lmk when you think you are 'ready' enough on this

@jreback I will try to finish some things of this thursday of friday. Is that OK?

sure

thxs


Resolving patch:
diff --git a/pandas/core/generic.py b/pandas/core/generic.py
--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -928,10 +928,11 @@ def to_sql(self, name, con, flavor='sqlite', if_exists='fail', index=True,
         con : SQLAlchemy engine or DBAPI2 connection (legacy mode)
             Using SQLAlchemy makes it possible to use any DB supported by that
             library.
-            If a DBAPI2 object is given, a supported SQL flavor must also be provided
+            If a DBAPI2 object, only sqlite3 is supported.
         flavor : {'sqlite', 'mysql'}, default 'sqlite'
             The flavor of SQL to use. Ignored when using SQLAlchemy engine.
-            Required when using DBAPI2 connection.
+            'mysql' is deprecated and will be removed in future versions, but it
+            will be further supported through SQLAlchemy engines.
         if_exists : {'fail', 'replace', 'append'}, default 'fail'
             - fail: If table exists, do nothing.
             - replace: If table exists, drop it, recreate it, and insert data.
diff --git a/pandas/io/sql.py b/pandas/io/sql.py
--- a/pandas/io/sql.py
+++ b/pandas/io/sql.py
@@ -221,8 +221,8 @@ def uquery(sql, con=None, cur=None, retry=True, params=None):
 #------------------------------------------------------------------------------
 #--- Read and write to DataFrames
 
-def read_sql_table(table_name, con, meta=None, index_col=None,
-                   coerce_float=True, parse_dates=None, columns=None):
+def read_sql_table(table_name, con, index_col=None, coerce_float=True,
+                   parse_dates=None, columns=None):
     """Read SQL database table into a DataFrame.
 
     Given a table name and an SQLAlchemy engine, returns a DataFrame.
@@ -234,8 +234,6 @@ def read_sql_table(table_name, con, meta=None, index_col=None,
         Name of SQL table in database
     con : SQLAlchemy engine
         Sqlite DBAPI conncection mode not supported
-    meta : SQLAlchemy meta, optional
-        If omitted MetaData is reflected from engine
     index_col : string, optional
         Column to set as index
     coerce_float : boolean, default True
@@ -264,7 +262,7 @@ def read_sql_table(table_name, con, meta=None, index_col=None,
 
 
     """
-    pandas_sql = PandasSQLAlchemy(con, meta=meta)
+    pandas_sql = PandasSQLAlchemy(con)
     table = pandas_sql.read_table(
         table_name, index_col=index_col, coerce_float=coerce_float,
         parse_dates=parse_dates, columns=columns)
@@ -292,11 +290,10 @@ def read_sql_query(sql, con, index_col=None, coerce_float=True, params=None,
         library.
         If a DBAPI2 object, only sqlite3 is supported.
     index_col : string, optional
-        column name to use for the returned DataFrame object.
+        Column name to use as index for the returned DataFrame object.
     coerce_float : boolean, default True
         Attempt to convert values to non-string, non-numeric objects (like
         decimal.Decimal) to floating point, useful for SQL result sets
-    cur : depreciated, cursor is obtained from connection
     params : list, tuple or dict, optional
         List of parameters to pass to execute method.
     parse_dates : list or dict
@@ -325,8 +322,8 @@ def read_sql_query(sql, con, index_col=None, coerce_float=True, params=None,
         parse_dates=parse_dates)
 
 
-def read_sql(sql, con, index_col=None, flavor='sqlite', coerce_float=True,
-             params=None, parse_dates=None, columns=None):
+def read_sql(sql, con, index_col=None, coerce_float=True, params=None,
+             parse_dates=None, columns=None):
     """
     Read SQL query or database table into a DataFrame.
 
@@ -339,15 +336,10 @@ def read_sql(sql, con, index_col=None, flavor='sqlite', coerce_float=True,
         library.
         If a DBAPI2 object, only sqlite3 is supported.
     index_col : string, optional
-        column name to use for the returned DataFrame object.
-    flavor : string, {'sqlite', 'mysql'}
-        The flavor of SQL to use. Ignored when using
-        SQLAlchemy engine. Required when using DBAPI2 connection.
-        'mysql' is still supported, but will be removed in future versions.
+        column name to use as index for the returned DataFrame object.
     coerce_float : boolean, default True
         Attempt to convert values to non-string, non-numeric objects (like
         decimal.Decimal) to floating point, useful for SQL result sets
-    cur : depreciated, cursor is obtained from connection
     params : list, tuple or dict, optional
         List of parameters to pass to execute method.
     parse_dates : list or dict
@@ -360,7 +352,8 @@ def read_sql(sql, con, index_col=None, flavor='sqlite', coerce_float=True,
           Especially useful with databases without native Datetime support,
           such as SQLite
     columns : list
-        List of column names to select from sql table
+        List of column names to select from sql table (only used when reading
+        a table).
 
     Returns
     -------
@@ -379,7 +372,7 @@ def read_sql(sql, con, index_col=None, flavor='sqlite', coerce_float=True,
     read_sql_query : Read SQL query into a DataFrame
 
     """
-    pandas_sql = pandasSQL_builder(con, flavor=flavor)
+    pandas_sql = pandasSQL_builder(con)
 
     if 'select' in sql.lower():
         try:
@@ -419,8 +412,8 @@ def to_sql(frame, name, con, flavor='sqlite', if_exists='fail', index=True,
         If a DBAPI2 object, only sqlite3 is supported.
     flavor : {'sqlite', 'mysql'}, default 'sqlite'
         The flavor of SQL to use. Ignored when using SQLAlchemy engine.
-        Required when using DBAPI2 connection.
-        'mysql' is still supported, but will be removed in future versions.
+        'mysql' is deprecated and will be removed in future versions, but it
+        will be further supported through SQLAlchemy engines.
     if_exists : {'fail', 'replace', 'append'}, default 'fail'
         - fail: If table exists, do nothing.
         - replace: If table exists, drop it, recreate it, and insert data.
@@ -461,8 +454,8 @@ def has_table(table_name, con, flavor='sqlite'):
         If a DBAPI2 object, only sqlite3 is supported.
     flavor: {'sqlite', 'mysql'}, default 'sqlite'
         The flavor of SQL to use. Ignored when using SQLAlchemy engine.
-        Required when using DBAPI2 connection.
-        'mysql' is still supported, but will be removed in future versions.
+        'mysql' is deprecated and will be removed in future versions, but it
+        will be further supported through SQLAlchemy engines.
 
     Returns
     -------
@@ -1090,15 +1083,23 @@ def _create_sql_schema(self, frame, table_name):
 
 def get_schema(frame, name, flavor='sqlite', keys=None, con=None):
     """
-    Get the SQL db table schema for the given frame
+    Get the SQL db table schema for the given frame.
 
     Parameters
     ----------
     frame : DataFrame
-    name : name of SQL table
+    name : string
+        name of SQL table
     flavor : {'sqlite', 'mysql'}, default 'sqlite'
-    keys : columns to use a primary key
+        The flavor of SQL to use. Ignored when using SQLAlchemy engine.
+        'mysql' is deprecated and will be removed in future versions, but it
+        will be further supported through SQLAlchemy engines.
+    keys : string or sequence
+        columns to use a primary key
     con: an open SQL database connection object or an SQLAlchemy engine
+        Using SQLAlchemy makes it possible to use any DB supported by that
+        library.
+        If a DBAPI2 object, only sqlite3 is supported.
 
     """
 


Test patch:
diff --git a/pandas/io/tests/test_sql.py b/pandas/io/tests/test_sql.py
--- a/pandas/io/tests/test_sql.py
+++ b/pandas/io/tests/test_sql.py
@@ -340,7 +340,7 @@ def test_read_sql_iris(self):
     def test_legacy_read_frame(self):
         with tm.assert_produces_warning(FutureWarning):
             iris_frame = sql.read_frame(
-                "SELECT * FROM iris", self.conn, flavor='sqlite')
+                "SELECT * FROM iris", self.conn)
         self._check_iris_loaded_frame(iris_frame)
 
     def test_to_sql(self):