Repository: pandas-dev/pandas

Issue:
SQL: Status of the legacy mode?
To discuss: do we regard the sql legacy mode (using a DBAPI2 connection, for sqlite and mysql) as a _'first-class citizen'_ in pandas? Or is it mainly for backwards compatibility? 
Is it useful to have something that is not dependent on sqlalchemy? And consequently we alo try to enhance it /accept enhancements to it? Or is it really just 'legacy' and we don't develop it further (only bug fixes)?

I think it is important to discuss this (and 'formally' decide on this), as there are some issues with this:
- If it is mainly for backwards compatibility, are we also planning to deprecate it? (which is not necessarily the consequence, if we just regard it as what the name says: legacy)
- If it is not only for backwards compatibility, why do we raise a warning?:
  
  ```
  UserWarning: Not an SQLAlchemy engine,
  attempting to use as legacy DBAPI connection
  ```
  
  As this implies to me that it is not really preferred to use this.
- Do we want to keep the functionality equal to the sqlalchemy version? (as we are using the same functions for both, it would make it complex if it is not)
  - Eg, we could easily provide a `read_table` functionality to legacy mode, by just doing a `read_sql("SELECT * FROM specified_table", ..)`.
- We should clarify the docs on this status: http://pandas-docs.github.io/pandas-docs-travis/io.html#legacy (as it is not really clear now)
- If we decide we want to fully support this, we should maybe try to find another name than 'legacy' .. :-)

Maybe the bottom line we should ask ourselves is: do we want that we can really _recommend_ it to users to use the legacy sqlite without sqlalchemy version? (and in that case I think it should be fully supported and not regarded as 'legacy')



Discussion:
We already started the discussion here (https://github.com/pydata/pandas/issues/6881#issuecomment-40406736), to quote @danielballan:

> In the long term, I'd like to keep full-fledged support for connections generated by sqlite3. For light applications, I still favor those over sqlalchemy for code portability (e.g., to collaborators who might not have sqlaclemy). Since all other flavors already require dependencies that are not built in to Python, I think it is reasonable to expect users to also obtain sqlalchemy if they want the very best support. One man's opinion.
> 
> So, I'd like to build MultiIndex support for legacy connections.

I think it might make sense to have full-fledged support for `sqlite3` DBAPI only as it doesn't have any deps, while all other flavors have the python driver deps at the very least.

So support sqlite3 thru direct DBAPI and sqlalchemy, whilst other flavors ONLY with sqlalchemy. If at some point their is sufficient interest / PR's for supporting other direct DBAPI then could be added later.

@hayd @danielballan @mangecoeur @jtratner 

And if we decide for a full-fledged support of `sqlite3`, this does also mean we deprecate the MySQL support via DBAPI connection (without sqlalchemy)? 
Also, only supporting sqlite3 without sqlalchemy makes sense in the sense that it is the simplest flavor to support fully. 

I kind of thing we should try and bring back the DBAPI support for this release (and potentially depreciate it, at least for mysql/postgres etc), it's going to **shock** people that their sql code is broken without warning/dep (which is what's happening atm??).

I've tried a few times to get the old tests working, but haven't been able to fix the mysql issues, so would be great if someone else could have a try!

@hayd There is nothing to 'bring back', as DBAPI is still supported via legacy mode. Normally all code that did run previously, should also run now (as only sqlite and mysql flavors were supported, postgresql was never supported). That is certainly something we have to guarantee. But of course if the tests say otherwise, we should look at that ..

Previously, only `'sqlite'` and `'mysql'` flavors were really supported, and those are now both supported via the legacy mode. Of course, some things previously also worked for other flavors, eg the `read_sql` to read a query is rather flavor independent and did also work for eg postgresql.
So the question is also if we continue to 'allow' this, or explicitly raise an error and point to sqlalchemy mode if a DPAPI connection is provided that is not sqlite (or mysql at this moment).

Yes, let's make sure the legacy tests are in order but deprecate legacy connections in future releases.

This way, we will not waste effort trying to fix bugs in the legacy code -- the bugs that motivated us to adopt sqlalchemy to begin with -- but neither will we pull out the rug from people who are using the legacy connections now and finding them adequate.

@danielballan With 'deprecate legacy connections in future releases', do you also mean sqlite? Or only the others, because in the other thread you were in favor of a fully supported sqlite fallback?

I will take a look at the old tests, and try to get them running with the new code.

Yes, that was ambiguous. I am still in favor of continued fully sqlite3 support. I mean that we should deprecate the others.

(Incidentally, supporting sqlite3 is a little less work because it only supports five datatypes.)

yeah, that is what I meant with it being the 'simplest flavor to support fully.'. Actually just int, real and text (and null and blob). No bool, no datetime in different forms, ..

so `sqlalchemy` prob writes datetimes as string and does the conversions? (for `sqlite3`)

@jreback Yes, that's right. As I explored that, I discovered that sqlalchemy cannot write timedeltas. It raises a StatementError. I'll open a separate issue.

And actually (I didn't realize before), sqlalchemy converts it to string, but keeps it as a 'datetime' column in the database (in sqlite, you can use declared data type as `datetime` as an alias, it's just not a datetime but a string), but in this way, also when reading that frame back in, sqlalchemy automatically converts it back to a datetime

@jorisvandenbossche bring back the old test file (to ensure we're not breaking too much stuff), sorry.

@hayd Is the latest copy of `test_legacy_sql.py` sitting the history of the master branch? I know I have an old copy in one of my SQL branches. Have you done recent work on it?

@danielballan I haven't done any recent work on it, although there was a couple of additional tests (e.g. not deleteing the database when in append mode) which could be added. I had tweaked it a little (e.g remove removed arguments), but mysql was choking/taking forever on the tests.

@hayd @danielballan shouldn't we just look at the `test_sql.py` from the 0.13.1 tagged version? Or were there changes done to the tests after that?

@jorisvandenbossche that's right, no changes were added so we should just use that version's tests (potentially could add on for table deleting when trying to append ?).

I looked at the old tests, and most are running now (submitted PR #6987 to fix some issues).

What do the others think of deprecating the `'mysql'` flavor for the legacy code with DBAPI connections? And only keep full support for sqlite without sqlalchemy? 
Because if we would do that, we could better do it now I think.

@jorisvandenbossche Yes.

We want to get out of the business of supporting flavors' various idiosyncrasies. We've established that making an exception for sqlite3 makes sense. But if we support 'mysql' and not postgresql, oracle, etc. it will only be for historical reasons, and I worry that the code won't be well maintained.

Speaking as someone who will have a lot of his code broken by this change, I still think it's the right choice. If the deprecation warning incites a riot we can always reevaluate....

So I will go forward:
- deprecate 'mysql' flavor (and point to use sqlalchemy)
- remove the UserWarning of not being a sqlalchemy engine

Somebody a proposal for another name than 'legacy'? Something like sqlite3 'fallback'?

@danielballan Normally your code should not be broken (yet), as the changes should be almost fully backwards compatible. And to adapt, I think it changing the connection an sqlalchemy engine will be enough in most cases as the interface stayed the same (but of course, this can also be a lot of work ..)

OK, then going forward with:
- deprecate `mysql` flavor
- remove the UserWarning of not using an sqlalchemy engine

Somebody a good idea how to name the 'legacy' mode, eg sqlite 'fallback'?

Normally, your code should not be broken (yet), as the changes are almost fully backwards compatible. And to adapt, the main thing to change is replacing the mysql connections with sqlalchemy engines as the interface should be the same for the rest (but of course, not saying that this + some of the corner cases could not be a lot of work ...)


Resolving patch:
diff --git a/pandas/io/sql.py b/pandas/io/sql.py
--- a/pandas/io/sql.py
+++ b/pandas/io/sql.py
@@ -76,7 +76,7 @@ def _parse_date_columns(data_frame, parse_dates):
     return data_frame
 
 
-def execute(sql, con, cur=None, params=None, flavor='sqlite'):
+def execute(sql, con, cur=None, params=None):
     """
     Execute the given SQL query using the provided connection object.
 
@@ -84,24 +84,22 @@ def execute(sql, con, cur=None, params=None, flavor='sqlite'):
     ----------
     sql : string
         Query to be executed
-    con : SQLAlchemy engine or DBAPI2 connection (legacy mode)
+    con : SQLAlchemy engine or sqlite3 DBAPI2 connection
         Using SQLAlchemy makes it possible to use any DB supported by that
         library.
-        If a DBAPI2 object, a supported SQL flavor must also be provided
+        If a DBAPI2 object, only sqlite3 is supported.
     cur : depreciated, cursor is obtained from connection
     params : list or tuple, optional
         List of parameters to pass to execute method.
-    flavor : string "sqlite", "mysql"
-        Specifies the flavor of SQL to use.
-        Ignored when using SQLAlchemy engine. Required when using DBAPI2 connection.
+
     Returns
     -------
     Results Iterable
     """
     if cur is None:
-        pandas_sql = pandasSQL_builder(con, flavor=flavor)
+        pandas_sql = pandasSQL_builder(con)
     else:
-        pandas_sql = pandasSQL_builder(cur, flavor=flavor, is_cursor=True)
+        pandas_sql = pandasSQL_builder(cur, is_cursor=True)
     args = _convert_params(sql, params)
     return pandas_sql.execute(*args)
 
@@ -235,7 +233,7 @@ def read_sql_table(table_name, con, meta=None, index_col=None,
     table_name : string
         Name of SQL table in database
     con : SQLAlchemy engine
-        Legacy mode not supported
+        Sqlite DBAPI conncection mode not supported
     meta : SQLAlchemy meta, optional
         If omitted MetaData is reflected from engine
     index_col : string, optional
@@ -277,8 +275,8 @@ def read_sql_table(table_name, con, meta=None, index_col=None,
         raise ValueError("Table %s not found" % table_name, con)
 
 
-def read_sql_query(sql, con, index_col=None, flavor='sqlite',
-                   coerce_float=True, params=None, parse_dates=None):
+def read_sql_query(sql, con, index_col=None, coerce_float=True, params=None,
+                   parse_dates=None):
     """Read SQL query into a DataFrame.
 
     Returns a DataFrame corresponding to the result set of the query
@@ -289,15 +287,12 @@ def read_sql_query(sql, con, index_col=None, flavor='sqlite',
     ----------
     sql : string
         SQL query to be executed
-    con : SQLAlchemy engine or DBAPI2 connection (legacy mode)
+    con : SQLAlchemy engine or sqlite3 DBAPI2 connection
         Using SQLAlchemy makes it possible to use any DB supported by that
         library.
-        If a DBAPI2 object is given, a supported SQL flavor must also be provided
+        If a DBAPI2 object, only sqlite3 is supported.
     index_col : string, optional
         column name to use for the returned DataFrame object.
-    flavor : string, {'sqlite', 'mysql'}
-        The flavor of SQL to use. Ignored when using
-        SQLAlchemy engine. Required when using DBAPI2 connection.
     coerce_float : boolean, default True
         Attempt to convert values to non-string, non-numeric objects (like
         decimal.Decimal) to floating point, useful for SQL result sets
@@ -324,7 +319,7 @@ def read_sql_query(sql, con, index_col=None, flavor='sqlite',
     read_sql
 
     """
-    pandas_sql = pandasSQL_builder(con, flavor=flavor)
+    pandas_sql = pandasSQL_builder(con)
     return pandas_sql.read_sql(
         sql, index_col=index_col, params=params, coerce_float=coerce_float,
         parse_dates=parse_dates)
@@ -342,12 +337,13 @@ def read_sql(sql, con, index_col=None, flavor='sqlite', coerce_float=True,
     con : SQLAlchemy engine or DBAPI2 connection (legacy mode)
         Using SQLAlchemy makes it possible to use any DB supported by that
         library.
-        If a DBAPI2 object is given, a supported SQL flavor must also be provided
+        If a DBAPI2 object, only sqlite3 is supported.
     index_col : string, optional
         column name to use for the returned DataFrame object.
     flavor : string, {'sqlite', 'mysql'}
         The flavor of SQL to use. Ignored when using
         SQLAlchemy engine. Required when using DBAPI2 connection.
+        'mysql' is still supported, but will be removed in future versions.
     coerce_float : boolean, default True
         Attempt to convert values to non-string, non-numeric objects (like
         decimal.Decimal) to floating point, useful for SQL result sets
@@ -417,13 +413,14 @@ def to_sql(frame, name, con, flavor='sqlite', if_exists='fail', index=True,
     frame : DataFrame
     name : string
         Name of SQL table
-    con : SQLAlchemy engine or DBAPI2 connection (legacy mode)
+    con : SQLAlchemy engine or sqlite3 DBAPI2 connection
         Using SQLAlchemy makes it possible to use any DB supported by that
         library.
-        If a DBAPI2 object is given, a supported SQL flavor must also be provided
+        If a DBAPI2 object, only sqlite3 is supported.
     flavor : {'sqlite', 'mysql'}, default 'sqlite'
         The flavor of SQL to use. Ignored when using SQLAlchemy engine.
         Required when using DBAPI2 connection.
+        'mysql' is still supported, but will be removed in future versions.
     if_exists : {'fail', 'replace', 'append'}, default 'fail'
         - fail: If table exists, do nothing.
         - replace: If table exists, drop it, recreate it, and insert data.
@@ -458,13 +455,14 @@ def has_table(table_name, con, flavor='sqlite'):
     ----------
     table_name: string
         Name of SQL table
-    con: SQLAlchemy engine or DBAPI2 connection (legacy mode)
+    con: SQLAlchemy engine or sqlite3 DBAPI2 connection
         Using SQLAlchemy makes it possible to use any DB supported by that
         library.
-        If a DBAPI2 object is given, a supported SQL flavor name must also be provided
+        If a DBAPI2 object, only sqlite3 is supported.
     flavor: {'sqlite', 'mysql'}, default 'sqlite'
         The flavor of SQL to use. Ignored when using SQLAlchemy engine.
         Required when using DBAPI2 connection.
+        'mysql' is still supported, but will be removed in future versions.
 
     Returns
     -------
@@ -476,6 +474,10 @@ def has_table(table_name, con, flavor='sqlite'):
 table_exists = has_table
 
 
+_MYSQL_WARNING = ("The 'mysql' flavor with DBAPI connection is deprecated "
+                  "and will be removed in future versions. "
+                  "MySQL will be further supported with SQLAlchemy engines.")
+
 def pandasSQL_builder(con, flavor=None, meta=None, is_cursor=False):
     """
     Convenience function to return the correct PandasSQL subclass based on the
@@ -489,21 +491,14 @@ def pandasSQL_builder(con, flavor=None, meta=None, is_cursor=False):
         if isinstance(con, sqlalchemy.engine.Engine):
             return PandasSQLAlchemy(con, meta=meta)
         else:
-            warnings.warn("Not an SQLAlchemy engine, "
-                          "attempting to use as legacy DBAPI connection")
-            if flavor is None:
-                raise ValueError(
-                    "PandasSQL must be created with an SQLAlchemy engine "
-                    "or a DBAPI2 connection and SQL flavor")
-            else:
-                return PandasSQLLegacy(con, flavor, is_cursor=is_cursor)
+            if flavor == 'mysql':
+                warnings.warn(_MYSQL_WARNING, FutureWarning)
+            return PandasSQLLegacy(con, flavor, is_cursor=is_cursor)
 
     except ImportError:
-        warnings.warn("SQLAlchemy not installed, using legacy mode")
-        if flavor is None:
-            raise SQLAlchemyRequired
-        else:
-            return PandasSQLLegacy(con, flavor, is_cursor=is_cursor)
+        if flavor == 'mysql':
+            warnings.warn(_MYSQL_WARNING, FutureWarning)
+        return PandasSQLLegacy(con, flavor, is_cursor=is_cursor)
 
 
 class PandasSQLTable(PandasObject):
@@ -893,7 +888,7 @@ def _create_sql_schema(self, frame, table_name):
 }
 
 
-_SAFE_NAMES_WARNING = ("The spaces in these column names will not be changed."
+_SAFE_NAMES_WARNING = ("The spaces in these column names will not be changed. "
                        "In pandas versions < 0.14, spaces were converted to "
                        "underscores.")
 
@@ -991,6 +986,8 @@ class PandasSQLLegacy(PandasSQL):
     def __init__(self, con, flavor, is_cursor=False):
         self.is_cursor = is_cursor
         self.con = con
+        if flavor is None:
+            flavor = 'sqlite'
         if flavor not in ['sqlite', 'mysql']:
             raise NotImplementedError
         else:
@@ -1098,6 +1095,8 @@ def get_schema(frame, name, flavor='sqlite', keys=None, con=None):
     """
 
     if con is None:
+        if flavor == 'mysql':
+            warnings.warn(_MYSQL_WARNING, FutureWarning)
         return _get_schema_legacy(frame, name, flavor, keys)
 
     pandas_sql = pandasSQL_builder(con=con, flavor=flavor)


Test patch:
diff --git a/pandas/io/tests/test_sql.py b/pandas/io/tests/test_sql.py
--- a/pandas/io/tests/test_sql.py
+++ b/pandas/io/tests/test_sql.py
@@ -332,7 +332,7 @@ def setUp(self):
 
     def test_read_sql_iris(self):
         iris_frame = sql.read_sql_query(
-            "SELECT * FROM iris", self.conn, flavor='sqlite')
+            "SELECT * FROM iris", self.conn)
         self._check_iris_loaded_frame(iris_frame)
 
     def test_legacy_read_frame(self):
@@ -391,8 +391,7 @@ def test_to_sql_append(self):
     def test_to_sql_series(self):
         s = Series(np.arange(5, dtype='int64'), name='series')
         sql.to_sql(s, "test_series", self.conn, flavor='sqlite', index=False)
-        s2 = sql.read_sql_query("SELECT * FROM test_series", self.conn,
-                                flavor='sqlite')
+        s2 = sql.read_sql_query("SELECT * FROM test_series", self.conn)
         tm.assert_frame_equal(s.to_frame(), s2)
 
     def test_to_sql_panel(self):
@@ -416,8 +415,7 @@ def test_roundtrip(self):
                    con=self.conn, flavor='sqlite')
         result = sql.read_sql_query(
             'SELECT * FROM test_frame_roundtrip',
-            con=self.conn,
-            flavor='sqlite')
+            con=self.conn)
 
         # HACK!
         result.index = self.test_frame1.index
@@ -428,41 +426,38 @@ def test_roundtrip(self):
 
     def test_execute_sql(self):
         # drop_sql = "DROP TABLE IF EXISTS test"  # should already be done
-        iris_results = sql.execute(
-            "SELECT * FROM iris", con=self.conn, flavor='sqlite')
+        iris_results = sql.execute("SELECT * FROM iris", con=self.conn)
         row = iris_results.fetchone()
         tm.equalContents(row, [5.1, 3.5, 1.4, 0.2, 'Iris-setosa'])
 
     def test_date_parsing(self):
         # Test date parsing in read_sq
         # No Parsing
-        df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
-                                flavor='sqlite')
+        df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn)
         self.assertFalse(
             issubclass(df.DateCol.dtype.type, np.datetime64),
             "DateCol loaded with incorrect type")
 
         df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
-                                flavor='sqlite', parse_dates=['DateCol'])
+                                parse_dates=['DateCol'])
         self.assertTrue(
             issubclass(df.DateCol.dtype.type, np.datetime64),
             "DateCol loaded with incorrect type")
 
         df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
-                                flavor='sqlite',
                                 parse_dates={'DateCol': '%Y-%m-%d %H:%M:%S'})
         self.assertTrue(
             issubclass(df.DateCol.dtype.type, np.datetime64),
             "DateCol loaded with incorrect type")
 
         df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
-                                flavor='sqlite', parse_dates=['IntDateCol'])
+                                parse_dates=['IntDateCol'])
 
         self.assertTrue(issubclass(df.IntDateCol.dtype.type, np.datetime64),
                         "IntDateCol loaded with incorrect type")
 
         df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
-                                flavor='sqlite', parse_dates={'IntDateCol': 's'})
+                                parse_dates={'IntDateCol': 's'})
 
         self.assertTrue(issubclass(df.IntDateCol.dtype.type, np.datetime64),
                         "IntDateCol loaded with incorrect type")
@@ -471,7 +466,7 @@ def test_date_and_index(self):
         # Test case where same column appears in parse_date and index_col
 
         df = sql.read_sql_query("SELECT * FROM types_test_data", self.conn,
-                                flavor='sqlite', index_col='DateCol',
+                                index_col='DateCol',
                                 parse_dates=['DateCol', 'IntDateCol'])
 
         self.assertTrue(issubclass(df.index.dtype.type, np.datetime64),
@@ -651,22 +646,19 @@ def test_sql_open_close(self):
 
             conn = self.connect(name)
             result = sql.read_sql_query("SELECT * FROM test_frame2_legacy;",
-                                        conn, flavor="sqlite")
+                                        conn)
             conn.close()
 
         tm.assert_frame_equal(self.test_frame2, result)
 
     def test_read_sql_delegate(self):
-        iris_frame1 = sql.read_sql_query(
-            "SELECT * FROM iris", self.conn, flavor=self.flavor)
-        iris_frame2 = sql.read_sql(
-            "SELECT * FROM iris", self.conn, flavor=self.flavor)
+        iris_frame1 = sql.read_sql_query("SELECT * FROM iris", self.conn)
+        iris_frame2 = sql.read_sql("SELECT * FROM iris", self.conn)
         tm.assert_frame_equal(iris_frame1, iris_frame2,
                               "read_sql and read_sql_query have not the same"
                               " result with a query")
 
-        self.assertRaises(ValueError, sql.read_sql, 'iris', self.conn,
-                          flavor=self.flavor)
+        self.assertRaises(ValueError, sql.read_sql, 'iris', self.conn)
 
     def test_safe_names_warning(self):
         # GH 6798
@@ -1109,6 +1101,14 @@ def tearDown(self):
         self.conn.commit()
         self.conn.close()
 
+    def test_a_deprecation(self):
+        with tm.assert_produces_warning(FutureWarning):
+            sql.to_sql(self.test_frame1, 'test_frame1', self.conn,
+                       flavor='mysql')
+        self.assertTrue(
+            sql.has_table('test_frame1', self.conn, flavor='mysql'),
+            'Table not written to DB')
+
 
 #------------------------------------------------------------------------------
 #--- Old tests from 0.13.1 (before refactor using sqlalchemy)
@@ -1277,8 +1277,6 @@ def _check_roundtrip(self, frame):
         expected = frame.copy()
         expected.index = Index(lrange(len(frame2))) + 10
         expected.index.name = 'Idx'
-        print(expected.index.names)
-        print(result.index.names)
         tm.assert_frame_equal(expected, result)
 
     def test_tquery(self):