Repository: pandas-dev/pandas

Issue:
SQL: read_sql functions should not inspect all tables of database
Related: comment of @balancap: https://github.com/pydata/pandas/issues/6416#issuecomment-45343694 and issue #7380 where you get a warning about a not understood type (probably in another table).

**Situation now**: in the creation of a `PandasSQLAlchemy` object (https://github.com/pydata/pandas/blob/v0.14.0/pandas/io/sql.py#L777), a full `MetaData` object is created and reflected (this means: all tables in the database are inspected and the schema's are stored as sqlalchemy `Table` objects). This is done each time `read_sql/read_sql_query/read_sql_table` is called.

**Consequence**: 
- this can be costly when having a very large database or having a distant server (https://github.com/pydata/pandas/issues/6416#issuecomment-45343694).
- this can trigger warnings that does not have to do anything with your current query, or the current table you want to read, when eg one of the types in other tables is not known to sqlalchemy (eg a postgis geometry column).

**Possible solution**: 
- I think the `read_sql` functions never should inspect all tables, but only the specified table (and `read_sql_query` even not that table, as with a query this information is not used, only for `read_sql_table`)
- This can maybe be achieved with using the `only` keyword in `meta.reflect(engine, only=...)`
- For the OO API interface, we can discuss what should be the default (inspect all tables or not)

@mangecoeur @danielballan @hayd



Discussion:
Sounds good. As for the OO API, if we take a cue from `HDFStore` it should _not_ inspect all tables. `HDFStore.keys()` can run slow for > 10 keys, so it is not run on instantiation.

In the case of HDFStore, the keys can be inspected once and cached, because only one user at a time can open the file for writing. Pandas doesn't cache them, but it's possible. (I'll mention that @nkeim has implemented some code for this; maybe others have too.) But for SQL tables, with multi-user access as a full feature and a common use case, any relevant inspection should be done at query-execution time, not at connection time. Maybe the `only` keyword can make this fast.

OK, for the query functions it is clear I think they should not inspect the full table. I have a PR coming to fix that. 

For the OO API, we should discuss that further together with the rest of the interface how this should look like.

Sounds good, I think the OO api can be stabilised after the functional one is sorted.


Resolving patch:
diff --git a/pandas/io/sql.py b/pandas/io/sql.py
--- a/pandas/io/sql.py
+++ b/pandas/io/sql.py
@@ -76,6 +76,17 @@ def _parse_date_columns(data_frame, parse_dates):
     return data_frame
 
 
+def _is_sqlalchemy_engine(con):
+    try:
+        import sqlalchemy
+        if isinstance(con, sqlalchemy.engine.Engine):
+            return True
+        else:
+            return False
+    except ImportError:
+        return False
+
+
 def execute(sql, con, cur=None, params=None):
     """
     Execute the given SQL query using the provided connection object.
@@ -262,7 +273,15 @@ def read_sql_table(table_name, con, index_col=None, coerce_float=True,
 
 
     """
-    pandas_sql = PandasSQLAlchemy(con)
+    import sqlalchemy
+    from sqlalchemy.schema import MetaData
+    meta = MetaData(con)
+    try:
+        meta.reflect(only=[table_name])
+    except sqlalchemy.exc.InvalidRequestError:
+        raise ValueError("Table %s not found" % table_name)
+
+    pandas_sql = PandasSQLAlchemy(con, meta=meta)
     table = pandas_sql.read_table(
         table_name, index_col=index_col, coerce_float=coerce_float,
         parse_dates=parse_dates, columns=columns)
@@ -380,6 +399,7 @@ def read_sql(sql, con, index_col=None, coerce_float=True, params=None,
             coerce_float=coerce_float, parse_dates=parse_dates)
 
     if pandas_sql.has_table(sql):
+        pandas_sql.meta.reflect(only=[sql])
         return pandas_sql.read_table(
             sql, index_col=index_col, coerce_float=coerce_float,
             parse_dates=parse_dates, columns=columns)
@@ -471,17 +491,9 @@ def pandasSQL_builder(con, flavor=None, meta=None, is_cursor=False):
     """
     # When support for DBAPI connections is removed,
     # is_cursor should not be necessary.
-    try:
-        import sqlalchemy
-
-        if isinstance(con, sqlalchemy.engine.Engine):
-            return PandasSQLAlchemy(con, meta=meta)
-        else:
-            if flavor == 'mysql':
-                warnings.warn(_MYSQL_WARNING, FutureWarning)
-            return PandasSQLLegacy(con, flavor, is_cursor=is_cursor)
-
-    except ImportError:
+    if _is_sqlalchemy_engine(con):
+        return PandasSQLAlchemy(con, meta=meta)
+    else:
         if flavor == 'mysql':
             warnings.warn(_MYSQL_WARNING, FutureWarning)
         return PandasSQLLegacy(con, flavor, is_cursor=is_cursor)
@@ -767,7 +779,6 @@ def __init__(self, engine, meta=None):
         if not meta:
             from sqlalchemy.schema import MetaData
             meta = MetaData(self.engine)
-            meta.reflect(self.engine)
 
         self.meta = meta
 
@@ -812,19 +823,16 @@ def tables(self):
         return self.meta.tables
 
     def has_table(self, name):
-        if self.meta.tables.get(name) is not None:
-            return True
-        else:
-            return False
+        return self.engine.has_table(name)
 
     def get_table(self, table_name):
         return self.meta.tables.get(table_name)
 
     def drop_table(self, table_name):
         if self.engine.has_table(table_name):
+            self.meta.reflect(only=[table_name])
             self.get_table(table_name).drop()
             self.meta.clear()
-            self.meta.reflect()
 
     def _create_sql_schema(self, frame, table_name):
         table = PandasSQLTable(table_name, self, frame=frame)


Test patch:
diff --git a/pandas/io/tests/test_sql.py b/pandas/io/tests/test_sql.py
--- a/pandas/io/tests/test_sql.py
+++ b/pandas/io/tests/test_sql.py
@@ -623,6 +623,22 @@ def test_read_sql_delegate(self):
         iris_frame2 = sql.read_sql('iris', self.conn)
         tm.assert_frame_equal(iris_frame1, iris_frame2)
 
+    def test_not_reflect_all_tables(self):
+        # create invalid table
+        qry = """CREATE TABLE invalid (x INTEGER, y UNKNOWN);"""
+        self.conn.execute(qry)
+        qry = """CREATE TABLE other_table (x INTEGER, y INTEGER);"""
+        self.conn.execute(qry)
+        
+        with warnings.catch_warnings(record=True) as w:
+            # Cause all warnings to always be triggered.
+            warnings.simplefilter("always")
+            # Trigger a warning.
+            sql.read_sql_table('other_table', self.conn)
+            sql.read_sql_query('SELECT * FROM other_table', self.conn)
+            # Verify some things
+            self.assertEqual(len(w), 0, "Warning triggered for other table")
+
 
 class TestSQLLegacyApi(_TestSQLApi):
     """
@@ -736,6 +752,8 @@ def setup_connect(self):
         try:
             self.conn = self.connect()
             self.pandasSQL = sql.PandasSQLAlchemy(self.conn)
+            # to test if connection can be made:
+            self.conn.connect()
         except sqlalchemy.exc.OperationalError:
             raise nose.SkipTest("Can't connect to {0} server".format(self.flavor))