Repository: weecology/retriever

Issue:
`fetch` errors when run after doing a flat file install in the same session
Hello,

Following https://github.com/weecology/retriever/issues/1262#issuecomment-469062884

```python
In [1]: import retriever as rt

In [2]: rt.install_csv("wine-composition")
Installing ./wine_composition_WineComposition.csv: 100%|████████████████████████████████████████████████████████████| 178/178 [00:00<00:00, 6408.68rows/s]
Out[2]: <retriever.engines.csvengine.engine at 0x114eeac18>

In [3]: wine_data = rt.fetch("wine-composition")
Installing wine_composition_WineComposition:  50%|█████████████████████████████████                                 | 178/356 [00:00<00:00, 4464.16rows/s]
---------------------------------------------------------------------------
OperationalError                          Traceback (most recent call last)
~/anaconda3/lib/python3.6/site-packages/pandas/io/sql.py in execute(self, *args, **kwargs)
   1377             else:
-> 1378                 cur.execute(*args)
   1379             return cur

OperationalError: near ".": syntax error

During handling of the above exception, another exception occurred:

DatabaseError                             Traceback (most recent call last)
<ipython-input-3-5b818d622ba9> in <module>
----> 1 wine_data = rt.fetch("wine-composition")

~/anaconda3/lib/python3.6/site-packages/retriever/lib/fetch.py in fetch(dataset, file, table_name)
     12     sqlite = engine()
     13     sqlite.opts = {"file": file, "table_name": table_name}
---> 14     df = sqlite.fetch_tables(dataset, db_table_names)
     15     return df

~/anaconda3/lib/python3.6/site-packages/retriever/engines/sqlite.py in fetch_tables(self, dataset, table_names)
     46                                                             "FROM {};".format(table),
     47                                                             connection)
---> 48                 for table in table_names}
     49         return data
     50

~/anaconda3/lib/python3.6/site-packages/retriever/engines/sqlite.py in <dictcomp>(.0)
     46                                                             "FROM {};".format(table),
     47                                                             connection)
---> 48                 for table in table_names}
     49         return data
     50

~/anaconda3/lib/python3.6/site-packages/pandas/io/sql.py in read_sql_query(sql, con, index_col, coerce_float, params, parse_dates, chunksize)
    312     return pandas_sql.read_query(
    313         sql, index_col=index_col, params=params, coerce_float=coerce_float,
--> 314         parse_dates=parse_dates, chunksize=chunksize)
    315
    316

~/anaconda3/lib/python3.6/site-packages/pandas/io/sql.py in read_query(self, sql, index_col, coerce_float, params, parse_dates, chunksize)
   1411
   1412         args = _convert_params(sql, params)
-> 1413         cursor = self.execute(*args)
   1414         columns = [col_desc[0] for col_desc in cursor.description]
   1415

~/anaconda3/lib/python3.6/site-packages/pandas/io/sql.py in execute(self, *args, **kwargs)
   1388             ex = DatabaseError(
   1389                 "Execution failed on sql '%s': %s" % (args[0], exc))
-> 1390             raise_with_traceback(ex)
   1391
   1392     @staticmethod

~/anaconda3/lib/python3.6/site-packages/pandas/compat/__init__.py in raise_with_traceback(exc, traceback)
    402         if traceback == Ellipsis:
    403             _, _, traceback = sys.exc_info()
--> 404         raise exc.with_traceback(traceback)
    405 else:
    406     # this version of raise is a syntax error in Python 3

~/anaconda3/lib/python3.6/site-packages/pandas/io/sql.py in execute(self, *args, **kwargs)
   1376                 cur.execute(*args, **kwargs)
   1377             else:
-> 1378                 cur.execute(*args)
   1379             return cur
   1380         except Exception as exc:

DatabaseError: Execution failed on sql 'SELECT * FROM ./wine_composition_WineComposition.csv;': near ".": syntax error
```

Maybe a message should warm user that `fetch` can't be use after installing dataset as a CSV file.

Kind regards


Discussion:
Thanks @scls19fr, was about to create an issue for a similar problem.
I think it's quite different... this is a bug... not a feature request (or doc improvement request)
@scls19fr I'm happy to see you again Sir. @ethanwhite @henrykironde Can I work on it?
Yes you can work on it. Also people can corroborate and get this fixed
Thanks for reporting this @scls19fr.

The issue is that `lib.engine.script_table_registry` is accumulating all tables created in a session. It is then being used in `fetch` to get a list of tables to load from the SQLite database. If tables with names that don't match those in the SQLite database get added during the session (by installing into one of the flat file engines) then this causes the error.

I'm not immediately clear on whether we should change how `lib.engine` works or change the usage in `fetch`. @henrykironde it looks like the `lib.engine.script_table_registry` code is yours so I think you might have the best idea regarding which approach to take. My initial thought is that I'm not clear why we'd want to accumulate a list of all the tables created in a session irrespective of engine. So maybe `script_table_registry` should either be keyed by `(dataset, engine)` tuples or be a nested dictionary?

Resolving patch:
diff --git a/retriever/lib/install.py b/retriever/lib/install.py
--- a/retriever/lib/install.py
+++ b/retriever/lib/install.py
@@ -2,6 +2,7 @@
 from __future__ import print_function
 
 import os
+from collections import OrderedDict
 
 from retriever.engines import choose_engine
 from retriever.lib.defaults import DATA_DIR, SCRIPT_WRITE_PATH
@@ -23,6 +24,7 @@ def _install(args, use_cache, debug):
     if data_sets_scripts:
         for data_sets_script in data_sets_scripts:
             try:
+                engine.script_table_registry = OrderedDict()
                 data_sets_script.download(engine, debug=debug)
                 data_sets_script.engine.final_cleanup()
             except Exception as e:


Test patch:
diff --git a/test/test_regression.py b/test/test_regression.py
--- a/test/test_regression.py
+++ b/test/test_regression.py
@@ -246,3 +246,12 @@ def test_fetch():
                 first_row_data = list(data_frame[table_i].iloc[0])
                 assert expected_data == first_row_data
                 assert expected_column_values == column_values
+
+
+def test_interface_table_registry():
+    # Test if script_table_registry keeps only the latest
+    # table names of the installed data packages in
+    # script_table_registry
+    install_csv("iris")
+    wine_data = fetch("wine-composition")
+    assert "iris" not in wine_data.keys()