Repository: weecology/retriever

Issue:
Python interface errors and needs more tests.
We have currently added the `data_dir` parameter. This has caused some changes that introduced some errors. 

```python
Import retriever as rt
rt.install_json("iris", data_dir="test)
Installing iris_Iris.json: 100%|██████████████████████████████████████████| 151/151 [00:00<00:00, 18683.70rows/s]
[Errno 2] No such file or directory: 'iris_Iris.json'
```

To solve this we need to rewrite the disconnect functions for some of the engines like xml, json, 



Discussion:
This seems to be working for me on `master`:

```python
In [1]: import retriever as rt                                                                                                                                                                                     

In [2]: rt.install_json('iris')                                                                                                                                                                                    
Downloading bezdekIris.data: 3.00B [00:00, 14.3B/s]                                                                                                                                                                
Installing iris_Iris.json: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 151/151 [00:00<00:00, 14006.06rows/s]
Out[2]: <retriever.engines.jsonengine.engine at 0x7f038cf8c978>

In [3]: rt.install_json('iris', data_dir='mytest')                                                                                                                                                                 
Installing iris_Iris.json:  50%|██████████████████████████████████████████████████████████████████████▏                                                                     | 151/301 [00:00<00:00, 22524.36rows/s]
Out[3]: <retriever.engines.jsonengine.engine at 0x7f038cf8c978>
```
Thanks Ethan, I will do a clean test and let you know

Am still getting the error 

```python

Python 3.6.5 (default, Apr 25 2018, 14:26:36) 
[GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.39.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import retriever as rt 
>>> rt.install_json('iris', data_dir='mytest') 
Installing iris_Iris.json: 100%|██████████████████████████████████████████| 151/151 [00:00<00:00, 19701.98rows/s]
[Errno 2] No such file or directory: 'iris_Iris.json'
<retriever.engines.jsonengine.engine object at 0x115ca5e80>
>>> 
```

@henrykironde I'm not getting the error too, just like @ethanwhite
The package I'm using was **installed from source (master)**
Platform: Windows 10 (64 bit)

```python
D:\Projects\GSOC>python
Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import retriever as rt
>>> rt.install_json('iris')
Installing iris_Iris.json: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████| 151/151 [00:00<00:00, 22146.30rows/s]
<retriever.engines.jsonengine.engine object at 0x0000011F957305C0>
>>> rt.install_json('iris',data_dir="test")
Installing iris_Iris.json:  50%|██████████████████████████████████████████████████████▋                                                      | 151/301 [00:00<00:00, 25778.01rows/s]
<retriever.engines.jsonengine.engine object at 0x0000011F957305C0>
>>>
```

Although I don't know why its showing 50% when I added the "data_dir" argument even though the file was fully downloaded inside the directory.
Thanks for the test @Jaffrey98, That is probably from running `tqdm` with out having asleep. But it does not affect the functionality of the retriever.
We could open an issue, to find a better way to represent the bars well.

OK, I did see a version of this error a minute ago. I think it's related to #1264 as @henrykironde noted. Don't have a fully reproducible example yet, but wanted to say that I'd seen it happen.
OK, I've got this one on the run. The new `data_dir` didn't get added to the `table_names` list that we're using for `disconnect`. So, not related to #1264 and @henrykironde had the source of the error right (no surprise). The reason @Jaffrey98 and I didn't see the error is because we ran the install without `data_dir` first, which put the file in the working directory, which allow `disconnect` to work.

@henrykironde - I'm working on this now. I'm guessing I should expect the same issue for the other flat file engines?

Resolving patch:
diff --git a/retriever/engines/csvengine.py b/retriever/engines/csvengine.py
--- a/retriever/engines/csvengine.py
+++ b/retriever/engines/csvengine.py
@@ -40,11 +40,12 @@ def create_db(self):
     def create_table(self):
         """Create the table by creating an empty csv file"""
         self.auto_column_number = 1
-        self.file = open_fw(os.path.join(self.opts["data_dir"], self.table_name()))
+        table_path = os.path.join(self.opts["data_dir"], self.table_name())
+        self.file = open_fw(table_path)
         self.output_file = open_csvw(self.file)
         column_list = self.table.get_insert_columns(join=False, create=True)
         self.output_file.writerow([u'{}'.format(val) for val in column_list])
-        self.table_names.append((self.file, self.table_name()))
+        self.table_names.append((self.file, table_path))
 
         # Register all tables created to enable
         # testing python files having custom download function
diff --git a/retriever/engines/jsonengine.py b/retriever/engines/jsonengine.py
--- a/retriever/engines/jsonengine.py
+++ b/retriever/engines/jsonengine.py
@@ -43,9 +43,10 @@ def create_db(self):
 
     def create_table(self):
         """Create the table by creating an empty json file"""
-        self.output_file = open_fw(os.path.join(self.opts["data_dir"], self.table_name()))
+        table_path = os.path.join(self.opts["data_dir"], self.table_name())
+        self.output_file = open_fw(table_path)
         self.output_file.write("[")
-        self.table_names.append((self.output_file, self.table_name()))
+        self.table_names.append((self.output_file, table_path))
         self.auto_column_number = 1
 
         # Register all tables created to enable
diff --git a/retriever/engines/xmlengine.py b/retriever/engines/xmlengine.py
--- a/retriever/engines/xmlengine.py
+++ b/retriever/engines/xmlengine.py
@@ -39,10 +39,11 @@ def create_db(self):
 
     def create_table(self):
         """Create the table by creating an empty XML file."""
-        self.output_file = open_fw(os.path.join(self.opts["data_dir"], self.table_name()))
+        table_path = os.path.join(self.opts["data_dir"], self.table_name())
+        self.output_file = open_fw(table_path)
         self.output_file.write(u'<?xml version="1.0" encoding="UTF-8"?>')
         self.output_file.write(u'\n<root>')
-        self.table_names.append((self.output_file, self.table_name()))
+        self.table_names.append((self.output_file, table_path))
         self.auto_column_number = 1
 
         # Register all tables created to enable


Test patch: