Repository: weecology/retriever

Issue:
Cross-tab functionality errors with delimiters inside enclosed fields
The retriever normally handles enclosed fields that include delimiters properly, but when these fields coincide with cross-tab data that needs to be reorganized something goes wrong.

This can be seen in https://github.com/weecology/retriever/pull/356 where if you use the first 14756 rows of the raw data file everything works fine, but including the next line:
`Reptilia,Crocodilia,Alligatoridae,Alligator,mississippiensis,-999,"Alligator, Gator, American alligator, Florida alligator, Mississippi alligator, Louisiana alligator.",4605,39.7,1,62000,85,-999,-999,50,-999,76.6,65,-999,56,4380,-999,47800,-999,47800,-999,76,-999,-999,-999,-999,-999,-999,-999,200,3652.5`

causes the associated script to error:

```
ethan@croryx:~/RetrieverCode (amniotelh)$ retriever install --debug csv AmnioteDB
=> Installing Amniote life History database
list index out of range
Traceback (most recent call last):
  File "/home/ethan/anaconda/bin/retriever", line 9, in <module>
    load_entry_point('retriever==1.7.0', 'console_scripts', 'retriever')()
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/__main__.py", line 118, in main
    dataset.download(engine, debug=debug)
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/lib/templates.py", line 93, in download
    self.engine.insert_data_from_url(value)
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/lib/engine.py", line 571, in insert_data_from_url
    self.insert_data_from_file(find)
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/lib/engine.py", line 563, in insert_data_from_file
    self.add_to_table(data_source)
  File "/home/ethan/anaconda/lib/python2.7/site-packages/retriever/lib/engine.py", line 69, in add_to_table
    name = [self.table.ct_names[n]]
IndexError: list index out of range
```



Discussion:

Resolving patch:
diff --git a/lib/engine.py b/lib/engine.py
--- a/lib/engine.py
+++ b/lib/engine.py
@@ -59,7 +59,7 @@ def add_to_table(self, data_source):
             lines = gen_from_source(data_source)
             real_lines = []
             for line in lines:
-                split_line = line.strip('\n\r\t ').split(self.table.delimiter)
+                split_line = self.table.split_on_delimiter(line)
                 initial_cols = len(self.table.columns) - (3 if hasattr(self.table, "ct_names") else 2)
                 begin = split_line[:initial_cols]
                 rest = split_line[initial_cols:]
@@ -70,7 +70,7 @@ def add_to_table(self, data_source):
                         n += 1
                     else:
                         name = []
-                    real_lines.append(self.table.delimiter.join(begin + name + [item]))
+                    real_lines.append(self.table.combine_on_delimiter(begin + name + [item]))
             real_line_length = len(real_lines)
         else:
             # this function returns a generator that iterates over the lines in
diff --git a/lib/table.py b/lib/table.py
--- a/lib/table.py
+++ b/lib/table.py
@@ -1,5 +1,6 @@
 from retriever.lib.cleanup import *
 import csv
+import StringIO
 
 
 class Table:
@@ -75,6 +76,15 @@ def split_on_delimiter(self, line):
         r = csv.reader([line], dialect=dialect, delimiter=self.delimiter)
         return r.next()
 
+    def combine_on_delimiter(self, line_as_list):
+        """Combine a list of values into a line of csv data"""
+        dialect = csv.excel
+        dialect.escapechar = "\\"
+        writer_file =  StringIO.StringIO()
+        writer = csv.writer(writer_file, dialect=dialect, delimiter=self.delimiter)
+        writer.writerow(line_as_list)
+        return writer_file.getvalue()
+
     def values_from_line(self, line):
         linevalues = []
         if (self.pk and self.contains_pk == False):


Test patch: