Repository: datahq/dataflows

Issue:
Local test runs create many files not present in codebase or gitignore
# Overview

After `py.test` we end up with a lot neither checked-in nor ignored files.

Discussion:
roll: @akariv 
Please take a look
coveralls: ## Pull Request Test Coverage Report for [Build 347](https://coveralls.io/builds/24390133)

* **0** of **0**   changed or added relevant lines in **0** files are covered.
* No unchanged relevant lines lost coverage.
* Overall coverage remained the same at **84.469%**

---



|  Totals | [![Coverage Status](https://coveralls.io/builds/24390133/badge)](https://coveralls.io/builds/24390133) |
| :-- | --: |
| Change from base [Build 346](https://coveralls.io/builds/24324339): |  0.0% |
| Covered Lines: | 1599 |
| Relevant Lines: | 1893 |

---
##### 💛  - [Coveralls](https://coveralls.io)

coveralls: ## Pull Request Test Coverage Report for [Build 348](https://coveralls.io/builds/24390132)

* **0** of **0**   changed or added relevant lines in **0** files are covered.
* No unchanged relevant lines lost coverage.
* Overall coverage remained the same at **84.469%**

---



|  Totals | [![Coverage Status](https://coveralls.io/builds/24390132/badge)](https://coveralls.io/builds/24390132) |
| :-- | --: |
| Change from base [Build 346](https://coveralls.io/builds/24324339): |  0.0% |
| Covered Lines: | 1599 |
| Relevant Lines: | 1893 |

---
##### 💛  - [Coveralls](https://coveralls.io)

coveralls: ## Pull Request Test Coverage Report for [Build 348](https://coveralls.io/builds/24390132)

* **0** of **0**   changed or added relevant lines in **0** files are covered.
* No unchanged relevant lines lost coverage.
* Overall coverage remained the same at **84.469%**

---



|  Totals | [![Coverage Status](https://coveralls.io/builds/24390132/badge)](https://coveralls.io/builds/24390132) |
| :-- | --: |
| Change from base [Build 346](https://coveralls.io/builds/24324339): |  0.0% |
| Covered Lines: | 1599 |
| Relevant Lines: | 1893 |

---
##### 💛  - [Coveralls](https://coveralls.io)

akariv: Would be good to move all outputs to `out/` instead of keeping them in root / data, no?
roll: @akariv 
I agree. I'll update. I wasn't sure why it uses the current way
roll: @akariv 
I've updated the tests. So now only one thing I had to add to `.gitignore` is `.checkpoints/`

Patch:
diff --git a/.gitignore b/.gitignore
index 1f4ba80..bafb34c 100644
--- a/.gitignore
+++ b/.gitignore
@@ -85,6 +85,5 @@ celerybeat-schedule
 
 # Resources created by our tests
 .coverage.*
-
 todo/
-out/
+.checkpoints/
diff --git a/PROCESSORS.md b/PROCESSORS.md
index 736a336..bfa8a9a 100644
--- a/PROCESSORS.md
+++ b/PROCESSORS.md
@@ -42,6 +42,7 @@ Loads data from various source types (local files, remote URLS, Google Spreadshe
 def load(source, name=None, resources=None, strip=True, limit_rows=None,
          infer_strategy=None, cast_strategy=None,
          override_schema=None, override_fields=None,
+         deduplicate_headers=False,
          on_error=raise_exception,
          **options)
     pass
@@ -76,6 +77,7 @@ Relevant only when _not_ loading data from a datapackage:
     - `load.CAST_WITH_SCHEMA` - Data will be parsed and casted using the schema and will error in case of faulty data
 - `override_schema` - Provided dictionary will be merged into the inferred schema. If `fields` key is set its contents will fully replace the inferred fields array. The same behavior will be applied for all other nested structures.
 - `override_fields` - Provided mapping will patch the inferred `schema.fields` array. In the mapping keys must be field names and values must be dictionaries intended to be merged into the corresponding field's metadata.
+- `deduplicate_headers` - (default `False`) If there are duplicate headers and the flag is set to `True` it will rename them using a `header (1), header (2), etc` approach. If there are duplicate headers and the flag is set to `False` it will raise an error.
 - `on_error` - Dictates how `load` will behave in case of a validation error.
     Options are identical to `on_error` in `set_type` and `validate`
 
diff --git a/data/duplicate_headers.csv b/data/duplicate_headers.csv
new file mode 100644
index 0000000..1f72e28
--- /dev/null
+++ b/data/duplicate_headers.csv
@@ -0,0 +1,2 @@
+header1,header2,header2
+value1,value2,value3
diff --git a/dataflows/processors/load.py b/dataflows/processors/load.py
index 6a88b55..a674882 100644
--- a/dataflows/processors/load.py
+++ b/dataflows/processors/load.py
@@ -106,8 +106,10 @@ class load(DataStreamProcessor):
     ERRORS_RAISE = raise_exception
 
     def __init__(self, load_source, name=None, resources=None, strip=True, limit_rows=None,
-                 infer_strategy=None, cast_strategy=None, on_error=raise_exception,
+                 infer_strategy=None, cast_strategy=None,
                  override_schema=None, override_fields=None,
+                 deduplicate_headers=False,
+                 on_error=raise_exception,
                  **options):
         super(load, self).__init__()
         self.load_source = load_source
@@ -119,6 +121,7 @@ class load(DataStreamProcessor):
         self.resources = resources
         self.override_schema = override_schema
         self.override_fields = override_fields
+        self.deduplicate_headers = deduplicate_headers
 
         self.load_dp = None
         self.resource_descriptors = []
@@ -194,6 +197,11 @@ class load(DataStreamProcessor):
                 self.options.setdefault('ignore_blank_headers', True)
                 self.options.setdefault('headers', 1)
                 stream: Stream = Stream(self.load_source, **self.options).open()
+                if len(stream.headers) != len(set(stream.headers)):
+                    if not self.deduplicate_headers:
+                        raise ValueError(
+                            'Found duplicate headers. Use the `deduplicate_headers` flag')
+                    stream.headers = self.rename_duplicate_headers(stream.headers)
                 schema = Schema().infer(
                     stream.sample, headers=stream.headers,
                     confidence=1, guesser_cls=self.guesser)
@@ -241,3 +249,16 @@ class load(DataStreamProcessor):
             if self.limit_rows:
                 it = self.limiter(it)
             yield it
+
+    def rename_duplicate_headers(self, duplicate_headers):
+        counter = {}
+        headers = []
+        for header in duplicate_headers:
+            counter.setdefault(header, 0)
+            counter[header] += 1
+            if counter[header] > 1:
+                if counter[header] == 2:
+                    headers[headers.index(header)] = '%s (%s)' % (header, 1)
+                header = '%s (%s)' % (header, counter[header])
+            headers.append(header)
+        return headers
diff --git a/out/.gitignore b/out/.gitignore
new file mode 100644
index 0000000..d6b7ef3
--- /dev/null
+++ b/out/.gitignore
@@ -0,0 +1,2 @@
+*
+!.gitignore
diff --git a/setup.py b/setup.py
index e6b2477..e920cf6 100644
--- a/setup.py
+++ b/setup.py
@@ -21,6 +21,7 @@ def read(*paths):
 PACKAGE = 'dataflows'
 NAME = PACKAGE.replace('_', '-')
 INSTALL_REQUIRES = [
+    'tabulator>=1.23.0',
     'datapackage>=1.5.0',
     'tableschema>=1.5',
     'kvfile>=0.0.6',
diff --git a/tox.ini b/tox.ini
index 739870d..22f82fe 100644
--- a/tox.ini
+++ b/tox.ini
@@ -22,6 +22,12 @@ commands=
     --cov-config tox.ini \
     --cov-report term-missing \
     {posargs}
+
 [pytest]
 # pytest.ini configuration here
 testpaths = tests
+
+[coverage:run]
+# .coveragerc configuration here
+omit =
+    */.tox/*


Test patch:
diff --git a/tests/test_cli.py b/tests/test_cli.py
index f6d20ae..ad10b07 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -1,5 +1,5 @@
 import subprocess
 
 def test_init_remote():
-    subprocess.check_output('dataflows init https://raw.githubusercontent.com/datahq/dataflows/master/data/academy.csv',
-                            shell=True)
\ No newline at end of file
+    subprocess.check_output('cd ./out && dataflows init https://raw.githubusercontent.com/datahq/dataflows/master/data/academy.csv',
+                            shell=True)
diff --git a/tests/test_lib.py b/tests/test_lib.py
index 036fa6d..d082723 100644
--- a/tests/test_lib.py
+++ b/tests/test_lib.py
@@ -1,3 +1,4 @@
+import pytest
 from dataflows import Flow
 
 data = [
@@ -19,12 +20,12 @@ def test_dump_to_sql():
                     'resource-name': 'res_1'
                 }
             ),
-            engine='sqlite:///test.db')
+            engine='sqlite:///out/test.db')
     )
     f.process()
 
     # Check validity
-    engine = create_engine('sqlite:///test.db')
+    engine = create_engine('sqlite:///out/test.db')
     result = list(dict(x) for x in engine.execute('select * from output_table'))
     assert result == data
 
@@ -438,11 +439,11 @@ def test_load_from_package():
 
     Flow(
         [{'foo': 'bar', 'moo': 12}],
-        dump_to_path('data/load_from_package')
+        dump_to_path('out/load_from_package')
     ).process()
 
     ds = Flow(
-        load('data/load_from_package/datapackage.json')
+        load('out/load_from_package/datapackage.json')
     ).datastream()
 
     assert len(ds.dp.resources) == 1
@@ -455,10 +456,10 @@ def test_load_from_env_var():
 
     Flow(
         [{'foo': 'bar'}],
-        dump_to_path('data/load_from_env_var')
+        dump_to_path('out/load_from_env_var')
     ).process()
 
-    os.environ['MY_DATAPACKAGE'] = 'data/load_from_env_var/datapackage.json'
+    os.environ['MY_DATAPACKAGE'] = 'out/load_from_env_var/datapackage.json'
     results, dp, _ = Flow(
         load('env://MY_DATAPACKAGE')
     ).results()
@@ -473,11 +474,11 @@ def test_load_from_package_resource_matching():
     Flow(
         [{'foo': 'bar'}],
         [{'foo': 'baz'}],
-        dump_to_path('data/load_from_package_resource_matching(')
+        dump_to_path('out/load_from_package_resource_matching(')
     ).process()
 
     ds = Flow(
-        load('data/load_from_package_resource_matching(/datapackage.json', resources=['res_2'])
+        load('out/load_from_package_resource_matching(/datapackage.json', resources=['res_2'])
     ).datastream()
 
     assert len(ds.dp.resources) == 1
@@ -705,10 +706,10 @@ def test_dump_to_path_use_titles():
         [{'hello': 'world', 'hola': 'mundo'}, {'hello': 'עולם', 'hola': 'عالم'}],
         *(set_type(name, resources=['res_1'], title=title) for name, title
           in (('hello', 'שלום'), ('hola', 'aloha'))),
-        dump_to_path('data/dump_with_titles', use_titles=True)
+        dump_to_path('out/dump_with_titles', use_titles=True)
     ).process()
 
-    with tabulator.Stream('data/dump_with_titles/res_1.csv') as stream:
+    with tabulator.Stream('out/dump_with_titles/res_1.csv') as stream:
         assert stream.read() == [['שלום',   'aloha'],
                                  ['world',  'mundo'],
                                  ['עולם',   'عالم']]
@@ -727,7 +728,7 @@ def test_load_dates():
             [{'today': str(_today), 'now': str(_now)}],
             set_type('today', type='date'),
             set_type('now', type='datetime', format=datetime_format),
-            dump_to_path('data/dump_dates')
+            dump_to_path('out/dump_dates')
         ).process()
 
     try:
@@ -748,7 +749,7 @@ def test_load_dates():
     out_now = datetime.datetime(_now.year, _now.month, _now.day, _now.hour, _now.minute, _now.second)
 
     assert Flow(
-        load('data/dump_dates/datapackage.json'),
+        load('out/dump_dates/datapackage.json'),
     ).results()[0] == [[{'today': _today, 'now': out_now}]]
 
 
@@ -900,11 +901,11 @@ def test_save_load_dates():
         [{'id': 1, 'ts': datetime.datetime.now()},
          {'id': 2, 'ts': datetime.datetime.now()}],
         set_type('ts', type='datetime', format='%Y-%m-%d/%H:%M:%S'),
-        dump_to_path('data/test_save_load_dates')
+        dump_to_path('out/test_save_load_dates')
     ).process()
 
     res, _, _ = Flow(
-        load('data/test_save_load_dates/datapackage.json'),
+        load('out/test_save_load_dates/datapackage.json'),
         printer()
     ).results()
 
@@ -1236,7 +1237,7 @@ def test_load_override_schema_and_fields():
         {'name': 'george', 'age': '17'},
         {'name': None, 'age': '22'},
     ]]
-      
+
 def test_delete_fields_regex():
     from dataflows import load, delete_fields
     flow = Flow(
@@ -1271,3 +1272,29 @@ def test_join_full_outer():
         {'id': 3, 'city': 'rome', 'population': None},
         {'id': 4, 'city': None, 'population': 3},
     ]]
+
+
+def test_load_duplicate_headers():
+    from dataflows import load
+    flow = Flow(
+        load('data/duplicate_headers.csv'),
+    )
+    with pytest.raises(ValueError) as excinfo:
+        flow.results()
+    assert 'duplicate headers' in str(excinfo.value)
+
+
+def test_load_duplicate_headers_with_deduplicate_headers_flag():
+    from dataflows import load
+    flow = Flow(
+        load('data/duplicate_headers.csv', deduplicate_headers=True),
+    )
+    data, package, stats = flow.results()
+    assert package.descriptor['resources'][0]['schema']['fields'] == [
+        {'name': 'header1', 'type': 'string', 'format': 'default'},
+        {'name': 'header2 (1)', 'type': 'string', 'format': 'default'},
+        {'name': 'header2 (2)', 'type': 'string', 'format': 'default'},
+    ]
+    assert data == [[
+        {'header1': 'value1', 'header2 (1)': 'value2', 'header2 (2)': 'value3'},
+    ]]