Repository: ansible/ansible

Issue:
lookup csvfile doesn't work with cyrillic and other unicode symbols
##### ISSUE TYPE
- Bug Report
##### ANSIBLE VERSION

```
ansible 2.0.1.0
  config file =
  configured module search path = Default w/o overrides
```
##### CONFIGURATION

Default
##### OS / ENVIRONMENT

N/A
##### SUMMARY

Lookup csvfile doesn't work with cyrillic and other unicode symbols. Csv module in python2 doesn't work with any symbols except ascii if utf-8 encoding is used (unicode strings). As result using cyrillic in csv file leads to errors.

The simplest way to fix this issue is to open file without encoding. Now I have following little fix: https://github.com/puzan/ansible/commit/ef1869b59945324bf0c4c9c46be3a308ed3d614d. I'm going to add test for this case and open pull request.
##### STEPS TO REPRODUCE

```
ansible-playbook csv_cyrillic.yml
```

csv_cyrillic.yml:

``` yml

---
- name: Test Cyrillic
  hosts: localhost
  tasks:
    - debug: msg="{{ lookup('csvfile', '1 file=cyrillic.csv delimiter=;') }}"
    - debug: msg="{{ lookup('csvfile', '1 file=cyrillic.csv delimiter=; col=2') }}"
```

cyrillic.csv:

``` csv
# id;name;city
1;Илья;Горький
```
##### EXPECTED RESULTS

```
PLAY [Test Cyrillic] ***********************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "msg": "Илья"
}

TASK [debug] *******************************************************************
ok: [localhost] => {
    "msg": "Горький"
}

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0
```
##### ACTUAL RESULTS

```
PLAY [Test Cyrillic] ***********************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [debug] *******************************************************************
fatal: [localhost]: FAILED! => {"failed": true, "msg": "csvfile: 'ascii' codec can't encode characters in position 2-5: ordinal not in range(128)"}

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1
```



Discussion:
I am afraid that Ansible cannot know what character-set a file was encoded with, and at the moment there is no way to tell the csvfile lookup plugin what the used character set is. (It is impossible to detect the used character set, unless it is part of the content/formatting)

Personally I think the best solution is to use UTF-8 everywhere, and convert all files to UTF-8. But maybe a pull-request to add support for custom character sets is acceptable as well ?  E.g. `lookup('csvfile', '1 file=cyrillic.csv delimiter=; col=2 charset=cp1251')`


Resolving patch:
diff --git a/lib/ansible/plugins/lookup/csvfile.py b/lib/ansible/plugins/lookup/csvfile.py
--- a/lib/ansible/plugins/lookup/csvfile.py
+++ b/lib/ansible/plugins/lookup/csvfile.py
@@ -23,13 +23,43 @@
 from ansible.errors import AnsibleError
 from ansible.plugins.lookup import LookupBase
 
+class CSVRecoder:
+    """
+    Iterator that reads an encoded stream and reencodes the input to UTF-8
+    """
+    def __init__(self, f, encoding='utf-8'):
+        self.reader = codecs.getreader(encoding)(f)
+
+    def __iter__(self):
+        return self
+
+    def next(self):
+        return self.reader.next().encode("utf-8")
+
+class CSVReader:
+    """
+    A CSV reader which will iterate over lines in the CSV file "f",
+    which is encoded in the given encoding.
+    """
+
+    def __init__(self, f, dialect=csv.excel, encoding='utf-8', **kwds):
+        f = CSVRecoder(f, encoding)
+        self.reader = csv.reader(f, dialect=dialect, **kwds)
+
+    def next(self):
+        row = self.reader.next()
+        return [unicode(s, 'utf-8') for s in row]
+
+    def __iter__(self):
+        return self
+
 class LookupModule(LookupBase):
 
     def read_csv(self, filename, key, delimiter, encoding='utf-8', dflt=None, col=1):
 
         try:
-            f = codecs.open(filename, 'r', encoding=encoding)
-            creader = csv.reader(f, delimiter=str(delimiter))
+            f = open(filename, 'r')
+            creader = CSVReader(f, delimiter=str(delimiter), encoding=encoding)
 
             for row in creader:
                 if row[0] == key:


Test patch: