Repository: ansible/ansible

Issue:
lookup csvfile doesn't work with cyrillic and other unicode symbols
##### ISSUE TYPE
- Bug Report
##### ANSIBLE VERSION

```
ansible 2.0.1.0
  config file =
  configured module search path = Default w/o overrides
```
##### CONFIGURATION

Default
##### OS / ENVIRONMENT

N/A
##### SUMMARY

Lookup csvfile doesn't work with cyrillic and other unicode symbols. Csv module in python2 doesn't work with any symbols except ascii if utf-8 encoding is used (unicode strings). As result using cyrillic in csv file leads to errors.

The simplest way to fix this issue is to open file without encoding. Now I have following little fix: https://github.com/puzan/ansible/commit/ef1869b59945324bf0c4c9c46be3a308ed3d614d. I'm going to add test for this case and open pull request.
##### STEPS TO REPRODUCE

```
ansible-playbook csv_cyrillic.yml
```

csv_cyrillic.yml:

``` yml

---
- name: Test Cyrillic
  hosts: localhost
  tasks:
    - debug: msg="{{ lookup('csvfile', '1 file=cyrillic.csv delimiter=;') }}"
    - debug: msg="{{ lookup('csvfile', '1 file=cyrillic.csv delimiter=; col=2') }}"
```

cyrillic.csv:

``` csv
# id;name;city
1;Илья;Горький
```
##### EXPECTED RESULTS

```
PLAY [Test Cyrillic] ***********************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [debug] *******************************************************************
ok: [localhost] => {
    "msg": "Илья"
}

TASK [debug] *******************************************************************
ok: [localhost] => {
    "msg": "Горький"
}

PLAY RECAP *********************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=0
```
##### ACTUAL RESULTS

```
PLAY [Test Cyrillic] ***********************************************************

TASK [setup] *******************************************************************
ok: [localhost]

TASK [debug] *******************************************************************
fatal: [localhost]: FAILED! => {"failed": true, "msg": "csvfile: 'ascii' codec can't encode characters in position 2-5: ordinal not in range(128)"}

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1
```



Discussion:
I am afraid that Ansible cannot know what character-set a file was encoded with, and at the moment there is no way to tell the csvfile lookup plugin what the used character set is. (It is impossible to detect the used character set, unless it is part of the content/formatting)

Personally I think the best solution is to use UTF-8 everywhere, and convert all files to UTF-8. But maybe a pull-request to add support for custom character sets is acceptable as well ?  E.g. `lookup('csvfile', '1 file=cyrillic.csv delimiter=; col=2 charset=cp1251')`


Resolving patch:
diff --git a/lib/ansible/plugins/lookup/csvfile.py b/lib/ansible/plugins/lookup/csvfile.py
--- a/lib/ansible/plugins/lookup/csvfile.py
+++ b/lib/ansible/plugins/lookup/csvfile.py
@@ -25,10 +25,10 @@
 
 class LookupModule(LookupBase):
 
-    def read_csv(self, filename, key, delimiter, dflt=None, col=1):
+    def read_csv(self, filename, key, delimiter, encoding='utf-8', dflt=None, col=1):
 
         try:
-            f = codecs.open(filename, 'r', encoding='utf-8')
+            f = codecs.open(filename, 'r', encoding=encoding)
             creader = csv.reader(f, delimiter=str(delimiter))
 
             for row in creader:
@@ -50,10 +50,11 @@ def run(self, terms, variables=None, **kwargs):
             key = params[0]
 
             paramvals = {
-                'file' : 'ansible.csv',
+                'col' : "1",          # column to return
                 'default' : None,
                 'delimiter' : "TAB",
-                'col' : "1",          # column to return
+                'file' : 'ansible.csv',
+                'encoding' : 'utf-8',
             }
 
             # parameters specified?
@@ -69,7 +70,7 @@ def run(self, terms, variables=None, **kwargs):
                 paramvals['delimiter'] = "\t"
 
             lookupfile = self._loader.path_dwim_relative(basedir, 'files', paramvals['file'])
-            var = self.read_csv(lookupfile, key, str(paramvals['delimiter']), paramvals['default'], paramvals['col'])
+            var = self.read_csv(lookupfile, key, str(paramvals['delimiter']), paramvals['encoding'], paramvals['default'], paramvals['col'])
             if var is not None:
                 if type(var) is list:
                     for v in var:


Test patch: