Repository: ansible-collections/community.general

Issue:
read_csv - Key 'Name' was not found in the CSV header fields
##### SUMMARY
The `read_csv` module fails to identify a field, yet displaces the field in the list of available fields.

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
read_csv

##### ANSIBLE VERSION

```
ansible 2.9.10
  config file = /home/anton/git/ansible-deploy-vmware-vm/ansible.cfg
  configured module search path = ['/home/anton/git/ansible-deploy-vmware-vm/library']
  ansible python module location = /home/anton/.local/lib/python3.6/site-packages/ansible
  executable location = /home/anton/.local/bin/ansible
  python version = 3.6.9 (default, Apr 18 2020, 01:56:04) [GCC 8.4.0]

```

##### CONFIGURATION

```
# config file for ansible -- http://ansible.com/
# ==============================================

# nearly all parameters can be overridden in ansible-playbook
# or with command line flags. ansible will read ANSIBLE_CONFIG,
# ansible.cfg in the current working directory, .ansible.cfg in
# the home directory or /etc/ansible/ansible.cfg, whichever it
# finds first

[defaults]
host_key_checking = False
host_key_check = False
ansible_python_interpreter=/usr/bin/python3
log_path = ./ansible.log
#bin_ansible_callbacks=True
#stdout_callback = debug


# some basic default values...
library        = ./library

# additional paths to search for roles in, colon separated
roles_path    = ./roles

[ssh_connection]
# ssh arguments to use
ssh_args = -o StrictHostKeyChecking=no
timeout=60


```

##### OS / ENVIRONMENT
Ubuntu 20:04


##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->

<!--- Paste example playbooks or commands between quotes below -->
```
---
- name: Right-size VMs
  gather_facts: false
  hosts: all
  connection: local
  tasks:
    # Read a CSV file and access the first item
  - name: Read users from CSV file and return a list
    read_csv:
        path: "files/vms/6-19-20 Optimization Report - Oversized Virtual Machines Prod2.csv"
        key: Name
    register: users

  - debug:
      msg: 'User {{ users.list.2.Name}}'
        # msg: 'User {{ users.list.2.Name}} has UID {{ users.list.2.ReclaimablevCPUs}} and GID {{ users.list.2.ReclaimableMemory}}'
        # msg: "{{ users }}"



```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
Expect to be able to read CSV values by col name (field) as based on module documentation.


##### ACTUAL RESULTS
```
fatal: [localhost]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": false, "msg": "Key 'Name' was not found in the CSV header fields: ﻿Name, Configured-vCPU, ReclaimablevCPUs, ConfiguredMemory, ReclaimableMemory, ParentvCenter"}
```





Discussion:
Files identified in the description:
* [`plugins/modules/files/read_csv.py`](https://github.com/ansible-collections/community.general/blob/master/plugins/modules/files/read_csv.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @dagwieers
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I also get the same error
fatal: [localhost]: FAILED! => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": false, "msg": "Key 'Name' was not found in the CSV header fields: ﻿Name, Configured-vCPU, ReclaimablevCPUs, ConfiguredMemory, ReclaimableMemory, ParentvCenter"}

I found '\ufeffName' in the output
I needed to convert the file to `ascii`encoding  in order to use it with `read_csv`
Thanks magic93, yea that works 
I did dos2unix file.csv first
The `\ufeff` is a [BOM](https://github.com/ansible-collections/community.general/issues/544#issuecomment-647823285). Excel has the bad habbit of adding them to CSV files...
Just removing the first two bytes of that file should also work.
> Just removing the first two bytes of that file should also work.

How do you recommend doing that?
You can use dd to do that (https://unix.stackexchange.com/questions/6852/best-way-to-remove-bytes-from-the-start-of-a-file). 

Since this happens more often (if Excel-generated CSV files are used), adding BOM handling to the read_csv module might also make sense.
Same issue here, when the key column have a space in it. Without a space as key file no problem in reading in.

```
{"changed": false, "msg": "Key 'Object ID' was not found in the CSV header fields: Object ID, Soort, Merk,
```
If i change the import to key 'Soort' no problem in the module and al the rows are read.

**edit**
indeed when removing the first bytes (looks like space when editing the CSV) the import was succesfull

Doing some googling I've found:
- https://tobywf.com/2017/08/unicode-csv-excel/
- https://stackoverflow.com/questions/38015124/issue-with-utf-encoding-on-csv-file-for-excel
- https://docs.python.org/3/library/csv.html#examples (there's a mention to encoding in this section)

It *seems* that using the encoding `utf-8-sig` solves the BOM problem (apparently not so straightforwardly for py2, so some tweaking and wrapping might be required). If we go that way, it brings another issue to the table: should we contemplate the possibility of choosing any encoding? I got mixed feelings there, between the flexibility that would give us and cluttering the interface of the module with too many low level options.
@dagwieers As maintainer of this module, what do you think about this?

Is this a feature which is fine to add to module or would lead to other problems?
Im having the same issue and would prefer to not have to encode or convert the file in order to get the desired output for this project. Is this possible? I am learning how to use `dd` for linux at the moment as a possible workaround.
Just ran into this issue myself and spent more time than I'd like to admit trying to figure out why a key was in the dict, but was showing as "variable not defined" when trying to print that value out with debug.  

If you're using Excel to produce a csv file, you can fix this by saving the file as regular csv or MS-DOS csv format instead of UTF-8.  Excel defaults to UTF-8 which is in the common file formats section, while the other two options are in the "specialty" file formats section.  

Alternatively, you can specify the fieldnames parmeter, but if you actually end up having a header row you end up with index 0 being a set of keys that are the same as the value (header / header), so then you would have to ignore index 0 if you're parsing the data.  
Files identified in the description:
* [`plugins/modules/read_csv.py`](https://github.com/ansible-collections/community.general/blob/main/plugins/modules/read_csv.py)

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
There is now a PR to fix this, please take a look! #6662

Resolving patch:
diff --git a/plugins/module_utils/csv.py b/plugins/module_utils/csv.py
--- a/plugins/module_utils/csv.py
+++ b/plugins/module_utils/csv.py
@@ -55,8 +55,10 @@ class unix_dialect(csv.Dialect):
 
 
 def read_csv(data, dialect, fieldnames=None):
-
+    BOM = to_native(u'\ufeff')
     data = to_native(data, errors='surrogate_or_strict')
+    if data.startswith(BOM):
+        data = data[len(BOM):]
 
     if PY3:
         fake_fh = StringIO(data)


Test patch:
diff --git a/tests/integration/targets/read_csv/meta/main.yml b/tests/integration/targets/read_csv/meta/main.yml
new file mode 100644
--- /dev/null
+++ b/tests/integration/targets/read_csv/meta/main.yml
@@ -0,0 +1,7 @@
+---
+# Copyright (c) Ansible Project
+# GNU General Public License v3.0+ (see LICENSES/GPL-3.0-or-later.txt or https://www.gnu.org/licenses/gpl-3.0.txt)
+# SPDX-License-Identifier: GPL-3.0-or-later
+
+dependencies:
+  - setup_remote_tmp_dir
diff --git a/tests/integration/targets/read_csv/tasks/main.yml b/tests/integration/targets/read_csv/tasks/main.yml
--- a/tests/integration/targets/read_csv/tasks/main.yml
+++ b/tests/integration/targets/read_csv/tasks/main.yml
@@ -11,16 +11,16 @@
 # Create basic CSV file
 - name: Create unique CSV file
   copy:
-    content: |
+    content: &users_content |
       name,uid,gid,gecos
       dag,500,500,Dag Wieërs
       jeroen,501,500,Jeroen Hoekx
-    dest: users_unique.csv
+    dest: "{{ remote_tmp_dir }}/users_unique.csv"
 
 # Read a CSV file and access user 'dag'
 - name: Read users from CSV file and return a dictionary
   read_csv:
-    path: users_unique.csv
+    path: "{{ remote_tmp_dir }}/users_unique.csv"
     key: name
   register: users_unique
 
@@ -35,10 +35,10 @@
     - users_unique.dict.jeroen.uid == '501'
     - users_unique.dict.jeroen.gid == '500'
 
- # Read a CSV file and access the first item
+# Read a CSV file and access the first item
 - name: Read users from CSV file and return a list
   read_csv:
-    path: users_unique.csv
+    path: "{{ remote_tmp_dir }}/users_unique.csv"
   register: users_unique
 
 - assert:
@@ -61,12 +61,12 @@
       dag;500;500;Dag Wieërs
       jeroen;501;500;Jeroen Hoekx
       dag;502;500;Dag Wieers
-    dest: users_nonunique.csv
+    dest: "{{ remote_tmp_dir }}/users_nonunique.csv"
 
 # Read a CSV file and access user 'dag'
 - name: Read users from CSV file and return a dictionary
   read_csv:
-    path: users_nonunique.csv
+    path: "{{ remote_tmp_dir }}/users_nonunique.csv"
     key: name
     unique: false
     delimiter: ';'
@@ -87,7 +87,7 @@
 # Read a CSV file using an non-existing dialect
 - name: Read users from CSV file and return a dictionary
   read_csv:
-    path: users_nonunique.csv
+    path: "{{ remote_tmp_dir }}/users_nonunique.csv"
     dialect: placebo
   register: users_placebo
   ignore_errors: true
@@ -104,12 +104,12 @@
     content: |
       dag,500,500,Dag Wieërs
       jeroen,501,500,Jeroen Hoekx
-    dest: users_noheader.csv
+    dest: "{{ remote_tmp_dir }}/users_noheader.csv"
 
 # Read a CSV file and access user 'dag'
 - name: Read users from CSV file and return a dictionary
   read_csv:
-    path: users_noheader.csv
+    path: "{{ remote_tmp_dir }}/users_noheader.csv"
     key: name
     fieldnames: name,uid,gid,gecos
   register: users_noheader
@@ -133,12 +133,12 @@
       name,uid,gid,gecos
       dag,500,500,Dag Wieërs
       jeroen,501,500,"Jeroen"Hoekx"
-    dest: users_broken.csv
+    dest: "{{ remote_tmp_dir }}/users_broken.csv"
 
 # Read a broken CSV file using strict
 - name: Read users from a broken CSV file
   read_csv:
-    path: users_broken.csv
+    path: "{{ remote_tmp_dir }}/users_broken.csv"
     key: name
     strict: true
   register: users_broken
@@ -148,3 +148,29 @@
     that:
     - users_broken is failed
     - "'Unable to process file' in users_broken.msg"
+
+# Create basic CSV file with BOM
+- name: Create unique CSV file with BOM
+  copy:
+    content: "{{ bom + content }}"
+    dest: "{{ remote_tmp_dir }}/users_bom.csv"
+  vars:
+    content: *users_content
+    bom: "{{ '\ufeff' }}"
+
+ # Read a CSV file and access the first item
+- name: Read users from CSV file and return a list
+  read_csv:
+    path: "{{ remote_tmp_dir }}/users_bom.csv"
+  register: users_bom
+
+- assert:
+    that:
+    - users_bom.list.0.name == 'dag'
+    - users_bom.list.0.gecos == 'Dag Wieërs'
+    - users_bom.list.0.uid == '500'
+    - users_bom.list.0.gid == '500'
+    - users_bom.list.1.name == 'jeroen'
+    - users_bom.list.1.gecos == 'Jeroen Hoekx'
+    - users_bom.list.1.uid == '501'
+    - users_bom.list.1.gid == '500'