Repository: ansible-collections/community.general

Issue:
ini_file is thrown off by BOM
### Summary

I noticed that an INI file I'm modifying is getting its first section split up. (I'm setting an option which doesn't already exist.) The software that uses the file does not support duplicate sections, and fails to launch as a result.

After investigation, the cause appears to be the presence of BOM at the beginning of the file, which causes the first section's header to be ignored.
EDIT: Further investigation revealed that the BOM was introduced by modifying the file within Windows Powershell (v5.1) using cmdlets from the [PSIni 3.1.2](https://lipkau.github.io/PsIni/) module. From what I'm reading about Powershell's handling of text files, upgrading to the newer, cross-platform version of powershell should solve the issue for no-explicit encoding set ("Utf8NoBom"), and the old version may be able to dodge the issue by specifying non-UTF encoding, such as ASCII, assuming the content allows for it.

I've only found reference to BOM in issue 26048, where the resolution says BOM files are unsupported in inventories - I think this is fair, but shouldn't BOM files be supported in INI module? In many cases the files which this module is to modify, come from Windows environments.

### Issue Type

Bug Report

### Component Name

ini_file

### Ansible Version

```console (paste below)
$ ansible --version
ansible 2.9.9
  config file = None
  configured module search path = ['/home/jenkins/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.9/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.9.0+ (default, Oct 19 2020, 09:51:18) [GCC 10.2.0]
```


### Configuration

```console (paste below)
$ ansible-config dump --only-changed

```


### OS / Environment

Debian 10-based docker container

### Steps to Reproduce

Run the following playbook
```
ansible-playbook test.yaml
```

<!--- Paste example playbooks or commands between quotes below -->
```yaml (paste below)
---
- hosts: localhost
  gather_facts: false
  tasks:
  - name: Create starting ini file
    copy:
      # The content below is the following text file with BOM:
      # [section1]
      # var1=aaa
      # var2=bbb
      # [section2]
      # var3=ccc
      content: !!binary |
        77u/W3NlY3Rpb24xXQp2YXIxPWFhYQp2YXIyPWJiYgpbc2VjdGlvbjJdCnZhcjM9Y2NjCg==
      dest: ./sandbox.ini
  - name: Test ini breakage
    ini_file:
      path: ./sandbox.ini
      section: "{{ item.section }}"
      option: "{{ item.option }}"
      value: "{{ item.value | default(omit) }}"
      state: "{{ item.state | default(omit) }}"
    with_items:
    - { section: "section1", option: "var4", value: "0" }

```

### Expected Results

Option _var4_ should be added to _section1_ at the beginning of the file.

### Actual Results

A new _section1_ is created, containing only _var4_
```
[section1]
var1=aaa
var2=bbb
[section2]
var3=ccc
[section1]
var4 = 0
```

### Code of Conduct

I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* [`plugins/modules/files/ini_file.py`](https://github.com/ansible-collections/community.general/blob/master/plugins/modules/files/ini_file.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @jpmens @noseka1
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
This sounds more like a feature request for me. If the module never said it supports BOM, this isn't a bug.

It's still something that's worth implementing though :)
I don't think the module ever said it supported BOM (isn't that a "windows-only illness"? :-) 


@jpmens Windows and programs trying to be too compatible to Microsoft programs tend to add BOMs, I haven't seen them anywhere else...
@felixfontein right, and glancing at [this](https://bugs.python.org/issue7651) makes me want to forget the idea of even looking closer at it.
This is a python 2 compatible way of addressing the issue, but this module doesn't seem like the right place to address it. (Unless  the presence of BOM's in ini files is so ubiquitous that it is a critical feature)
Seems more like a use case suited for a reusable utility.
```python
 144         with open(filename, 'rb') as ini_file:
 145             ini_lines = list(map(lambda s: s.decode('utf-8-sig'), ini_file.readlines()))
```
I guess it's also ok to open the file in binary mode, read the first couple of bytes, and if they are a BOM use [`io.open(encoding='utf-8-sig')`](https://docs.python.org/2/library/io.html#io.open) to read the lines.
This problem also happens in `read_csv`. 

@Ajpantuso I agree, we shoukd make a `module_utils` for that (or add to an existing one).
Searching a bit on this, it seems we should be considering utf-16 and -32 as well. 
> Searching a bit on this, it seems we should be considering utf-16 and -32 as well.

I did spend some time playing around with python's unicode implementation and I believe (as per usual) the only special case is for UTF-8 BOM's.

The UTF-16 and UTF-32 BOM's are handled by python's encodings by default and will be silently dropped while reading. (See the  end of this [section](https://docs.python.org/3/howto/unicode.html#reading-and-writing-unicode-data) in the python how-to's)

For the affected modules (ini_file, xml, and read_csv) there are three different scenarios:
- ini_file - this one should probably explicitly check for a UTF-8 BOM and treat it as a "token" to pop while descending into the ini file and push it back before writing
- read_csv - The UTF-8 BOM should just be discarded with a simple `str.lstrip(BOM_UTF8)` since the data is never written back to disk
- xml - BOM's and xml files don't play nice and I don't think it makes sense to add explicit support. (This is the only scenario I've encountered in the wild and that's because small businesses like to use Microsoft business integration software against "real" api's)

```python
   1 from codecs import BOM_UTF8, BOM_UTF16, BOM_UTF32
   2
   3 # The UTF-8 BOM decodes to '\ufeff'
   4 assert BOM_UTF8.decode('UTF-8') != ''
   5
   6 # The UTF-16/UTF-32 BOM's decode to '' (codecs handles endianness of 16/32 BOM's automatically)
   7 assert BOM_UTF16.decode('UTF-16') == BOM_UTF32.decode('UTF-32') == ''
   8
   9 # Dummy text to simiulate a file read in with encoding 'UTF-8'
  10 text = BOM_UTF8.decode('UTF-8') + 'start_of_text'
  11
  12 # Demonstrating that a UTF-8 decoded UTF-8 BOM works as expected in str operations
  13 assert text.lstrip(BOM_UTF8.decode('UTF-8')) == text.encode('UTF-8').decode('UTF-8-SIG')
```
>     * read_csv - The UTF-8 BOM should just be discarded with a simple `str.lstrip(BOM_UTF8)` since the data is never written back to disk

Please note that `lstrip` (for byte sequences) does NOT do "remove the whole string as many times as it appears at the beginning of the data", but "remove all bytes from the beginning of this data which appear anywhere in this byte sequence". So for example `'aaabbb'.lstrip('cba') == ''`.

So `str.lstrip(BOM_UTF8)` does something else than we want (we also want to strip it at most once, and not more than once).

> > ```
> > * read_csv - The UTF-8 BOM should just be discarded with a simple `str.lstrip(BOM_UTF8)` since the data is never written back to disk
> > ```
> 
> Please note that `lstrip` (for byte sequences) does NOT do "remove the whole string as many times as it appears at the beginning of the data", but "remove all bytes from the beginning of this data which appear anywhere in this byte sequence". So for example `'aaabbb'.lstrip('cba') == ''`.
> 
> So `str.lstrip(BOM_UTF8)` does something else than we want (we also want to strip it at most once, and not more than once).

Yep, not sure what language I was thinking, but the intention would be whatever most optimally does the following:
```python
binary_string = binary_string[len(BOM_UTF8):] if binary_string.startswith(BOM_UTF8) else binary_string
```

Resolving patch:
diff --git a/plugins/modules/files/ini_file.py b/plugins/modules/files/ini_file.py
--- a/plugins/modules/files/ini_file.py
+++ b/plugins/modules/files/ini_file.py
@@ -104,6 +104,7 @@
     backup: yes
 '''
 
+import io
 import os
 import re
 import tempfile
@@ -141,7 +142,7 @@ def do_ini(module, filename, section=None, option=None, value=None,
             os.makedirs(destpath)
         ini_lines = []
     else:
-        with open(filename, 'r') as ini_file:
+        with io.open(filename, 'r', encoding="utf-8-sig") as ini_file:
             ini_lines = ini_file.readlines()
 
     if module._diff:


Test patch:
diff --git a/tests/integration/targets/ini_file/tasks/main.yml b/tests/integration/targets/ini_file/tasks/main.yml
--- a/tests/integration/targets/ini_file/tasks/main.yml
+++ b/tests/integration/targets/ini_file/tasks/main.yml
@@ -480,3 +480,37 @@
   assert:
     that:
       - content15 == expected15
+
+- name: Create starting ini file
+  copy:
+    # The content below is the following text file with BOM:
+    # [section1]
+    # var1=aaa
+    # var2=bbb
+    # [section2]
+    # var3=ccc
+    content: !!binary |
+      77u/W3NlY3Rpb24xXQp2YXIxPWFhYQp2YXIyPWJiYgpbc2VjdGlvbjJdCnZhcjM9Y2NjCg==
+    dest: "{{ output_file }}"
+- name: Test ini breakage
+  ini_file:
+    path: "{{ output_file }}"
+    section: section1
+    option: var4
+    value: 0
+
+- name: read content from output file
+  slurp:
+    src: "{{ output_file }}"
+  register: output_content
+
+- name: set expected content and get current ini file content
+  set_fact:
+    expected16: "[section1]\nvar1=aaa\nvar2=bbb\nvar4 = 0\n[section2]\nvar3=ccc\n"
+    content16: "{{ output_content.content | b64decode }}"
+- debug:
+    var: content16
+- name: Verify content of ini file is as expected
+  assert:
+    that:
+      - content16 == expected16