Repository: ansible/ansible

Issue:
lineinfile not working with `insertbefore: BOF` in Ansible 2.5.0
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
lineinfile

##### ANSIBLE VERSION
ansible 2.5.0
  config file = /Volumes/WebServer/Projects/ansible.local/ansible.cfg
  configured module search path = [u'/Users/craig/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Library/Python/2.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 2.7.10 (default, Oct  6 2017, 22:29:07) [GCC 4.2.1 Compatible Apple LLVM 9.0.0 (clang-900.0.31)]

##### CONFIGURATION
ANSIBLE_PIPELINING(/Users/craig/Downloads/ansible/ansible.cfg) = True
DEFAULT_HOST_LIST(/Users/craig/Downloads/ansible/ansible.cfg) = [u'/Users/craig/Downloads/ansible/inventory']
RETRY_FILES_ENABLED(/Users/craig/Downloads/ansible/ansible.cfg) = False

##### OS / ENVIRONMENT
MacOS 10.13.4 (17E199)

##### SUMMARY
When looking for a line that is at the beginning of the file (BOF), it doesn't always detect it exists, or insert it in the correct location.

##### STEPS TO REPRODUCE

```yaml
    - name: "Example BOF"
      lineinfile:
        dest: "/path/to/file.txt"
        insertbefore: BOF
        line: "Hello"
```

Create the `dest` file so it's empty: First run, the line is added (good); Second time, "An exception occurred during task execution" ... "The error was: IndexError: list index out of range".

Replace the contents of this file with a newline, followed by some text (i.e. 2 lines, but the first line is blank): First run, the line is added (good); Second time the line is added again.

Replace the contents of this file with 3 lines of random text: First run, the line is added (good); Second time the line is added again (on the 4th line).

##### EXPECTED RESULTS
When the line is missing, it should be added to the file... but once it exists, it shouldn't be added again.

##### ACTUAL RESULTS
Depends on the contents of the file before hand.

[Video of results](https://github.com/ansible/ansible/files/1871936/Ansible-LineInFile.mp4.zip)


Discussion:
Files identified in the description:
* [lib/ansible/modules/files/lineinfile.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/files/lineinfile.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @ahtik @dhozac
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Full playbook to reproduce:
```
---

- name: test
  hosts: all
  gather_facts: no

  tasks:
    - shell: echo 123 > test_file creates=test_file

    - lineinfile:
        dest: test_file
        line: 123
        insertbefore: BOF
```
Error:
```
ansible-playbook 2.5.0
  config file = None
  configured module search path = ['/home/florian/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/florian/.local/venvs/ansible/lib/python3.6/site-packages/ansible
  executable location = /home/florian/.local/bin/ansible-playbook
  python version = 3.6.4 (default, Jan  5 2018, 02:13:53) [GCC 7.2.0]
No config file found; using defaults
Parsed /home/florian/sources/test/inventory inventory source with ini plugin

PLAYBOOK: site.yml *********************************************************************************************************************************************************************************************
1 plays in site.yml

PLAY [test] ****************************************************************************************************************************************************************************************************
META: ran handlers

TASK [shell] ***************************************************************************************************************************************************************************************************
task path: /home/florian/sources/test/site.yml:8
Using module file /home/florian/.local/venvs/ansible/lib/python3.6/site-packages/ansible/modules/commands/command.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: florian
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/florian/.ansible/tmp/ansible-tmp-1523274750.470957-227748945958289 `" && echo ansible-tmp-1523274750.470957-227748945958289="` echo /home/florian/.ansible/tmp/ansible-tmp-1523274750.470957-227748945958289 `" ) && sleep 0'
<localhost> PUT /home/florian/.ansible/tmp/ansible-local-15912i0yzblad/tmpnyoyhojz TO /home/florian/.ansible/tmp/ansible-tmp-1523274750.470957-227748945958289/command.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/florian/.ansible/tmp/ansible-tmp-1523274750.470957-227748945958289/ /home/florian/.ansible/tmp/ansible-tmp-1523274750.470957-227748945958289/command.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/florian/.ansible/tmp/ansible-tmp-1523274750.470957-227748945958289/command.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/florian/.ansible/tmp/ansible-tmp-1523274750.470957-227748945958289/ > /dev/null 2>&1 && sleep 0'
ok: [localhost] => {
    "changed": false,
    "cmd": "echo 123 > test_file",
    "invocation": {
        "module_args": {
            "_raw_params": "echo 123 > test_file",
            "_uses_shell": true,
            "chdir": null,
            "creates": "test_file",
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "rc": 0,
    "stdout": "skipped, since test_file exists",
    "stdout_lines": [
        "skipped, since test_file exists"
    ]
}

TASK [lineinfile] **********************************************************************************************************************************************************************************************
task path: /home/florian/sources/test/site.yml:10
Using module file /home/florian/.local/venvs/ansible/lib/python3.6/site-packages/ansible/modules/files/lineinfile.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: florian
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/florian/.ansible/tmp/ansible-tmp-1523274750.7862477-66171061825838 `" && echo ansible-tmp-1523274750.7862477-66171061825838="` echo /home/florian/.ansible/tmp/ansible-tmp-1523274750.7862477-66171061825838 `" ) && sleep 0'
<localhost> PUT /home/florian/.ansible/tmp/ansible-local-15912i0yzblad/tmp9acry5n4 TO /home/florian/.ansible/tmp/ansible-tmp-1523274750.7862477-66171061825838/lineinfile.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/florian/.ansible/tmp/ansible-tmp-1523274750.7862477-66171061825838/ /home/florian/.ansible/tmp/ansible-tmp-1523274750.7862477-66171061825838/lineinfile.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/florian/.ansible/tmp/ansible-tmp-1523274750.7862477-66171061825838/lineinfile.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/florian/.ansible/tmp/ansible-tmp-1523274750.7862477-66171061825838/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_Ly5cPo/ansible_module_lineinfile.py", line 512, in <module>
    main()
  File "/tmp/ansible_Ly5cPo/ansible_module_lineinfile.py", line 503, in main
    ins_aft, ins_bef, create, backup, backrefs, firstmatch)
  File "/tmp/ansible_Ly5cPo/ansible_module_lineinfile.py", line 337, in present
    elif b_lines[index[1] - 1].rstrip(b('\r\n')) != b_line:
IndexError: list index out of range

fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_Ly5cPo/ansible_module_lineinfile.py\", line 512, in <module>\n    main()\n  File \"/tmp/ansible_Ly5cPo/ansible_module_lineinfile.py\", line 503, in main\n    ins_aft, ins_bef, create, backup, backrefs, firstmatch)\n  File \"/tmp/ansible_Ly5cPo/ansible_module_lineinfile.py\", line 337, in present\n    elif b_lines[index[1] - 1].rstrip(b('\\r\\n')) != b_line:\nIndexError: list index out of range\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 1
}
	to retry, use: --limit @/home/florian/sources/test/site.retry

PLAY RECAP *****************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1   

➜  test ansible-playbook -vvvDi inventory site.yml 
ansible-playbook 2.5.0
  config file = None
  configured module search path = ['/home/florian/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/florian/.local/venvs/ansible/lib/python3.6/site-packages/ansible
  executable location = /home/florian/.local/bin/ansible-playbook
  python version = 3.6.4 (default, Jan  5 2018, 02:13:53) [GCC 7.2.0]
No config file found; using defaults
Parsed /home/florian/sources/test/inventory inventory source with ini plugin

PLAYBOOK: site.yml *********************************************************************************************************************************************************************************************
1 plays in site.yml

PLAY [test] ****************************************************************************************************************************************************************************************************
META: ran handlers

TASK [shell] ***************************************************************************************************************************************************************************************************
task path: /home/florian/sources/test/site.yml:8
Using module file /home/florian/.local/venvs/ansible/lib/python3.6/site-packages/ansible/modules/commands/command.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: florian
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/florian/.ansible/tmp/ansible-tmp-1523274866.5070899-41694213921926 `" && echo ansible-tmp-1523274866.5070899-41694213921926="` echo /home/florian/.ansible/tmp/ansible-tmp-1523274866.5070899-41694213921926 `" ) && sleep 0'
<localhost> PUT /home/florian/.ansible/tmp/ansible-local-160322t4x8g9x/tmpsgmu62r6 TO /home/florian/.ansible/tmp/ansible-tmp-1523274866.5070899-41694213921926/command.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/florian/.ansible/tmp/ansible-tmp-1523274866.5070899-41694213921926/ /home/florian/.ansible/tmp/ansible-tmp-1523274866.5070899-41694213921926/command.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/florian/.ansible/tmp/ansible-tmp-1523274866.5070899-41694213921926/command.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/florian/.ansible/tmp/ansible-tmp-1523274866.5070899-41694213921926/ > /dev/null 2>&1 && sleep 0'
ok: [localhost] => {
    "changed": false,
    "cmd": "echo 123 > test_file",
    "invocation": {
        "module_args": {
            "_raw_params": "echo 123 > test_file",
            "_uses_shell": true,
            "chdir": null,
            "creates": "test_file",
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "rc": 0,
    "stdout": "skipped, since test_file exists",
    "stdout_lines": [
        "skipped, since test_file exists"
    ]
}

TASK [lineinfile] **********************************************************************************************************************************************************************************************
task path: /home/florian/sources/test/site.yml:10
Using module file /home/florian/.local/venvs/ansible/lib/python3.6/site-packages/ansible/modules/files/lineinfile.py
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: florian
<localhost> EXEC /bin/sh -c 'echo ~ && sleep 0'
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/florian/.ansible/tmp/ansible-tmp-1523274866.8191464-173365190910096 `" && echo ansible-tmp-1523274866.8191464-173365190910096="` echo /home/florian/.ansible/tmp/ansible-tmp-1523274866.8191464-173365190910096 `" ) && sleep 0'
<localhost> PUT /home/florian/.ansible/tmp/ansible-local-160322t4x8g9x/tmp9s3cjrn2 TO /home/florian/.ansible/tmp/ansible-tmp-1523274866.8191464-173365190910096/lineinfile.py
<localhost> EXEC /bin/sh -c 'chmod u+x /home/florian/.ansible/tmp/ansible-tmp-1523274866.8191464-173365190910096/ /home/florian/.ansible/tmp/ansible-tmp-1523274866.8191464-173365190910096/lineinfile.py && sleep 0'
<localhost> EXEC /bin/sh -c '/usr/bin/python /home/florian/.ansible/tmp/ansible-tmp-1523274866.8191464-173365190910096/lineinfile.py && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /home/florian/.ansible/tmp/ansible-tmp-1523274866.8191464-173365190910096/ > /dev/null 2>&1 && sleep 0'
The full traceback is:
Traceback (most recent call last):
  File "/tmp/ansible_aNUD7p/ansible_module_lineinfile.py", line 512, in <module>
    main()
  File "/tmp/ansible_aNUD7p/ansible_module_lineinfile.py", line 503, in main
    ins_aft, ins_bef, create, backup, backrefs, firstmatch)
  File "/tmp/ansible_aNUD7p/ansible_module_lineinfile.py", line 337, in present
    elif b_lines[index[1] - 1].rstrip(b('\r\n')) != b_line:
IndexError: list index out of range

fatal: [localhost]: FAILED! => {
    "changed": false,
    "module_stderr": "Traceback (most recent call last):\n  File \"/tmp/ansible_aNUD7p/ansible_module_lineinfile.py\", line 512, in <module>\n    main()\n  File \"/tmp/ansible_aNUD7p/ansible_module_lineinfile.py\", line 503, in main\n    ins_aft, ins_bef, create, backup, backrefs, firstmatch)\n  File \"/tmp/ansible_aNUD7p/ansible_module_lineinfile.py\", line 337, in present\n    elif b_lines[index[1] - 1].rstrip(b('\\r\\n')) != b_line:\nIndexError: list index out of range\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE",
    "rc": 1
}
	to retry, use: --limit @/home/florian/sources/test/site.retry

PLAY RECAP *****************************************************************************************************************************************************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1   
```
@samdoran I think this is related to https://github.com/ansible/ansible/pull/36474
The interesting behavior I see is that running this again on a file that has previously been updated to insert a line at BOF using insertbefore causes a second line to be inserted on the line just above the last line in the file.
Yes also it added the line at the end of a file instead of at the beginning... very weird
`lineinfile` now inserts at the end of file instead of start of file, breaking configuration files.
I can also reproduce this behavior: 

```
lineinfile:
  path: /tmp/test
  create: yes
  insertbefore: BOF
  line: 'InsertThisLine'
```

- nonexistent file with `create: yes` creates one-line file, as expected.
- re-running against this one-line file with the correct line crashes with `IndexError`.
- empty but existent file becomes one-line file, as expected.
- 5-line file *without* target line correctly inserts line at BOF.
- 6- line file *with* target line at BOF inserts the line, incorrectly, before the last line, making it the *second to last line* in the now 7-line file.
- 7-line file created above is treated as `OK` by `lineinfile`.

*6-line file produced above*
```
InsertThisLine
one
two
three
four
five
```

*7-line file produced above and read by `lineinfile` as `ok`*
```
InsertThisLine
one
two
three
four
InsertThisLine
five
```
Hey, this has been open for more than a month. It's breaking playbooks. What's the story?
I am working on a fix. Apologies for the delay. I see why this is happening but it's at odds with the changes I made in #36474. I'm trying to balance the two. I've also added tests for this to prevent a regression.
This may actually be more related to what was done in #33393. Anyway, still working through it.

Resolving patch:
diff --git a/lib/ansible/modules/files/lineinfile.py b/lib/ansible/modules/files/lineinfile.py
--- a/lib/ansible/modules/files/lineinfile.py
+++ b/lib/ansible/modules/files/lineinfile.py
@@ -291,8 +291,8 @@ def present(module, dest, regexp, line, insertafter, insertbefore, create,
 
     msg = ''
     changed = False
-    # Regexp matched a line in the file
     b_linesep = to_bytes(os.linesep, errors='surrogate_or_strict')
+    # Regexp matched a line in the file
     if index[0] != -1:
         if backrefs:
             b_new_line = m.expand(b_line)
@@ -303,13 +303,12 @@ def present(module, dest, regexp, line, insertafter, insertbefore, create,
         if not b_new_line.endswith(b_linesep):
             b_new_line += b_linesep
 
-        # If a regexp is specified and a match is found anywhere in the file, do
-        # not insert the line before or after.
+        # If no regexp was given and a line match is found anywhere in the file,
+        # insert the line appropriately if using insertbefore or insertafter
         if regexp is None and m:
 
             # Insert lines
             if insertafter and insertafter != 'EOF':
-
                 # Ensure there is a line separator after the found string
                 # at the end of the file.
                 if b_lines and not b_lines[-1][-1:] in (b('\n'), b('\r')):
@@ -327,7 +326,7 @@ def present(module, dest, regexp, line, insertafter, insertbefore, create,
                     msg = 'line added'
                     changed = True
 
-            elif insertbefore:
+            elif insertbefore and insertbefore != 'BOF':
                 # If the line to insert before is at the beginning of the file
                 # use the appropriate index value.
                 if index[1] == 0:


Test patch:
diff --git a/test/integration/targets/lineinfile/tasks/main.yml b/test/integration/targets/lineinfile/tasks/main.yml
--- a/test/integration/targets/lineinfile/tasks/main.yml
+++ b/test/integration/targets/lineinfile/tasks/main.yml
@@ -36,18 +36,27 @@
     line: "New line at the beginning"
     insertbefore: "BOF"
     backup: yes
-  register: result
+  register: result1
+
+- name: insert a line at the beginning of the file again
+  lineinfile:
+    dest: "{{ output_dir }}/test.txt"
+    state: present
+    line: "New line at the beginning"
+    insertbefore: "BOF"
+  register: result2
 
 - name: assert that the line was inserted at the head of the file
   assert:
     that:
-      - result is changed
-      - "result.msg == 'line added'"
-      - "result.backup != ''"
+      - result1 is changed
+      - result2 is not changed
+      - result1.msg == 'line added'
+      - result1.backup != ''
 
 - name: stat the backup file
   stat:
-    path: "{{ result.backup }}"
+    path: "{{ result1.backup }}"
   register: result
 
 - name: assert the backup file matches the previous hash