Repository: ansible/ansible

Issue:
lineinfile "regexp" ignored when using "insertafter"
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
lineinfile

##### ANSIBLE VERSION
```
ansible 2.6.0
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/home/jsteel/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.14 (default, Jan  5 2018, 10:41:29) [GCC 7.2.1 20171224]
```

##### CONFIGURATION
N/A

##### OS / ENVIRONMENT
N/A

##### SUMMARY
Recent change #28721 breaks my config where I use both regexp AND insertafter; the regexp doesn't seem to match any more, so on every subsequent run it inserts duplicate lines.

##### STEPS TO REPRODUCE
<!---
For bugs, show exactly how to reproduce the problem, using a minimal test-case.
For new features, show how the feature would be used.
-->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: my.cnf options
  lineinfile:
    dest: "{{ mariadb_config }}"
    regexp: "^{{ item.option }}="
    line: "{{ item.option }}={{ item.value }}"
    insertafter: '^\[{{ item.section }}\]'
  with_items:
     - { section: mysqld, option: innodb_log_file_size, value: "{{ innodb_log_file_size }}" }
     - { section: mysqld, option: default-storage-engine, value: innodb }
     - { section: mysqld, option: max_allowed_packet, value: 128M }
```

<!--- You can also paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- What did you expect to happen when running the steps above? -->
On subsequent runs, nothing should change.

##### ACTUAL RESULTS
<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->
A duplicate line is inserted for each item, for eg:
<!--- Paste verbatim command output between quotes below -->
```
--- before: /etc/my.cnf (content)
+++ after: /etc/my.cnf (content)
@@ -1,4 +1,5 @@
 [mysqld]
+default-storage-engine=innodb
 skip-name-resolve
 max_allowed_packet=128M
 default-storage-engine=innodb
```



Discussion:
Files identified in the description:
* [lib/ansible/modules/files/lineinfile.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/files/lineinfile.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @ahtik @dhozac
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Looks like https://github.com/ansible/ansible/pull/33393 went into `stable-2.4`, `stable-2.5` and `devel`, so the fix will need to go there as well.

Would be good to an integration test for this
@jsteel44 Thanks for opening this issue. I'll work on getting a fix in shortly.
@jsteel44 That took me a lot longer than anticipated. Please test #36474 as soon as you can since we have releases going out soon. Thanks!

Resolving patch:
diff --git a/lib/ansible/modules/files/lineinfile.py b/lib/ansible/modules/files/lineinfile.py
--- a/lib/ansible/modules/files/lineinfile.py
+++ b/lib/ansible/modules/files/lineinfile.py
@@ -42,10 +42,10 @@
   regexp:
     description:
       - The regular expression to look for in every line of the file. For
-        C(state=present), the pattern to replace if found; only the last line
+        C(state=present), the pattern to replace if found. Only the last line
         found will be replaced. For C(state=absent), the pattern of the line(s)
-        to remove.  Uses Python regular expressions; see
-        U(http://docs.python.org/2/library/re.html).
+        to remove. Uses Python regular expressions.
+        See U(http://docs.python.org/2/library/re.html).
     version_added: '1.7'
   state:
     description:
@@ -282,7 +282,7 @@ def present(module, dest, regexp, line, insertafter, insertbefore, create,
                 if firstmatch:
                     break
             if insertbefore:
-                # + 1 for the previous line
+                # index[1] for the previous line
                 index[1] = lineno
                 if firstmatch:
                     break
@@ -301,44 +301,49 @@ def present(module, dest, regexp, line, insertafter, insertbefore, create,
         if not b_new_line.endswith(b_linesep):
             b_new_line += b_linesep
 
-        # Add lines when the regexp match already exists somewhere else in the file
-        if insertafter and insertafter != 'EOF':
-
-            # Ensure there is a line separator after the found string
-            # at the end of the file.
-            if b_lines and not b_lines[-1][-1:] in (b('\n'), b('\r')):
-                b_lines[-1] = b_lines[-1] + b_linesep
-
-            # If the line to insert after is at the end of the file
-            # use the appropriate index value.
-            if len(b_lines) == index[1]:
-                if b_lines[index[1] - 1].rstrip(b('\r\n')) != b_line:
-                    b_lines.append(b_line + b_linesep)
+        # If a regexp is specified and a match is found anywhere in the file, do
+        # not insert the line before or after.
+        if regexp is None and m:
+
+            # Insert lines
+            if insertafter and insertafter != 'EOF':
+
+                # Ensure there is a line separator after the found string
+                # at the end of the file.
+                if b_lines and not b_lines[-1][-1:] in (b('\n'), b('\r')):
+                    b_lines[-1] = b_lines[-1] + b_linesep
+
+                # If the line to insert after is at the end of the file
+                # use the appropriate index value.
+                if len(b_lines) == index[1]:
+                    if b_lines[index[1] - 1].rstrip(b('\r\n')) != b_line:
+                        b_lines.append(b_line + b_linesep)
+                        msg = 'line added'
+                        changed = True
+                elif b_lines[index[1]].rstrip(b('\r\n')) != b_line:
+                    b_lines.insert(index[1], b_line + b_linesep)
                     msg = 'line added'
                     changed = True
-            elif b_lines[index[1]].rstrip(b('\r\n')) != b_line:
-                b_lines.insert(index[1], b_line + b_linesep)
-                msg = 'line added'
-                changed = True
-
-        elif insertbefore:
-            # If the line to insert before is at the beginning of the file
-            # use the appropriate index value.
-            if index[1] == 0:
-                if b_lines[index[1]].rstrip(b('\r\n')) != b_line:
+
+            elif insertbefore:
+                # If the line to insert before is at the beginning of the file
+                # use the appropriate index value.
+                if index[1] == 0:
+                    if b_lines[index[1]].rstrip(b('\r\n')) != b_line:
+                        b_lines.insert(index[1], b_line + b_linesep)
+                        msg = 'line replaced'
+                        changed = True
+
+                elif b_lines[index[1] - 1].rstrip(b('\r\n')) != b_line:
                     b_lines.insert(index[1], b_line + b_linesep)
                     msg = 'line replaced'
                     changed = True
 
-            elif b_lines[index[1] - 1].rstrip(b('\r\n')) != b_line:
-                b_lines.insert(index[1], b_line + b_linesep)
-                msg = 'line replaced'
-                changed = True
-
         elif b_lines[index[0]] != b_new_line:
             b_lines[index[0]] = b_new_line
             msg = 'line replaced'
             changed = True
+
     elif backrefs:
         # Do absolutely nothing, since it's not safe generating the line
         # without the regexp matching to populate the backrefs.


Test patch:
diff --git a/test/integration/targets/lineinfile/files/test.conf b/test/integration/targets/lineinfile/files/test.conf
new file mode 100644
--- /dev/null
+++ b/test/integration/targets/lineinfile/files/test.conf
@@ -0,0 +1,5 @@
+[section_one]
+
+[section_two]
+
+[section_three]
diff --git a/test/integration/targets/lineinfile/tasks/main.yml b/test/integration/targets/lineinfile/tasks/main.yml
--- a/test/integration/targets/lineinfile/tasks/main.yml
+++ b/test/integration/targets/lineinfile/tasks/main.yml
@@ -544,3 +544,167 @@
   assert:
     that:
       - result.stat.checksum == 'a8452bb3643be8d18ba3fc212632b1633bd9f885'
+
+###################################################################
+# Issue 36156
+# Test insertbefore and insertafter with regexp
+
+- name: Deploy the test.conf file
+  copy:
+    src: test.conf
+    dest: "{{ output_dir }}/test.conf"
+  register: result
+
+- name: Assert that the test.conf file was deployed
+  assert:
+    that:
+      - result is changed
+      - result.checksum == '6037f13e419b132eb3fd20a89e60c6c87a6add38'
+      - result.state == 'file'
+
+# Test instertafter
+- name: Insert lines after with regexp
+  lineinfile:
+    path: "{{ output_dir }}/test.conf"
+    regexp: "{{ item.regexp }}"
+    line: "{{ item.line }}"
+    insertafter: "{{ item.after }}"
+  with_items: "{{ test_befaf_regexp }}"
+  register: _multitest_5
+
+- name: Do the same thing again and check for changes
+  lineinfile:
+    path: "{{ output_dir }}/test.conf"
+    regexp: "{{ item.regexp }}"
+    line: "{{ item.line }}"
+    insertafter: "{{ item.after }}"
+  with_items: "{{ test_befaf_regexp }}"
+  register: _multitest_6
+
+- name: Assert that the file was changed the first time but not the second time
+  assert:
+    that:
+      - item.0 is changed
+      - item.1 is not changed
+  with_together:
+    - "{{ _multitest_5.results }}"
+    - "{{ _multitest_6.results }}"
+
+- name: Stat the file
+  stat:
+    path: "{{ output_dir }}/test.conf"
+  register: result
+
+- name: Assert that the file contents match what is expected
+  assert:
+    that:
+      - result.stat.checksum == '06e2c456e5028dd7bcd0b117b5927a1139458c82'
+
+- name: Do the same thing a third time without regexp and check for changes
+  lineinfile:
+    path: "{{ output_dir }}/test.conf"
+    line: "{{ item.line }}"
+    insertafter: "{{ item.after }}"
+  with_items: "{{ test_befaf_regexp }}"
+  register: _multitest_7
+
+- name: Stat the file
+  stat:
+    path: "{{ output_dir }}/test.conf"
+  register: result
+
+- name: Assert that the file was changed when no regexp was provided
+  assert:
+    that:
+      - item is changed
+  with_items: "{{ _multitest_7.results }}"
+
+- name: Stat the file
+  stat:
+    path: "{{ output_dir }}/test.conf"
+  register: result
+
+- name: Assert that the file contents match what is expected
+  assert:
+    that:
+      - result.stat.checksum == '5bf50f3d74afd20de4010ca5c04bc7037b062d30'
+
+# Test insertbefore
+- name: Deploy the test.conf file
+  copy:
+    src: test.conf
+    dest: "{{ output_dir }}/test.conf"
+  register: result
+
+- name: Assert that the test.conf file was deployed
+  assert:
+    that:
+      - result is changed
+      - result.checksum == '6037f13e419b132eb3fd20a89e60c6c87a6add38'
+      - result.state == 'file'
+
+- name: Insert lines before with regexp
+  lineinfile:
+    path: "{{ output_dir }}/test.conf"
+    regexp: "{{ item.regexp }}"
+    line: "{{ item.line }}"
+    insertbefore: "{{ item.before }}"
+  with_items: "{{ test_befaf_regexp }}"
+  register: _multitest_8
+
+- name: Do the same thing again and check for changes
+  lineinfile:
+    path: "{{ output_dir }}/test.conf"
+    regexp: "{{ item.regexp }}"
+    line: "{{ item.line }}"
+    insertbefore: "{{ item.before }}"
+  with_items: "{{ test_befaf_regexp }}"
+  register: _multitest_9
+
+- name: Assert that the file was changed the first time but not the second time
+  assert:
+    that:
+      - item.0 is changed
+      - item.1 is not changed
+  with_together:
+    - "{{ _multitest_8.results }}"
+    - "{{ _multitest_9.results }}"
+
+- name: Stat the file
+  stat:
+    path: "{{ output_dir }}/test.conf"
+  register: result
+
+- name: Assert that the file contents match what is expected
+  assert:
+    that:
+      - result.stat.checksum == 'c3be9438a07c44d4c256cebfcdbca15a15b1db91'
+
+- name: Do the same thing a third time without regexp and check for changes
+  lineinfile:
+    path: "{{ output_dir }}/test.conf"
+    line: "{{ item.line }}"
+    insertbefore: "{{ item.before }}"
+  with_items: "{{ test_befaf_regexp }}"
+  register: _multitest_10
+
+- name: Stat the file
+  stat:
+    path: "{{ output_dir }}/test.conf"
+  register: result
+
+- name: Assert that the file was changed when no regexp was provided
+  assert:
+    that:
+      - item is changed
+  with_items: "{{ _multitest_10.results }}"
+
+- name: Stat the file
+  stat:
+    path: "{{ output_dir }}/test.conf"
+  register: result
+
+- name: Assert that the file contents match what is expected
+  assert:
+    that:
+      - result.stat.checksum == 'eca8d8ea089d4ea57a3b87d4091599ca8b60dfd2'
diff --git a/test/integration/targets/lineinfile/vars/main.yml b/test/integration/targets/lineinfile/vars/main.yml
--- a/test/integration/targets/lineinfile/vars/main.yml
+++ b/test/integration/targets/lineinfile/vars/main.yml
@@ -10,3 +10,20 @@ test_regexp:
 
   - regex: '4'
     replace: 'bar'
+
+
+test_befaf_regexp:
+  - before: section_three
+    after: section_one
+    regexp: option_one=
+    line: option_one=1
+
+  - before: section_three
+    after: section_one
+    regexp: option_two=
+    line: option_two=2
+
+  - before: section_three
+    after: section_one
+    regexp: option_three=
+    line: option_three=3