Repository: ansible-collections/community.general

Issue:
iptables_state: race condition detected, random and reproducible
##### SUMMARY

When restoring iptables ruleset from a file with this module (`iptables_state`),
it fails with some hosts as the action plugin reaches ssh timeout.

> Timeout (12s) waiting for privilege escalation prompt:

This `AnsibleError` message comes from the `ssh` connection plugin, that is
used by the `async_status` module, which itself is called by `iptables_state`
action plugin, in:

```python
    # Catch early errors due to missing mandatory option, bad
    # option type/value, missing required system command, etc.
    result = merge_hash(result, self._async_result(async_status_args, task_vars, 0))
```

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME

`iptables_state`:
- plugins/action/system/iptables_state.py
- plugins/mudules/system/iptables_state.py

##### ANSIBLE VERSION
```paste below
ansible 2.10.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/quidame/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/quidame/.venvs/py3/lib/python3.7/site-packages/ansible
  executable location = /home/quidame/.venvs/py3/bin/ansible
  python version = 3.7.3 (default, Apr  3 2019, 05:39:12) [GCC 8.3.0]
```

##### CONFIGURATION
```paste below
ANSIBLE_FORCE_COLOR(/home/quidame/github.com/ansible.cfg) = True
ANSIBLE_PIPELINING(/home/quidame/github.com/ansible.cfg) = True
ANSIBLE_SSH_ARGS(/home/quidame/github.com/ansible.cfg) = -o ControlMaster=auto -o ControlPersist=60s -o ProxyCommand=none -q
DEFAULT_ACTION_PLUGIN_PATH(/home/quidame/github.com/ansible.cfg) = ['/home/quidame/github.com/action_plugins']
DEFAULT_CALLBACK_WHITELIST(/home/quidame/github.com/ansible.cfg) = ['profile_roles']
DEFAULT_FILTER_PLUGIN_PATH(/home/quidame/github.com/ansible.cfg) = ['/home/quidame/github.com/filter_plugins']
DEFAULT_HOST_LIST(/home/quidame/github.com/ansible.cfg) = ['/home/quidame/github.com/hosts']
DEFAULT_MODULE_PATH(/home/quidame/github.com/ansible.cfg) = ['/home/quidame/github.com/library']
DEFAULT_ROLES_PATH(/home/quidame/github.com/ansible.cfg) = ['/home/quidame/github.com/roles']
DEFAULT_VAULT_PASSWORD_FILE(/home/quidame/github.com/ansible.cfg) = /home/quidame/github.com/vault_password
INTERPRETER_PYTHON(/home/quidame/github.com/ansible.cfg) = auto_legacy_silent
```

##### OS / ENVIRONMENT

Target OS:
* Debian 10.6 (*buster*)
* Debian testing (*bullseye*)
* CentOS 7.8.2003
* CentOS 8.2.2004

All are Virtual Machines (KVM guests, on the same KVM host) with a minimal system (ssh server with console tools) and with:
* 1024-16384 Mb RAM
* 1-4 vCPU
* x86_64
* python2 (2.7) installed
* python3 (3.7 or 3.8) installed

##### STEPS TO REPRODUCE

I don't know why it happens with some hosts and not with others. But when it happens, this is with:

- ssh `pipelining = True` (in *ansible.cfg* or from environment var `ANSIBLE_PIPELINING`)
- we start from *NULL* state (i.e. iptables related kernel modules are not loaded and then `iptables-save` output is empty).
  This is the case for our freshly provisionned, unconfigured VM. To reproduce: disable your firewall service and reboot.
  Note: maybe some stateless configurations may lead to the same issue (not tested).
- and this kind of rule in the ruleset to restore:
  ```
  # a shortcut:
  -A INPUT -p tcp -m tcp ! --syn -m conntrack --ctstate NEW -j DROP
  # that is expanded by iptables to:
  -A INPUT -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m conntrack --ctstate NEW -j DROP
  ```
  This rule says that incoming TCP packets that are seen as *NEW* by the kernel (by *conntrack* netfilter module) but don't come with the SYN flag - and *only* with the SYN flag - are dropped. This is a common sanity rule.

When all these conditions are met, some hosts fail to execute the module, as if the ssh connection was dropped (or the ethernet wire unplugged), even if we are sure the iptables ruleset is safe. And we are. And it is.

This is a playbook to reproduce the issue. Just find a host that fails, so it will fail almost every time (while others will never fail)
```yaml
---
# test_iptables_state.yml

- name: TEST iptables_state MODULE
  hosts: centos
  gather_facts: no
  become: yes
  tasks:
    - name: disable iptables service
      systemd:
        name: iptables
        state: stopped
        enabled: false

    - name: reboot hosts
      reboot:

    - name: copy firewall ruleset
      copy:
        dest: /var/tmp/iptables.ansible
        content: |
          # Ansible managed
          *filter
          :INPUT DROP
          :FORWARD DROP
          :OUTPUT ACCEPT
          -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
          -A INPUT -m conntrack --ctstate INVALID -j DROP
          # All subsequent rules are for NEW packets
          -A INPUT -i lo -m comment --comment loopback -j ACCEPT
          -A INPUT -p icmp -m comment --comment ICMP -j ACCEPT
          -A INPUT -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m comment --comment "fake NEWs" -j DROP
          -A INPUT -p tcp -m tcp --dport 22 -m comment --comment SSH -j ACCEPT
          -A OUTPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
          -A OUTPUT -m conntrack --ctstate INVALID -j DROP
          COMMIT

    - name: restore state of the firewall from the file
      iptables_state:
        path: /var/tmp/iptables.ansible
        state: restored
      async: "{{ ansible_timeout }}"
      poll: 0
```

##### EXPECTED RESULTS

With ssh pipelining disabled, I got:
```bash
ANSIBLE_PIPELINING=false ansible-playbook test_iptables_state.yml
```
```
PLAY [TEST iptables_state MODULE] ******************************************************************************************************

TASK [disable iptables service] ********************************************************************************************************
Tuesday 20 October 2020  14:23:27 +0200 (0:00:00.019)       0:00:00.019 ******* 
ok: [test4]
ok: [test5]

TASK [reboot hosts] ********************************************************************************************************************
Tuesday 20 October 2020  14:23:29 +0200 (0:00:01.426)       0:00:01.446 ******* 
changed: [test4]
changed: [test5]

TASK [copy firewall ruleset] ***********************************************************************************************************
Tuesday 20 October 2020  14:23:58 +0200 (0:00:29.280)       0:00:30.727 ******* 
ok: [test4]
ok: [test5]

TASK [restore state of the firewall from the file] *************************************************************************************
Tuesday 20 October 2020  14:23:59 +0200 (0:00:01.006)       0:00:31.734 ******* 
changed: [test4]
changed: [test5]

PLAY RECAP *****************************************************************************************************************************
test4                      : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
test5                      : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Tuesday 20 October 2020  14:24:04 +0200 (0:00:04.630)       0:00:36.365 ******* 
=============================================================================== 
reboot ----------------------------------------------------------------- 29.28s
iptables_state ---------------------------------------------------------- 4.63s
systemd ----------------------------------------------------------------- 1.43s
copy -------------------------------------------------------------------- 1.01s
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
total ------------------------------------------------------------------ 36.35s
```

##### ACTUAL RESULTS

With ssh pipelining enabled, I got:
```bash
ANSIBLE_PIPELINING=true ansible-playbook test_iptables_state.yml
```
```
PLAY [TEST iptables_state MODULE] ******************************************************************************************************

TASK [disable iptables service] ********************************************************************************************************
Tuesday 20 October 2020  14:20:00 +0200 (0:00:00.019)       0:00:00.019 ******* 
ok: [test4]
ok: [test5]

TASK [reboot hosts] ********************************************************************************************************************
Tuesday 20 October 2020  14:20:01 +0200 (0:00:01.203)       0:00:01.222 ******* 
changed: [test4]
changed: [test5]

TASK [copy firewall ruleset] ***********************************************************************************************************
Tuesday 20 October 2020  14:20:29 +0200 (0:00:28.212)       0:00:29.435 ******* 
ok: [test4]
ok: [test5]

TASK [restore state of the firewall from the file] *************************************************************************************
Tuesday 20 October 2020  14:20:30 +0200 (0:00:00.811)       0:00:30.247 ******* 
changed: [test5]
fatal: [test4]: FAILED! => {"msg": "Timeout (12s) waiting for privilege escalation prompt: "}

PLAY RECAP *****************************************************************************************************************************
test4                      : ok=3    changed=1    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   
test5                      : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Tuesday 20 October 2020  14:20:43 +0200 (0:00:12.847)       0:00:43.094 ******* 
=============================================================================== 
reboot ----------------------------------------------------------------- 28.21s
iptables_state --------------------------------------------------------- 12.85s
systemd ----------------------------------------------------------------- 1.20s
copy -------------------------------------------------------------------- 0.81s
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 
total ------------------------------------------------------------------ 43.08s
```

This erratic behaviour is due to a race condition between the action plugin and the module, that are supposed to share the same time referencial but do not.

Say I want to restore a stateful firewall configuration on newly provisionned hosts, and the *TCP-SYN-is-NEW-and-NEW-TCP-is-SYN* rule is part of the ruleset to deploy, with ssh pipelining enabled. What happens is that when querying the async status of the module on the target host, the module may already have loaded the new ruleset into the kernel, so when starting from a stateless firewall, loading stateful firewall settings leads to this situation:

Connection tracking counters start from 0 with already established connections, so at the time the new settings are applied, all incoming packets are seen as NEW, but those (fake NEWs) ssh packets don't come with the TCP SYN flag alone, since they're not new connections at all (but a previous connection maintained with *pipelining*).

In a few words, the new settings may drop incoming packets of an established connection without closing this connection, as can do a network issue.

This doesn't happen when at least one of these conditions is met:
- ssh `pipelining = False` in *ansible.cfg* (or in environment). This has been verified in my tests.
  Note: `ssh_args` *ControlPath* option doesn't matter (enabled or not).
- The described SYN-is-NEW rule is not in the ruleset to restore. This has been verified in my tests.
- The initial state is already a stateful firewall, so existing ssh connection will still be accepted as `ESTABLISHED` traffic. This has been verified in my tests, by prepending `iptables_state` task with `iptables` as shown here:
  ```yaml
  - name: set stateful firewall
    iptables:
      table: filter
      chain: INPUT
      state: present
      action: insert
      ctstate: ESTABLISHED,RELATED
      jump: ACCEPT

  - name: restore iptables rules from a file
    iptables_state:
      path: /var/tmp/iptables.ansible
      state: restored
    async: "{{ ansible_timeout }}"
    poll: 0
  ```

##### POSSIBLE SOLUTIONS

On the way to understand the issue, the following changes have solved it (either the one or the other):

1. Add a delay in the module (`time.sleep(0.1)` did the job in my tests) right before to execute the *iptables-restore* command.
   Oh no, please ! don't do that (there is no *good* default delay, and no *good* - simple to explain, simple to use - option for that).

2. Remove the line that times out from the action plugin, i.e. do not try to catch early errors so early, since they will be retrieved anyway, in the last `async_status` call. But it doesn't seem consensual (see #271).

3. Build a kind of handshake between plugin (controller) and module (hosts), to ensure module doesn't execute its main command before plugin is ready. For example, from the plugin, query async status, then create a cookie file on the host; from the module, wait for the cookie file to be created before restoring iptables state.


Discussion:
Files identified in the description:
* [`plugins/action/__init__.py`](https://github.com/ansible-collections/community.general/blob/master/plugins/action/__init__.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Files identified in the description:
* [`plugins/modules/system/iptables_state.py`](https://github.com/ansible-collections/community.general/blob/master/plugins/modules/system/iptables_state.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @MorrisA @bcoca @d-little @flynn1973 @gforster @kairoaraujo @marvin-sinister @mator @molekuul @ramooncamacho @wtcross
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/plugins/action/system/iptables_state.py b/plugins/action/system/iptables_state.py
--- a/plugins/action/system/iptables_state.py
+++ b/plugins/action/system/iptables_state.py
@@ -125,15 +125,18 @@ def run(self, tmp=None, task_vars=None):
                     module_args['_back'] = '%s/iptables.state' % async_dir
                     async_status_args = dict(_async_dir=async_dir)
                     confirm_cmd = 'rm -f %s' % module_args['_back']
+                    starter_cmd = 'touch %s.starter' % module_args['_back']
                     remaining_time = max(task_async, max_timeout)
 
             # do work!
             result = merge_hash(result, self._execute_module(module_args=module_args, task_vars=task_vars, wrap_async=wrap_async))
 
             # Then the 3-steps "go ahead or rollback":
-            # - reset connection to ensure a persistent one will not be reused
-            # - confirm the restored state by removing the backup on the remote
-            # - retrieve the results of the asynchronous task to return them
+            # 1. Catch early errors of the module (in asynchronous task) if any.
+            #    Touch a file on the target to signal the module to process now.
+            # 2. Reset connection to ensure a persistent one will not be reused.
+            # 3. Confirm the restored state by removing the backup on the remote.
+            #    Retrieve the results of the asynchronous task to return them.
             if '_back' in module_args:
                 async_status_args['jid'] = result.get('ansible_job_id', None)
                 if async_status_args['jid'] is None:
@@ -143,12 +146,18 @@ def run(self, tmp=None, task_vars=None):
                 # option type/value, missing required system command, etc.
                 result = merge_hash(result, self._async_result(async_status_args, task_vars, 0))
 
+                # The module is aware to not process the main iptables-restore
+                # command before finding (and deleting) the 'starter' cookie on
+                # the host, so the previous query will not reach ssh timeout.
+                garbage = self._low_level_execute_command(starter_cmd, sudoable=self.DEFAULT_SUDOABLE)
+
+                # As the main command is not yet executed on the target, here
+                # 'finished' means 'failed before main command be executed'.
                 if not result['finished']:
                     try:
                         self._connection.reset()
-                        display.v("%s: reset connection" % (module_name))
                     except AttributeError:
-                        display.warning("Connection plugin does not allow to reset the connection.")
+                        pass
 
                     for x in range(max_timeout):
                         time.sleep(1)
diff --git a/plugins/modules/system/iptables_state.py b/plugins/modules/system/iptables_state.py
--- a/plugins/modules/system/iptables_state.py
+++ b/plugins/modules/system/iptables_state.py
@@ -551,6 +551,18 @@ def main():
             restored_state = state_to_restore
 
     else:
+        # Let time enough to the plugin to retrieve async status of the module
+        # in case of bad option type/value and the like.
+        if _back is not None:
+            b_starter = to_bytes('%s.starter' % _back, errors='surrogate_or_strict')
+            while True:
+                if os.path.exists(b_starter):
+                    os.remove(b_starter)
+                    break
+                else:
+                    time.sleep(0.01)
+                    continue
+
         (rc, stdout, stderr) = module.run_command(MAINCOMMAND)
         if 'Another app is currently holding the xtables lock' in stderr:
             module.fail_json(


Test patch: