Repository: ansible-collections/community.general

Issue:
iptables_state fails with Unknown arguments found on commandline
### Summary

```
fatal: [router]: FAILED! => changed=true 
  cmd: /usr/sbin/iptables-restore /tmp/tmpx4gy9cya /tmp/tmpgf9e7lpr
  msg: Unknown arguments found on commandline
  rc: 1
  stderr: |-
    Unknown arguments found on commandline
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
```

### Issue Type

Bug Report

### Component Name

iptables_state

### Ansible Version

```console
$ ansible --version
ansible [core 2.11.0] 
  config file = /home/rgl/Projects/debian-router-ansible-vagrant/ansible.cfg
  configured module search path = ['/home/rgl/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/rgl/Projects/debian-router-ansible-vagrant/.ansible-venv/lib/python3.8/site-packages/ansible
  ansible collection location = /home/rgl/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/rgl/Projects/debian-router-ansible-vagrant/.ansible-venv/bin/ansible
  python version = 3.8.5 (default, Jan 27 2021, 15:41:15) [GCC 9.3.0]
  jinja version = 3.0.0
  libyaml = True
```


### Configuration

```console (paste below)
$ ansible-config dump --only-changed
DEFAULT_HOST_LIST(/home/rgl/Projects/debian-router-ansible-vagrant/ansible.cfg) = ['/home/rgl/Projects/debian-router-ansible-vagrant/.vagrant/provisioners/ansible/inventory']
DEFAULT_STDOUT_CALLBACK(/home/rgl/Projects/debian-router-ansible-vagrant/ansible.cfg) = community.general.yaml
INTERPRETER_PYTHON(/home/rgl/Projects/debian-router-ansible-vagrant/ansible.cfg) = python3
```


### OS / Environment

Ubuntu 20.04 Host.
Debian 10 Guest/Target.

### Steps to Reproduce

Playbook:

```yaml (paste below)
- name: Install iptables
  block:
    # these anwsers were obtained (after installing iptables-persistent) with:
    #   #sudo debconf-show iptables-persistent
    #   sudo apt-get install debconf-utils
    #   # this way you can see the comments:
    #   sudo debconf-get-selections
    #   # this way you can just see the values needed for debconf-set-selections:
    #   sudo debconf-get-selections | grep -E '^iptables-persistent\s+' | sort
    - name: Configure iptables autosave answers
      debconf:
        name: iptables-persistent
        question: "iptables-persistent/{{ item }}"
        vtype: boolean
        value: false
      loop:
        - autosave_v4
        - autosave_v6
    - name: Install iptables
      apt:
        name: iptables-persistent

- name: Create the iptables rules
  template:
    src: iptables-rules.v4.j2
    dest: /etc/iptables/rules.v4
  register: iptables_rules_v4

# NB this equivalent of doing iptables-restore /etc/iptables/rules.v4
- name: Apply the iptables rules
  community.general.iptables_state:
    state: restored
    path: /etc/iptables/rules.v4
  async: "{{ ansible_timeout }}"
  poll: 0
  when: iptables_rules_v4.changed

- name: Enable IPv4 forwarding
  ansible.posix.sysctl:
    name: net.ipv4.ip_forward
    value: 1

```


The `iptables-rules.v4.j2` template:

```plain
# {{ ansible_managed }}
# see https://help.ubuntu.com/community/IptablesHowTo
# TODO log rejected connections (and rate limit the log)
*filter
:INPUT DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
# NAT through eth0.
# NB use something like -s 10.10.10/24 to limit to a specific network.
-A POSTROUTING -o eth0 -j MASQUERADE
COMMIT
```

### Expected Results

It should have worked. :-)

### Actual Results

```console (paste below)
fatal: [router]: FAILED! => changed=true 
cmd: /usr/sbin/iptables-restore /tmp/tmpzu6wp1wl /tmp/tmpw6_j2_a0
invocation:
  module_args:
    counters: false
    ip_version: ipv4
    modprobe: null
    noflush: false
    path: /etc/iptables/rules.v4
    state: restored
    table: null
    wait: null
msg: Unknown arguments found on commandline
rc: 1
stderr: |-
  Unknown arguments found on commandline
stderr_lines: <omitted>
stdout: ''
stdout_lines: <omitted>
```

Those temporary files have this content:

```console
root@router:~# cat /tmp/tmpzu6wp1wl  
*filter
COMMIT
root@router:~# cat /tmp/tmpw6_j2_a0
*nat
COMMIT
root@router:~# 
```

Please note that manually calling `iptables-restore /etc/iptables/rules.v4` worked without problems.

It seems `iptables-restore` can only be passed a single argument, but it seems `iptables_state` is passing it two files arguments. Have a look at this console output:

```console
root@router:~# iptables-restore /etc/iptables/rules.v4 
root@router:~# iptables-restore /etc/iptables/rules.v4 /etc/iptables/rules.v4
Unknown arguments found on commandline
root@router:~# 

```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* [`plugins/modules/system/iptables_state.py`](https://github.com/ansible-collections/community.general/blob/master/plugins/modules/system/iptables_state.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @quidame
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
This seems to be related with the `Configure iptables autosave answers` task. When that task is omitted, `iptables_state` works, but I believe it should work even with autosave disabled.
@rgl thanks for this detailed report ! I've found the culprit piece of code, it will be fixed soon :)

For the record: the issue is due to the function `initialize_from_null_state()`, whose the purpose is to get a reliable `iptables-save` output, that means *usable for rollback*. It is called when first `iptables-save` output is empty or doesn't contain a required table (in your case, both `filter` and `nat` tables), and runs an iptables-family command to load all (per-table) needed kernel modules. The error comes from a variable that is reused at each call of the function, that makes it cumulative (and so, iptables-restore has first file as argument at first iteration, and at second iteration, it has two files as arguments rather than the second file).

Note that the reason why it works when `Configure iptables autosave answers` task is omitted is that iptables-persistent package initializes iptables from null state on its own (for the `filter` table) when *autosave* is set to *true* , that is the default. In that case, `iptables_state` module has only to initialize `nat` table, i.e. only one iteration in the loop == bug not triggered. It would still fail if you'd add another table in your configuration.
Thanks for looking into this! I really appreciate it.

I have a nagging doubt, I'm currently writing directly to `/etc/iptables/rules.v4` with the `template` module, but, if for some reason, the template generates a bad rules file, the machine will be busted at the next reboot (regardless of this `iptables_state` module doing the correct thing of failing due to invalid rules), because at the next reboot, the `iptables-persistent` machinery will try to load that rules.v4 file.

How do you recommend we should protect ourselves against this?

Maybe I am currently using this module incorrectly and I should actually write to another place?
> How do you recommend we should protect ourselves against this?

Maybe this:

```yaml
# This could be achieved with a temporary file too, with an obvious lack of idempotency
- name: Create the iptables rules
  ansible.builtin.template:
    src: iptables-rules.v4.j2
    dest: /etc/iptables/rules.v4.in

# NB this is the ansible way of doing iptables-apply /etc/iptables/rules.v4.in
- name: Apply the iptables rules (or rollback and fail)
  community.general.iptables_state:
    state: restored
    path: /etc/iptables/rules.v4.in
  async: "{{ ansible_timeout }}"
  poll: 0

# NB this is the ansible way of doing iptables-save > /etc/iptables/rules.v4
- name: Save the iptables rules to be loaded at next boot
  community.general.iptables_state:
    state: saved
    path: /etc/iptables/rules.v4
```

(this is my own interpretation of [a store last good configuration mechanism](https://man7.org/linux/man-pages/man8/iptables-apply.8.html))

> Maybe I am currently using this module incorrectly and I should actually write to another place?

Not really, but I would avoid conditionals (`when` statements). In this case: once the playbook has failed due to a buggy `iptables_state` module, if you replay it with a fixed version of the module on the same hosts it has failed the first time, the previous task will notice no changes, and then the repaired task will be skipped, that is probably not what you want.
I was also thinking in those lines of backstaging the rules, but that backstage file, made me wonder for a better way.

But anyhow, its better than what I was using, as it does not install broken rules. Thanks for pointing it out :-)

BTW, do you known why `iptables_state` does not support the ansible-playbook --diff mode?

PS: This backstaging solution still leaves me with a funny feeling that I'm missing something in ansible. Maybe some kind of stateful optimistic-locking/retry mechanism (e.g. some way to declare a block of tasks that share a persisted generation number, which stays the same unless something fails; that way, when the block is re-run, I could use a more sane `when` expression to run the task between ansible-playbook executions). This because, I'm not satisfied with the `when` or the `handler` mechanisms of ansible. They seem to have pretty odd error failure modes, but I don't known enough about ansible to come up with a better solution.
> I was also thinking in those lines of backstaging the rules, but that backstage file, made me wonder for a better way.

Maybe change your point of view and forget the backstage file. Think more about system states. Currently, the module's `state=restored` applies to system runtime/dynamic/in-memory state, that will be lost at shutdown; its `state=saved` applies to system static/on-disk state, that will survive to reboots. This is similar to the differences between mounting a device and recording the mount in /etc/fstab, or starting/stopping a service and enabling/disabling it, or use a one-shot configuration for a network interface (from DHCP) or store this configuration in whatever network manager conffiles, and so on.

That would lead to questions like:
Maybe this module could be able to set in-memory and on-disk states at once ?
Maybe this module could not need a file to restore, but only data from stdin ?
And why not both ?
Feel free to open a new issue :) 

> BTW, do you known why `iptables_state` does not support the ansible-playbook --diff mode?

...because it ***has to be*** implemented :) I was probably too busy/lazy to do it at time, but this kind of missing feature is clearly a hole in the ansible stack. Feel free to open a new issue.
Indeed it leads us to that :-)

Maybe something along the following lines?

```yml
- name: Set the iptables rules
  community.general.iptables_state:
    state: restored_and_saved
    rules: "{{ lookup('template', 'iptables-rules.v4.j2') }}"
    path: /etc/iptables/rules.v4
  async: "{{ ansible_timeout }}"
  poll: 0
```

Maybe this could also open the door for saving the rules file as-is they were passed to the `rules` property? I mean, currently, the `saved` state will use `iptables-save` which, understandably, looses all of our rules comments. But maybe this is starting to be too much responsibilities for a single module, would complicate the future diff mode, would introduce too many changes, and just having a `restored_and_saved` and `rules` property would be enough.

What do you think about this (or another module) also handling `ipset`s? I'm asking because, it seems there's a dependency between iptables rules and ipsets and a requirement that a ipset must exist before being used by a rule and they cannot be deleted until they are still used (or the deletion might fail with `Set cannot be destroyed: it is in use by a kernel component`); having a module to handle this complexity would be pretty awesome, thou, from what I've read, this might have race conditions before/after adding/deleting ipset/rules, e.g. https://github.com/weaveworks/weave/issues/3847.

I've opened the https://github.com/ansible-collections/community.general/issues/2551 issue about the diff mode :-)

Resolving patch:
diff --git a/plugins/action/system/iptables_state.py b/plugins/action/system/iptables_state.py
--- a/plugins/action/system/iptables_state.py
+++ b/plugins/action/system/iptables_state.py
@@ -7,7 +7,7 @@
 import time
 
 from ansible.plugins.action import ActionBase
-from ansible.errors import AnsibleError, AnsibleActionFail, AnsibleConnectionFailure
+from ansible.errors import AnsibleActionFail, AnsibleConnectionFailure
 from ansible.utils.vars import merge_hash
 from ansible.utils.display import Display
 
@@ -46,7 +46,7 @@ def _async_result(self, module_args, task_vars, timeout):
         the async wrapper results (those with the ansible_job_id key).
         '''
         # At least one iteration is required, even if timeout is 0.
-        for i in range(max(1, timeout)):
+        for dummy in range(max(1, timeout)):
             async_result = self._execute_module(
                 module_name='ansible.builtin.async_status',
                 module_args=module_args,
@@ -76,7 +76,6 @@ def run(self, tmp=None, task_vars=None):
             task_async = self._task.async_val
             check_mode = self._play_context.check_mode
             max_timeout = self._connection._play_context.timeout
-            module_name = self._task.action
             module_args = self._task.args
 
             if module_args.get('state', None) == 'restored':
@@ -133,7 +132,7 @@ def run(self, tmp=None, task_vars=None):
                 # The module is aware to not process the main iptables-restore
                 # command before finding (and deleting) the 'starter' cookie on
                 # the host, so the previous query will not reach ssh timeout.
-                garbage = self._low_level_execute_command(starter_cmd, sudoable=self.DEFAULT_SUDOABLE)
+                dummy = self._low_level_execute_command(starter_cmd, sudoable=self.DEFAULT_SUDOABLE)
 
                 # As the main command is not yet executed on the target, here
                 # 'finished' means 'failed before main command be executed'.
@@ -143,7 +142,7 @@ def run(self, tmp=None, task_vars=None):
                     except AttributeError:
                         pass
 
-                    for x in range(max_timeout):
+                    for dummy in range(max_timeout):
                         time.sleep(1)
                         remaining_time -= 1
                         # - AnsibleConnectionFailure covers rejected requests (i.e.
@@ -151,7 +150,7 @@ def run(self, tmp=None, task_vars=None):
                         # - ansible_timeout is able to cover dropped requests (due
                         #   to a rule or policy DROP) if not lower than async_val.
                         try:
-                            garbage = self._low_level_execute_command(confirm_cmd, sudoable=self.DEFAULT_SUDOABLE)
+                            dummy = self._low_level_execute_command(confirm_cmd, sudoable=self.DEFAULT_SUDOABLE)
                             break
                         except AnsibleConnectionFailure:
                             continue
@@ -164,12 +163,12 @@ def run(self, tmp=None, task_vars=None):
                         del result[key]
 
                 if result.get('invocation', {}).get('module_args'):
-                    if '_timeout' in result['invocation']['module_args']:
-                        del result['invocation']['module_args']['_back']
-                        del result['invocation']['module_args']['_timeout']
+                    for key in ('_back', '_timeout', '_async_dir', 'jid'):
+                        if result['invocation']['module_args'].get(key):
+                            del result['invocation']['module_args'][key]
 
                 async_status_args['mode'] = 'cleanup'
-                garbage = self._execute_module(
+                dummy = self._execute_module(
                     module_name='ansible.builtin.async_status',
                     module_args=async_status_args,
                     task_vars=task_vars,
diff --git a/plugins/modules/system/iptables_state.py b/plugins/modules/system/iptables_state.py
--- a/plugins/modules/system/iptables_state.py
+++ b/plugins/modules/system/iptables_state.py
@@ -232,7 +232,7 @@
 import shutil
 
 from ansible.module_utils.basic import AnsibleModule
-from ansible.module_utils._text import to_bytes, to_native, to_text
+from ansible.module_utils._text import to_bytes, to_native
 
 
 IPTABLES = dict(
@@ -262,7 +262,7 @@ def read_state(b_path):
     lines = text.splitlines()
     while '' in lines:
         lines.remove('')
-    return (lines)
+    return lines
 
 
 def write_state(b_path, lines, changed):
@@ -282,9 +282,9 @@ def write_state(b_path, lines, changed):
         if b_destdir and not os.path.exists(b_destdir) and not module.check_mode:
             try:
                 os.makedirs(b_destdir)
-            except Exception as e:
+            except Exception as err:
                 module.fail_json(
-                    msg='Error creating %s. Error code: %s. Error description: %s' % (destdir, e[0], e[1]),
+                    msg='Error creating %s: %s' % (destdir, to_native(err)),
                     initial_state=lines)
         changed = True
 
@@ -295,10 +295,10 @@ def write_state(b_path, lines, changed):
     if changed and not module.check_mode:
         try:
             shutil.copyfile(tmpfile, b_path)
-        except Exception as e:
+        except Exception as err:
             path = to_native(b_path, errors='surrogate_or_strict')
             module.fail_json(
-                msg='Error saving state into %s. Error code: %s. Error description: %s' % (path, e[0], e[1]),
+                msg='Error saving state into %s: %s' % (path, to_native(err)),
                 initial_state=lines)
 
     return changed
@@ -313,14 +313,11 @@ def initialize_from_null_state(initializer, initcommand, table):
     if table is None:
         table = 'filter'
 
-    tmpfd, tmpfile = tempfile.mkstemp()
-    with os.fdopen(tmpfd, 'w') as f:
-        f.write('*%s\nCOMMIT\n' % table)
-
-    initializer.append(tmpfile)
-    (rc, out, err) = module.run_command(initializer, check_rc=True)
+    commandline = list(initializer)
+    commandline += ['-t', table]
+    (rc, out, err) = module.run_command(commandline, check_rc=True)
     (rc, out, err) = module.run_command(initcommand, check_rc=True)
-    return (rc, out, err)
+    return rc, out, err
 
 
 def filter_and_format_state(string):
@@ -328,13 +325,13 @@ def filter_and_format_state(string):
     Remove timestamps to ensure idempotence between runs. Also remove counters
     by default. And return the result as a list.
     '''
-    string = re.sub('((^|\n)# (Generated|Completed)[^\n]*) on [^\n]*', '\\1', string)
+    string = re.sub(r'((^|\n)# (Generated|Completed)[^\n]*) on [^\n]*', r'\1', string)
     if not module.params['counters']:
-        string = re.sub('[[][0-9]+:[0-9]+[]]', '[0:0]', string)
+        string = re.sub(r'\[[0-9]+:[0-9]+\]', r'[0:0]', string)
     lines = string.splitlines()
     while '' in lines:
         lines.remove('')
-    return (lines)
+    return lines
 
 
 def per_table_state(command, state):
@@ -347,14 +344,14 @@ def per_table_state(command, state):
         COMMAND = list(command)
         if '*%s' % t in state.splitlines():
             COMMAND.extend(['--table', t])
-            (rc, out, err) = module.run_command(COMMAND, check_rc=True)
-            out = re.sub('(^|\n)(# Generated|# Completed|[*]%s|COMMIT)[^\n]*' % t, '', out)
-            out = re.sub(' *[[][0-9]+:[0-9]+[]] *', '', out)
+            dummy, out, dummy = module.run_command(COMMAND, check_rc=True)
+            out = re.sub(r'(^|\n)(# Generated|# Completed|[*]%s|COMMIT)[^\n]*' % t, r'', out)
+            out = re.sub(r' *\[[0-9]+:[0-9]+\] *', r'', out)
             table = out.splitlines()
             while '' in table:
                 table.remove('')
             tables[t] = table
-    return (tables)
+    return tables
 
 
 def main():
@@ -402,7 +399,7 @@ def main():
     changed = False
     COMMANDARGS = []
     INITCOMMAND = [bin_iptables_save]
-    INITIALIZER = [bin_iptables_restore]
+    INITIALIZER = [bin_iptables, '-L', '-n']
     TESTCOMMAND = [bin_iptables_restore, '--test']
 
     if counters:
@@ -502,7 +499,7 @@ def main():
 
     if _back is not None:
         b_back = to_bytes(_back, errors='surrogate_or_strict')
-        garbage = write_state(b_back, initref_state, changed)
+        dummy = write_state(b_back, initref_state, changed)
         BACKCOMMAND = list(MAINCOMMAND)
         BACKCOMMAND.append(_back)
 
@@ -559,9 +556,7 @@ def main():
                 if os.path.exists(b_starter):
                     os.remove(b_starter)
                     break
-                else:
-                    time.sleep(0.01)
-                    continue
+                time.sleep(0.01)
 
         (rc, stdout, stderr) = module.run_command(MAINCOMMAND)
         if 'Another app is currently holding the xtables lock' in stderr:
@@ -579,7 +574,7 @@ def main():
         (rc, stdout, stderr) = module.run_command(SAVECOMMAND, check_rc=True)
         restored_state = filter_and_format_state(stdout)
 
-    if restored_state != initref_state and restored_state != initial_state:
+    if restored_state not in (initref_state, initial_state):
         if module.check_mode:
             changed = True
         else:
@@ -609,7 +604,7 @@ def main():
     #   timeout
     # * task attribute 'poll' equals 0
     #
-    for x in range(_timeout):
+    for dummy in range(_timeout):
         if os.path.exists(b_back):
             time.sleep(1)
             continue


Test patch: