Repository: ansible-collections/community.general

Issue:
lxd_container: do not ignore volatile configs

##### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
Treat `volatile.` configs as any other configs. This prevents reporting change every time a `volatile.` config option is set in the task. That happens for instance when one wants to setup stable MAC address for a container, using config option `volatile.eth0.hwaddr`.

<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
- Bugfix Pull Request

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below -->
lxd_container

##### ADDITIONAL INFORMATION
<!--- Include additional information to help people understand the change here -->
<!--- A step-by-step reproduction of the problem is helpful if there is no related issue -->
Lets have this playbook:
```yaml
- hosts: localhost
  tasks:
    - name: Setup lxd containers
      lxd_container:
        name: test
        state: started
        config:
          volatile.eth0.hwaddr: "00:16:3e:ee:ca:fe"
        source:
          type: image
          mode: pull
          server: https://images.linuxcontainers.org
          protocol: simplestreams
          alias: ubuntu/focal
        profiles:
          - default
        wait_for_ipv4_addresses: true
```
Let's run it for the first time:
```console
$ lxd init --auto
$ ansible-playbook site.yaml 
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
[WARNING]: packaging Python module unavailable; unable to validate collection
Ansible version requirements

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [Setup lxd containers] ****************************************************
changed: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
```

Then run it again:
```console
$ ansible-playbook site.yaml 
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
[WARNING]: packaging Python module unavailable; unable to validate collection
Ansible version requirements

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [Setup lxd containers] ****************************************************
changed: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
```

<!--- Paste verbatim command output below, e.g. before and after your change -->
After this change, any subsequent runs will not report change, like this:
```console
$ ansible-playbook site.yaml 
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
[WARNING]: packaging Python module unavailable; unable to validate collection
Ansible version requirements

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [Setup lxd containers] ****************************************************
ok: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
```
lxd_container: do not ignore volatile configs

##### SUMMARY
<!--- Describe the change below, including rationale and design decisions -->
Treat `volatile.` configs as any other configs. This prevents reporting change every time a `volatile.` config option is set in the task. That happens for instance when one wants to setup stable MAC address for a container, using config option `volatile.eth0.hwaddr`.

<!--- HINT: Include "Fixes #nnn" if you are fixing an existing issue -->

##### ISSUE TYPE
<!--- Pick one below and delete the rest -->
- Bugfix Pull Request

##### COMPONENT NAME
<!--- Write the short name of the module, plugin, task or feature below -->
lxd_container

##### ADDITIONAL INFORMATION
<!--- Include additional information to help people understand the change here -->
<!--- A step-by-step reproduction of the problem is helpful if there is no related issue -->
Lets have this playbook:
```yaml
- hosts: localhost
  tasks:
    - name: Setup lxd containers
      lxd_container:
        name: test
        state: started
        config:
          volatile.eth0.hwaddr: "00:16:3e:ee:ca:fe"
        source:
          type: image
          mode: pull
          server: https://images.linuxcontainers.org
          protocol: simplestreams
          alias: ubuntu/focal
        profiles:
          - default
        wait_for_ipv4_addresses: true
```
Let's run it for the first time:
```console
$ lxd init --auto
$ ansible-playbook site.yaml 
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
[WARNING]: packaging Python module unavailable; unable to validate collection
Ansible version requirements

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [Setup lxd containers] ****************************************************
changed: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
```

Then run it again:
```console
$ ansible-playbook site.yaml 
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
[WARNING]: packaging Python module unavailable; unable to validate collection
Ansible version requirements

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [Setup lxd containers] ****************************************************
changed: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
```

<!--- Paste verbatim command output below, e.g. before and after your change -->
After this change, any subsequent runs will not report change, like this:
```console
$ ansible-playbook site.yaml 
[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
[WARNING]: packaging Python module unavailable; unable to validate collection
Ansible version requirements

PLAY [localhost] ***************************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]

TASK [Setup lxd containers] ****************************************************
ok: [localhost]

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
```


Discussion:
Hmm, I'm not sure whether this is a good idea. This is changing behavior, thus this is a breaking change, which cannot be merged as a bugfix.

I don't know why the original behavior was done this way. How about instead adding an option which allows to not ignore volatile keys?

@conloos since you know more about lxd, maybe you can take a look at this?
I traced the origin to [this commit](https://github.com/ansible/ansible/commit/53fe20f1dabc59590ff55522bcc94ca37587a1da) by @hnakamur. I don't see any logical reason for the current behaviour, that's why I consider this a bugfix.

I cannot see a scenario where my change breaks anything:

 - If you don't try to set any `volatile.` container options in your task, then there is no change in behaviour.
 - If you specify some `volatile.` container options and their values differ from current values of the container, then there is no change in behaviour, they are **overwritten on every task run**.
 - If you specify some `volatile.` options and their values **are the same as current values of the container**, current behaviour is to overwrite them with the exact same values and report change. There is no value in this dummy change so there is no harm caused by not making it.

My guess is that the idea behind the current *illogical* behaviour was to be able to provide volatile options only for the first time when the container is created and do not touch them later during subsequent task runs. But that is **not what it does now**. Current behaviour really does not make any sense to me.
Hi @all,

I agree with oskar456 that the implementation is still wrong and that the patch produces the expected behavior. I checked that the keyword "volatile" has been in lxd since 2015.
Unfortunately, the behavior has been unchanged for 5 years or has not been objected to.
I also follow oskar456's reasoning, but I cannot rule out that ansible-scripts use this behavior in some way.

In the interests of stability, I suggest installing a switch, e.g. "new_behavior". I would then also incorporate some changes. And than we could switch the default_behavior in a later release.
Alternatively, we could fork the plugin and enter the corrections there. The existing one will be replaced later.

@felixfontein - how are these changes usually made at ansible?

Frank
> but I cannot rule out that ansible-scripts use this behavior in some way.

Yep, that's the main problem with such changes :) I've seen my share of such problems in the docker_container module...

> @felixfontein - how are these changes usually made at ansible?

The best way is to add a new option whose default value leads to the current behavior, deprecate its default value (i.e. announce it will be changed in a deprecation warning, allowing users to turn the warning off by explicitly setting the value themselves to whichever one they prefer), and eventually (when the removal version comes up) flip the default value (and remove the deprecation warning). That way users are informed (assuming they don't ignore the deprecation warning), and users can make their own choice which behavior they want.

(To add an option with deprecated default value, add an option without a default value and only write in the documentation "The default value is currently C(...), but will change to C(...) in community.general 5.0.0.", and in code check for `None`, and if it is present, use `module.deprecate()` to emit the deprecation warning.)

I don't get it. What would be the outcome of this module doing GET-not_modify-PUT on LXD API enpoint every time the task is run? This is just a plain bug and your position reminds me [xkcd 1172](https://xkcd.com/1172/) ;)

If somebody wanted to have this task report change all the time, they would probably use `changed_when: true` and not depend on this erradic behaviour.
> What would be the outcome of this module doing GET-not_modify-PUT on LXD API enpoint every time the task is run?

I'm not using lxd and I don't know the API, so I have no idea whether what you say is correct or not. I personally avoid merging PRs where I'm not reasonably convinced that they do the right thing (or are at least better then the previous state).
needs_info
What kind of info should be provided here?
Hi oskar456,

i think, if there will be a patch to fix that behavior?
I did not see anyting, so i rewrite the module.

@felixfontein: I think i need a week.

Thanks Frank
Hmm, I'm not sure whether this is a good idea. This is changing behavior, thus this is a breaking change, which cannot be merged as a bugfix.

I don't know why the original behavior was done this way. How about instead adding an option which allows to not ignore volatile keys?

@conloos since you know more about lxd, maybe you can take a look at this?
I traced the origin to [this commit](https://github.com/ansible/ansible/commit/53fe20f1dabc59590ff55522bcc94ca37587a1da) by @hnakamur. I don't see any logical reason for the current behaviour, that's why I consider this a bugfix.

I cannot see a scenario where my change breaks anything:

 - If you don't try to set any `volatile.` container options in your task, then there is no change in behaviour.
 - If you specify some `volatile.` container options and their values differ from current values of the container, then there is no change in behaviour, they are **overwritten on every task run**.
 - If you specify some `volatile.` options and their values **are the same as current values of the container**, current behaviour is to overwrite them with the exact same values and report change. There is no value in this dummy change so there is no harm caused by not making it.

My guess is that the idea behind the current *illogical* behaviour was to be able to provide volatile options only for the first time when the container is created and do not touch them later during subsequent task runs. But that is **not what it does now**. Current behaviour really does not make any sense to me.
Hi @all,

I agree with oskar456 that the implementation is still wrong and that the patch produces the expected behavior. I checked that the keyword "volatile" has been in lxd since 2015.
Unfortunately, the behavior has been unchanged for 5 years or has not been objected to.
I also follow oskar456's reasoning, but I cannot rule out that ansible-scripts use this behavior in some way.

In the interests of stability, I suggest installing a switch, e.g. "new_behavior". I would then also incorporate some changes. And than we could switch the default_behavior in a later release.
Alternatively, we could fork the plugin and enter the corrections there. The existing one will be replaced later.

@felixfontein - how are these changes usually made at ansible?

Frank
> but I cannot rule out that ansible-scripts use this behavior in some way.

Yep, that's the main problem with such changes :) I've seen my share of such problems in the docker_container module...

> @felixfontein - how are these changes usually made at ansible?

The best way is to add a new option whose default value leads to the current behavior, deprecate its default value (i.e. announce it will be changed in a deprecation warning, allowing users to turn the warning off by explicitly setting the value themselves to whichever one they prefer), and eventually (when the removal version comes up) flip the default value (and remove the deprecation warning). That way users are informed (assuming they don't ignore the deprecation warning), and users can make their own choice which behavior they want.

(To add an option with deprecated default value, add an option without a default value and only write in the documentation "The default value is currently C(...), but will change to C(...) in community.general 5.0.0.", and in code check for `None`, and if it is present, use `module.deprecate()` to emit the deprecation warning.)

I don't get it. What would be the outcome of this module doing GET-not_modify-PUT on LXD API enpoint every time the task is run? This is just a plain bug and your position reminds me [xkcd 1172](https://xkcd.com/1172/) ;)

If somebody wanted to have this task report change all the time, they would probably use `changed_when: true` and not depend on this erradic behaviour.
> What would be the outcome of this module doing GET-not_modify-PUT on LXD API enpoint every time the task is run?

I'm not using lxd and I don't know the API, so I have no idea whether what you say is correct or not. I personally avoid merging PRs where I'm not reasonably convinced that they do the right thing (or are at least better then the previous state).
needs_info
What kind of info should be provided here?
Hi oskar456,

i think, if there will be a patch to fix that behavior?
I did not see anyting, so i rewrite the module.

@felixfontein: I think i need a week.

Thanks Frank

Resolving patch:
diff --git a/plugins/modules/cloud/lxd/lxd_container.py b/plugins/modules/cloud/lxd/lxd_container.py
--- a/plugins/modules/cloud/lxd/lxd_container.py
+++ b/plugins/modules/cloud/lxd/lxd_container.py
@@ -31,16 +31,24 @@
         description:
           - 'The config for the container (e.g. {"limits.cpu": "2"}).
             See U(https://github.com/lxc/lxd/blob/master/doc/rest-api.md#post-1)'
-          - If the container already exists and its "config" value in metadata
-            obtained from
-            GET /1.0/containers/<name>
+          - If the container already exists and its "config" values in metadata
+            obtained from GET /1.0/containers/<name>
             U(https://github.com/lxc/lxd/blob/master/doc/rest-api.md#10containersname)
-            are different, they this module tries to apply the configurations.
-          - The key starts with 'volatile.' are ignored for this comparison.
-          - Not all config values are supported to apply the existing container.
-            Maybe you need to delete and recreate a container.
+            are different, this module tries to apply the configurations.
+          - The keys starting with C(volatile.) are ignored for this comparison when I(ignore_volatile_options=true).
         type: dict
         required: false
+    ignore_volatile_options:
+        description:
+          - If set to C(true), options starting with C(volatile.) are ignored. As a result,
+            they are reapplied for each execution.
+          - This default behavior can be changed by setting this option to C(false).
+          - The default value C(true) will be deprecated in community.general 4.0.0,
+            and will change to C(false) in community.general 5.0.0.
+        type: bool
+        default: true
+        required: false
+        version_added: 3.7.0
     profiles:
         description:
           - Profile to be used by the container
@@ -176,6 +184,7 @@
     - name: Create a started container
       community.general.lxd_container:
         name: mycontainer
+        ignore_volatile_options: true
         state: started
         source:
           type: image
@@ -209,6 +218,7 @@
     - name: Create a started container
       community.general.lxd_container:
         name: mycontainer
+        ignore_volatile_options: true
         state: started
         source:
           type: image
@@ -279,6 +289,7 @@
     - name: Create LXD container
       community.general.lxd_container:
         name: new-container-1
+        ignore_volatile_options: true
         state: started
         source:
           type: image
@@ -289,6 +300,7 @@
     - name: Create container on another node
       community.general.lxd_container:
         name: new-container-2
+        ignore_volatile_options: true
         state: started
         source:
           type: image
@@ -557,7 +569,7 @@ def _frozen(self):
     def _needs_to_change_container_config(self, key):
         if key not in self.config:
             return False
-        if key == 'config':
+        if key == 'config' and self.ignore_volatile_options:  # the old behavior is to ignore configurations by keyword "volatile"
             old_configs = dict((k, v) for k, v in self.old_container_json['metadata'][key].items() if not k.startswith('volatile.'))
             for k, v in self.config['config'].items():
                 if k not in old_configs:
@@ -565,6 +577,14 @@ def _needs_to_change_container_config(self, key):
                 if old_configs[k] != v:
                     return True
             return False
+        elif key == 'config':  # next default behavior
+            old_configs = dict((k, v) for k, v in self.old_container_json['metadata'][key].items())
+            for k, v in self.config['config'].items():
+                if k not in old_configs:
+                    return True
+                if old_configs[k] != v:
+                    return True
+            return False
         else:
             old_configs = self.old_container_json['metadata'][key]
             return self.config[key] != old_configs
@@ -606,6 +626,7 @@ def run(self):
         try:
             if self.trust_password is not None:
                 self.client.authenticate(self.trust_password)
+            self.ignore_volatile_options = self.module.params.get('ignore_volatile_options')
 
             self.old_container_json = self._get_container_json()
             self.old_state = self._container_json_to_module_state(self.old_container_json)
@@ -651,6 +672,10 @@ def main():
             config=dict(
                 type='dict',
             ),
+            ignore_volatile_options=dict(
+                type='bool',
+                default=True
+            ),
             devices=dict(
                 type='dict',
             ),
@@ -703,7 +728,13 @@ def main():
         ),
         supports_check_mode=False,
     )
-
+    # if module.params['ignore_volatile_options'] is None:
+    #     module.params['ignore_volatile_options'] = True
+    #     module.deprecate(
+    #         'If the keyword "volatile" is used in a playbook in the config section, a
+    #         "changed" message will appear with every run, even without a change to the playbook.
+    #         This will change in the future.
+    #         Please test your scripts by "ignore_volatile_options: false"', version='5.0.0', collection_name='community.general')
     lxd_manage = LXDContainerManagement(module=module)
     lxd_manage.run()
 


Test patch: