Repository: ansible-collections/community.general

Issue:
gitlab_group_variable deletes all variables when purge is enabled and GitLab API changes
### Summary

GitLab recently added new parameters to their variable objects. When purge is activated, the full variable objects are compared. This results in removing all variables whenever some parameters are not included. As the list of parameters in the GitLab objects differ with GitLab version simply enhancing the defaults will not be sufficient to fix this issue. 

This also affects idempotency (add or update) as the same issue occurs here, so with the enhanced object (raw Parameter) the variables will always be changed.

At least for the purge part it would be a solution to not compare the whole objects but only compare the keys.

### Issue Type

Bug Report

### Component Name

- gitlab_group_variable 
- gitlab_project_variable

### Ansible Version

all as this is due to an implementation issue in the collection

### Community.general Version

>=4.5.0 has the code that not only compares key name but whole key for purge. 
```
/ansible # ansible-galaxy collection list community.general

# /usr/lib/python3.10/site-packages/ansible_collections
Collection        Version
----------------- -------
community.general 5.7.0

# /home/ansible/.ansible/collections/ansible_collections
Collection        Version
----------------- -------
community.general 5.7.0
```

### Configuration

N/A

### OS / Environment

N/A

### Steps to Reproduce

- Add at least one variable to gitlab using the module while enabling purge mode

### Expected Results

Variables from input list are present in GitLab

### Actual Results

All GitLab variables are deleted

### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* [`plugins/modules/gitlab_group_variable.py`](https://github.com/ansible-collections/community.general/blob/main/plugins/modules/gitlab_group_variable.py)
* [`plugins/modules/gitlab_project_variable.py`](https://github.com/ansible-collections/community.general/blob/main/plugins/modules/gitlab_project_variable.py)

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @Lunik @Shaps @dj-wasabi @lgatellier @markuman @marwatk @metanovii @nejch @scodeman @sh0shin @suukit @waheedi @zanssa
[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: notify --->
I think both the variable key and the environment scope would be relevant in this case, as you could have the same key across environments.
@MaxBidlingmaier  @nejch  I cannot find documentation about the new `purge` parameters ...  
https://docs.gitlab.com/ee/api/group_level_variables.html

@markuman the `purge` parameter is part of the Ansible gitlab variable modules, not the GitLab API :)

E.g.:

https://github.com/ansible-collections/community.general/blob/b52a7a6b0edcf5f73904259f6a21777cd94a2d49/plugins/modules/gitlab_group_variable.py#L42-L46
Ah, yeah. Of course it is an ansible feature.

> GitLab recently added new parameters to their variable objects

But it is not added "recently". Imo it exists as long as the module exists. And its default value is `false`.  
So it's by design, that when you set `purge: true`, it will remove all variables that are not defined in the task.  

Or maybe I still don't understand the issue here.
...maybe the description of the `purge` value is not sufficient....  
it will remove all variables that are not given in the task. that's why its default value is `false`.
the problem is with https://github.com/ansible-collections/community.general/blob/b52a7a6b0edcf5f73904259f6a21777cd94a2d49/plugins/modules/gitlab_group_variable.py#L305-L315 not featureing new fields introduced in newest version of gitlab.
'raw' attribute was introduced - which isn't featured in  in `requested_variables`
Ah, now I get it.  
This also affects `gitlab_project_variable` next to the group variable.
Is someone of you @vandlol @nejch @MaxBidlingmaier willing to create a PR to fix `gitlab_group_variable` and `gitlab_project_variable`?
defaults in requested variables should be dropped completely as older versions will face the same issue otherwise

> defaults in requested variables should be dropped completely as older versions will face the same issue otherwise

You mean dropping the default values from module?  
That's not that easy and would take some deprecated releases before dropping.  
Furthermore the module uses the API default values: https://docs.gitlab.com/ee/api/project_level_variables.html#create-a-variable
i mean its enough to compare variable names for deletion - not comparing all of the options
as @nejch mentioned. the environment scope is also important.
that would mean another breaking change for older versions

 @vandlol @nejch @MaxBidlingmaier  can you checkout the PR https://github.com/ansible-collections/community.general/pull/5667  
In a quick test, this solved the issue.

> https://github.com/ansible-collections/community.general/pull/5667

I don't think so ...
The entire variable thing has a long history and goes back to times, where there were no properties in gitlab. There is much room for improvments I think.   
Some behaviours must be dropped from the module, like the `vars` property. But every step in this direction needs to goes through some deprecation circles until we can drop it.  
This PR fixes the current but, where variables get drops accidentally.

Resolving patch:
diff --git a/plugins/module_utils/gitlab.py b/plugins/module_utils/gitlab.py
--- a/plugins/module_utils/gitlab.py
+++ b/plugins/module_utils/gitlab.py
@@ -110,3 +110,14 @@ def gitlab_authentication(module):
             GitLab remove Session API now that private tokens are removed from user API endpoints since version 10.2." % to_native(e))
 
     return gitlab_instance
+
+
+def filter_returned_variables(gitlab_variables):
+    # pop properties we don't know
+    existing_variables = [dict(x.attributes) for x in gitlab_variables]
+    KNOWN = ['key', 'value', 'masked', 'protected', 'variable_type', 'environment_scope']
+    for item in existing_variables:
+        for key in list(item.keys()):
+            if key not in KNOWN:
+                item.pop(key)
+    return existing_variables
diff --git a/plugins/modules/gitlab_group_variable.py b/plugins/modules/gitlab_group_variable.py
--- a/plugins/modules/gitlab_group_variable.py
+++ b/plugins/modules/gitlab_group_variable.py
@@ -165,7 +165,7 @@
 from ansible.module_utils.six import integer_types
 
 from ansible_collections.community.general.plugins.module_utils.gitlab import (
-    auth_argument_spec, gitlab_authentication, ensure_gitlab_package
+    auth_argument_spec, gitlab_authentication, ensure_gitlab_package, filter_returned_variables
 )
 
 
@@ -296,11 +296,7 @@ def native_python_main(this_gitlab, purge, requested_variables, state, module):
     before = [x.attributes for x in gitlab_keys]
 
     gitlab_keys = this_gitlab.list_all_group_variables()
-    existing_variables = [x.attributes for x in gitlab_keys]
-
-    # preprocessing:filter out and enrich before compare
-    for item in existing_variables:
-        item.pop('group_id')
+    existing_variables = filter_returned_variables(gitlab_keys)
 
     for item in requested_variables:
         item['key'] = item.pop('name')
@@ -331,9 +327,7 @@ def native_python_main(this_gitlab, purge, requested_variables, state, module):
         if purge:
             # refetch and filter
             gitlab_keys = this_gitlab.list_all_group_variables()
-            existing_variables = [x.attributes for x in gitlab_keys]
-            for item in existing_variables:
-                item.pop('group_id')
+            existing_variables = filter_returned_variables(gitlab_keys)
 
             remove = [x for x in existing_variables if x not in requested_variables]
             for item in remove:
diff --git a/plugins/modules/gitlab_project_variable.py b/plugins/modules/gitlab_project_variable.py
--- a/plugins/modules/gitlab_project_variable.py
+++ b/plugins/modules/gitlab_project_variable.py
@@ -189,7 +189,7 @@
     HAS_GITLAB_PACKAGE = False
 
 from ansible_collections.community.general.plugins.module_utils.gitlab import (
-    auth_argument_spec, gitlab_authentication, ensure_gitlab_package
+    auth_argument_spec, gitlab_authentication, ensure_gitlab_package, filter_returned_variables
 )
 
 
@@ -255,9 +255,11 @@ def create_variable(self, var_obj):
             return True
 
         var = {
-            "key": var_obj.get('key'), "value": var_obj.get('value'),
-            "masked": var_obj.get('masked'), "protected": var_obj.get('protected'),
-            "variable_type": var_obj.get('variable_type')
+            "key": var_obj.get('key'),
+            "value": var_obj.get('value'),
+            "masked": var_obj.get('masked'),
+            "protected": var_obj.get('protected'),
+            "variable_type": var_obj.get('variable_type'),
         }
 
         if var_obj.get('environment_scope') is not None:
@@ -319,12 +321,9 @@ def native_python_main(this_gitlab, purge, requested_variables, state, module):
     before = [x.attributes for x in gitlab_keys]
 
     gitlab_keys = this_gitlab.list_all_project_variables()
-    existing_variables = [x.attributes for x in gitlab_keys]
-
-    # preprocessing:filter out and enrich before compare
-    for item in existing_variables:
-        item.pop('project_id')
+    existing_variables = filter_returned_variables(gitlab_keys)
 
+    # filter out and enrich before compare
     for item in requested_variables:
         item['key'] = item.pop('name')
         item['value'] = str(item.get('value'))
@@ -354,9 +353,7 @@ def native_python_main(this_gitlab, purge, requested_variables, state, module):
         if purge:
             # refetch and filter
             gitlab_keys = this_gitlab.list_all_project_variables()
-            existing_variables = [x.attributes for x in gitlab_keys]
-            for item in existing_variables:
-                item.pop('project_id')
+            existing_variables = filter_returned_variables(gitlab_keys)
 
             remove = [x for x in existing_variables if x not in requested_variables]
             for item in remove:
@@ -409,7 +406,7 @@ def main():
             masked=dict(type='bool', default=False),
             protected=dict(type='bool', default=False),
             environment_scope=dict(type='str', default='*'),
-            variable_type=dict(type='str', default='env_var', choices=["env_var", "file"])
+            variable_type=dict(type='str', default='env_var', choices=["env_var", "file"]),
         )),
         state=dict(type='str', default="present", choices=["absent", "present"]),
     )


Test patch: