Repository: ansible-collections/community.general

Issue:
gitlab_project_variable add support for new `expanded` key
### Summary

On 2022-11-22, GitLab released support for a new flag: `expand`, see [here](https://docs.gitlab.com/ee/ci/yaml/#variablesexpand).

This also introduced a [new toggle for project CI vars](https://docs.gitlab.com/ee/ci/variables/index.html#expand-cicd-variables):

![grafik](https://user-images.githubusercontent.com/241552/207574327-12faf44d-6e16-45e7-a4a8-558c7052d64b.png)
(Ignore the green border)
![grafik](https://user-images.githubusercontent.com/241552/207577020-bf0b2263-5455-4a57-86f5-8b1126ce0acf.png)


GitLab claims, this has been there since GitLab 13.7 (which released 2 years ago) but I have never seen this before.

Anyway, my ansible playbook that sets CI vars now claims all my variables are missing and it would like to add them all.
I guess this is because GitLab now puts the expand flag an all of them by default and ansible sees that as a difference?
It's also not possible to set this from ansible as the module does not support the flag.

So this is kinda a feature request to add the flag but also a bug report because currently it incorrectly thinks everything is different.

### Issue Type

Feature Idea

### Component Name

gitlab_project_variable

### Additional Information

Example taken from the docs and altered:

```yaml (paste below)
- name: Set or update some CI/CD variables
  community.general.gitlab_project_variable:
    api_url: https://gitlab.com
    api_token: secret_access_token
    project: markuman/dotfiles
    purge: false
    variables:
      - name: ACCESS_KEY_ID
        value: abc123
      - name: SECRET_ACCESS_KEY
        value: dassgrfaeui8989
        masked: true
        protected: true
        expanded: false  # This would be a new key to disable variable expansion, GitLab default is true
        environment_scope: production
```


### Code of Conduct

- [X] I agree to follow the Ansible Code of Conduct


Discussion:
Files identified in the description:
* [`plugins/modules/gitlab_project_variable.py`](https://github.com/ansible-collections/community.general/blob/main/plugins/modules/gitlab_project_variable.py)

If these files are incorrect, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @Lunik @Shaps @dj-wasabi @lgatellier @markuman @marwatk @metanovii @nejch @scodeman @sh0shin @suukit @waheedi @zanssa
[click here for bot help](https://github.com/ansible/ansibullbot/blob/devel/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Some notes. `expanded` it the name in the frontend. in the backend, the parameter is called `raw`
Does that also mean the value is swapped? `expanded: true` is actually `raw: false` ?
> GitLab claims, this has been there since GitLab 13.7 (which released 2 years ago) but I have never seen this before.

It was introduced with GitLab 15.6: https://about.gitlab.com/releases/2022/11/22/gitlab-15-6-released/#support-for-special-characters-in-cicd-variables

> Anyway, my ansible playbook that sets CI vars now claims all my variables are missing and it would like to add them all.
I guess this is because GitLab now puts the expand flag an all of them by default and ansible sees that as a difference?
It's also not possible to set this from ansible as the module does not support the flag.

This bug-behaviour is already fixed, but not released yet: https://github.com/ansible-collections/community.general/pull/5667

> So this is kinda a feature request to add the flag but also a bug report because currently it incorrectly thinks everything is different.

Yes, the new feature/parameter must still be added for both
* gitlab_project_variable
* gitlab_group_variable

> Does that also mean the value is swapped? expanded: true is actually raw: false ?

I don't know that atm. But you can read it here: https://docs.gitlab.com/ee/api/project_level_variables.html#create-a-variable

> | Attribute | Type | Required | Description |
> | -- | -- | -- | -- |
> | raw | boolean | no | Whether the variable is expandable. Default: false |




> It was introduced with GitLab 15.6: https://about.gitlab.com/releases/2022/11/22/gitlab-15-6-released/#support-for-special-characters-in-cicd-variables

When you click on the (?) beside "Expand variable reference" in my screenshot above it takes you [here](https://docs.gitlab.com/ee/ci/variables/index.html#expand-cicd-variables). And there is says 13.7 which is just plain wrong.

> This bug-behaviour is already fixed, but not released yet: https://github.com/ansible-collections/community.general/pull/5667

Nice, thanks!

> I don't know that atm. But you can read it here: https://docs.gitlab.com/ee/api/project_level_variables.html#create-a-variable

Well, they say that the default value for "expanded" is true, and there they say for "raw" it's false, so I guess it's swapped.
> When you click on the (?) beside "Expand variable reference" in my screenshot above it takes you [here](https://docs.gitlab.com/ee/ci/variables/index.html#expand-cicd-variables). And there is says 13.7 which is just plain wrong.

I guess they are addressing here the entire variable feature ...
I think it's a docs typo from a copy/paste, although they've been working on the topic for a while.

I agree the module should follow the api attributes as it already does for most settings, so `raw=False` by default. I think @markuman you actually implemented the fix initially like that :grinning: 
> Hi @ltraveler,
> 
> Thanks for your feedback !
> 
> community.general's GitLab-related modules rely on [python-gitlab](https://github.com/python-gitlab/python-gitlab), which doesn't support "Expand variable reference" option yet, as far as I know.
> 
> I suggest you to open an issue on python-gitlab GitHub project first.

I made some tests.

```python
>>> var = project.variables.create({'key': 'key2', 'value': 'value1', 'raw': False})
>>> var = project.variables.create({'key': 'key3', 'value': 'value1', 'raw': True})
>>> project.variables.get('key2').to_json()
'{"variable_type": "env_var", "key": "key2", "value": "value1", "protected": false, "masked": false, "raw": false, "environment_scope": "*", "description": null}'
>>> project.variables.get('key3').to_json()
'{"variable_type": "env_var", "key": "key3", "value": "value1", "protected": false, "masked": false, "raw": true, "environment_scope": "*", "description": null}'
```

`python-gitlab` supports the property already. You can use any key. It depends on the GitLab Server version, if the keys are accepted/supported.  
Imo adding a new parameter to support `expanded key`, should be easy.   
Caution. Frontend naming is `expanded key`, backend naming is `raw`.

Is someone of you interested in providing this feature via a pull request? @nejch @envy @lgatellier @ltraveler 
Thanks for the ping @markuman I just posted https://github.com/ansible-collections/community.general/issues/6973#issuecomment-1670947076 before I noticed this.

I've just relocated so was away for a while but I'm catching up on GitHub issues. I can try to work on it in a few days unless someone else jumps in first ;)

Resolving patch:
diff --git a/plugins/module_utils/gitlab.py b/plugins/module_utils/gitlab.py
--- a/plugins/module_utils/gitlab.py
+++ b/plugins/module_utils/gitlab.py
@@ -135,6 +135,7 @@ def vars_to_variables(vars, module):
                     "value": str(value),
                     "masked": False,
                     "protected": False,
+                    "raw": False,
                     "variable_type": "env_var",
                 }
             )
@@ -145,6 +146,7 @@ def vars_to_variables(vars, module):
                 "value": value.get('value'),
                 "masked": value.get('masked'),
                 "protected": value.get('protected'),
+                "raw": value.get('raw'),
                 "variable_type": value.get('variable_type'),
             }
 
diff --git a/plugins/modules/gitlab_group_variable.py b/plugins/modules/gitlab_group_variable.py
--- a/plugins/modules/gitlab_group_variable.py
+++ b/plugins/modules/gitlab_group_variable.py
@@ -53,13 +53,14 @@
     type: bool
   vars:
     description:
-      - When the list element is a simple key-value pair, set masked and protected to false.
-      - When the list element is a dict with the keys C(value), C(masked) and C(protected), the user can
-        have full control about whether a value should be masked, protected or both.
+      - When the list element is a simple key-value pair, masked, raw and protected will be set to false.
+      - When the list element is a dict with the keys C(value), C(masked), C(raw) and C(protected), the user can
+        have full control about whether a value should be masked, raw, protected or both.
       - Support for group variables requires GitLab >= 9.5.
       - Support for environment_scope requires GitLab Premium >= 13.11.
       - Support for protected values requires GitLab >= 9.3.
       - Support for masked values requires GitLab >= 11.10.
+      - Support for raw values requires GitLab >= 15.7.
       - A C(value) must be a string or a number.
       - Field C(variable_type) must be a string with either V(env_var), which is the default, or V(file).
       - When a value is masked, it must be in Base64 and have a length of at least 8 characters.
@@ -95,6 +96,13 @@
           - Wether variable value is protected or not.
         type: bool
         default: false
+      raw:
+        description:
+          - Wether variable value is raw or not.
+          - Support for raw values requires GitLab >= 15.7.
+        type: bool
+        default: false
+        version_added: '7.4.0'
       variable_type:
         description:
           - Wether a variable is an environment variable (V(env_var)) or a file (V(file)).
@@ -126,6 +134,38 @@
         variable_type: env_var
         environment_scope: production
 
+- name: Set or update some CI/CD variables with raw value
+  community.general.gitlab_group_variable:
+    api_url: https://gitlab.com
+    api_token: secret_access_token
+    group: scodeman/testgroup/
+    purge: false
+    vars:
+      ACCESS_KEY_ID: abc123
+      SECRET_ACCESS_KEY:
+        value: 3214cbad
+        masked: true
+        protected: true
+        raw: true
+        variable_type: env_var
+        environment_scope: '*'
+
+- name: Set or update some CI/CD variables with expandable value
+  community.general.gitlab_group_variable:
+    api_url: https://gitlab.com
+    api_token: secret_access_token
+    group: scodeman/testgroup/
+    purge: false
+    vars:
+      ACCESS_KEY_ID: abc123
+      SECRET_ACCESS_KEY:
+        value: '$MY_OTHER_VARIABLE'
+        masked: true
+        protected: true
+        raw: false
+        variable_type: env_var
+        environment_scope: '*'
+
 - name: Delete one variable
   community.general.gitlab_group_variable:
     api_url: https://gitlab.com
@@ -199,6 +239,7 @@ def create_variable(self, var_obj):
             "value": var_obj.get('value'),
             "masked": var_obj.get('masked'),
             "protected": var_obj.get('protected'),
+            "raw": var_obj.get('raw'),
             "variable_type": var_obj.get('variable_type'),
         }
         if var_obj.get('environment_scope') is not None:
@@ -267,6 +308,8 @@ def native_python_main(this_gitlab, purge, requested_variables, state, module):
         item['value'] = str(item.get('value'))
         if item.get('protected') is None:
             item['protected'] = False
+        if item.get('raw') is None:
+            item['raw'] = False
         if item.get('masked') is None:
             item['masked'] = False
         if item.get('environment_scope') is None:
@@ -343,6 +386,7 @@ def main():
             value=dict(type='str', no_log=True),
             masked=dict(type='bool', default=False),
             protected=dict(type='bool', default=False),
+            raw=dict(type='bool', default=False),
             environment_scope=dict(type='str', default='*'),
             variable_type=dict(type='str', default='env_var', choices=["env_var", "file"])
         )),
diff --git a/plugins/modules/gitlab_project_variable.py b/plugins/modules/gitlab_project_variable.py
--- a/plugins/modules/gitlab_project_variable.py
+++ b/plugins/modules/gitlab_project_variable.py
@@ -51,11 +51,12 @@
     type: bool
   vars:
     description:
-      - When the list element is a simple key-value pair, masked and protected will be set to false.
-      - When the list element is a dict with the keys C(value), C(masked) and C(protected), the user can
-        have full control about whether a value should be masked, protected or both.
+      - When the list element is a simple key-value pair, masked, raw and protected will be set to false.
+      - When the list element is a dict with the keys C(value), C(masked), C(raw) and C(protected), the user can
+        have full control about whether a value should be masked, raw, protected or both.
       - Support for protected values requires GitLab >= 9.3.
       - Support for masked values requires GitLab >= 11.10.
+      - Support for raw values requires GitLab >= 15.7.
       - Support for environment_scope requires GitLab Premium >= 13.11.
       - Support for variable_type requires GitLab >= 11.11.
       - A C(value) must be a string or a number.
@@ -96,6 +97,13 @@
           - Support for protected values requires GitLab >= 9.3.
         type: bool
         default: false
+      raw:
+        description:
+          - Wether variable value is raw or not.
+          - Support for raw values requires GitLab >= 15.7.
+        type: bool
+        default: false
+        version_added: '7.4.0'
       variable_type:
         description:
           - Wether a variable is an environment variable (V(env_var)) or a file (V(file)).
@@ -143,6 +151,38 @@
         variable_type: env_var
         environment_scope: '*'
 
+- name: Set or update some CI/CD variables with raw value
+  community.general.gitlab_project_variable:
+    api_url: https://gitlab.com
+    api_token: secret_access_token
+    project: markuman/dotfiles
+    purge: false
+    vars:
+      ACCESS_KEY_ID: abc123
+      SECRET_ACCESS_KEY:
+        value: 3214cbad
+        masked: true
+        protected: true
+        raw: true
+        variable_type: env_var
+        environment_scope: '*'
+
+- name: Set or update some CI/CD variables with expandable value
+  community.general.gitlab_project_variable:
+    api_url: https://gitlab.com
+    api_token: secret_access_token
+    project: markuman/dotfiles
+    purge: false
+    vars:
+      ACCESS_KEY_ID: abc123
+      SECRET_ACCESS_KEY:
+        value: '$MY_OTHER_VARIABLE'
+        masked: true
+        protected: true
+        raw: false
+        variable_type: env_var
+        environment_scope: '*'
+
 - name: Delete one variable
   community.general.gitlab_project_variable:
     api_url: https://gitlab.com
@@ -220,6 +260,7 @@ def create_variable(self, var_obj):
             "value": var_obj.get('value'),
             "masked": var_obj.get('masked'),
             "protected": var_obj.get('protected'),
+            "raw": var_obj.get('raw'),
             "variable_type": var_obj.get('variable_type'),
         }
 
@@ -290,6 +331,8 @@ def native_python_main(this_gitlab, purge, requested_variables, state, module):
         item['value'] = str(item.get('value'))
         if item.get('protected') is None:
             item['protected'] = False
+        if item.get('raw') is None:
+            item['raw'] = False
         if item.get('masked') is None:
             item['masked'] = False
         if item.get('environment_scope') is None:
@@ -366,6 +409,7 @@ def main():
             value=dict(type='str', no_log=True),
             masked=dict(type='bool', default=False),
             protected=dict(type='bool', default=False),
+            raw=dict(type='bool', default=False),
             environment_scope=dict(type='str', default='*'),
             variable_type=dict(type='str', default='env_var', choices=["env_var", "file"]),
         )),


Test patch: