Repository: ansible-collections/community.general

Issue:
Gitlab_runner is broken with GitLab.com (gitlab v13+)
<!--- Verify first that your issue is not already reported on GitHub -->
<!--- Also test if the latest release and devel branch are affected too -->
<!--- Complete *all* sections as described, this form is processed automatically -->

##### SUMMARY
<!--- Explain the problem briefly below -->
(copy pasta from https://github.com/ansible/ansible/issues/69778)
When registering a third party runner for the official gitlab.com site, you will have a user-token, never an admin token. It is not possible to do so using Ansible 2.9.3+ due to:
-  https://github.com/ansible/ansible/issues/67159
- https://github.com/ansible/ansible/issues/68224

However, since GitLab upgraded the official site to 13, containing [breaking changes](https://about.gitlab.com/releases/2020/05/06/gitlab-com-13-0-breaking-changes/) which [affect the runner API](https://gitlab.com/gitlab-org/gitlab/-/issues/214322), it is now not possible any more to use this with previously working Ansible 2.9.2, because `token` is no longer returned and you will not be able to insert the required token in the `/etc/gitlab-runner/config.toml` required to run the software.

**It is currently not possible to configure a gitlab_runner at all with Ansible, even taking workarounds into account.**

##### ISSUE TYPE
- Bug Report

##### COMPONENT NAME
gitlab_runner

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes -->
```paste below
ansible 2.9.9
  config file = /Users/ghostlyrics/Repositories/ttansible/ansible.cfg
  configured module search path = ['/Users/ghostlyrics/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/2.9.9/libexec/lib/python3.8/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.3 (default, May 27 2020, 20:54:22) [Clang 11.0.3 (clang-1103.0.32.59)]
```

```
ansible 2.9.2
  config file = /Users/ghostlyrics/Repositories/ttansible/ansible.cfg
  configured module search path = ['/Users/ghostlyrics/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible@2.9.2/2.9.2_1/libexec/lib/python3.8/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.3 (default, May 27 2020, 20:54:22) [Clang 11.0.3 (clang-1103.0.32.59)]
```

##### CONFIGURATION
Not relevant.

##### OS / ENVIRONMENT
<!--- Provide all relevant information below, e.g. target OS versions, network device firmware, etc. -->
GitLab 13.0
Debian 10 (Buster)
macOS 10.15 (Catalina)

##### STEPS TO REPRODUCE
<!--- Describe exactly how to reproduce the problem, using a minimal test-case -->

<!--- Paste example playbooks or commands between quotes below -->
```yaml
- name: register docker runner
  gitlab_runner:
    api_url: https://gitlab.com/
    api_token: YOUR_NON_ADMIN_API_TOKEN
    registration_token: YOUR_RUNNER_REGISTRATION_TOKEN
    description: "{{ ansible_hostname }}: docker"
    state: present
    active: True
    tag_list: ['docker']
    run_untagged: False
    locked: False
    access_level: not_protected
  register: gitlab_runner_registration

- name: create gitlab-runner configuration file
  template:
    src: config.toml.j2
    dest: /etc/gitlab-runner/config.toml
    owner: root
    group: root
    mode: '0600'
```

```toml
# config.toml.j2

concurrent = 2
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "{{ ansible_hostname }}: docker"
  url = "https://gitlab.com/"
  token = "{{ gitlab_runner_registration.runner.token }}"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "buildpack-deps:buster"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = [
        "/cache",
    ]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

```

<!--- HINT: You can paste gist.github.com links for larger files -->

##### EXPECTED RESULTS
<!--- Describe what you expected to happen when running the steps above -->
- Runners can be registered.
- Runners can be registered with **user** tokens.
- There is **some** way to fill the config.toml with the tokens that **must** be there. Calling `gitlab_runner` does apparently not use the `gitlab-runner` command line util.
##### ACTUAL RESULTS
<!--- Describe what actually happened. If possible run with extra verbosity (-vvvv) -->
<!--- Paste verbatim command output between quotes -->
Depending on which Ansible version you try with, you either get 403:
```paste below
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: gitlab.exceptions.GitlabListError: 403: 403 Forbidden
fatal: [runner1]: FAILED! => changed=false
  module_stderr: |-
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/dist-packages/gitlab/exceptions.py", line 259, in wrapped_f
        return f(*args, **kwargs)
      File "/usr/local/lib/python3.7/dist-packages/gitlab/v4/objects.py", line 4637, in all
        return self.gitlab.http_list(path, query_data, **kwargs)
      File "/usr/local/lib/python3.7/dist-packages/gitlab/__init__.py", line 660, in http_list
        return GitlabList(self, url, query_data, **kwargs)
      File "/usr/local/lib/python3.7/dist-packages/gitlab/__init__.py", line 780, in __init__
        self._query(url, query_data, **kwargs)
      File "/usr/local/lib/python3.7/dist-packages/gitlab/__init__.py", line 785, in _query
        result = self._gl.http_request("get", url, query_data=query_data, **kwargs)
      File "/usr/local/lib/python3.7/dist-packages/gitlab/__init__.py", line 581, in http_request
        response_body=result.content,
    gitlab.exceptions.GitlabHttpError: 403: 403 Forbidden

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<stdin>", line 102, in <module>
      File "<stdin>", line 94, in _ansiballz_main
      File "<stdin>", line 40, in invoke_module
      File "/usr/lib/python3.7/runpy.py", line 205, in run_module
        return _run_module_code(code, init_globals, run_name, mod_spec)
      File "/usr/lib/python3.7/runpy.py", line 96, in _run_module_code
        mod_name, mod_spec, pkg_name, script_name)
      File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
        exec(code, run_globals)
      File "/tmp/ansible_gitlab_runner_payload_rbhtqvmd/ansible_gitlab_runner_payload.zip/ansible/modules/source_control/gitlab_runner.py", line 392, in <module>
      File "/tmp/ansible_gitlab_runner_payload_rbhtqvmd/ansible_gitlab_runner_payload.zip/ansible/modules/source_control/gitlab_runner.py", line 366, in main
      File "/tmp/ansible_gitlab_runner_payload_rbhtqvmd/ansible_gitlab_runner_payload.zip/ansible/modules/source_control/gitlab_runner.py", line 271, in existsRunner
      File "/tmp/ansible_gitlab_runner_payload_rbhtqvmd/ansible_gitlab_runner_payload.zip/ansible/modules/source_control/gitlab_runner.py", line 261, in findRunner
      File "/usr/local/lib/python3.7/dist-packages/gitlab/cli.py", line 43, in wrapped_f
        return f(*args, **kwargs)
      File "/usr/local/lib/python3.7/dist-packages/gitlab/exceptions.py", line 261, in wrapped_f
        raise error(e.error_message, e.response_code, e.response_body)
    gitlab.exceptions.GitlabListError: 403: 403 Forbidden
  module_stdout: ''
  msg: |-
    MODULE FAILURE
    See stdout/stderr for the exact error
  rc: 1
```

Or you get token not available:
```
fatal: [runner1]: FAILED! => changed=false
  msg: 'AnsibleUndefinedVariable: ''dict object'' has no attribute ''token'''
```
---
@Shrews migrated.


Discussion:
Files identified in the description:
* [`plugins/modules/source_control/gitlab/gitlab_runner.py`](https://github.com/ansible-collections/community.general/blob/master/plugins/modules/source_control/gitlab/gitlab_runner.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @Lunik @SamyCoenen @Shaps @dj-wasabi @marwatk @waheedi
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
@GhostLyrics can you try again?
It's working for me with Ansible 2.9.2 
```
samy ) ansible -m gitlab_runner -a  "registration_token=REDACTED api_token=REDACTED description=my-gitlab-run
ner-ansible api_url=https://gitlab.com" localhost --connection=local
[WARNING]: Platform linux on host localhost is using the discovered Python interpreter at /usr/bin/python, but future installation
of another Python interpreter could change this. See              
https://docs.ansible.com/ansible/2.9/reference_appendices/interpreter_discovery.html for more information.
                                                                  
localhost | CHANGED => {                                                                                                             
    "ansible_facts": {                                                                                                               
        "discovered_interpreter_python": "/usr/bin/python"        
    },                                                                                                                               
    "changed": true,                                                                                                                 
    "msg": "Successfully created or updated the runner my-gitlab-runner-ansible",
    "runner": {                
        "id": 2208307,                                            
        "token": "REDACTED"
    }                                                             
} 
```
@SamyCoenen with your input I was able to narrow the issue down.

`token` is available when the runner is created. It is not available when the runner is updated (ansible=changed) or found to be correct already (ansible=ok).
I achieve a workaround by reading the token IDs from the toml file if the file exists.
This might be related to #484 (unit tests for gitlab_runner do not work with latest gitlab-python dependency).
@SamyCoenen see my comment on the old issue [here](https://github.com/ansible/ansible/issues/67159#issuecomment-590357692). Runner registration works in <=2.9.2 but breaks with a 403 in >2.9.2 because of a patch to request _all_ runners which requires an admin-level token.
@bbatliner do I understand you correctly that the change in PR #611 would also break the gitlab_runners inventory plugin for users without an admin token?
@felixfontein I judge that very likely.
On an only slightly related role - would including `to_toml` and `from_toml` filters here help a bit?

I noticed https://github.com/ansible/ansible/issues/69120 with an advice to use community `sivel/toiletwater` however its description wasn't very reassuring ;-)
@weakcamel haha! I have no idea whether they would help here, but I guess we would accept such filters in community.general if someone wants to provide them.
> have no idea whether they would help here

I was under the impression that the gitlab-runner module handles `config.toml` as well which means having a TOML filter could be helpful... but after digging deeper turned out I was wrong. Never mind then :-)
cc @scodeman @zanssa
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/plugins/modules/source_control/gitlab/gitlab_runner.py b/plugins/modules/source_control/gitlab/gitlab_runner.py
--- a/plugins/modules/source_control/gitlab/gitlab_runner.py
+++ b/plugins/modules/source_control/gitlab/gitlab_runner.py
@@ -57,6 +57,12 @@
       - The registration token is used to register new runners.
     required: True
     type: str
+  owned:
+    description:
+      - Searches only runners available to the user when searching for existing, when false admin token required.
+    default: no
+    type: bool
+    version_added: 2.0.0
   active:
     description:
       - Define if the runners is immediately active after creation.
@@ -114,6 +120,14 @@
     api_token: "{{ access_token }}"
     description: Docker Machine t1
     state: absent
+
+- name: Delete an owned runner as a non-admin
+  community.general.gitlab_runner:
+    api_url: https://gitlab.example.com/
+    api_token: "{{ access_token }}"
+    description: Docker Machine t1
+    owned: yes
+    state: absent
 '''
 
 RETURN = '''
@@ -246,18 +260,28 @@ def updateRunner(self, runner, arguments):
     '''
     @param description Description of the runner
     '''
-    def findRunner(self, description):
-        runners = self._gitlab.runners.all(as_list=False)
+    def findRunner(self, description, owned=False):
+        if owned:
+            runners = self._gitlab.runners.list(as_list=False)
+        else:
+            runners = self._gitlab.runners.all(as_list=False)
+
         for runner in runners:
-            if (runner['description'] == description):
-                return self._gitlab.runners.get(runner['id'])
+            # python-gitlab 2.2 through at least 2.5 returns a list of dicts for list() instead of a Runner
+            # object, so we need to handle both
+            if hasattr(runner, "description"):
+                if (runner.description == description):
+                    return self._gitlab.runners.get(runner.id)
+            else:
+                if (runner['description'] == description):
+                    return self._gitlab.runners.get(runner['id'])
 
     '''
     @param description Description of the runner
     '''
-    def existsRunner(self, description):
+    def existsRunner(self, description, owned=False):
         # When runner exists, object will be stored in self.runnerObject.
-        runner = self.findRunner(description)
+        runner = self.findRunner(description, owned)
 
         if runner:
             self.runnerObject = runner
@@ -279,6 +303,7 @@ def main():
         api_token=dict(type='str', no_log=True),
         description=dict(type='str', required=True, aliases=["name"]),
         active=dict(type='bool', default=True),
+        owned=dict(type='bool', default=False),
         tag_list=dict(type='list', default=[]),
         run_untagged=dict(type='bool', default=True),
         locked=dict(type='bool', default=False),
@@ -304,6 +329,7 @@ def main():
     )
 
     state = module.params['state']
+    owned = module.params['owned']
     runner_description = module.params['description']
     runner_active = module.params['active']
     tag_list = module.params['tag_list']
@@ -319,7 +345,7 @@ def main():
     gitlab_instance = gitlabAuthentication(module)
 
     gitlab_runner = GitLabRunner(module, gitlab_instance)
-    runner_exists = gitlab_runner.existsRunner(runner_description)
+    runner_exists = gitlab_runner.existsRunner(runner_description, owned)
 
     if state == 'absent':
         if runner_exists:


Test patch:
diff --git a/tests/unit/plugins/modules/source_control/gitlab/gitlab.py b/tests/unit/plugins/modules/source_control/gitlab/gitlab.py
--- a/tests/unit/plugins/modules/source_control/gitlab/gitlab.py
+++ b/tests/unit/plugins/modules/source_control/gitlab/gitlab.py
@@ -524,9 +524,14 @@ def resp_delete_project_hook(url, request):
 '''
 
 
-@urlmatch(scheme="http", netloc="localhost", path="/api/v4/runners/all", method="get")
+@urlmatch(scheme="http", netloc="localhost", path=r'/api/v4/runners/all$', method="get")
 def resp_find_runners_all(url, request):
-    headers = {'content-type': 'application/json'}
+    headers = {'content-type': 'application/json',
+               "X-Page": 1,
+               "X-Next-Page": 2,
+               "X-Per-Page": 1,
+               "X-Total-Pages": 1,
+               "X-Total": 2}
     content = ('[{"active": true,"description": "test-1-20150125","id": 1,'
                '"is_shared": false,"ip_address": "127.0.0.1","name": null,'
                '"online": true,"status": "online"},{"active": true,'
@@ -536,7 +541,7 @@ def resp_find_runners_all(url, request):
     return response(200, content, headers, None, 5, request)
 
 
-@urlmatch(scheme="http", netloc="localhost", path="/api/v4/runners", method="get")
+@urlmatch(scheme="http", netloc="localhost", path=r'/api/v4/runners$', method="get")
 def resp_find_runners_list(url, request):
     headers = {'content-type': 'application/json',
                "X-Page": 1,
@@ -544,16 +549,33 @@ def resp_find_runners_list(url, request):
                "X-Per-Page": 1,
                "X-Total-Pages": 1,
                "X-Total": 2}
-    content = ('[{"active": true,"description": "test-1-20150125","id": 1,'
+    content = ('[{"active": true,"description": "test-1-20201214","id": 1,'
                '"is_shared": false,"ip_address": "127.0.0.1","name": null,'
                '"online": true,"status": "online"},{"active": true,'
-               '"description": "test-2-20150125","id": 2,"ip_address": "127.0.0.1",'
+               '"description": "test-2-20201214","id": 2,"ip_address": "127.0.0.1",'
+               '"is_shared": false,"name": null,"online": false,"status": "offline"}]')
+    content = content.encode("utf-8")
+    return response(200, content, headers, None, 5, request)
+
+
+@urlmatch(scheme="http", netloc="localhost", path=r'/api/v4/runners/1$', method="put")
+def resp_update_runner(url, request):
+    headers = {'content-type': 'application/json',
+               "X-Page": 1,
+               "X-Next-Page": 2,
+               "X-Per-Page": 1,
+               "X-Total-Pages": 1,
+               "X-Total": 2}
+    content = ('[{"active": true,"description": "test-1-20201214","id": 1,'
+               '"is_shared": false,"ip_address": "127.0.0.1","name": null,'
+               '"online": true,"status": "online"},{"active": true,'
+               '"description": "test-2-20201214","id": 2,"ip_address": "127.0.0.1",'
                '"is_shared": false,"name": null,"online": false,"status": "offline"}]')
     content = content.encode("utf-8")
     return response(200, content, headers, None, 5, request)
 
 
-@urlmatch(scheme="http", netloc="localhost", path="/api/v4/runners/1", method="get")
+@urlmatch(scheme="http", netloc="localhost", path=r'/api/v4/runners/1$', method="get")
 def resp_get_runner(url, request):
     headers = {'content-type': 'application/json'}
     content = ('{"active": true,"description": "test-1-20150125","id": 1,'
@@ -563,7 +585,7 @@ def resp_get_runner(url, request):
     return response(200, content, headers, None, 5, request)
 
 
-@urlmatch(scheme="http", netloc="localhost", path="/api/v4/runners", method="post")
+@urlmatch(scheme="http", netloc="localhost", path=r'/api/v4/runners$', method="post")
 def resp_create_runner(url, request):
     headers = {'content-type': 'application/json'}
     content = ('{"active": true,"description": "test-1-20150125","id": 1,'
@@ -573,7 +595,7 @@ def resp_create_runner(url, request):
     return response(201, content, headers, None, 5, request)
 
 
-@urlmatch(scheme="http", netloc="localhost", path="/api/v4/runners/1", method="delete")
+@urlmatch(scheme="http", netloc="localhost", path=r'/api/v4/runners/1$', method="delete")
 def resp_delete_runner(url, request):
     headers = {'content-type': 'application/json'}
     content = ('{}')
diff --git a/tests/unit/plugins/modules/source_control/gitlab/test_gitlab_runner.py b/tests/unit/plugins/modules/source_control/gitlab/test_gitlab_runner.py
--- a/tests/unit/plugins/modules/source_control/gitlab/test_gitlab_runner.py
+++ b/tests/unit/plugins/modules/source_control/gitlab/test_gitlab_runner.py
@@ -21,6 +21,7 @@ def _dummy(x):
 try:
     from .gitlab import (GitlabModuleTestCase,
                          python_version_match_requirement,
+                         resp_find_runners_all,
                          resp_find_runners_list, resp_get_runner,
                          resp_create_runner, resp_delete_runner)
 
@@ -50,9 +51,9 @@ def setUp(self):
 
         self.moduleUtil = GitLabRunner(module=self.mock_module, gitlab_instance=self.gitlab_instance)
 
-    @with_httmock(resp_find_runners_list)
+    @with_httmock(resp_find_runners_all)
     @with_httmock(resp_get_runner)
-    def test_runner_exist(self):
+    def test_runner_exist_all(self):
         rvalue = self.moduleUtil.existsRunner("test-1-20150125")
 
         self.assertEqual(rvalue, True)
@@ -61,6 +62,17 @@ def test_runner_exist(self):
 
         self.assertEqual(rvalue, False)
 
+    @with_httmock(resp_find_runners_list)
+    @with_httmock(resp_get_runner)
+    def test_runner_exist_owned(self):
+        rvalue = self.moduleUtil.existsRunner("test-1-20201214", True)
+
+        self.assertEqual(rvalue, True)
+
+        rvalue = self.moduleUtil.existsRunner("test-3-00000000", True)
+
+        self.assertEqual(rvalue, False)
+
     @with_httmock(resp_create_runner)
     def test_create_runner(self):
         runner = self.moduleUtil.createRunner({"token": "token", "description": "test-1-20150125"})
@@ -68,7 +80,7 @@ def test_create_runner(self):
         self.assertEqual(type(runner), Runner)
         self.assertEqual(runner.description, "test-1-20150125")
 
-    @with_httmock(resp_find_runners_list)
+    @with_httmock(resp_find_runners_all)
     @with_httmock(resp_get_runner)
     def test_update_runner(self):
         runner = self.moduleUtil.findRunner("test-1-20150125")
@@ -84,7 +96,7 @@ def test_update_runner(self):
         self.assertEqual(changed, False)
         self.assertEqual(newRunner.description, "Runner description")
 
-    @with_httmock(resp_find_runners_list)
+    @with_httmock(resp_find_runners_all)
     @with_httmock(resp_get_runner)
     @with_httmock(resp_delete_runner)
     def test_delete_runner(self):
diff --git a/tests/unit/requirements.txt b/tests/unit/requirements.txt
--- a/tests/unit/requirements.txt
+++ b/tests/unit/requirements.txt
@@ -12,7 +12,7 @@ linode-python  # APIv3
 linode_api4 ; python_version > '2.6'  # APIv4
 
 # requirement for the gitlab module
-python-gitlab < 2.3.0  # version 2.3.0 makes gitlab_runner tests fail
+python-gitlab
 httmock
 
 # requirement for kubevirt modules