Repository: ansible/ansible

Issue:
github_deploy_key removal giving key already exists error
##### ISSUE TYPE
 - Bug Report

##### COMPONENT NAME
github_deploy_key

##### ANSIBLE VERSION
<!--- Paste verbatim output from "ansible --version" between quotes below -->
```
ansible 2.5.0rc1
  config file = /var/www/deployment/irflow-deploy/ansible.cfg
  configured module search path = [u'/var/www/deployment/irflow-deploy/plugins/modules', u'/usr/share/ansible']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Aug  4 2017, 00:39:18) [GCC 4.8.5 20150623 (Red Hat 4.8.5-16)]
```
also tried 2.4.2 and 2.4.1
##### CONFIGURATION
```
ANSIBLE_FORCE_COLOR(/var/www/deployment/irflow-deploy/ansible.cfg) = True
ANSIBLE_NOCOWS(/var/www/deployment/irflow-deploy/ansible.cfg) = False
ANSIBLE_PIPELINING(/var/www/deployment/irflow-deploy/ansible.cfg) = True
ANSIBLE_SSH_ARGS(/var/www/deployment/irflow-deploy/ansible.cfg) = -o ForwardAgent=yes
ANSIBLE_SSH_CONTROL_PATH(/var/www/deployment/irflow-deploy/ansible.cfg) = config/tmp/ssh/%%h-%%p-%%r
COMMAND_WARNINGS(/var/www/deployment/irflow-deploy/ansible.cfg) = True
DEFAULT_ACTION_PLUGIN_PATH(/var/www/deployment/irflow-deploy/ansible.cfg) = [u'/var/www/deployment/irflow-deploy/plugins/ac
DEFAULT_CALLBACK_PLUGIN_PATH(/var/www/deployment/irflow-deploy/ansible.cfg) = [u'/var/www/deployment/irflow-deploy/plugins/
DEFAULT_CONNECTION_PLUGIN_PATH(/var/www/deployment/irflow-deploy/ansible.cfg) = [u'/var/www/deployment/irflow-deploy/plugin
DEFAULT_FILTER_PLUGIN_PATH(/var/www/deployment/irflow-deploy/ansible.cfg) = [u'/var/www/deployment/irflow-deploy/plugins/fi
DEFAULT_FORCE_HANDLERS(/var/www/deployment/irflow-deploy/ansible.cfg) = False
DEFAULT_FORKS(/var/www/deployment/irflow-deploy/ansible.cfg) = 20
DEFAULT_GATHERING(/var/www/deployment/irflow-deploy/ansible.cfg) = smart
DEFAULT_HASH_BEHAVIOUR(/var/www/deployment/irflow-deploy/ansible.cfg) = replace
DEFAULT_INTERNAL_POLL_INTERVAL(/var/www/deployment/irflow-deploy/ansible.cfg) = 0.001
DEFAULT_LOG_PATH(/var/www/deployment/irflow-deploy/ansible.cfg) = /var/www/deployment/irflow-deploy/config/tmp/ansible.log
DEFAULT_LOOKUP_PLUGIN_PATH(/var/www/deployment/irflow-deploy/ansible.cfg) = [u'/var/www/deployment/irflow-deploy/plugins/lo
DEFAULT_MANAGED_STR(/var/www/deployment/irflow-deploy/ansible.cfg) = Ansible managed - ansible-skel
DEFAULT_MODULE_LANG(/var/www/deployment/irflow-deploy/ansible.cfg) = en_US.UTF-8
DEFAULT_MODULE_PATH(/var/www/deployment/irflow-deploy/ansible.cfg) = [u'/var/www/deployment/irflow-deploy/plugins/modules',
DEFAULT_MODULE_SET_LOCALE(/var/www/deployment/irflow-deploy/ansible.cfg) = True
DEFAULT_NO_LOG(/var/www/deployment/irflow-deploy/ansible.cfg) = False
DEFAULT_NO_TARGET_SYSLOG(/var/www/deployment/irflow-deploy/ansible.cfg) = True
DEFAULT_POLL_INTERVAL(/var/www/deployment/irflow-deploy/ansible.cfg) = 15
DEFAULT_REMOTE_PORT(/var/www/deployment/irflow-deploy/ansible.cfg) = 22
DEFAULT_REMOTE_USER(/var/www/deployment/irflow-deploy/ansible.cfg) = root
DEFAULT_ROLES_PATH(/var/www/deployment/irflow-deploy/ansible.cfg) = [u'/var/www/deployment/irflow-deploy/roles/vendors', u'
DEFAULT_SCP_IF_SSH(/var/www/deployment/irflow-deploy/ansible.cfg) = False
DEFAULT_STRATEGY(/var/www/deployment/irflow-deploy/ansible.cfg) = free
DEFAULT_STRATEGY_PLUGIN_PATH(/var/www/deployment/irflow-deploy/ansible.cfg) = [u'/var/www/deployment/irflow-deploy/plugins/
DEFAULT_TIMEOUT(/var/www/deployment/irflow-deploy/ansible.cfg) = 5
DEFAULT_TRANSPORT(/var/www/deployment/irflow-deploy/ansible.cfg) = ssh
DEFAULT_UNDEFINED_VAR_BEHAVIOR(/var/www/deployment/irflow-deploy/ansible.cfg) = True
DEFAULT_VARS_PLUGIN_PATH(/var/www/deployment/irflow-deploy/ansible.cfg) = [u'/var/www/deployment/irflow-deploy/plugins/vars
DEFAULT_VAULT_PASSWORD_FILE(/var/www/deployment/irflow-deploy/ansible.cfg) = /home/irflow/.ansible/default.vault
DEPRECATION_WARNINGS(/var/www/deployment/irflow-deploy/ansible.cfg) = False
DISPLAY_ARGS_TO_STDOUT(/var/www/deployment/irflow-deploy/ansible.cfg) = False
DISPLAY_SKIPPED_HOSTS(/var/www/deployment/irflow-deploy/ansible.cfg) = False
HOST_KEY_CHECKING(/var/www/deployment/irflow-deploy/ansible.cfg) = True
RETRY_FILES_ENABLED(/var/www/deployment/irflow-deploy/ansible.cfg) = True
RETRY_FILES_SAVE_PATH(/var/www/deployment/irflow-deploy/ansible.cfg) = /var/www/deployment/irflow-deploy/config/tmp/retry
(END)
```

##### OS / ENVIRONMENT
CentOS 7.4

##### SUMMARY
Unable to delete github deploy key via "absent". 

##### STEPS TO REPRODUCE
After successfully creating a github deploykey unable to delete it. Seems the API might be configured wrong. In order for this error to show itself, you need to have > 30 deploy keys. Github's API returns them paginated in groups of 30.  

```
Play: 

- name: remove deploy key in test runs
      github_deploy_key:
        owner: "<someorg>"
        repository: "integrations"
        name: "{{ <some>_customer_name }}_{{server_environment}}"
        key: "ssh-rsa AAAAB3N...
        force: yes
        token: "{{ github_access_token }}"
        state: absent
      when: server_environment  == "test"
```

##### EXPECTED RESULTS
I expect to see the deploy key "absent" or be removed successfully

##### ACTUAL RESULTS

```
Read vars_file '../host_vars/all.yml'
Read vars_file '../host_vars/vault.yml'
Read vars_file '../host_vars/staging.yml'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/source_control/github_deploy_key.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: irflow
<127.0.0.1> EXEC /bin/sh -c 'sudo -H -S -n -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-....; /usr/bin/python2'"'"' && sleep 0'

TASK [remove deploy key in test runs] *************************************************************************************
task path: deploy/playbooks/setup_prep_scrap.yml:77
ok: [localhost] => {
    "changed": false, 
    "invocation": {
        "module_args": {
            "force": true, 
            "key": "ssh-rsa AAAAB3N...
            "name": "devops_testing_test", 
            "otp": null, 
            "owner": "<org>", 
            "password": null, 
            "read_only": true, 
            "repo": "integrations", 
            "repository": "integrations", 
            "state": "absent", 
            "token": "VALUE_SPECIFIED_IN_NO_LOG_PARAMETER", 
            "username": null
        }
    }, 
    "msg": "Deploy key already exists"
}
```



Discussion:
Files identified in the description:
* [lib/ansible/modules/source_control/github_deploy_key.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/source_control/github_deploy_key.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @bincyber
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->
Looking at github_deploy_key.py, I think the module is trying to always add a key even for 'state=absent'

https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/source_control/github_deploy_key.py#L314

If I read that correctly, the key will be deleted, then line 314 will run anyway and try to add a
new key and cause the error missing.

ie, looks like there is a missing else clause in there.

pinging @bincyber 
After reading the source code, somehow the code is getting an error message I don't think we should get for state=absent. Because that code is only accessible via the [add_new_key function](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/source_control/github_deploy_key.py#L237)

```
"msg": "Deploy key already exists"
```

Taking this a step further - it looks like if we don't get a http 204 we do print the error, but we don't exit, resulting in line 314 getting executed. 
Thanks for reporting this issue. I just tested this and I'm not able to duplicate what you're experiencing. 

When `state="absent"`, L314 should not be executed because the module exits here: https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/source_control/github_deploy_key.py#L250

Adding an `else` clause after L250 won't make any difference:
```
if status_code == 204:
    if self.state == 'absent':
        self.module.exit_json(changed=True, msg="Deploy key successfully deleted", id=key_id)
    elif self.state == "present":
        return
```

The reason it doesn't exit for `state="present"` is to support `force=True` which requires deleting the existing key and re-adding it.

L314 could be modified to ensure `state=present`:
```
if state == "present":
    deploy_key.add_new_key({"title": name, "key": key, "read_only": read_only})
```
@bincyber Are there instructions on how I could test? And maybe debug this in pycharm...  It doesnt make sense to me why I’m getting neither a key removed or key remove failed error - but a 422 - and if I understand correctly - the code should never fire... e.g. I should never get a 422 if state is absent. 
@bincyber - I figured out how to debug this. What it appears is happening in this situation is the github API is not returning the key created. Will get back with more soon. Trying some other stuff. 

@bincyber Update: This is a pagination issue. If there are more than 30 deploy keys, the current code will never see it. I updated the issue it self also.

I still think the code should also not create a new key if it's not explicitly requested. (e.g. your suggestion to modify 314 to state present. )
@punkrokk thanks for debugging the issue further. I will modify the code to support pagination.
Thanks man. Lmk if you need help. We started a PR but realized it needed some refactoring we couldn’t do today. 

Cheers,

JP

> On Feb 27, 2018, at 8:39 PM, Ali <notifications@github.com> wrote:
> 
> @punkrokk thanks for debugging the issue further. I will modify the code to support pagination.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.
> 


Resolving patch:
diff --git a/lib/ansible/modules/source_control/github/github_deploy_key.py b/lib/ansible/modules/source_control/github/github_deploy_key.py
--- a/lib/ansible/modules/source_control/github/github_deploy_key.py
+++ b/lib/ansible/modules/source_control/github/github_deploy_key.py
@@ -69,8 +69,6 @@
     description:
       - The 6 digit One Time Password for 2-Factor Authentication. Required together with I(username) and I(password).
     aliases: ['2fa_token']
-requirements:
-   - python-requests
 notes:
    - "Refer to GitHub's API documentation here: https://developer.github.com/v3/repos/keys/."
 '''
@@ -152,82 +150,89 @@
     sample: 24381901
 '''
 
-import json
-
 from ansible.module_utils.basic import AnsibleModule
 from ansible.module_utils.urls import fetch_url
+from re import findall
 
 
 class GithubDeployKey(object):
-    def __init__(self, module=None, url=None, state=None, username=None, password=None, token=None, otp=None):
+    def __init__(self, module):
         self.module = module
-        self.url = url
-        self.state = state
-        self.username = username
-        self.password = password
-        self.token = token
-        self.otp = otp
-        self.timeout = 5
-        self.auth = None
-        self.headers = None
-
-        if username is not None and password is not None:
-            self.module.params['url_username'] = module.params['username']
-            self.module.params['url_password'] = module.params['password']
+
+        self.name = module.params['name']
+        self.key = module.params['key']
+        self.state = module.params['state']
+        self.read_only = module.params.get('read_only', True)
+        self.force = module.params.get('force', False)
+        self.username = module.params.get('username', None)
+        self.password = module.params.get('password', None)
+        self.token = module.params.get('token', None)
+        self.otp = module.params.get('otp', None)
+
+    @property
+    def url(self):
+        owner = self.module.params['owner']
+        repo = self.module.params['repo']
+        return "https://api.github.com/repos/{0}/{1}/keys".format(owner, repo)
+
+    @property
+    def headers(self):
+        if self.username is not None and self.password is not None:
+            self.module.params['url_username'] = self.username
+            self.module.params['url_password'] = self.password
             self.module.params['force_basic_auth'] = True
             if self.otp is not None:
-                self.headers = {"X-GitHub-OTP": self.otp}
+                return {"X-GitHub-OTP": self.otp}
+        elif self.token is not None:
+            return {"Authorization": "token {0}".format(self.token)}
         else:
-            self.headers = {"Authorization": "token {0}".format(self.token)}
+            return None
 
-    def get_existing_key(self, key, title, force):
-        resp, info = fetch_url(self.module, self.url, headers=self.headers, method="GET")
+    def paginate(self, url):
+        while url:
+            resp, info = fetch_url(self.module, url, headers=self.headers, method="GET")
 
-        status_code = info["status"]
+            if info["status"] == 200:
+                yield self.module.from_json(resp.read())
+
+                links = {}
+                for x, y in findall(r'<([^>]+)>;\s*rel="(\w+)"', info["link"]):
+                    links[y] = x
 
-        if status_code == 200:
-            response_body = json.loads(resp.read())
+                url = links.get('next')
+            else:
+                self.handle_error(method="GET", info=info)
 
-            if response_body:
-                for i in response_body:
+    def get_existing_key(self):
+        for keys in self.paginate(self.url):
+            if keys:
+                for i in keys:
                     existing_key_id = str(i["id"])
-                    if i["key"].split() == key.split()[:2]:
+                    if i["key"].split() == self.key.split()[:2]:
                         return existing_key_id
-                    elif i['title'] == title and force:
+                    elif i['title'] == self.name and self.force:
                         return existing_key_id
             else:
-                if self.state == 'absent':
-                    self.module.exit_json(changed=False, msg="Deploy key does not exist")
-                else:
-                    return None
-        elif status_code == 401:
-            self.module.fail_json(msg="Failed to connect to github.com due to invalid credentials", http_status_code=status_code)
-        elif status_code == 404:
-            self.module.fail_json(msg="GitHub repository does not exist", http_status_code=status_code)
-        else:
-            self.module.fail_json(msg="Failed to retrieve existing deploy keys", http_status_code=status_code)
+                return None
+
+    def add_new_key(self):
+        request_body = {"title": self.name, "key": self.key, "read_only": self.read_only}
 
-    def add_new_key(self, request_body):
-        resp, info = fetch_url(self.module, self.url, data=json.dumps(request_body), headers=self.headers, method="POST")
+        resp, info = fetch_url(self.module, self.url, data=self.module.jsonify(request_body), headers=self.headers, method="POST", timeout=30)
 
         status_code = info["status"]
 
         if status_code == 201:
-            response_body = json.loads(resp.read())
+            response_body = self.module.from_json(resp.read())
             key_id = response_body["id"]
             self.module.exit_json(changed=True, msg="Deploy key successfully added", id=key_id)
-        elif status_code == 401:
-            self.module.fail_json(msg="Failed to connect to github.com due to invalid credentials", http_status_code=status_code)
-        elif status_code == 404:
-            self.module.fail_json(msg="GitHub repository does not exist", http_status_code=status_code)
         elif status_code == 422:
             self.module.exit_json(changed=False, msg="Deploy key already exists")
         else:
-            err = info["body"]
-            self.module.fail_json(msg="Failed to add deploy key", http_status_code=status_code, error=err)
+            self.handle_error(method="POST", info=info)
 
     def remove_existing_key(self, key_id):
-        resp, info = fetch_url(self.module, self.url + "/{0}".format(key_id), headers=self.headers, method="DELETE")
+        resp, info = fetch_url(self.module, "{0}/{1}".format(self.url, key_id), headers=self.headers, method="DELETE")
 
         status_code = info["status"]
 
@@ -235,11 +240,28 @@ def remove_existing_key(self, key_id):
             if self.state == 'absent':
                 self.module.exit_json(changed=True, msg="Deploy key successfully deleted", id=key_id)
         else:
-            self.module.fail_json(msg="Failed to delete existing deploy key", id=key_id, http_status_code=status_code)
+            self.handle_error(method="DELETE", info=info, key_id=key_id)
 
+    def handle_error(self, method, info, key_id=None):
+        status_code = info['status']
+        body = info.get('body')
+        if body:
+            err = self.module.from_json(body)['message']
 
-def main():
+        if status_code == 401:
+            self.module.fail_json(msg="Failed to connect to github.com due to invalid credentials", http_status_code=status_code, error=err)
+        elif status_code == 404:
+            self.module.fail_json(msg="GitHub repository does not exist", http_status_code=status_code, error=err)
+        else:
+            if method == "GET":
+                self.module.fail_json(msg="Failed to retrieve existing deploy keys", http_status_code=status_code, error=err)
+            elif method == "POST":
+                self.module.fail_json(msg="Failed to add deploy key", http_status_code=status_code, error=err)
+            elif method == "DELETE":
+                self.module.fail_json(msg="Failed to delete existing deploy key", id=key_id, http_status_code=status_code, error=err)
 
+
+def main():
     module = AnsibleModule(
         argument_spec=dict(
             owner=dict(required=True, type='str', aliases=['account', 'organization']),
@@ -267,37 +289,26 @@ def main():
         supports_check_mode=True,
     )
 
-    owner = module.params['owner']
-    repo = module.params['repo']
-    name = module.params['name']
-    key = module.params['key']
-    state = module.params['state']
-    read_only = module.params.get('read_only', True)
-    force = module.params.get('force', False)
-    username = module.params.get('username', None)
-    password = module.params.get('password', None)
-    token = module.params.get('token', None)
-    otp = module.params.get('otp', None)
-
-    GITHUB_API_URL = "https://api.github.com/repos/{0}/{1}/keys".format(owner, repo)
-
-    deploy_key = GithubDeployKey(module, GITHUB_API_URL, state, username, password, token, otp)
+    deploy_key = GithubDeployKey(module)
 
     if module.check_mode:
-        key_id = deploy_key.get_existing_key(key, name, force)
-        if state == "present" and key_id is None:
+        key_id = deploy_key.get_existing_key()
+        if deploy_key.state == "present" and key_id is None:
             module.exit_json(changed=True)
-        elif state == "present" and key_id is not None:
+        elif deploy_key.state == "present" and key_id is not None:
             module.exit_json(changed=False)
 
     # to forcefully modify an existing key, the existing key must be deleted first
-    if state == 'absent' or force:
-        key_id = deploy_key.get_existing_key(key, name, force)
+    if deploy_key.state == 'absent' or deploy_key.force:
+        key_id = deploy_key.get_existing_key()
 
         if key_id is not None:
             deploy_key.remove_existing_key(key_id)
+        elif deploy_key.state == 'absent':
+            module.exit_json(changed=False, msg="Deploy key does not exist")
 
-    deploy_key.add_new_key({"title": name, "key": key, "read_only": read_only})
+    if deploy_key.state == "present":
+        deploy_key.add_new_key()
 
 
 if __name__ == '__main__':


Test patch: