Repository: zulip/zulip

Issue:
Error with AWX 7.0.0
Hello.
I'm testing integration between **AWX 7.0.0** (Ansible Tower) by sending notifications in **Zulip 2.0.4**.
During testing, I encounter an error from Ansible :

![image](https://user-images.githubusercontent.com/2016950/66895321-c5baf980-eff2-11e9-8649-06e5f2433133.png)

And I immediatly receive an email warning from Zulip with the following content :

```Logger root, from module zerver.middleware line 291:
Error generated by Ansible (user42@zulip.******.**) on zulip.******.** deployment

Traceback (most recent call last):
  File "/srv/zulip-venv-cache/ebe617662f96425113e5a75344bbe5a0593f634a/zulip-py3-venv/lib/python3.7/site-packages/django/core/handlers/base.py", line 185, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/srv/zulip-venv-cache/ebe617662f96425113e5a75344bbe5a0593f634a/zulip-py3-venv/lib/python3.7/site-packages/django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "./zerver/lib/request.py", line 289, in _wrapped_view_func
    return view_func(request, *args, **kwargs)
  File "./zerver/decorator.py", line 375, in _wrapped_func_arguments
    raise err
  File "./zerver/decorator.py", line 361, in _wrapped_func_arguments
    return view_func(request, user_profile, *args, **kwargs)
  File "./zerver/lib/request.py", line 289, in _wrapped_view_func
    return view_func(request, *args, **kwargs)
  File "./zerver/webhooks/ansibletower/view.py", line 27, in api_ansibletower_webhook
    body = get_body(payload)
  File "./zerver/webhooks/ansibletower/view.py", line 34, in get_body
    if (payload['friendly_name'] == 'Job'):
KeyError: 'friendly_name'


Deployed code:
- git: 2.0.0-2546-ga1fa0b011
- ZULIP_VERSION: 2.0.4+git


Request info:
- path: /api/v1/external/ansibletower
- POST: {}
- REMOTE_ADDR: "['10.10.36.6']"
- QUERY_STRING: "['api_key=******&topic=******&stream=******&topic=******"
- SERVER_NAME: "['']"
```

I have already disable the "Disable SSL checking" but it seems also that the new version of AWX (the 7.0.0) contains new options for webhook like "HTTP Headers" and "HTTP Method".

![image](https://user-images.githubusercontent.com/2016950/66895546-5db8e300-eff3-11e9-9954-8e1df5485e17.png)

Note that I have already notifications from GitLab so the notification service works in my self-hosted Zulip configuration.


Discussion:
Hello @zulip/server-integrations members, this issue was labeled with the "area: integrations" label, so you may want to check it out!

<!-- areaLabelAddition -->
@Hypro999 do you have time to look into this?  It may be that our Ansible Tower integration needs to be updated for upstream API changes.
@zulipbot claim
@zulipbot abandon
@sulian if it's okay with you, could you please share a (redacted) version of the payload that you received that caused this?

If you're not sure of how, here's how you can do this in 5 easy steps:
1. Go to https://webhook.site
2. The site will give you a target URL. Something like `https://webhook.site/fec9a205-ecf8-422e-97a2-d42d16048e1d` - copy this to your clipboard.
3. Then go to your AWX installation and replace the URL field with the target URL that https://webhook.site gave you (paste it here).
4. Trigger the webhook event.
5. Now if you go back to webhook.site it will show you the payload that it just received.

If you could just redact that (remove personal info by replacing with `*`s for example) and post the results here, I would greatly appreciate it since setting up a fresh AWX installation seems like it'll be **really** tedious (a worst case measure if at all avoidable). If you'd like to refrain from doing this, then that's totally fine and understandable - so the decision is up to you.

Also, sorry for the really late response, I remember that I planned on doing this a little later after seeing the @-mention, but then it slipped my mind before I could get around to it.
@danielepesciallo that's great! Can you trigger a payload for events which involve a "job" so that we can see if the keys have been renamed/changed? Thanks for the help! 🙂
Finding change logs for the AWX project was quite a bit of hassle, but after a fair amount of digging around, I was able to find:
https://groups.google.com/forum/#!searchin/awx-project/7.0.0%7Csort:date/awx-project/lemzchJocjk/HAOXI0DnAAAJ

Posting here for future reference. I'll try to see what progress I can make on this issue over the course of this week (or whatever's left of it) - it may take a few days though. I may need to dig around a bit more and perhaps skim through their source code to better understand what's going on on their end and how they're sending webhook payloads since they sadly don't have any documentation for this apart from https://docs.ansible.com/ansible-tower/latest/html/userguide/notifications.html#webhook (which makes this a lot harder). I also need to determine exactly when they made this change so that we can have this integration maintain backwards compatibility.
Ok, so today I took the time to set up my own installation of Ansible + AWX so that I can take a closer look at what's going on and to test things out. Here are the installation details:
1. OS: Ubuntu 18.04.3 LTS x86_64
2. Ansible: ansible 2.5.1 (using python 2.7.17 by default)
3. AWX: awx 7.0.0. (commit: 4edfe7e5fc8b31b71e53091cb9271210fbee641b)

Then I ran the demo job template and I found that I was unable to reproduce the error condition that @sulian faced. Here's what I got:

When running the job:
```json
{
  "id": 5,
  "name": "Demo Job Template",
  "url": "https://towerhost/#/jobs/playbook/5",
  "created_by": "admin",
  "started": "2020-01-23T08:37:51.227591+00:00",
  "finished": null,
  "status": "running",
  "traceback": "",
  "inventory": "Demo Inventory",
  "project": "Demo Project",
  "playbook": "hello_world.yml",
  "credential": "Demo Credential",
  "limit": "",
  "extra_vars": "{}",
  "hosts": {},
  "friendly_name": "Job"
}
```

After successful completion of the job:
```json
{
  "id": 5,
  "name": "Demo Job Template",
  "url": "https://towerhost/#/jobs/playbook/5",
  "created_by": "admin",
  "started": "2020-01-23T08:37:51.227591+00:00",
  "finished": "2020-01-23T08:38:01.452808+00:00",
  "status": "successful",
  "traceback": "",
  "inventory": "Demo Inventory",
  "project": "Demo Project",
  "playbook": "hello_world.yml",
  "credential": "Demo Credential",
  "limit": "",
  "extra_vars": "{}",
  "hosts": {
    "localhost": {
      "failed": false,
      "changed": 0,
      "dark": 0,
      "failures": 0,
      "ok": 2,
      "processed": 1,
      "skipped": 0
    }
  },
  "friendly_name": "Job"
} 
```

So I'm not too sure of what's going on on your end.
Now, I've reconfigured my server to use the latest versions of both Ansible and AWX since @danielepesciallo that's what you seem to be using. Here are the new installation details:
    OS: Ubuntu 18.04.3 LTS x86_64
    Ansible: ansible 2.9.4 (using python 2.7.17 by default)
    AWX: awx 9.1.1 (commit: 4edfe7e5fc8b31b71e53091cb9271210fbee641b)

And the good news here (depending on how you look at it I guess) is that I was able to reproduce the same data (and thus the error).

When running the job:
```json
{
  "id": 1,
  "name": "Demo Job Template",
  "url": "https://towerhost/#/jobs/playbook/1",
  "created_by": "admin",
  "started": "2020-01-23T09:15:45.741213+00:00",
  "finished": null,
  "status": "running",
  "traceback": "",
  "inventory": "Demo Inventory",
  "project": "Demo Project",
  "playbook": "hello_world.yml",
  "credential": "Demo Credential",
  "limit": "",
  "extra_vars": "{}",
  "hosts": {}
}
```

After successful completion of the job:
```json
{
  "id": 1,
  "name": "Demo Job Template",
  "url": "https://towerhost/#/jobs/playbook/1",
  "created_by": "admin",
  "started": "2020-01-23T09:15:45.741213+00:00",
  "finished": "2020-01-23T09:15:49.729882+00:00",
  "status": "successful",
  "traceback": "",
  "inventory": "Demo Inventory",
  "project": "Demo Project",
  "playbook": "hello_world.yml",
  "credential": "Demo Credential",
  "limit": "",
  "extra_vars": "{}",
  "hosts": {
    "localhost": {
      "failed": false,
      "changed": 0,
      "dark": 0,
      "failures": 0,
      "ok": 2,
      "processed": 1,
      "skipped": 0,
      "rescued": 0,
      "ignored": 0
    }
  }
}
```
Just an update: I've tried out a few more versions of AWX to see if I can determine where the change happened and this issue seems to have been introduced from AWX v9.0.0 onward. Though sadly the release notes at https://groups.google.com/forum/#!searchin/awx-project/AWX$209.0.0%7Csort:date/awx-project/YZqcSmspqL0/tiyQqZhbBAAJ don't seem to ever suggest this change.
I think that the simplest and most efficient way to handle this would be to just use AWX's [custom notifications templates functionality](https://docs.ansible.com/ansible-tower/latest/html/userguide/notifications.html#create-a-notification-template) to re-introduce the missing key in the webhook payloads.

Particularly using the following template:
```json
{
    "id": "{{ job.id }}",
    "name": "{{ job.name }}",
    "friendly_name": "{{ job_friendly_name }}",
    "status": "{{ job.status }}",
    "url": "{{ url }}",
    "hosts": {}
}
```
should do the trick. Of course, this template is the bare minimum, any extra information that the recipient is interested in can be added, but these are the bare minimum keys.
Note: I effectively eliminated the hosts key in this example because after an hour or poking around, I found that reconstructing the hosts field might not be possible because AWX currently does not support using `host_name` in their custom notification templates (either that or I didn't get what "hosts" exactly are) despite having `host_status_counts`. Sources:
1. https://docs.ansible.com/ansible-tower/latest/html/userguide/notifications.html#create-a-notification-template
2. https://docs.ansible.com/ansible-tower/3.6.2/html/installandreference/notification_parameters_supported.html#ir-notifications-reference
3. https://github.com/ansible/awx/blob/b7a064b05d9724ad88b6fce4ec7c0f776a7c51bb/awx/main/models/jobs.py#L692

Sample response from AWX:
```json
{
  "id": "13",
  "name": "Demo Job Template",
  "friendly_name": "Job",
  "status": "running",
  "url": "https://towerhost/#/jobs/playbook/13",
  "hosts": {}
}
```

@danielepesciallo can you give this a try?
Also, @timabbott, can I get your opinions on this based on the presented findings?
I think we might be able to just stop using the `friendly_name` field from the payload and instead just use the `name` element -- that is potentially better output in any case.  And I think the conditionals we have in the code on `friendly_name` are mostly conditionals on whether a `hosts` element are there plus some duplicated code.  So we might be able to unify the templates as part of this.
> I think that the simplest and most efficient way to handle this would be to just use AWX's [custom notifications templates functionality](https://docs.ansible.com/ansible-tower/latest/html/userguide/notifications.html#create-a-notification-template) to re-introduce the missing key in the webhook payloads.
> 
> Particularly using the following template:
> 
> ```json
> {
>     "id": "{{ job.id }}",
>     "name": "{{ job.name }}",
>     "friendly_name": "{{ job_friendly_name }}",
>     "status": "{{ job.status }}",
>     "url": "{{ url }}",
>     "hosts": {}
> }
> ```
> 
> should do the trick. Of course, this template is the bare minimum, any extra information that the recipient is interested in can be added, but these are the bare minimum keys.
> Note: I effectively eliminated the hosts key in this example because after an hour or poking around, I found that reconstructing the hosts field might not be possible because AWX currently does not support using `host_name` in their custom notification templates (either that or I didn't get what "hosts" exactly are) despite having `host_status_counts`. Sources:
> 
> 1. https://docs.ansible.com/ansible-tower/latest/html/userguide/notifications.html#create-a-notification-template
> 2. https://docs.ansible.com/ansible-tower/3.6.2/html/installandreference/notification_parameters_supported.html#ir-notifications-reference
> 3. https://github.com/ansible/awx/blob/b7a064b05d9724ad88b6fce4ec7c0f776a7c51bb/awx/main/models/jobs.py#L692
> 
> Sample response from AWX:
> 
> ```json
> {
>   "id": "13",
>   "name": "Demo Job Template",
>   "friendly_name": "Job",
>   "status": "running",
>   "url": "https://towerhost/#/jobs/playbook/13",
>   "hosts": {}
> }
> ```
> 
> @danielepesciallo can you give this a try?
> Also, @timabbott, can I get your opinions on this based on the presented findings?

Hi @Hypro999,
I tried your trick and now i'm able to receive AWX Notification!

Thanks a lot!! 😄 

Daniele

Resolving patch:
diff --git a/zerver/webhooks/ansibletower/view.py b/zerver/webhooks/ansibletower/view.py
--- a/zerver/webhooks/ansibletower/view.py
+++ b/zerver/webhooks/ansibletower/view.py
@@ -30,8 +30,19 @@ def api_ansibletower_webhook(request: HttpRequest, user_profile: UserProfile,
     check_send_webhook_message(request, user_profile, subject, body)
     return json_success()
 
+def extract_friendly_name(payload: Dict[str, Any]) -> str:
+    tentative_job_name = payload.get("friendly_name", "")
+    if not tentative_job_name:
+        url = payload["url"]
+        segments = url.split("/")
+        tentative_job_name = segments[-3]
+        if tentative_job_name == "jobs":
+            tentative_job_name = "Job"
+    return tentative_job_name
+
 def get_body(payload: Dict[str, Any]) -> str:
-    if (payload['friendly_name'] == 'Job'):
+    friendly_name = extract_friendly_name(payload)
+    if (friendly_name == 'Job'):
         hosts_list_data = payload['hosts']
         hosts_data = []
         for host in payload['hosts']:
@@ -51,7 +62,7 @@ def get_body(payload: Dict[str, Any]) -> str:
 
         return ANSIBLETOWER_JOB_MESSAGE_TEMPLATE.format(
             name=payload['name'],
-            friendly_name=payload['friendly_name'],
+            friendly_name=friendly_name,
             id=payload['id'],
             url=payload['url'],
             status=status,
@@ -67,7 +78,7 @@ def get_body(payload: Dict[str, Any]) -> str:
 
         data = {
             "name": payload['name'],
-            "friendly_name": payload['friendly_name'],
+            "friendly_name": friendly_name,
             "id": payload['id'],
             "url": payload['url'],
             "status": status


Test patch:
diff --git a/zerver/webhooks/ansibletower/tests.py b/zerver/webhooks/ansibletower/tests.py
--- a/zerver/webhooks/ansibletower/tests.py
+++ b/zerver/webhooks/ansibletower/tests.py
@@ -55,6 +55,19 @@ def test_ansibletower_job_successful_message(self) -> None:
 
         self.send_and_test_stream_message('job_successful', expected_topic, expected_message)
 
+    def test_ansibletower_nine_job_successful_message(self) -> None:
+        """
+        Test to see if awx/ansibletower 9.x.x job successful notifications are
+        handled just as successfully as prior to 9.x.x.
+        """
+        expected_topic = "Demo Job Template"
+        expected_message = """
+Job: [#1 Demo Job Template](https://towerhost/#/jobs/playbook/1) was successful:
+* localhost: Success
+""".strip()
+
+        self.send_and_test_stream_message('job_complete_successful_awx_9.1.1', expected_topic, expected_message)
+
     def test_ansibletower_job_failed_message(self) -> None:
         """
         Tests if ansibletower job failed notification is handled correctly