Repository: ansible/ansible

Issue:
grafana annotation callback
##### ISSUE TYPE

 - Feature Idea

##### COMPONENT NAME

plugins/callback/grafana.py

##### ANSIBLE VERSION
```
ansible 2.4.2.0
  config file = None
  configured module search path = [u'/Users/remirey/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/remirey/.venv/reminder/lib/python2.7/site-packages/ansible
  executable location = /Users/remirey/.venv/reminder/bin/ansible
  python version = 2.7.13 (default, Apr  4 2017, 08:47:57) [GCC 4.2.1 Compatible Apple LLVM 8.1.0 (clang-802.0.38)]
```

##### CONFIGURATION
N/A

##### OS / ENVIRONMENT
N/A

##### SUMMARY

Grafana v4.6+ comes with a native annotation store and the ability to add annotation events directly from the graph panel or via the HTTP API.

An Ansible callback could publish global annotations to Grafana through the HTTP API when the callback is whitelisted.

The annotations are extremely useful to notify of an intervention through ansible playbook.
With the callback, each playbook execution would automatically imply a new annotation to be published on the grafana dashboard, allowing to see immediately on the monitoring side that something was done through playbook.

It could accelerate investigations and allow to associate metrics variations to a playbook execution.


Discussion:
Files identified in the description:
* [lib/ansible/plugins/callback/__init__.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/__init__.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
Files identified in the description:
* [lib/ansible/plugins/callback/__init__.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/__init__.py)
* [lib/ansible/plugins/callback/actionable.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/actionable.py)
* [lib/ansible/plugins/callback/context_demo.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/context_demo.py)
* [lib/ansible/plugins/callback/debug.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/debug.py)
* [lib/ansible/plugins/callback/default.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/default.py)
* [lib/ansible/plugins/callback/dense.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/dense.py)
* [lib/ansible/plugins/callback/foreman.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/foreman.py)
* [lib/ansible/plugins/callback/full_skip.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/full_skip.py)
* [lib/ansible/plugins/callback/hipchat.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/hipchat.py)
* [lib/ansible/plugins/callback/jabber.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/jabber.py)
* [lib/ansible/plugins/callback/json.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/json.py)
* [lib/ansible/plugins/callback/junit.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/junit.py)
* [lib/ansible/plugins/callback/log_plays.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/log_plays.py)
* [lib/ansible/plugins/callback/logentries.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/logentries.py)
* [lib/ansible/plugins/callback/logstash.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/logstash.py)
* [lib/ansible/plugins/callback/mail.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/mail.py)
* [lib/ansible/plugins/callback/minimal.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/minimal.py)
* [lib/ansible/plugins/callback/null.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/null.py)
* [lib/ansible/plugins/callback/oneline.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/oneline.py)
* [lib/ansible/plugins/callback/osx_say.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/osx_say.py)
* [lib/ansible/plugins/callback/profile_roles.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/profile_roles.py)
* [lib/ansible/plugins/callback/profile_tasks.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/profile_tasks.py)
* [lib/ansible/plugins/callback/selective.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/selective.py)
* [lib/ansible/plugins/callback/skippy.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/skippy.py)
* [lib/ansible/plugins/callback/slack.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/slack.py)
* [lib/ansible/plugins/callback/stderr.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/stderr.py)
* [lib/ansible/plugins/callback/syslog_json.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/syslog_json.py)
* [lib/ansible/plugins/callback/timer.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/timer.py)
* [lib/ansible/plugins/callback/tree.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/tree.py)
* [lib/ansible/plugins/callback/unixy.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/unixy.py)
* [lib/ansible/plugins/callback/yaml.py](https://github.com/ansible/ansible/blob/devel/lib/ansible/plugins/callback/yaml.py)
* [test/units/plugins/callback/__init__.py](https://github.com/ansible/ansible/blob/devel/test/units/plugins/callback/__init__.py)
* [test/units/plugins/callback/test_callback.py](https://github.com/ansible/ansible/blob/devel/test/units/plugins/callback/test_callback.py)

If these files are inaccurate, please update the `component name` section of the description or use the `!component` bot command.

[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: components_banner --->
cc @akatch
[click here for bot help](https://github.com/ansible/ansibullbot/blob/master/ISSUE_HELP.md)
<!--- boilerplate: notify --->

Resolving patch:
diff --git a/lib/ansible/plugins/callback/grafana_annotations.py b/lib/ansible/plugins/callback/grafana_annotations.py
new file mode 100644
--- /dev/null
+++ b/lib/ansible/plugins/callback/grafana_annotations.py
@@ -0,0 +1,261 @@
+# -*- coding: utf-8 -*-
+# This file is part of Ansible
+#
+# Ansible is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Ansible is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Ansible.  If not, see <http://www.gnu.org/licenses/>.
+
+from __future__ import (absolute_import, division, print_function)
+__metaclass__ = type
+
+import os
+import json
+import socket
+import getpass
+from base64 import b64encode
+from datetime import datetime
+
+from ansible.module_utils.urls import open_url
+from ansible.plugins.callback import CallbackBase
+
+
+DOCUMENTATION = """
+    callback: grafana_annotations
+    callback_type: notification
+    short_description: send ansible events as annotations on charts to grafana over http api.
+    author: "Rémi REY (@rrey)"
+    description:
+      - This callback will report start, failed and stats events to Grafana as annotations (https://grafana.com)
+    version_added: "2.6"
+    requirements:
+      - whitelisting in configuration
+    options:
+      grafana_url:
+        description: Grafana annotations api URL
+        required: True
+        env:
+          - name: GRAFANA_URL
+        ini:
+          - section: callback_grafana_annotations
+            key: grafana_url
+      validate_grafana_certs:
+        description: (bool) validate the SSL certificate of the Grafana server. (For HTTPS url)
+        env:
+          - name: GRAFANA_VALIDATE_CERT
+        ini:
+          - section: callback_grafana_annotations
+            key: validate_grafana_certs
+        default: True
+      http_agent:
+        description: The HTTP 'User-agent' value to set in HTTP requets.
+        env:
+          - name: HTTP_AGENT
+        ini:
+          - section: callback_grafana_annotations
+            key: http_agent
+        default: 'Ansible (grafana_annotations callback)'
+      grafana_api_key:
+        description: Grafana API key, allowing to authenticate when posting on the HTTP API.
+                     If not provided, grafana_login and grafana_password will
+                     be required.
+        env:
+          - name: GRAFANA_API_KEY
+        ini:
+          - section: callback_grafana_annotations
+            key: grafana_api_key
+      grafana_user:
+        description: Grafana user used for authentication. Ignored if grafana_api_key is provided.
+        env:
+          - name: GRAFANA_USER
+        ini:
+          - section: callback_grafana_annotations
+            key: grafana_user
+        default: ansible
+      grafana_password:
+        description: Grafana password used for authentication. Ignored if grafana_api_key is provided.
+        env:
+          - name: GRAFANA_PASSWORD
+        ini:
+          - section: callback_grafana_annotations
+            key: grafana_password
+        default: ansible
+      grafana_dashboard_id:
+        description: The grafana dashboard id where the annotation shall be created.
+        env:
+          - name: GRAFANA_DASHBOARD_ID
+        ini:
+          - section: callback_grafana_annotations
+            key: grafana_dashboard_id
+      grafana_panel_id:
+        description: The grafana panel id where the annotation shall be created.
+        env:
+          - name: GRAFANA_PANEL_ID
+        ini:
+          - section: callback_grafana_annotations
+            key: grafana_panel_id
+"""
+
+
+PLAYBOOK_START_TXT = """\
+Started playbook {playbook}
+
+From '{hostname}'
+By user '{username}'
+"""
+
+PLAYBOOK_ERROR_TXT = """\
+Playbook {playbook} Failure !
+
+From '{hostname}'
+By user '{username}'
+
+'{task}' failed on {host}
+
+debug: {result}
+"""
+
+PLAYBOOK_STATS_TXT = """\
+Playbook {playbook}
+Duration: {duration}
+Status: {status}
+
+From '{hostname}'
+By user '{username}'
+
+Result:
+{summary}
+"""
+
+
+def to_millis(dt):
+    return int(dt.strftime('%s')) * 1000
+
+
+class CallbackModule(CallbackBase):
+    """
+    ansible grafana callback plugin
+    ansible.cfg:
+        callback_plugins   = <path_to_callback_plugins_folder>
+        callback_whitelist = grafana_annotations
+    and put the plugin in <path_to_callback_plugins_folder>
+    """
+
+    CALLBACK_VERSION = 1.0
+    CALLBACK_TYPE = 'aggregate'
+    CALLBACK_NAME = 'grafana_annotations'
+    CALLBACK_NEEDS_WHITELIST = True
+
+    def __init__(self, display=None):
+
+        super(CallbackModule, self).__init__(display=display)
+
+        self.headers = {'Content-Type': 'application/json'}
+        self.force_basic_auth = False
+        self.hostname = socket.gethostname()
+        self.username = getpass.getuser()
+        self.start_time = datetime.now()
+        self.errors = 0
+
+    def set_options(self, task_keys=None, var_options=None, direct=None):
+
+        super(CallbackModule, self).set_options(task_keys=task_keys, var_options=var_options, direct=direct)
+
+        self.grafana_api_key = self.get_option('grafana_api_key')
+        self.grafana_url = self.get_option('grafana_url')
+        self.validate_grafana_certs = self.get_option('validate_grafana_certs')
+        self.http_agent = self.get_option('http_agent')
+        self.grafana_user = self.get_option('grafana_user')
+        self.grafana_password = self.get_option('grafana_password')
+        self.dashboard_id = self.get_option('grafana_dashboard_id')
+        self.panel_id = self.get_option('grafana_panel_id')
+
+        if self.grafana_api_key:
+            self.headers['Authorization'] = "Bearer %s" % self.grafana_api_key
+        else:
+            self.force_basic_auth = True
+
+        if self.grafana_url is None:
+            self.disabled = True
+            self._display.warning('Grafana URL was not provided. The '
+                                  'Grafana URL can be provided using '
+                                  'the `GRAFANA_URL` environment variable.')
+        self._display.info('Grafana URL: %s' % self.grafana_url)
+
+    def v2_playbook_on_start(self, playbook):
+        self.playbook = playbook._file_name
+        text = PLAYBOOK_START_TXT.format(playbook=self.playbook, hostname=self.hostname,
+                                         username=self.username)
+        data = {
+            'time': to_millis(self.start_time),
+            'text': text,
+            'tags': ['ansible', 'ansible_event_start', self.playbook]
+        }
+        if self.dashboard_id:
+            data["dashboardId"] = int(self.dashboard_id)
+        if self.panel_id:
+            data["panelId"] = int(self.panel_id)
+        self._send_annotation(json.dumps(data))
+
+    def v2_playbook_on_stats(self, stats):
+        end_time = datetime.now()
+        duration = end_time - self.start_time
+        summarize_stat = {}
+        for host in stats.processed.keys():
+            summarize_stat[host] = stats.summarize(host)
+
+        status = "FAILED"
+        if self.errors == 0:
+            status = "OK"
+
+        text = PLAYBOOK_STATS_TXT.format(playbook=self.playbook, hostname=self.hostname,
+                                         duration=duration.total_seconds(),
+                                         status=status, username=self.username,
+                                         summary=json.dumps(summarize_stat))
+
+        data = {
+            'time': to_millis(self.start_time),
+            'timeEnd': to_millis(end_time),
+            'isRegion': True,
+            'text': text,
+            'tags': ['ansible', 'ansible_report', self.playbook]
+        }
+        if self.dashboard_id:
+            data["dashboardId"] = int(self.dashboard_id)
+        if self.panel_id:
+            data["panelId"] = int(self.panel_id)
+        self._send_annotation(json.dumps(data))
+
+    def v2_runner_on_failed(self, result, **kwargs):
+        text = PLAYBOOK_ERROR_TXT.format(playbook=self.playbook, hostname=self.hostname,
+                                         username=self.username, task=result._task,
+                                         host=result._host.name, result=self._dump_results(result._result))
+        data = {
+            'time': to_millis(datetime.now()),
+            'text': text,
+            'tags': ['ansible', 'ansible_event_failure', self.playbook]
+        }
+        self.errors += 1
+        if self.dashboard_id:
+            data["dashboardId"] = int(self.dashboard_id)
+        if self.panel_id:
+            data["panelId"] = int(self.panel_id)
+        self._send_annotation(json.dumps(data))
+
+    def _send_annotation(self, annotation):
+        try:
+            response = open_url(self.grafana_url, data=annotation, headers=self.headers,
+                                method="POST",
+                                validate_certs=self.validate_grafana_certs,
+                                url_username=self.grafana_user, url_password=self.grafana_password,
+                                http_agent=self.http_agent, force_basic_auth=self.force_basic_auth)
+        except Exception as e:
+            self._display.error('Could not submit message to Grafana: %s' % str(e))


Test patch: