Repository: DataDog/dd-agent

Issue:
[rabbitmq] make ssl verification optional
<!--

This issue queue is primarily intended for tracking features, bugs and work items associated with
the dd-agent open-source project.

Prior to submitting an issue please review the following:

- [ ] [Troubleshooting](https://datadog.zendesk.com/hc/en-us/sections/200766955-Troubleshooting) section of our [Knowledge base](https://datadog.zendesk.com/hc/en-us).
- [ ] Contact our [support](http://docs.datadoghq.com/help/) and [send them your logs](https://github.com/DataDog/dd-agent/wiki/Send-logs-to-support).
- [ ] Finally, you can open a Github issue respecting this [title convention](https://github.com/DataDog/dd-agent/blob/master/CONTRIBUTING.md#commits-titles) (it helps us triage).

If you are reporting a new issue, make sure that we do not have any duplicates
already open. You can ensure this by searching the issue list for this
repository. If there is a duplicate, please close your issue and add a comment
to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to
include the BUG REPORT INFORMATION shown below.

---------------------------------------------------
BUG REPORT INFORMATION
---------------------------------------------------
Use the commands below to provide key information from your environment:
You do NOT have to include this information if this is a FEATURE REQUEST
-->

*_Output of the [info page](https://help.datadoghq.com/hc/en-us/articles/203764635-Agent-Status-and-Information) *_

```
[root@myhost01 conf.d]# service datadog-agent info
===================
Collector (v 5.9.1)
===================

  Status date: 2016-10-11 12:14:09 (3s ago)
  Pid: 12084
  Platform: Linux-2.6.32-642.4.2.el6.x86_64-x86_64-with-redhat-6.8-Santiago
  Python Version: 2.7.12, 64bit
  Logs: <stderr>, /var/log/datadog/collector.log

  Clocks
  ======

    NTP offset: 0.0005 s
    System UTC time: 2016-10-11 19:14:12.603287

  Paths
  =====

    conf.d: /etc/dd-agent/conf.d
    checks.d: /opt/datadog-agent/agent/checks.d

  Hostnames
  =========

    agent-hostname: myhost01
    hostname: myhost01
    socket-fqdn: MYHOST01.MYDOMAIN.COM

  Checks
  ======

    nginx
    -----
      - instance #0 [OK]
      - Collected 7 metrics, 0 events & 2 service checks

    ntp
    ---
      - Collected 0 metrics, 0 events & 1 service check

    disk
    ----
      - instance #0 [OK]
      - Collected 32 metrics, 0 events & 1 service check

    rabbitmq
    --------
      - instance #0 [OK]
      - Collected 0 metrics, 0 events & 2 service checks

    network
    -------
      - instance #0 [OK]
      - Collected 15 metrics, 0 events & 1 service check


  Emitters
  ========

    - http_emitter [OK]

===================
Dogstatsd (v 5.9.1)
===================

  Status date: 2016-10-11 12:14:04 (9s ago)
  Pid: 12081
  Platform: Linux-2.6.32-642.4.2.el6.x86_64-x86_64-with-redhat-6.8-Santiago
  Python Version: 2.7.12, 64bit
  Logs: <stderr>, /var/log/datadog/dogstatsd.log

  Flush count: 3
  Packet Count: 28
  Packets per second: 0.0
  Metric count: 14
  Event count: 0
  Service check count: 0

===================
Forwarder (v 5.9.1)
===================

  Status date: 2016-10-11 12:14:14 (0s ago)
  Pid: 12083
  Platform: Linux-2.6.32-642.4.2.el6.x86_64-x86_64-with-redhat-6.8-Santiago
  Python Version: 2.7.12, 64bit
  Logs: <stderr>, /var/log/datadog/forwarder.log

  Queue Size: 540 bytes
  Queue Length: 1
  Flush Count: 10
  Transactions received: 8
  Transactions flushed: 7
  Transactions rejected: 0

```

**Additional environment details (Operating System, Cloud provider, etc):**

**Steps to reproduce the issue:**
1. Stand up RabbitMQ server with management plugin enabled
2. Create a self-signed SSL certificate and configure the management plugin to use it
3. Configure datadog's rabbitmq integration on host, using `https://localhost:15672/api/` as `rabbitmq_api_url`
4. Manually execute check for rabbitmq integration

**Describe the results you received:**

```
[root@myhost01 conf.d]# sudo -u dd-agent dd-agent check rabbitmq
2016-10-11 12:16:39,975 | WARNING | dd.collector | checks.disk(disk.py:87) | Using `use_mount` in datadog.conf has been deprecated in favor of `use_mount` in disk.yaml
2016-10-11 12:16:39,983 | INFO | dd.collector | config(config.py:1061) | initialized checks.d checks: ['nginx', 'ntp', 'disk', 'rabbitmq', 'network']
2016-10-11 12:16:39,984 | INFO | dd.collector | config(config.py:1062) | initialization failed checks.d checks: []
2016-10-11 12:16:39,984 | INFO | dd.collector | checks.collector(collector.py:526) | Running check rabbitmq
2016-10-11 12:16:40,014 | ERROR | dd.collector | checks.rabbitmq(rabbitmq.py:163) | Error executing check: Cannot open RabbitMQ API url: https://localhost:15672/api/queues [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
Metrics: 
[]
Events: 
[]
Service Checks: 
[{'check': 'rabbitmq.status',
  'host_name': 'myhost01',
  'id': 1,
  'message': u'Error executing check: Cannot open RabbitMQ API url: https://localhost:15672/api/queues [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)',
  'status': 2,
  'tags': None,
  'timestamp': 1476213400.014164}]
Service Metadata: 
[{}]
    rabbitmq
    --------
      - instance #0 [OK]
      - Collected 0 metrics, 0 events & 1 service check
```

**Describe the results you expected:**
That the code should support the ability to selectively enable/disable ssl verification on a per-integration or per-instance basis. Currently the python `requests` library is called, but no `verify=` option is passed, so this is not configurable.

**Additional information you deem important (e.g. issue happens only occasionally):**
N/A

[rabbitmq] make ssl verification optional
<!--

This issue queue is primarily intended for tracking features, bugs and work items associated with
the dd-agent open-source project.

Prior to submitting an issue please review the following:

- [ ] [Troubleshooting](https://datadog.zendesk.com/hc/en-us/sections/200766955-Troubleshooting) section of our [Knowledge base](https://datadog.zendesk.com/hc/en-us).
- [ ] Contact our [support](http://docs.datadoghq.com/help/) and [send them your logs](https://github.com/DataDog/dd-agent/wiki/Send-logs-to-support).
- [ ] Finally, you can open a Github issue respecting this [title convention](https://github.com/DataDog/dd-agent/blob/master/CONTRIBUTING.md#commits-titles) (it helps us triage).

If you are reporting a new issue, make sure that we do not have any duplicates
already open. You can ensure this by searching the issue list for this
repository. If there is a duplicate, please close your issue and add a comment
to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to
include the BUG REPORT INFORMATION shown below.

---------------------------------------------------
BUG REPORT INFORMATION
---------------------------------------------------
Use the commands below to provide key information from your environment:
You do NOT have to include this information if this is a FEATURE REQUEST
-->

*_Output of the [info page](https://help.datadoghq.com/hc/en-us/articles/203764635-Agent-Status-and-Information) *_

```
[root@myhost01 conf.d]# service datadog-agent info
===================
Collector (v 5.9.1)
===================

  Status date: 2016-10-11 12:14:09 (3s ago)
  Pid: 12084
  Platform: Linux-2.6.32-642.4.2.el6.x86_64-x86_64-with-redhat-6.8-Santiago
  Python Version: 2.7.12, 64bit
  Logs: <stderr>, /var/log/datadog/collector.log

  Clocks
  ======

    NTP offset: 0.0005 s
    System UTC time: 2016-10-11 19:14:12.603287

  Paths
  =====

    conf.d: /etc/dd-agent/conf.d
    checks.d: /opt/datadog-agent/agent/checks.d

  Hostnames
  =========

    agent-hostname: myhost01
    hostname: myhost01
    socket-fqdn: MYHOST01.MYDOMAIN.COM

  Checks
  ======

    nginx
    -----
      - instance #0 [OK]
      - Collected 7 metrics, 0 events & 2 service checks

    ntp
    ---
      - Collected 0 metrics, 0 events & 1 service check

    disk
    ----
      - instance #0 [OK]
      - Collected 32 metrics, 0 events & 1 service check

    rabbitmq
    --------
      - instance #0 [OK]
      - Collected 0 metrics, 0 events & 2 service checks

    network
    -------
      - instance #0 [OK]
      - Collected 15 metrics, 0 events & 1 service check


  Emitters
  ========

    - http_emitter [OK]

===================
Dogstatsd (v 5.9.1)
===================

  Status date: 2016-10-11 12:14:04 (9s ago)
  Pid: 12081
  Platform: Linux-2.6.32-642.4.2.el6.x86_64-x86_64-with-redhat-6.8-Santiago
  Python Version: 2.7.12, 64bit
  Logs: <stderr>, /var/log/datadog/dogstatsd.log

  Flush count: 3
  Packet Count: 28
  Packets per second: 0.0
  Metric count: 14
  Event count: 0
  Service check count: 0

===================
Forwarder (v 5.9.1)
===================

  Status date: 2016-10-11 12:14:14 (0s ago)
  Pid: 12083
  Platform: Linux-2.6.32-642.4.2.el6.x86_64-x86_64-with-redhat-6.8-Santiago
  Python Version: 2.7.12, 64bit
  Logs: <stderr>, /var/log/datadog/forwarder.log

  Queue Size: 540 bytes
  Queue Length: 1
  Flush Count: 10
  Transactions received: 8
  Transactions flushed: 7
  Transactions rejected: 0

```

**Additional environment details (Operating System, Cloud provider, etc):**

**Steps to reproduce the issue:**
1. Stand up RabbitMQ server with management plugin enabled
2. Create a self-signed SSL certificate and configure the management plugin to use it
3. Configure datadog's rabbitmq integration on host, using `https://localhost:15672/api/` as `rabbitmq_api_url`
4. Manually execute check for rabbitmq integration

**Describe the results you received:**

```
[root@myhost01 conf.d]# sudo -u dd-agent dd-agent check rabbitmq
2016-10-11 12:16:39,975 | WARNING | dd.collector | checks.disk(disk.py:87) | Using `use_mount` in datadog.conf has been deprecated in favor of `use_mount` in disk.yaml
2016-10-11 12:16:39,983 | INFO | dd.collector | config(config.py:1061) | initialized checks.d checks: ['nginx', 'ntp', 'disk', 'rabbitmq', 'network']
2016-10-11 12:16:39,984 | INFO | dd.collector | config(config.py:1062) | initialization failed checks.d checks: []
2016-10-11 12:16:39,984 | INFO | dd.collector | checks.collector(collector.py:526) | Running check rabbitmq
2016-10-11 12:16:40,014 | ERROR | dd.collector | checks.rabbitmq(rabbitmq.py:163) | Error executing check: Cannot open RabbitMQ API url: https://localhost:15672/api/queues [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
Metrics: 
[]
Events: 
[]
Service Checks: 
[{'check': 'rabbitmq.status',
  'host_name': 'myhost01',
  'id': 1,
  'message': u'Error executing check: Cannot open RabbitMQ API url: https://localhost:15672/api/queues [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)',
  'status': 2,
  'tags': None,
  'timestamp': 1476213400.014164}]
Service Metadata: 
[{}]
    rabbitmq
    --------
      - instance #0 [OK]
      - Collected 0 metrics, 0 events & 1 service check
```

**Describe the results you expected:**
That the code should support the ability to selectively enable/disable ssl verification on a per-integration or per-instance basis. Currently the python `requests` library is called, but no `verify=` option is passed, so this is not configurable.

**Additional information you deem important (e.g. issue happens only occasionally):**
N/A



Discussion:
I may be able to undertake the effort to add this feature; however, I'd want to know if it would be better if ssl verification should be a per-instance configuration option (like the [mesos integration](https://github.com/DataDog/dd-agent/pull/2809)) or a per-integration configuration (e.g. init_config option, like the [openstack integration](https://github.com/DataDog/dd-agent/blob/master/checks.d/openstack.py#L135)).

It looks like the mesos integration and the http check both use an instance variable: https://github.com/DataDog/dd-agent/blob/master/conf.d/http_check.yaml.example#L60

Hi @iancward and thanks for reporting!

Yes, I think the right place is putting the setting at the instance level. I've added this to our queue but if you want to give a shot to it and open a PR, that would speed up the process.

@masci I believe I have something together, but I'm trying to sort out how to add a test for it. I'll also need to go through my employer's review process in order to submit the code.

If you need any help with testing, please let me know!

I may be able to undertake the effort to add this feature; however, I'd want to know if it would be better if ssl verification should be a per-instance configuration option (like the [mesos integration](https://github.com/DataDog/dd-agent/pull/2809)) or a per-integration configuration (e.g. init_config option, like the [openstack integration](https://github.com/DataDog/dd-agent/blob/master/checks.d/openstack.py#L135)).

It looks like the mesos integration and the http check both use an instance variable: https://github.com/DataDog/dd-agent/blob/master/conf.d/http_check.yaml.example#L60

Hi @iancward and thanks for reporting!

Yes, I think the right place is putting the setting at the instance level. I've added this to our queue but if you want to give a shot to it and open a PR, that would speed up the process.

@masci I believe I have something together, but I'm trying to sort out how to add a test for it. I'll also need to go through my employer's review process in order to submit the code.

If you need any help with testing, please let me know!


Resolving patch:
diff --git a/checks.d/rabbitmq.py b/checks.d/rabbitmq.py
--- a/checks.d/rabbitmq.py
+++ b/checks.d/rabbitmq.py
@@ -113,6 +113,10 @@ def _get_config(self, instance):
             base_url += '/'
         username = instance.get('rabbitmq_user', 'guest')
         password = instance.get('rabbitmq_pass', 'guest')
+        parsed_url = urlparse.urlparse(base_url)
+        ssl_verify = _is_affirmative(instance.get('ssl_verify', True))
+        if not ssl_verify and parsed_url.scheme == 'https':
+            self.log.warning('Skipping SSL cert validation for %s based on configuration.' % (base_url))
 
         # Limit of queues/nodes to collect metrics from
         max_detailed = {
@@ -140,19 +144,19 @@ def _get_config(self, instance):
 
         auth = (username, password)
 
-        return base_url, max_detailed, specified, auth
+        return base_url, max_detailed, specified, auth, ssl_verify
 
     def check(self, instance):
-        base_url, max_detailed, specified, auth = self._get_config(instance)
+        base_url, max_detailed, specified, auth, ssl_verify = self._get_config(instance)
         try:
             # Generate metrics from the status API.
-            self.get_stats(instance, base_url, QUEUE_TYPE, max_detailed[QUEUE_TYPE], specified[QUEUE_TYPE], auth=auth)
-            self.get_stats(instance, base_url, NODE_TYPE, max_detailed[NODE_TYPE], specified[NODE_TYPE], auth=auth)
+            self.get_stats(instance, base_url, QUEUE_TYPE, max_detailed[QUEUE_TYPE], specified[QUEUE_TYPE], auth=auth, ssl_verify=ssl_verify)
+            self.get_stats(instance, base_url, NODE_TYPE, max_detailed[NODE_TYPE], specified[NODE_TYPE], auth=auth, ssl_verify=ssl_verify)
 
             # Generate a service check from the aliveness API. In the case of an invalid response
             # code or unparseable JSON this check will send no data.
             vhosts = instance.get('vhosts')
-            self._check_aliveness(base_url, vhosts, auth=auth)
+            self._check_aliveness(base_url, vhosts, auth=auth, ssl_verify=ssl_verify)
 
             # Generate a service check for the service status.
             self.service_check('rabbitmq.status', AgentCheck.OK)
@@ -162,9 +166,9 @@ def check(self, instance):
             self.service_check('rabbitmq.status', AgentCheck.CRITICAL, message=msg)
             self.log.error(msg)
 
-    def _get_data(self, url, auth=None):
+    def _get_data(self, url, auth=None, ssl_verify=True):
         try:
-            r = requests.get(url, auth=auth, timeout=self.default_integration_http_timeout)
+            r = requests.get(url, auth=auth, timeout=self.default_integration_http_timeout, verify=ssl_verify)
             r.raise_for_status()
             return r.json()
         except RequestException as e:
@@ -172,7 +176,7 @@ def _get_data(self, url, auth=None):
         except ValueError as e:
             raise RabbitMQException('Cannot parse JSON response from API url: {} {}'.format(url, str(e)))
 
-    def get_stats(self, instance, base_url, object_type, max_detailed, filters, auth=None):
+    def get_stats(self, instance, base_url, object_type, max_detailed, filters, auth=None, ssl_verify=True):
         """
         instance: the check instance
         base_url: the url of the rabbitmq management api (e.g. http://localhost:15672/api)
@@ -180,7 +184,7 @@ def get_stats(self, instance, base_url, object_type, max_detailed, filters, auth
         max_detailed: the limit of objects to collect for this type
         filters: explicit or regexes filters of specified queues or nodes (specified in the yaml file)
         """
-        data = self._get_data(urlparse.urljoin(base_url, object_type), auth=auth)
+        data = self._get_data(urlparse.urljoin(base_url, object_type), auth=auth, ssl_verify=ssl_verify)
 
         # Make a copy of this list as we will remove items from it at each
         # iteration
@@ -311,7 +315,7 @@ def alert(self, base_url, max_detailed, size, object_type):
 
         self.event(event)
 
-    def _check_aliveness(self, base_url, vhosts=None, auth=None):
+    def _check_aliveness(self, base_url, vhosts=None, auth=None, ssl_verify=True):
         """
         Check the aliveness API against all or a subset of vhosts. The API
         will return {"status": "ok"} and a 200 response code in the case
@@ -320,7 +324,7 @@ def _check_aliveness(self, base_url, vhosts=None, auth=None):
         if not vhosts:
             # Fetch a list of _all_ vhosts from the API.
             vhosts_url = urlparse.urljoin(base_url, 'vhosts')
-            vhosts_response = self._get_data(vhosts_url, auth=auth)
+            vhosts_response = self._get_data(vhosts_url, auth=auth, ssl_verify=ssl_verify)
             vhosts = [v['name'] for v in vhosts_response]
 
         for vhost in vhosts:
@@ -328,7 +332,7 @@ def _check_aliveness(self, base_url, vhosts=None, auth=None):
             # We need to urlencode the vhost because it can be '/'.
             path = u'aliveness-test/%s' % (urllib.quote_plus(vhost))
             aliveness_url = urlparse.urljoin(base_url, path)
-            aliveness_response = self._get_data(aliveness_url, auth=auth)
+            aliveness_response = self._get_data(aliveness_url, auth=auth, ssl_verify=ssl_verify)
             message = u"Response from aliveness API: %s" % aliveness_response
 
             if aliveness_response.get('status') == 'ok':


Test patch: