Repository: DataDog/dd-agent

Issue:
[etcd] `/stats/leader` from a follower returns a 403
The `/stats/leader` endpoint is only available on the leader etcd host.
On the followers, it returns a 403: it is illegitimately logged as an exception in the agent and triggers a CRITICAL service check.

`#0[ERROR]: 'Http status code 403 on url http://localhost:4002/v2/stats/leader`

Full stacktrace: 

```
2015-06-17 16:41:46 UTC | ERROR | dd.collector | checks.etcd(__init__.py:678) | Check 'etcd' instance #0 failed
Traceback (most recent call last):
  File "/opt/datadog-agent/agent/checks/__init__.py", line 661, in run
    self.check(copy.deepcopy(instance))
  File "/opt/datadog-agent/agent/checks.d/etcd.py", line 114, in check
    leader_response = self._get_leader_metrics(url, timeout)
  File "/opt/datadog-agent/agent/checks.d/etcd.py", line 143, in _get_leader_metrics
    return self._get_json(url + "/v2/stats/leader", timeout)
  File "/opt/datadog-agent/agent/checks.d/etcd.py", line 159, in _get_json
    raise Exception("Http status code {0} on url {1}".format(r.status_code, url))
Exception: Http status code 403 on url http://localhost:4002/v2/stats/leader
```



Discussion:

Resolving patch:
diff --git a/checks.d/etcd.py b/checks.d/etcd.py
--- a/checks.d/etcd.py
+++ b/checks.d/etcd.py
@@ -111,22 +111,22 @@ def check(self, instance):
                     self.log.warn("Missing key {0} in stats.".format(key))
 
         # Gather leader metrics
-        leader_response = self._get_leader_metrics(url, timeout)
-        if leader_response is not None and is_leader \
-                and len(leader_response.get("followers", {})) > 0:
-            # Get the followers
-            followers = leader_response.get("followers")
-            for fol in followers:
-                # counts
-                for key in self.LEADER_COUNTS:
-                    self.rate(self.LEADER_COUNTS[key],
-                              followers[fol].get("counts").get(key),
-                              tags=instance_tags + ['follower:{0}'.format(fol)])
-                # latency
-                for key in self.LEADER_LATENCY:
-                    self.gauge(self.LEADER_LATENCY[key],
-                               followers[fol].get("latency").get(key),
-                               tags=instance_tags + ['follower:{0}'.format(fol)])
+        if is_leader:
+            leader_response = self._get_leader_metrics(url, timeout)
+            if leader_response is not None and len(leader_response.get("followers", {})) > 0:
+                # Get the followers
+                followers = leader_response.get("followers")
+                for fol in followers:
+                    # counts
+                    for key in self.LEADER_COUNTS:
+                        self.rate(self.LEADER_COUNTS[key],
+                                  followers[fol].get("counts").get(key),
+                                  tags=instance_tags + ['follower:{0}'.format(fol)])
+                    # latency
+                    for key in self.LEADER_LATENCY:
+                        self.gauge(self.LEADER_LATENCY[key],
+                                   followers[fol].get("latency").get(key),
+                                   tags=instance_tags + ['follower:{0}'.format(fol)])
 
         # Service check
         if self_response is not None and store_response is not None:


Test patch: