Repository: DataDog/dd-agent

Issue:
Problem with Docker 1.2.0 and agent 5.0.0
I'm having a problem with `docker ps` hanging after starting the Docker agent on Ubuntu 14.04 and Docker 1.2.0. I'm not sure if this is a Docker 1.2.0 problem or a Datadog problem. But I'm only able to produce it with the Datadog agent.

After starting the agent with the Docker plugin enabled, the `/containers/json` endpoint stops responding:

``` console
$ echo -e "GET /containers/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock

```

If I stop the Datadog agent and restart Docker, the endpoint starts working again:

``` console
$ echo -e "GET /containers/json HTTP/1.0\r\n" | nc -U /var/run/docker.sock
HTTP/1.0 200 OK
Content-Type: application/json
Date: Sun, 31 Aug 2014 23:34:28 GMT
Content-Length: 2
```

Any ideas?



Discussion:
I was able to confirm it has something to do with Datadog and the Docker 1.2.0 upgrade. Here are some steps to reproduce on Ubuntu 14.04 (reproduced in EC2):
- Install Ubuntu 14.04
- Get the Docker APT repository installed:

``` shell
if [ ! -e /usr/lib/apt/methods/https ]; then
    sudo apt-get update
    sudo apt-get install -y apt-transport-https
fi

echo deb https://get.docker.io/ubuntu docker main sudo tee -a /etc/apt/sources.list.d/docker.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
sudo apt-get update
```
- Install Docker 1.1.2 - `sudo apt-get install -y lxc-docker-1.1.2`
- Start the Datadog container - `sudo docker run -d --privileged --name dd-agent -h $(hostname) -v /var/run/docker.sock:/var/run/docker.sock -v /proc/mounts:/host/proc/mounts:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY=<your_key> datadog/docker-dd-agent`
- Start some other container - `sudo docker run -d --name sleep ubuntu:trusty sleep 99999999`
- `sudo docker ps` works fine

Now we upgrade Docker to 1.2.0 and do the same test.
- Kill the containers - `sudo docker rm -f dd-agent sleep`
- Upgrade Docker - `sudo apt-get -y install lxc-docker-1.2.0`
- Restart daemon - `sudo service docker restart`
- Start the Datadog container - `sudo docker run -d --privileged --name dd-agent -h $(hostname) -v /var/run/docker.sock:/var/run/docker.sock -v /proc/mounts:/host/proc/mounts:ro -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -e API_KEY=<your_key> datadog/docker-dd-agent`
- Start some other container - `sudo docker run -d --name sleep ubuntu:trusty sleep 99999999`
- After a few moments `sudo docker ps` will stop responding

Thanks for all your work @andyshinn !

I managed to reproduce this same issue on both CoreOS 427 and Debian 7.
However this happened to me only when the Datadog Agent container was running: the issue I identifies doesn't occur when the Agent run outside of the container. Can you confirm me this?
- Run your Ubuntu 14.04, with Docker 1.2.0 installed, no other container.
- Run some container but the Agent: sudo docker run -d --name sleep ubuntu:trusty sleep 99999999
- Install the Agent on your host (According to https://app.datadoghq.com/account/settings#agent/ubuntu )
- Enable the Docker integration on your host https://app.datadoghq.com/account/settings#integrations/docker
- Restart your Datadog Agent `sudo service datadog-agent restart`
- Check if it works or not (try a `sudo docker ps`, check your metrics in Datadog, run the netcat command,  ...)

In my case, it works. 
And if it's good for you too, I'll be able to release a fix quickly.

I just tested this some more. It still hangs even using the agent outside of a container (on the Ubuntu 14.04 host directly). I performed the steps you outlined on a fresh Ubuntu 14.04 EC2 instance. Let me know what other debug information you need.

It looks like this could also be related to bad Docker behavior in docker/docker#7905 and docker/docker#8010. So even if there is a workaround in Datadog, the ultimate fix might be from Docker.


Resolving patch:
diff --git a/checks.d/docker.py b/checks.d/docker.py
--- a/checks.d/docker.py
+++ b/checks.d/docker.py
@@ -13,7 +13,6 @@
 # project
 from checks import AgentCheck
 
-DEFAULT_MAX_CONTAINERS = 20
 EVENT_TYPE = SOURCE_TYPE_NAME = 'docker'
 
 CGROUP_METRICS = [
@@ -36,19 +35,6 @@
             "rss": ("docker.mem.rss", "gauge"),
             "swap": ("docker.mem.swap", "gauge"),
             "unevictable": ("docker.mem.unevictable", "gauge"),
-            "total_active_anon": ("docker.mem.total_active_anon", "gauge"),
-            "total_active_file": ("docker.mem.total_active_file", "gauge"),
-            "total_cache": ("docker.mem.total_cache", "gauge"),
-            "total_inactive_anon": ("docker.mem.total_inactive_anon", "gauge"),
-            "total_inactive_file": ("docker.mem.total_inactive_file", "gauge"),
-            "total_mapped_file": ("docker.mem.total_mapped_file", "gauge"),
-            "total_pgfault": ("docker.mem.total_pgfault", "rate"),
-            "total_pgmajfault": ("docker.mem.total_pgmajfault", "rate"),
-            "total_pgpgin": ("docker.mem.total_pgpgin", "rate"),
-            "total_pgpgout": ("docker.mem.total_pgpgout", "rate"),
-            "total_rss": ("docker.mem.total_rss", "gauge"),
-            "total_swap": ("docker.mem.total_swap", "gauge"),
-            "total_unevictable": ("docker.mem.total_unevictable", "gauge"),
         }
     },
     {
@@ -70,19 +56,22 @@
     "Image",
 ]
 
-SOCKET_TIMEOUT = 5
+DEFAULT_SOCKET_TIMEOUT = 5
 
 
-class UnixHTTPConnection(httplib.HTTPConnection, object):
+class UnixHTTPConnection(httplib.HTTPConnection):
     """Class used in conjuction with UnixSocketHandler to make urllib2
     compatible with Unix sockets."""
+
+    socket_timeout = DEFAULT_SOCKET_TIMEOUT
+
     def __init__(self, unix_socket):
         self._unix_socket = unix_socket
 
     def connect(self):
         sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
         sock.connect(self._unix_socket)
-        sock.settimeout(SOCKET_TIMEOUT)
+        sock.settimeout(self.socket_timeout)
         self.sock = sock
 
     def __call__(self, *args, **kwargs):
@@ -111,63 +100,98 @@ def unix_open(self, req):
 
 
 class Docker(AgentCheck):
+    """Collect metrics and events from Docker API and cgroups"""
+
     def __init__(self, name, init_config, agentConfig):
         AgentCheck.__init__(self, name, init_config, agentConfig)
+
+        # Initialize a HTTP opener with Unix socket support
+        socket_timeout = int(init_config.get('socket_timeout', 0)) or DEFAULT_SOCKET_TIMEOUT
+        UnixHTTPConnection.socket_timeout = socket_timeout
+        self.url_opener = urllib2.build_opener(UnixSocketHandler())
+
+        # Locate cgroups directories
         self._mountpoints = {}
+        self._cgroup_filename_pattern = None
         docker_root = init_config.get('docker_root', '/')
         for metric in CGROUP_METRICS:
             self._mountpoints[metric["cgroup"]] = self._find_cgroup(metric["cgroup"], docker_root)
+
         self._last_event_collection_ts = defaultdict(lambda: None)
-        self.url_opener = urllib2.build_opener(UnixSocketHandler())
-        self.should_get_size = True
-        self._cgroup_filename_pattern = None
 
-    def _find_cgroup_filename_pattern(self):
-        if self._mountpoints:
-            # We try with different cgroups so that it works even if only one is properly working
-            for mountpoint in self._mountpoints.values():
-                stat_file_path_lxc = os.path.join(mountpoint, "lxc")
-                stat_file_path_docker = os.path.join(mountpoint, "docker")
-                stat_file_path_coreos = os.path.join(mountpoint, "system.slice")
+    def check(self, instance):
+        # Report image metrics
+        self._count_images(instance)
 
-                if os.path.exists(stat_file_path_lxc):
-                    return os.path.join('%(mountpoint)s/lxc/%(id)s/%(file)s')
-                elif os.path.exists(stat_file_path_docker):
-                    return os.path.join('%(mountpoint)s/docker/%(id)s/%(file)s')
-                elif os.path.exists(stat_file_path_coreos):
-                    return os.path.join('%(mountpoint)s/system.slice/docker-%(id)s.scope/%(file)s')
+        # Get the list of containers and the index of their names
+        containers, ids_to_names = self._get_and_count_containers(instance)
 
-        raise Exception("Cannot find Docker cgroup directory. Be sure your system is supported.")
+        # Report container metrics from cgroups
+        self._report_containers_metrics(containers, instance)
 
-    def _get_cgroup_file(self, cgroup, container_id, filename):
-        # This can't be initialized at startup because cgroups may not be mounted
-        if not self._cgroup_filename_pattern:
-            self._cgroup_filename_pattern = self._find_cgroup_filename_pattern()
+        # Send events from Docker API
+        if instance.get('collect_events', True):
+            self._process_events(instance, ids_to_names)
 
-        return self._cgroup_filename_pattern % (dict(
-                    mountpoint=self._mountpoints[cgroup],
-                    id=container_id,
-                    file=filename,
-                ))
 
-    def check(self, instance):
+    # Containers
+
+    def _count_images(self, instance):
+        # It's not an important metric, keep going if it fails
         try:
-            self._process_events(self._get_events(instance))
-        except (socket.timeout, urllib2.URLError):
-            self.warning('Timeout during socket connection. Events will be missing.')
+            tags = instance.get("tags", [])
+            active_images = len(self._get_images(instance, get_all=False))
+            all_images = len(self._get_images(instance, get_all=True))
 
-        self._count_images(instance)
-        containers = self._get_and_count_containers(instance)
+            self.gauge("docker.images.available", active_images, tags=tags)
+            self.gauge("docker.images.intermediate", (all_images - active_images), tags=tags)
+        except Exception, e:
+            self.warning("Failed to count Docker images. Exception: {0}".format(e))
 
-        max_containers = instance.get('max_containers', DEFAULT_MAX_CONTAINERS)
+    def _get_and_count_containers(self, instance):
+        tags = instance.get("tags", [])
+        with_size = instance.get('collect_container_size', False)
 
-        if not instance.get("exclude") or not instance.get("include"):
-            if len(containers) > max_containers:
-                self.warning("Too many containers to collect. Please refine the containers to collect"
-                    "by editing the configuration file. Truncating to %s containers" % max_containers)
-                containers = containers[:max_containers]
+        service_check_name = 'docker.service_up'
+        try:
+            containers = self._get_containers(instance, with_size=with_size)
+        except (socket.timeout, urllib2.URLError), e:
+            self.service_check(service_check_name, AgentCheck.CRITICAL, tags=tags)
+            raise Exception("Failed to collect the list of containers. Exception: {0}".format(e))
+        self.service_check(service_check_name, AgentCheck.OK, tags=tags)
 
-        collected_containers = 0
+        container_count_by_image = defaultdict(int)
+        for container in containers:
+            container_count_by_image[container['Image']] += 1
+
+        for image, count in container_count_by_image.iteritems():
+            self.gauge("docker.containers.running", count, tags=(tags + ['image:%s' % image]))
+
+        all_containers = self._get_containers(instance, get_all=True)
+        stopped_containers_count = len(all_containers) - len(containers)
+        self.gauge("docker.containers.stopped", stopped_containers_count, tags=tags)
+
+        ids_to_names = {}
+        for container in all_containers:
+            ids_to_names[container['Id']] = container['Names'][0].lstrip("/")
+
+        return containers, ids_to_names
+
+    def _is_container_included(self, instance, tags):
+        def _is_tag_included(tag):
+            for exclude_rule in instance.get("exclude") or []:
+                if re.match(exclude_rule, tag):
+                    for include_rule in instance.get("include") or []:
+                        if re.match(include_rule, tag):
+                            return True
+                    return False
+            return True
+        for tag in tags:
+            if _is_tag_included(tag):
+                return True
+        return False
+
+    def _report_containers_metrics(self, containers, instance):
         for container in containers:
             container_tags = instance.get("tags", [])
             for name in container["Names"]:
@@ -187,76 +211,79 @@ def check(self, instance):
                 stats = self._parse_cgroup_file(stat_file)
                 if stats:
                     for key, (dd_key, metric_type) in cgroup['metrics'].items():
-                        if key.startswith('total_') and not instance.get('collect_total'):
-                            continue
                         if key in stats:
                             getattr(self, metric_type)(dd_key, int(stats[key]), tags=container_tags)
 
-            collected_containers += 1
-            if collected_containers >= max_containers:
-                self.warning("Too many containers are matching the current configuration. Some containers will not be collected. Please refine your configuration")
-                break
-
-    def _process_events(self, events):
-        for ev in events:
-            self.log.debug("Creating event for %s" % ev)
-            self.event({
-                'timestamp': ev['time'],
-                'host': self.hostname,
-                'event_type': EVENT_TYPE,
-                'msg_title': "%s %s on %s" % (ev['from'], ev['status'], self.hostname),
-                'source_type_name': EVENT_TYPE,
-                'event_object': ev['from'],
-            })
-
-    def _count_images(self, instance):
-        tags = instance.get("tags", [])
-        active_images = len(self._get_images(instance, get_all=False))
-        all_images = len(self._get_images(instance, get_all=True))
+    def _make_tag(self, key, value):
+        return "%s:%s" % (key.lower(), value.strip())
 
-        self.gauge("docker.images.available", active_images, tags=tags)
-        self.gauge("docker.images.intermediate", (all_images - active_images), tags=tags)
 
-    def _get_and_count_containers(self, instance):
-        tags = instance.get("tags", [])
+    # Events
 
+    def _process_events(self, instance, ids_to_names):
         try:
-            containers = self._get_containers(instance, with_size=self.should_get_size)
+            api_events = self._get_events(instance)
+            aggregated_events = self._pre_aggregate_events(api_events)
+            events = self._format_events(aggregated_events, ids_to_names)
+            self._report_events(events)
         except (socket.timeout, urllib2.URLError):
-            # Probably because of: https://github.com/DataDog/dd-agent/issues/963
-            # Then we should stop trying to get size info
-            self.log.info("Cannot get container size because of API timeout. Stop collecting it.")
-            self.should_get_size = False
-            containers = self._get_containers(instance, with_size=self.should_get_size)
-
-        if not containers:
-            containers = []
-            self.warning("No containers are running.")
-
-        stopped_containers_count = len(self._get_containers(instance, get_all=True)) - len(containers)
-        self.gauge("docker.containers.running", len(containers), tags=tags)
-        self.gauge("docker.containers.stopped", stopped_containers_count, tags=tags)
+            self.warning('Timeout during socket connection. Events will be missing.')
 
-        return containers
+    def _pre_aggregate_events(self, api_events):
+        # Aggregate events, one per image. Put newer events first.
+        events = defaultdict(list)
+        for event in api_events:
+            # Known bug: from may be missing
+            if 'from' in event:
+                events[event['from']].insert(0, event)
+
+        return events
+
+    def _format_events(self, aggregated_events, ids_to_names):
+        events = []
+        for image_name, event_group in aggregated_events.iteritems():
+            max_timestamp = 0
+            status = defaultdict(int)
+            status_change = []
+            for event in event_group:
+                max_timestamp = max(max_timestamp, int(event['time']))
+                status[event['status']] += 1
+                container_name = event['id'][:12]
+                if event['id'] in ids_to_names:
+                    container_name = "%s %s" % (container_name, ids_to_names[event['id']])
+                status_change.append([container_name, event['status']])
+
+            status_text = ", ".join(["%d %s" % (count, st) for st, count in status.iteritems()])
+            msg_title = "%s %s on %s" % (image_name, status_text, self.hostname)
+            msg_body = ("%%%\n"
+                "{image_name} {status} on {hostname}\n"
+                "```\n{status_changes}\n```\n"
+                "%%%").format(
+                    image_name=image_name,
+                    status=status_text,
+                    hostname=self.hostname,
+                    status_changes="\n".join(
+                        ["%s \t%s" % (change[1].upper(), change[0]) for change in status_change])
+            )
+            events.append({
+                'timestamp': max_timestamp,
+                'host': self.hostname,
+                'event_type': EVENT_TYPE,
+                'msg_title': msg_title,
+                'msg_text': msg_body,
+                'source_type_name': EVENT_TYPE,
+                'event_object': 'docker:%s' % image_name,
+            })
 
+        return events
 
-    def _make_tag(self, key, value):
-        return "%s:%s" % (key.lower(), value.strip())
+    def _report_events(self, events):
+        for ev in events:
+            self.log.debug("Creating event: %s" % ev['msg_title'])
+            self.event(ev)
 
-    def _is_container_included(self, instance, tags):
-        def _is_tag_included(tag):
-            for exclude_rule in instance.get("exclude") or []:
-                if re.match(exclude_rule, tag):
-                    for include_rule in instance.get("include") or []:
-                        if re.match(include_rule, tag):
-                            return True
-                    return False
-            return True
-        for tag in tags:
-            if _is_tag_included(tag):
-                return True
-        return False
 
+    # Docker API
 
     def _get_containers(self, instance, with_size=False, get_all=False):
         """Gets the list of running/all containers in Docker."""
@@ -282,22 +309,16 @@ def _get_json(self, uri, params=None, multi=False):
         """Utility method to get and parse JSON streams."""
         if params:
             uri = "%s?%s" % (uri, urllib.urlencode(params))
-        self.log.debug("Connecting to: %s" % uri)
+        self.log.debug("Connecting to Docker API at: %s" % uri)
         req = urllib2.Request(uri, None)
 
-        service_check_name = 'docker.service_up'        
-        service_check_tags = ['host:%s' % self.hostname]
-
         try:
             request = self.url_opener.open(req)
-        except urllib2.URLError, e:            
-            self.service_check(service_check_name, AgentCheck.CRITICAL, tags=service_check_tags)
+        except urllib2.URLError, e:
             if "Errno 13" in str(e):
                 raise Exception("Unable to connect to socket. dd-agent user must be part of the 'docker' group")
             raise
 
-        self.service_check(service_check_name, AgentCheck.OK, tags=service_check_tags)
-
         response = request.read()
         if multi and "}{" in response: # docker api sometimes returns juxtaposed json dictionaries
             response = "[{0}]".format(response.replace("}{", "},{"))
@@ -307,6 +328,37 @@ def _get_json(self, uri, params=None, multi=False):
 
         return json.loads(response)
 
+
+    # Cgroups
+
+    def _find_cgroup_filename_pattern(self):
+        if self._mountpoints:
+            # We try with different cgroups so that it works even if only one is properly working
+            for mountpoint in self._mountpoints.values():
+                stat_file_path_lxc = os.path.join(mountpoint, "lxc")
+                stat_file_path_docker = os.path.join(mountpoint, "docker")
+                stat_file_path_coreos = os.path.join(mountpoint, "system.slice")
+
+                if os.path.exists(stat_file_path_lxc):
+                    return os.path.join('%(mountpoint)s/lxc/%(id)s/%(file)s')
+                elif os.path.exists(stat_file_path_docker):
+                    return os.path.join('%(mountpoint)s/docker/%(id)s/%(file)s')
+                elif os.path.exists(stat_file_path_coreos):
+                    return os.path.join('%(mountpoint)s/system.slice/docker-%(id)s.scope/%(file)s')
+
+        raise Exception("Cannot find Docker cgroup directory. Be sure your system is supported.")
+
+    def _get_cgroup_file(self, cgroup, container_id, filename):
+        # This can't be initialized at startup because cgroups may not be mounted yet
+        if not self._cgroup_filename_pattern:
+            self._cgroup_filename_pattern = self._find_cgroup_filename_pattern()
+
+        return self._cgroup_filename_pattern % (dict(
+                    mountpoint=self._mountpoints[cgroup],
+                    id=container_id,
+                    file=filename,
+                ))
+
     def _find_cgroup(self, hierarchy, docker_root):
         """Finds the mount point for a specified cgroup hierarchy. Works with
         old style and new style mounts."""
@@ -329,12 +381,12 @@ def _find_cgroup(self, hierarchy, docker_root):
     def _parse_cgroup_file(self, stat_file):
         """Parses a cgroup pseudo file for key/values."""
         fp = None
-        self.log.debug("Opening file: %s" % stat_file)
+        self.log.debug("Opening cgroup file: %s" % stat_file)
         try:
             fp = open(stat_file)
             return dict(map(lambda x: x.split(), fp.read().splitlines()))
         except IOError:
-            # Can be because the container got stopped
+            # It is possible that the container got stopped between the API call and now
             self.log.info("Can't open %s. Metrics for this container are skipped." % stat_file)
         finally:
             if fp is not None:


Test patch: