Repository: DataDog/dd-agent

Issue:
Issues with new docker_daemon.py check
I get a slew of warnings with the new docker_daemon.py check. I am running it using datadog/docker-dd-agent, latest version 5.5.0, and have mounted /proc as /host/proc and /sys/fs/cgroup as /host/sys/fs/cgroup. I am 90% sure data is still being collected, it is just annoying because the integration is marked as 'with issues' in the infrastructure list. Is any of this indicative to you all as a real error i can deal with?

```
    docker_daemon
    -------------
      - instance #0 [WARNING]
          Warning: Cannot parse /host/proc/1/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/2/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/3/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/5/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/7/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/8/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/9/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/10/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/11/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/12/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/13/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/15/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/16/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/17/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/18/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/19/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/20/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/21/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/22/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/23/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/24/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/25/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/26/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/27/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/30/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/31/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/38/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/39/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/40/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/42/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/103/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/259/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/271/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/272/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/290/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/323/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/324/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/328/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/408/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/410/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/436/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/474/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/475/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/506/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/534/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/536/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/537/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/538/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/543/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/570/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/580/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/583/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/585/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/665/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/705/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/706/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/822/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/12188/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/16639/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/16642/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/16643/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/16644/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/23083/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/23084/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/23100/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/23110/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/23123/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/23245/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/23262/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/23270/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/23282/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/25222/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/27745/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/27778/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/27819/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28233/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28303/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28395/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28442/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28536/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28560/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28645/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28655/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28753/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28766/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28786/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28812/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28813/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28817/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28818/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28819/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28933/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28948/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28950/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28951/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28979/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/28985/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Cannot parse /host/proc/30586/cgroup content: argument of type 'NoneType' is not iterable
          Warning: Couldn't find pid directory for container: (redacted, x4)
```

It's also possible this is a docker-specific issue, in which case let me know and I'll refile the issue there.



Discussion:

Resolving patch:
diff --git a/checks.d/docker_daemon.py b/checks.d/docker_daemon.py
--- a/checks.d/docker_daemon.py
+++ b/checks.d/docker_daemon.py
@@ -18,6 +18,7 @@
 SERVICE_CHECK_NAME = 'docker.service_up'
 SIZE_REFRESH_RATE = 5 # Collect container sizes every 5 iterations of the check
 MAX_CGROUP_LISTING_RETRIES = 3
+CONTAINER_ID_RE = re.compile('[0-9a-f]{64}')
 
 GAUGE = AgentCheck.gauge
 RATE = AgentCheck.rate
@@ -655,9 +656,16 @@ def _crawl_container_pids(self, container_dict):
                 continue
 
             try:
-                content = dict((line[1], line[2]) for line in content)
-                if 'docker/' in content.get('cpuacct'):
-                    container_id = content['cpuacct'].split('docker/')[1]
+                for line in content:
+                    if line[1] in ('cpu,cpuacct', 'cpuacct,cpu', 'cpuacct') and 'docker' in line[2]:
+                        cpuacct = line[2]
+                        break
+                else:
+                    continue
+
+                match = CONTAINER_ID_RE.search(cpuacct)
+                if match:
+                    container_id = match.group(0)
                     container_dict[container_id]['_pid'] = folder
                     container_dict[container_id]['_proc_root'] = os.path.join(proc_path, folder)
             except Exception, e:


Test patch: