Repository: DataDog/dd-agent

Issue:
[Service Discovery] %%host%% and %%port%% don't seem to work
<!--

This issue queue is primarily intended for tracking features, bugs and work items associated with
the dd-agent open-source project.

Prior to submitting an issue please review the following:

- [ ] [Troubleshooting](https://datadog.zendesk.com/hc/en-us/sections/200766955-Troubleshooting) section of our [Knowledge base](https://datadog.zendesk.com/hc/en-us).
- [ ] Contact our [support](http://docs.datadoghq.com/help/) and [send them your logs](https://github.com/DataDog/dd-agent/wiki/Send-logs-to-support).
- [ ] Finally, you can open a Github issue respecting this [title convention](https://github.com/DataDog/dd-agent/blob/master/CONTRIBUTING.md#commits-titles) (it helps us triage).

If you are reporting a new issue, make sure that we do not have any duplicates
already open. You can ensure this by searching the issue list for this
repository. If there is a duplicate, please close your issue and add a comment
to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to
include the BUG REPORT INFORMATION shown below.

---------------------------------------------------
BUG REPORT INFORMATION
---------------------------------------------------
Use the commands below to provide key information from your environment:
You do NOT have to include this information if this is a FEATURE REQUEST
-->

*_Output of the [info page](https://help.datadoghq.com/hc/en-us/articles/203764635-Agent-Status-and-Information) *_

```
2016-09-14 20:05:21,822 | WARNING | dd.collector | utils.service_discovery.config(config.py:32) | No configuration backend provided for service discovery. Only auto config templates will be used.
===================
Collector (v 5.8.5)
===================

  Status date: 2016-09-14 20:05:07 (14s ago)
  Pid: 16
  Platform: Linux-4.4.15-moby-x86_64-with-debian-8.5
  Python Version: 2.7.12, 64bit
  Logs: <stderr>, /var/log/datadog/collector.log

  Clocks
  ======

    NTP offset: -0.4848 s
    System UTC time: 2016-09-14 20:05:22.010079

  Paths
  =====

    conf.d: /etc/dd-agent/conf.d
    checks.d: /opt/datadog-agent/agent/checks.d

  Hostnames
  =========

    socket-hostname: 2291333eb983
    hostname: moby
    socket-fqdn: 2291333eb983

  Checks
  ======

    ntp
    ---
      - Collected 0 metrics, 0 events & 1 service check

    disk
    ----
      - instance #0 [OK]
      - Collected 32 metrics, 0 events & 1 service check

    docker_daemon
    -------------
      - instance #0 [OK]
      - Collected 50 metrics, 1 event & 2 service checks


  Emitters
  ========

    - http_emitter [OK]

===================
Dogstatsd (v 5.8.5)
===================

  Status date: 2016-09-14 20:05:15 (6s ago)
  Pid: 12
  Platform: Linux-4.4.15-moby-x86_64-with-debian-8.5
  Python Version: 2.7.12, 64bit
  Logs: <stderr>, /var/log/datadog/dogstatsd.log

  Flush count: 9
  Packet Count: 0
  Packets per second: 0.0
  Metric count: 1
  Event count: 0
  Service check count: 0


===================
Forwarder (v 5.8.5)
===================

  Status date: 2016-09-14 20:05:19 (2s ago)
  Pid: 14
  Platform: Linux-4.4.15-moby-x86_64-with-debian-8.5
  Python Version: 2.7.12, 64bit
  Logs: <stderr>, /var/log/datadog/forwarder.log

  Queue Size: 0 bytes
  Queue Length: 0
  Flush Count: 32
  Transactions received: 13
  Transactions flushed: 13
  Transactions rejected: 0
```

**Additional environment details (Operating System, Cloud provider, etc):**
Using datadog agent container and redis containers (for testing) via docker for mac (not docker-machine)

**Steps to reproduce the issue:**
1. Create a datadog agent container
2. Create some redis containers (unlinked from the agent)
3. Run `service datadog-agent configcheck` to see the results

**Describe the results you received:**

```
2016-09-14 20:08:26,719 | WARNING | dd.collector | utils.service_discovery.config(config.py:32) | No configuration backend provided for service discovery. Only auto config templates will be used.
docker_daemon.yaml is valid
All yaml files passed. You can now run the Datadog agent.

Loading check configurations...


2016-09-14 20:08:27,034 | WARNING | dd.collector | checks.disk(disk.py:80) | Using `use_mount` in datadog.conf has been deprecated in favor of `use_mount` in disk.yaml
2016-09-14 20:08:27,036 | INFO | dd.collector | config(config.py:834) | Fetching service discovery check configurations.
2016-09-14 20:08:27,044 | WARNING | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:289) | No supported configuration backend was provided, using auto-config only.
2016-09-14 20:08:27,049 | ERROR | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:82) | No IP address was found for container 6a53e83e14c8 (redis)
2016-09-14 20:08:27,049 | ERROR | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:349) | Could not find a value for the template variable host for container 6a53e83e14c8 (redis): Invalid value for variable host.
2016-09-14 20:08:27,049 | WARNING | dd.collector | utils.service_discovery.abstract_sd_backend(abstract_sd_backend.py:65) | Failed to interpolate variable %%host%% for the host parameter. Dropping this configuration.
2016-09-14 20:08:27,052 | WARNING | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:289) | No supported configuration backend was provided, using auto-config only.
2016-09-14 20:08:27,054 | ERROR | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:82) | No IP address was found for container 7643b63897cd (redis)
2016-09-14 20:08:27,054 | ERROR | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:349) | Could not find a value for the template variable host for container 7643b63897cd (redis): Invalid value for variable host.
2016-09-14 20:08:27,054 | WARNING | dd.collector | utils.service_discovery.abstract_sd_backend(abstract_sd_backend.py:65) | Failed to interpolate variable %%host%% for the host parameter. Dropping this configuration.
2016-09-14 20:08:27,056 | WARNING | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:289) | No supported configuration backend was provided, using auto-config only.
2016-09-14 20:08:27,057 | ERROR | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:82) | No IP address was found for container 368a7b0467f4 (redis)
2016-09-14 20:08:27,057 | ERROR | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:349) | Could not find a value for the template variable host for container 368a7b0467f4 (redis): Invalid value for variable host.
2016-09-14 20:08:27,058 | WARNING | dd.collector | utils.service_discovery.abstract_sd_backend(abstract_sd_backend.py:65) | Failed to interpolate variable %%host%% for the host parameter. Dropping this configuration.
2016-09-14 20:08:27,060 | WARNING | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:289) | No supported configuration backend was provided, using auto-config only.
2016-09-14 20:08:27,061 | ERROR | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:82) | No IP address was found for container 9a6aa097cdf9 (redis)
2016-09-14 20:08:27,061 | ERROR | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:349) | Could not find a value for the template variable host for container 9a6aa097cdf9 (redis): Invalid value for variable host.
2016-09-14 20:08:27,061 | WARNING | dd.collector | utils.service_discovery.abstract_sd_backend(abstract_sd_backend.py:65) | Failed to interpolate variable %%host%% for the host parameter. Dropping this configuration.
2016-09-14 20:08:27,063 | WARNING | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:289) | No supported configuration backend was provided, using auto-config only.
2016-09-14 20:08:27,065 | WARNING | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:289) | No supported configuration backend was provided, using auto-config only.
2016-09-14 20:08:27,066 | ERROR | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:82) | No IP address was found for container 294beeca0fdc (redis)
2016-09-14 20:08:27,066 | ERROR | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:349) | Could not find a value for the template variable host for container 294beeca0fdc (redis): Invalid value for variable host.
2016-09-14 20:08:27,067 | WARNING | dd.collector | utils.service_discovery.abstract_sd_backend(abstract_sd_backend.py:65) | Failed to interpolate variable %%host%% for the host parameter. Dropping this configuration.
2016-09-14 20:08:27,067 | INFO | dd.collector | config(config.py:1024) | initialized checks.d checks: ['ntp', 'disk', 'docker_daemon']
2016-09-14 20:08:27,067 | INFO | dd.collector | config(config.py:1025) | initialization failed checks.d checks: []

Source of the configuration objects built by the agent:

Check "agent_metrics":
  source --> YAML file
  config --> {'instances': [{}], 'init_config': {'process_metrics': [{'active': True, 'type': 'gauge', 'name': 'memory_info'}, {'active': True, 'type': 'rate', 'name': 'io_counters'}, {'active': True, 'type': 'gauge', 'name': 'num_threads'}, {'active': False, 'type': 'gauge', 'name': 'connections'}]}}

Check "ntp":
  source --> YAML file
  config --> {'instances': [{'offset_threshold': 60}], 'init_config': None}

Check "disk":
  source --> YAML file
  config --> {'instances': [{'use_mount': False}], 'init_config': None}

Check "docker_daemon":
  source --> YAML file
  config --> {'instances': [{'url': 'unix://var/run/docker.sock'}], 'init_config': {'docker_root': '/host'}}


Containers info:

Number of containers found: 6
    - ID: 6a53e83e14c8 image: redis name: datadogsrvdis_redis_3
    - ID: 7643b63897cd image: redis name: datadogsrvdis_redis_5
    - ID: 368a7b0467f4 image: redis name: datadogsrvdis_redis_4
    - ID: 9a6aa097cdf9 image: redis name: datadogsrvdis_redis_2
    - ID: 2291333eb983 image: datadog/docker-dd-agent name: datadogsrvdis_ddsvcdis_1
    - ID: 294beeca0fdc image: redis name: datadogsrvdis_redis_1
```

**Describe the results you expected:**
I expect a config instance for every redis container, propertly mapped to the containers IP and published randomized external port

**Additional information you deem important (e.g. issue happens only occasionally):**
To bypass the need for `%%host%%` I configure one datadog agent container and just one redis config and link the two containers.  This way I can use the link as the host name in the auto config file.  `%%port%%` still doesn't seem to resolve properly.  It only works if I publish the port externally as the same as it is exposed internally.  This obviously won't work if I am trying to monitor multiple containers of the same type on one node as the ports will bump into each other.



Discussion:
**Bypassing `%%host%%`**
As described above, here is how I bypassed the need for dynamic hostname.

**docker-compose file**

```
version: '2'

services:
  ddsvcdis:
    image: datadog/docker-dd-agent
    environment:
     - "API_KEY=abcNOPE"
     - "SD_BACKEND=docker"
    volumes:
      - /proc/:/host/proc/:ro
      - /sys/fs/cgroup/:/host/sys/fs/cgroup:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - $PWD/redisdb.yaml:/etc/dd-agent/conf.d/auto_conf/redisdb.yaml 
    links:
      - redis

  redis:
    image: redis
    ports:
      - 6379
```

**redisdb.yaml**

```
docker_images:
  - redis

init_config:

instances:
  - host: "redis"
    port: "%%port%%"
```

**config check**

```
2016-09-14 21:00:46,304 | WARNING | dd.collector | utils.service_discovery.config(config.py:32) | No configuration backend provided for service discovery. Only auto config templates will be used.
docker_daemon.yaml is valid
All yaml files passed. You can now run the Datadog agent.

Loading check configurations...


2016-09-14 21:00:46,620 | WARNING | dd.collector | checks.disk(disk.py:80) | Using `use_mount` in datadog.conf has been deprecated in favor of `use_mount` in disk.yaml
2016-09-14 21:00:46,622 | INFO | dd.collector | config(config.py:834) | Fetching service discovery check configurations.
2016-09-14 21:00:46,633 | WARNING | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:289) | No supported configuration backend was provided, using auto-config only.
2016-09-14 21:00:46,634 | WARNING | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:289) | No supported configuration backend was provided, using auto-config only.
2016-09-14 21:00:46,643 | INFO | dd.collector | config(config.py:1024) | initialized checks.d checks: ['ntp', 'disk', 'redisdb', 'docker_daemon']
2016-09-14 21:00:46,643 | INFO | dd.collector | config(config.py:1025) | initialization failed checks.d checks: []

Source of the configuration objects built by the agent:

Check "agent_metrics":
  source --> YAML file
  config --> {'instances': [{}], 'init_config': {'process_metrics': [{'active': True, 'type': 'gauge', 'name': 'memory_info'}, {'active': True, 'type': 'rate', 'name': 'io_counters'}, {'active': True, 'type': 'gauge', 'name': 'num_threads'}, {'active': False, 'type': 'gauge', 'name': 'connections'}]}}

Check "ntp":
  source --> YAML file
  config --> {'instances': [{'offset_threshold': 60}], 'init_config': None}

Check "disk":
  source --> YAML file
  config --> {'instances': [{'use_mount': False}], 'init_config': None}

Check "redisdb":
  source --> auto-configuration
  config --> {'instances': [{'host': 'redis', 'port': u'6379'}], 'init_config': {}}

Check "docker_daemon":
  source --> YAML file
  config --> {'instances': [{'url': 'unix://var/run/docker.sock'}], 'init_config': {'docker_root': '/host'}}


Containers info:

Number of containers found: 2
    - ID: 68ae79ccaac2 image: datadog/docker-dd-agent name: datadogsrvdis_ddsvcdis_1
    - ID: 470cbe263562 image: redis name: datadogsrvdis_redis_1
```

As you can see, port 6379 is being used, which isn't correct. It should be the randomized published port docker provides.

So the port bug is with this line of code:
https://github.com/DataDog/dd-agent/blob/master/utils/service_discovery/sd_docker_backend.py#L119
`ports = map(lambda x: x.split('/')[0], container_inspect['NetworkSettings']['Ports'].keys())`

Considering the following ports:

```
  redis:
    image: redis
    ports:
      - "6379"
      - "3000"
      - "3000-3005"
      - "8000:8000"
      - "9090-9091:8080-8081"
      - "49100:22"
      - "127.0.0.1:8001:8001"
      - "127.0.0.1:5000-5010:5000-5010"
```

`container_inspect['NetworkSettings']['Ports']` looks like:

```
u'Ports': {u'22/tcp': [{u'HostIp': u'0.0.0.0',
                                              u'HostPort': u'49100'}],
                                 u'3000/tcp': [{u'HostIp': u'0.0.0.0',
                                                u'HostPort': u'32815'},
                                               {u'HostIp': u'0.0.0.0',
                                                u'HostPort': u'32816'}],
                                 u'3001/tcp': [{u'HostIp': u'0.0.0.0',
                                                u'HostPort': u'32814'}],
                                 u'3002/tcp': [{u'HostIp': u'0.0.0.0',
                                                u'HostPort': u'32813'}],
                                 u'3003/tcp': [{u'HostIp': u'0.0.0.0',
                                                u'HostPort': u'32812'}],
                                 u'3004/tcp': [{u'HostIp': u'0.0.0.0',
                                                u'HostPort': u'32811'}],
                                 u'3005/tcp': [{u'HostIp': u'0.0.0.0',
                                                u'HostPort': u'32817'}],
                                 u'5000/tcp': [{u'HostIp': u'127.0.0.1',
                                                u'HostPort': u'5000'}],
                                 u'5001/tcp': [{u'HostIp': u'127.0.0.1',
                                                u'HostPort': u'5001'}],
                                 u'5002/tcp': [{u'HostIp': u'127.0.0.1',
                                                u'HostPort': u'5002'}],
                                 u'5003/tcp': [{u'HostIp': u'127.0.0.1',
                                                u'HostPort': u'5003'}],
                                 u'5004/tcp': [{u'HostIp': u'127.0.0.1',
                                                u'HostPort': u'5004'}],
                                 u'5005/tcp': [{u'HostIp': u'127.0.0.1',
                                                u'HostPort': u'5005'}],
                                 u'5006/tcp': [{u'HostIp': u'127.0.0.1',
                                                u'HostPort': u'5006'}],
                                 u'5007/tcp': [{u'HostIp': u'127.0.0.1',
                                                u'HostPort': u'5007'}],
                                 u'5008/tcp': [{u'HostIp': u'127.0.0.1',
                                                u'HostPort': u'5008'}],
                                 u'5009/tcp': [{u'HostIp': u'127.0.0.1',
                                                u'HostPort': u'5009'}],
                                 u'5010/tcp': [{u'HostIp': u'127.0.0.1',
                                                u'HostPort': u'5010'}],
                                 u'6379/tcp': [{u'HostIp': u'0.0.0.0',
                                                u'HostPort': u'32810'}],
                                 u'8000/tcp': [{u'HostIp': u'0.0.0.0',
                                                u'HostPort': u'8000'}],
                                 u'8001/tcp': [{u'HostIp': u'127.0.0.1',
                                                u'HostPort': u'8001'}],
                                 u'8080/tcp': [{u'HostIp': u'0.0.0.0',
                                                u'HostPort': u'9090'}],
                                 u'8081/tcp': [{u'HostIp': u'0.0.0.0',
                                                u'HostPort': u'9091'}]},
```

Only the keys are considered, which are the internal ports.  What is needed is the `HostPort` within the one of the values.

i think it needs to be
`ports = map(lambda x: x[1][0]['HostPort'], container_inspect['NetworkSettings']['Ports'].iteritems())`

Hi @hypergig 
Thanks a lot for the thorough report! The way ports are handled is not a bug. It works as long as the agent can reach the container directly, which is nice because ports are not always exposed to the host so we don't want to rely on that.

The issue I see here is the `host` variable. It should resolve but doesn't for some reason. Once it does the agent should be able to query `CONTAINER_IP:6379` successfully. Could you paste here a docker inspect of your redis container(s) please?

Thanks again

@hkaj it surprises me this is the way ports are handled.  Consider the scenario in which you have multiple containers of the same image running on one node.  Say nginx containers.  There is only one port `80` available on the host node.  What you are saying is you require me to statically map `80` on each individual nginx replica to a port on the host node. Docker actually already does this for you when you publish an internally `EXPOSED` port.  It provides a randomized host node port statically mapped to the internal `80`, for instance.  Why would I want to reinvent that process?

Publishing internal `EXPOSED` ports is exactly how you give external access to a container.  The agent will be able to reach the container directly through the published port.

They only other way is to link every single container you want to monitor to the agent container but that comes with much bigger issues. For one, the agent container (which holds the links) will have to be restarted for every new link as the links are created on container startup.  That makes autoscaling more complicated.  Also, what if you don't want the agent running in a container, because you want to monitor docker itself.

Regarding service discovery, what makes the most sense to me is:
- Ports used for monitoring need to be published
- The datadog agent, via the docker api, continually sniff out all containers matching the `image` name defined in the auto config file
- For each container matching, fetch the IP and published port(s) as defined in the auto config
  - `%%host%%` resolves to the IP
  - `%%port_n%%` resolves to the externally published `EXPOSED` port, numerically sorted by  `EXPOSED` port e.g if you expose `80` and `8080`, `%%port_0%%` will be whatever `80` was mapped to externally

If you want to connect via the internal port, then you might as well ship one linked agent with each app container, with the old school integration file that has the ports hardcoded as internal ports never change.

@hypergig what I mean is that the agent is expected to query your redis container directly, not through an exposed port.
For example (also with docker on mac) here's the network part of `docker inspect redis_container` I see:

``` json
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "b92c148352696aa2f6add174b58fb52d12c12d369ded6da6ebf9669e288183d4",
                    "EndpointID": "b393f0f43cd65e93d57d30d3c61c60122b32cbaeb194c245076c5a744b085275",
                    "Gateway": "172.17.0.1",
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:11:00:02"
                }
            }
```

The agent will use the `IPAddress` field of the `bridge` network as the host variable (bridge is selected by default, you can use `%%host_network-id%%` to specify another network as long as the agent has access to it).

With this IP address, connecting to your redis container on port 6379 should work.
Although apparently it doesn't on your setup, so could you please paste here the result of `docker inspect redis_container-id`, `docker --version` and `docker-machine --version`? That would help us investigate what's failing here.

Thanks

My apologies @hkaj, that is much more clever than my way!

**docker inspect redis_container-id**

```
[
    {
        "Id": "61177378e6d5d0448921d8ee060dcc175cd9430997d79c2d36fbf33b1240697d",
        "Created": "2016-09-16T14:49:55.745632572Z",
        "Path": "docker-entrypoint.sh",
        "Args": [
            "redis-server"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 2541,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2016-09-16T14:49:56.427019633Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:0d1cbfaa41da0776bb606103843ade3f26f4c4e3af7825e6f9ed1e8f152ba057",
        "ResolvConfPath": "/var/lib/docker/containers/61177378e6d5d0448921d8ee060dcc175cd9430997d79c2d36fbf33b1240697d/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/61177378e6d5d0448921d8ee060dcc175cd9430997d79c2d36fbf33b1240697d/hostname",
        "HostsPath": "/var/lib/docker/containers/61177378e6d5d0448921d8ee060dcc175cd9430997d79c2d36fbf33b1240697d/hosts",
        "LogPath": "/var/lib/docker/containers/61177378e6d5d0448921d8ee060dcc175cd9430997d79c2d36fbf33b1240697d/61177378e6d5d0448921d8ee060dcc175cd9430997d79c2d36fbf33b1240697d-json.log",
        "Name": "/datadogsrvdis_redis_1",
        "RestartCount": 0,
        "Driver": "aufs",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "json-file",
                "Config": {}
            },
            "NetworkMode": "datadogsrvdis_default",
            "PortBindings": {
                "6379/tcp": [
                    {
                        "HostIp": "",
                        "HostPort": ""
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": [],
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 0,
            "Memory": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Name": "aufs",
            "Data": null
        },
        "Mounts": [
            {
                "Name": "ad6186a6ce391b69284159c2b4dd1419d139111ceb8b5a23b6841c44b9629d3e",
                "Source": "/var/lib/docker/volumes/ad6186a6ce391b69284159c2b4dd1419d139111ceb8b5a23b6841c44b9629d3e/_data",
                "Destination": "/data",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "61177378e6d5",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "6379/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "no_proxy=*.local, 169.254/16",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GOSU_VERSION=1.7",
                "REDIS_VERSION=3.2.3",
                "REDIS_DOWNLOAD_URL=http://download.redis.io/releases/redis-3.2.3.tar.gz",
                "REDIS_DOWNLOAD_SHA1=92d6d93ef2efc91e595c8bf578bf72baff397507"
            ],
            "Cmd": [
                "redis-server"
            ],
            "Image": "redis",
            "Volumes": {
                "/data": {}
            },
            "WorkingDir": "/data",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.compose.config-hash": "ceec4b3cf663c932b2b5f5e60529cfcbc939b3849ba1c179ee5173d236b604fa",
                "com.docker.compose.container-number": "1",
                "com.docker.compose.oneoff": "False",
                "com.docker.compose.project": "datadogsrvdis",
                "com.docker.compose.service": "redis",
                "com.docker.compose.version": "1.8.0"
            }
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "88620b12a817455dc1792b9bee69166350e892b33c060d0b0f0aebf156237057",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "6379/tcp": [
                    {
                        "HostIp": "0.0.0.0",
                        "HostPort": "32776"
                    }
                ]
            },
            "SandboxKey": "/var/run/docker/netns/88620b12a817",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "datadogsrvdis_default": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": [
                        "redis",
                        "61177378e6d5"
                    ],
                    "NetworkID": "cbbee1b029f8ad0ea48acd68540f21b8c0fa51ffa45aaa9b77d423a774e4ed55",
                    "EndpointID": "c30e36476d3f05924788fc1a5e228c7142e8a657c191537b77723a52ce9f5c2e",
                    "Gateway": "172.19.0.1",
                    "IPAddress": "172.19.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:ac:13:00:02"
                }
            }
        }
    }
]
```

**docker --version**

```
Docker version 1.12.1, build 6f9534c
```

**docker-machine --version**
Not using docker machine, I am using docker for mac.

and from the datadog agent container

```
root@b3d802bfa584:/# nc -vzw 1 172.19.0.2 6379
datadogsrvdis_redis_1.datadogsrvdis_default [172.19.0.2] 6379 (?) open
```

The port is absolutely opened, just need to get the right ip... lets test it with the ip hardcoded:

```
docker_images:
  - redis

init_config:

instances:
  - host: "172.19.0.2"
    port: "%%port%%"
```

Checking...

```
root@823097c1cd73:/# 
root@823097c1cd73:/# service datadog-agent configcheck
2016-09-16 15:26:58,989 | WARNING | dd.collector | utils.service_discovery.config(config.py:32) | No configuration backend provided for service discovery. Only auto config templates will be used.
All yaml files passed. You can now run the Datadog agent.

Loading check configurations...


2016-09-16 15:26:59,185 | INFO | dd.collector | config(config.py:834) | Fetching service discovery check configurations.
2016-09-16 15:26:59,194 | WARNING | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:289) | No supported configuration backend was provided, using auto-config only.
2016-09-16 15:26:59,196 | WARNING | dd.collector | utils.service_discovery.sd_docker_backend(sd_docker_backend.py:289) | No supported configuration backend was provided, using auto-config only.
2016-09-16 15:26:59,204 | INFO | dd.collector | config(config.py:1024) | initialized checks.d checks: ['redisdb']
2016-09-16 15:26:59,204 | INFO | dd.collector | config(config.py:1025) | initialization failed checks.d checks: []

Source of the configuration objects built by the agent:

Check "redisdb":
  source --> auto-configuration
  config --> {'instances': [{'host': '172.19.0.2', 'port': u'6379'}], 'init_config': {}}


Containers info:

Number of containers found: 2
    - ID: 823097c1cd73 image: datadog/docker-dd-agent name: datadogsrvdis_ddsvcdis_1
    - ID: 014c73a5ddf7 image: redis name: datadogsrvdis_redis_1


root@823097c1cd73:/# 
root@823097c1cd73:/# 
root@823097c1cd73:/# 
root@823097c1cd73:/# service datadog-agent info       
2016-09-16 15:27:09,329 | WARNING | dd.collector | utils.service_discovery.config(config.py:32) | No configuration backend provided for service discovery. Only auto config templates will be used.
===================
Collector (v 5.8.5)
===================

  Status date: 2016-09-16 15:26:51 (17s ago)
  Pid: 18
  Platform: Linux-4.4.20-moby-x86_64-with-debian-8.5
  Python Version: 2.7.12, 64bit
  Logs: <stderr>, /var/log/datadog/collector.log

  Clocks
  ======

    NTP offset: 0.0018 s
    System UTC time: 2016-09-16 15:27:09.495978

  Paths
  =====

    conf.d: /opt/datadog-agent/agent/conf.d
    checks.d: /opt/datadog-agent/agent/checks.d

  Hostnames
  =========

    socket-hostname: 823097c1cd73
    hostname: moby
    socket-fqdn: 823097c1cd73

  Checks
  ======

    redisdb
    -------
      - instance #0 [OK]
      - Collected 24 metrics, 0 events & 2 service checks


  Emitters
  ========

    - http_emitter [OK]

===================
Dogstatsd (v 5.8.5)
===================

  Status date: 2016-09-16 15:27:07 (2s ago)
  Pid: 15
  Platform: Linux-4.4.20-moby-x86_64-with-debian-8.5
  Python Version: 2.7.12, 64bit
  Logs: <stderr>, /var/log/datadog/dogstatsd.log

  Flush count: 2
  Packet Count: 0
  Packets per second: 0.0
  Metric count: 1
  Event count: 0
  Service check count: 0

===================
Forwarder (v 5.8.5)
===================

  Status date: 2016-09-16 15:27:07 (2s ago)
  Pid: 17
  Platform: Linux-4.4.20-moby-x86_64-with-debian-8.5
  Python Version: 2.7.12, 64bit
  Logs: <stderr>, /var/log/datadog/forwarder.log

  Queue Size: 435 bytes
  Queue Length: 1
  Flush Count: 5
  Transactions received: 2
  Transactions flushed: 1
  Transactions rejected: 0


root@823097c1cd73:/# 
```

It works! So the issue is with the `%%host%%` var then.

Thank you for all these details @hypergig !
I localized the issue and will issue a fix today.


Resolving patch:
diff --git a/utils/service_discovery/sd_docker_backend.py b/utils/service_discovery/sd_docker_backend.py
--- a/utils/service_discovery/sd_docker_backend.py
+++ b/utils/service_discovery/sd_docker_backend.py
@@ -44,22 +44,20 @@ def __init__(self, agentConfig):
     def _get_host_address(self, c_inspect, tpl_var):
         """Extract the container IP from a docker inspect object, or the kubelet API."""
         c_id, c_img = c_inspect.get('Id', ''), c_inspect.get('Config', {}).get('Image', '')
-        tpl_parts = tpl_var.split('_')
-
-        # a specifier was given
-        if len(tpl_parts) > 1:
-            networks = c_inspect.get('NetworkSettings', {}).get('Networks') or {}
-            ip_dict = {}
-            for net_name, net_desc in networks.iteritems():
-                ip = net_desc.get('IPAddress')
-                if ip:
-                    ip_dict[net_name] = ip
-            ip_addr = self._extract_ip_from_networks(ip_dict, tpl_var)
-            if ip_addr:
-                return ip_addr
+
+        networks = c_inspect.get('NetworkSettings', {}).get('Networks') or {}
+        ip_dict = {}
+        for net_name, net_desc in networks.iteritems():
+            ip = net_desc.get('IPAddress')
+            if ip:
+                ip_dict[net_name] = ip
+        ip_addr = self._extract_ip_from_networks(ip_dict, tpl_var)
+        if ip_addr:
+            return ip_addr
 
         # try to get the bridge (default) IP address
-        log.debug("No network was specified for container %s (%s), trying with IPAddress field" % (c_id[:12], c_img))
+        log.debug("No IP address was found in container %s (%s) "
+            "networks, trying with the IPAddress field" % (c_id[:12], c_img))
         ip_addr = c_inspect.get('NetworkSettings', {}).get('IPAddress')
         if ip_addr:
             return ip_addr
@@ -87,7 +85,7 @@ def _extract_ip_from_networks(self, ip_dict, tpl_var):
         """Extract a single IP from a dictionary made of network names and IPs."""
         if not ip_dict:
             return None
-        tpl_parts = tpl_var.split('_')
+        tpl_parts = tpl_var.split('_', 1)
 
         # no specifier
         if len(tpl_parts) < 2:
@@ -108,7 +106,7 @@ def _get_fallback_ip(self, ip_dict):
             return ip_dict['bridge']
         else:
             last_key = sorted(ip_dict.iterkeys())[-1]
-            log.warning("Trying with the last key: '%s'." % last_key)
+            log.warning("Trying with the last (sorted) network: '%s'." % last_key)
             return ip_dict[last_key]
 
     def _get_port(self, container_inspect, tpl_var):
@@ -142,7 +140,7 @@ def _extract_port_from_list(self, ports, tpl_var):
         if not ports:
             return None
 
-        tpl_parts = tpl_var.split('_')
+        tpl_parts = tpl_var.split('_', 1)
 
         if len(tpl_parts) == 1:
             log.debug("No index was passed for template variable %s. "


Test patch:
diff --git a/tests/core/test_service_discovery.py b/tests/core/test_service_discovery.py
--- a/tests/core/test_service_discovery.py
+++ b/tests/core/test_service_discovery.py
@@ -209,7 +209,7 @@ def test_get_host_address(self, mock_check_yaml, mock_get):
                 'Networks': {
                     'bridge': {'IPAddress': '172.17.0.2'},
                     'foo': {'IPAddress': '192.168.0.2'}}}},
-             'host', '127.0.0.1'),
+             'host', '172.17.0.2'),
 
             ({'NetworkSettings': {'Networks': {}}}, 'host', None),
             ({'NetworkSettings': {'Networks': {}}}, 'host_bridge', None),
@@ -349,7 +349,7 @@ def test_fill_tpl(self):
                     'Networks': {'bridge': {'IPAddress': '172.17.0.2'}}}
                   },
                  {'host': '%%host%%', 'port': 1337}, ['host'], ['foo', 'bar:baz']),
-                ({'host': '%%host%%', 'port': 1337, 'tags': ['foo', 'bar:baz']}, {'host': '127.0.0.1'}),
+                ({'host': '%%host%%', 'port': 1337, 'tags': ['foo', 'bar:baz']}, {'host': '172.17.0.2'}),
             ),
             (
                 ({'NetworkSettings': {