Repository: DataDog/dd-agent

Issue:
Problem retrieving custom EC2 tags | HTTPError: HTTP Error 404: Not Found
Seeing the following logged for `dd-collector` using the AWS integration.

```
2015-09-02 18:49:56 UTC | ERROR | dd.collector | util(util.py:348) | Problem retrieving custom EC2 tags
Traceback (most recent call last):
  File "/opt/datadog-agent/agent/util.py", line 336, in get_tags
    iam_role = urllib2.urlopen(EC2.METADATA_URL_BASE + "/iam/security-credentials").read().strip()
  File "/opt/datadog-agent/embedded/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/datadog-agent/embedded/lib/python2.7/urllib2.py", line 437, in open
    response = meth(req, response)
  File "/opt/datadog-agent/embedded/lib/python2.7/urllib2.py", line 550, in http_response
    'http', request, response, code, msg, hdrs)
  File "/opt/datadog-agent/embedded/lib/python2.7/urllib2.py", line 475, in error
    return self._call_chain(*args)
  File "/opt/datadog-agent/embedded/lib/python2.7/urllib2.py", line 409, in _call_chain
    result = func(*args)
  File "/opt/datadog-agent/embedded/lib/python2.7/urllib2.py", line 558, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
HTTPError: HTTP Error 404: Not Found
```



Discussion:
Hi @nodesocket - I submitted a [PR](https://github.com/DataDog/dd-agent/pull/1332/files) in that area to support multiple AWS regions.  I will resubmit an updated one and would like to address this issue.  Can you provide a bit more information about your EC2 state?
1. Are you associating [IAM Roles](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html) with your EC2 instance(s)?
2. What is the output of `curl http://169.254.169.254/latest/meta-data/iam/info`
3. What is the output of `curl http://169.254.169.254/latest/meta-data/iam/info/` (The trailing slash [is important](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-instance-metadata.html))
4. What is the output of `curl http://169.254.169.254/latest/meta-data/iam/security-credentials/`

Thanks.

#### 1.) Yes, associating IAM roles on some instances, but the instances am seeing the error from no.

#### 2.) On the instances I am seeing the error from:

```
ubuntu@brandcast-0:~$ curl http://169.254.169.254/latest/meta-data/iam/info
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
```

#### 3.)

```
ubuntu@brandcast-0:~$ curl http://169.254.169.254/latest/meta-data/iam/info/
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
```

#### 4.)

```
ubuntu@brandcast-0:~$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>404 - Not Found</title>
 </head>
 <body>
  <h1>404 - Not Found</h1>
 </body>
</html>
```

Any update on this issue? Still seeing these errors.

> Sep 15 16:46:10 processing1 dd.collector:  ERROR (util.py:348): Problem retrieving custom EC2 tags#012Traceback (most recent call last):#012  File "/opt/datadog-agent/agent/util.py", line 343, in get_tags#012    tag_object = connection.get_all_tags({'resource-id': EC2.metadata['instance-id']})#012  File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/boto/ec2/connection.py", line 4177, in get_all_tags#012    [('item', Tag)], verb='POST')#012  File "/opt/datadog-agent/embedded/lib/python2.7/site-packages/boto/connection.py", line 1185, in get_list#012    raise self.ResponseError(response.status, response.reason, body)#012EC2ResponseError: EC2ResponseError: 403 Forbidden#012<?xml version="1.0" encoding="UTF-8"?>#012<Response><Errors><Error><Code>UnauthorizedOperation</Code><Message>You are not authorized to perform this operation.</Message></Error></Errors><RequestID>9bf53ee1-74d6-4fb2-84d4-0c04f88cd18d</RequestID></Response>

Sorry for the late answer @nodesocket.

Datadog agent relies on (among others) http://169.254.169.254/latest/meta-data/iam/info endpoint to retrieve the instance's metadata. If the endpoint is unavailable or if it returns an HTTP error status code,  then no EC2 tag is fetched.

We've been running some tests, and found that an **IAM role** assignation is required to be able to use this endpoint. The error message, I reckon, is quite confusing here: we will likely catch this error to display a more comprehensive one in a next version.

Is this error disrupting the proper functionning of Datadog agent on your instance ?

Thank you

Thanks for the reply. No the error is not disrupting anything, just noticed it in our logs. Perhaps if the instance does not have a IAM role assigned just return instead of showing the 404 error.

Thanks for the further details @nodesocket. 
I am assigning the ticket to our 5.6.0 agent milestone, so we'll make sure to handle this case more gracefully :art: 


Resolving patch:
diff --git a/util.py b/util.py
--- a/util.py
+++ b/util.py
@@ -340,12 +340,35 @@ class EC2(object):
     TIMEOUT = 0.1  # second
     metadata = {}
 
+    class NoIAMRole(Exception):
+        """
+        Instance has no associated IAM role.
+        """
+        pass
+
+    @staticmethod
+    def get_iam_role():
+        """
+        Retrieve instance's IAM role.
+        Raise `NoIAMRole` when unavailable.
+        """
+        try:
+            return urllib2.urlopen(EC2.METADATA_URL_BASE + "/iam/security-credentials/").read().strip()
+        except urllib2.HTTPError as err:
+            if err.code == 404:
+                raise EC2.NoIAMRole()
+            raise
+
     @staticmethod
     def get_tags(agentConfig):
+        """
+        Retrieve AWS EC2 tags.
+        """
         if not agentConfig['collect_instance_metadata']:
             log.info("Instance metadata collection is disabled. Not collecting it.")
             return []
 
+        EC2_tags = []
         socket_to = None
         try:
             socket_to = socket.getdefaulttimeout()
@@ -354,7 +377,7 @@ def get_tags(agentConfig):
             pass
 
         try:
-            iam_role = urllib2.urlopen(EC2.METADATA_URL_BASE + "/iam/security-credentials/").read().strip()
+            iam_role = EC2.get_iam_role()
             iam_params = json.loads(urllib2.urlopen(EC2.METADATA_URL_BASE + "/iam/security-credentials/" + unicode(iam_role)).read().strip())
             instance_identity = json.loads(urllib2.urlopen(EC2.INSTANCE_IDENTITY_URL).read().strip())
             region = instance_identity['region']
@@ -376,9 +399,13 @@ def get_tags(agentConfig):
             if agentConfig.get('collect_security_groups') and EC2.metadata.get('security-groups'):
                 EC2_tags.append(u"security-group-name:{0}".format(EC2.metadata.get('security-groups')))
 
+        except EC2.NoIAMRole:
+            log.warning(
+                u"Unable to retrieve AWS EC2 custom tags: "
+                u"an IAM role associated with the instance is required"
+            )
         except Exception:
             log.exception("Problem retrieving custom EC2 tags")
-            EC2_tags = []
 
         try:
             if socket_to is None:


Test patch: