Repository: open-telemetry/opentelemetry-python-contrib

Issue:
EC2 resource detector hangs for a long time outside of an EC2 instance
**Describe your environment** Describe any aspect of your environment relevant to the problem, including your Python version, [platform](https://docs.python.org/3/library/platform.html), version numbers of installed dependencies, information about your cloud hosting provider, etc. If you're reporting a problem with a specific version of a library in this repo, please check whether the problem has been fixed on main.

The environment I initially saw this in was a container running in Docker compose on an AWS EC2 instance but I've been able to reproduce it on my laptop as well. I think it will show up in anything not directly running in AWS.

**Steps to reproduce**
Describe exactly how to reproduce the error. Include a code sample if applicable.

The following code reproduced the issue on my laptop:

```python
from opentelemetry.sdk.extension.aws.resource.ec2 import AwsEc2ResourceDetector
from opentelemetry.sdk.resources import get_aggregated_resources

resource = get_aggregated_resources(
    detectors=[AwsEc2ResourceDetector()]
)
```

**What is the expected behavior?**

It should complete quickly (this is the behavior I see running on an EC2 instance).

**What is the actual behavior?**

What did you see instead?

On my laptop, it will hand ~indefinitely.

Note: one solution is just to remove the resource detector but we'd like to be able to include it and just have it fail silently, which is the behavior we've seen in other resource detectors.

**Additional context**

I think the problem is here: https://github.com/open-telemetry/opentelemetry-python-contrib/blob/80969a06da77d1e616124de178d12a1ebe3ffe7f/sdk-extension/opentelemetry-sdk-extension-aws/src/opentelemetry/sdk/extension/aws/resource/ec2.py#L37

It looks like the request is using a 1000 _second_ timeout which I suspect is intended to be a 1000 _millisecond_ timeout. At least with the server program I've been working on that will block the startup of the program until the request completes.

You can verify by running:

```
curl http://169.254.169.254/latest/api/token
```

Which is one of the requests that the resource detector makes -- it should hang indefinitely as well.



Discussion:

Resolving patch:
diff --git a/instrumentation/opentelemetry-instrumentation-botocore/setup.py b/instrumentation/opentelemetry-instrumentation-botocore/setup.py
new file mode 100644
--- /dev/null
+++ b/instrumentation/opentelemetry-instrumentation-botocore/setup.py
@@ -0,0 +1,31 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import os
+
+import setuptools
+
+BASE_DIR = os.path.dirname(__file__)
+VERSION_FILENAME = os.path.join(
+    BASE_DIR,
+    "src",
+    "opentelemetry",
+    "instrumentation",
+    "botocore",
+    "version.py",
+)
+PACKAGE_INFO = {}
+with open(VERSION_FILENAME) as f:
+    exec(f.read(), PACKAGE_INFO)
+
+setuptools.setup(version=PACKAGE_INFO["__version__"])
diff --git a/instrumentation/opentelemetry-instrumentation-botocore/src/opentelemetry/instrumentation/botocore/__init__.py b/instrumentation/opentelemetry-instrumentation-botocore/src/opentelemetry/instrumentation/botocore/__init__.py
new file mode 100644
--- /dev/null
+++ b/instrumentation/opentelemetry-instrumentation-botocore/src/opentelemetry/instrumentation/botocore/__init__.py
@@ -0,0 +1,222 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+Instrument `Botocore`_ to trace service requests.
+
+There are two options for instrumenting code. The first option is to use the
+``opentelemetry-instrument`` executable which will automatically
+instrument your Botocore client. The second is to programmatically enable
+instrumentation via the following code:
+
+.. _Botocore: https://pypi.org/project/botocore/
+
+Usage
+-----
+
+.. code:: python
+
+    from opentelemetry import trace
+    from opentelemetry.instrumentation.botocore import BotocoreInstrumentor
+    from opentelemetry.sdk.trace import TracerProvider
+    import botocore
+
+    trace.set_tracer_provider(TracerProvider())
+
+    # Instrument Botocore
+    BotocoreInstrumentor().instrument(
+        tracer_provider=trace.get_tracer_provider()
+    )
+
+    # This will create a span with Botocore-specific attributes
+    session = botocore.session.get_session()
+    session.set_credentials(
+        access_key="access-key", secret_key="secret-key"
+    )
+    ec2 = self.session.create_client("ec2", region_name="us-west-2")
+    ec2.describe_instances()
+
+API
+---
+"""
+
+import logging
+
+from botocore.client import BaseClient
+from wrapt import ObjectProxy, wrap_function_wrapper
+
+from opentelemetry.instrumentation.botocore.version import __version__
+from opentelemetry.instrumentation.instrumentor import BaseInstrumentor
+from opentelemetry.sdk.trace import Resource
+from opentelemetry.trace import SpanKind, get_tracer
+
+logger = logging.getLogger(__name__)
+
+
+class BotocoreInstrumentor(BaseInstrumentor):
+    """A instrumentor for Botocore
+
+    See `BaseInstrumentor`
+    """
+
+    def _instrument(self, **kwargs):
+
+        # FIXME should the tracer provider be accessed via Configuration
+        # instead?
+        # pylint: disable=attribute-defined-outside-init
+        self._tracer = get_tracer(
+            __name__, __version__, kwargs.get("tracer_provider")
+        )
+
+        wrap_function_wrapper(
+            "botocore.client",
+            "BaseClient._make_api_call",
+            self._patched_api_call,
+        )
+
+    def _uninstrument(self, **kwargs):
+        unwrap(BaseClient, "_make_api_call")
+
+    def _patched_api_call(self, original_func, instance, args, kwargs):
+
+        endpoint_name = deep_getattr(instance, "_endpoint._endpoint_prefix")
+
+        with self._tracer.start_as_current_span(
+            "{}.command".format(endpoint_name), kind=SpanKind.CONSUMER,
+        ) as span:
+
+            operation = None
+            if args and span.is_recording():
+                operation = args[0]
+                span.resource = Resource(
+                    attributes={
+                        "endpoint": endpoint_name,
+                        "operation": operation.lower(),
+                    }
+                )
+
+            else:
+                span.resource = Resource(
+                    attributes={"endpoint": endpoint_name}
+                )
+
+            add_span_arg_tags(
+                span,
+                endpoint_name,
+                args,
+                ("action", "params", "path", "verb"),
+                {"params", "path", "verb"},
+            )
+
+            if span.is_recording():
+                region_name = deep_getattr(instance, "meta.region_name")
+
+                meta = {
+                    "aws.agent": "botocore",
+                    "aws.operation": operation,
+                    "aws.region": region_name,
+                }
+                for key, value in meta.items():
+                    span.set_attribute(key, value)
+
+            result = original_func(*args, **kwargs)
+
+            if span.is_recording():
+                span.set_attribute(
+                    "http.status_code",
+                    result["ResponseMetadata"]["HTTPStatusCode"],
+                )
+                span.set_attribute(
+                    "retry_attempts",
+                    result["ResponseMetadata"]["RetryAttempts"],
+                )
+
+            return result
+
+
+def unwrap(obj, attr):
+    function = getattr(obj, attr, None)
+    if (
+        function
+        and isinstance(function, ObjectProxy)
+        and hasattr(function, "__wrapped__")
+    ):
+        setattr(obj, attr, function.__wrapped__)
+
+
+def add_span_arg_tags(span, endpoint_name, args, args_names, args_traced):
+    def truncate_arg_value(value, max_len=1024):
+        """Truncate values which are bytes and greater than `max_len`.
+        Useful for parameters like "Body" in `put_object` operations.
+        """
+        if isinstance(value, bytes) and len(value) > max_len:
+            return b"..."
+
+        return value
+
+    def flatten_dict(dict_, sep=".", prefix=""):
+        """
+        Returns a normalized dict of depth 1 with keys in order of embedding
+        """
+        # adapted from https://stackoverflow.com/a/19647596
+        return (
+            {
+                prefix + sep + k if prefix else k: v
+                for kk, vv in dict_.items()
+                for k, v in flatten_dict(vv, sep, kk).items()
+            }
+            if isinstance(dict_, dict)
+            else {prefix: dict_}
+        )
+
+    if not span.is_recording():
+        return
+
+    if endpoint_name not in {"kms", "sts"}:
+        tags = dict(
+            (name, value)
+            for (name, value) in zip(args_names, args)
+            if name in args_traced
+        )
+        tags = flatten_dict(tags)
+        for key, value in {
+            k: truncate_arg_value(v)
+            for k, v in tags.items()
+            if k not in {"s3": ["params.Body"]}.get(endpoint_name, [])
+        }.items():
+            span.set_attribute(key, value)
+
+
+def deep_getattr(obj, attr_string, default=None):
+    """
+    Returns the attribute of ``obj`` at the dotted path given by
+    ``attr_string``, if no such attribute is reachable, returns ``default``.
+
+    >>> deep_getattr(cass, "cluster")
+    <cassandra.cluster.Cluster object at 0xa20c350
+
+    >>> deep_getattr(cass, "cluster.metadata.partitioner")
+    u"org.apache.cassandra.dht.Murmur3Partitioner"
+
+    >>> deep_getattr(cass, "i.dont.exist", default="default")
+    "default"
+    """
+    attrs = attr_string.split(".")
+    for attr in attrs:
+        try:
+            obj = getattr(obj, attr)
+        except AttributeError:
+            return default
+
+    return obj
diff --git a/instrumentation/opentelemetry-instrumentation-botocore/src/opentelemetry/instrumentation/botocore/version.py b/instrumentation/opentelemetry-instrumentation-botocore/src/opentelemetry/instrumentation/botocore/version.py
new file mode 100644
--- /dev/null
+++ b/instrumentation/opentelemetry-instrumentation-botocore/src/opentelemetry/instrumentation/botocore/version.py
@@ -0,0 +1,15 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+__version__ = "0.15.dev0"


Test patch:
diff --git a/instrumentation/opentelemetry-instrumentation-botocore/tests/__init__.py b/instrumentation/opentelemetry-instrumentation-botocore/tests/__init__.py
new file mode 100644
diff --git a/instrumentation/opentelemetry-instrumentation-botocore/tests/test_botocore_instrumentation.py b/instrumentation/opentelemetry-instrumentation-botocore/tests/test_botocore_instrumentation.py
new file mode 100644
--- /dev/null
+++ b/instrumentation/opentelemetry-instrumentation-botocore/tests/test_botocore_instrumentation.py
@@ -0,0 +1,277 @@
+# Copyright The OpenTelemetry Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from unittest.mock import Mock, patch
+
+import botocore.session
+from botocore.exceptions import ParamValidationError
+from moto import (  # pylint: disable=import-error
+    mock_ec2,
+    mock_kinesis,
+    mock_kms,
+    mock_lambda,
+    mock_s3,
+    mock_sqs,
+)
+
+from opentelemetry.instrumentation.botocore import BotocoreInstrumentor
+from opentelemetry.sdk.resources import Resource
+from opentelemetry.test.test_base import TestBase
+
+
+def assert_span_http_status_code(span, code):
+    """Assert on the span"s "http.status_code" tag"""
+    tag = span.attributes["http.status_code"]
+    assert tag == code, "%r != %r" % (tag, code)
+
+
+class TestBotocoreInstrumentor(TestBase):
+    """Botocore integration testsuite"""
+
+    def setUp(self):
+        super().setUp()
+        BotocoreInstrumentor().instrument()
+
+        self.session = botocore.session.get_session()
+        self.session.set_credentials(
+            access_key="access-key", secret_key="secret-key"
+        )
+
+    def tearDown(self):
+        super().tearDown()
+        BotocoreInstrumentor().uninstrument()
+
+    @mock_ec2
+    def test_traced_client(self):
+        ec2 = self.session.create_client("ec2", region_name="us-west-2")
+
+        ec2.describe_instances()
+
+        spans = self.memory_exporter.get_finished_spans()
+        assert spans
+        span = spans[0]
+        self.assertEqual(len(spans), 1)
+        self.assertEqual(span.attributes["aws.agent"], "botocore")
+        self.assertEqual(span.attributes["aws.region"], "us-west-2")
+        self.assertEqual(span.attributes["aws.operation"], "DescribeInstances")
+        assert_span_http_status_code(span, 200)
+        self.assertEqual(
+            span.resource,
+            Resource(
+                attributes={
+                    "endpoint": "ec2",
+                    "operation": "describeinstances",
+                }
+            ),
+        )
+        self.assertEqual(span.name, "ec2.command")
+
+    @mock_ec2
+    def test_not_recording(self):
+        mock_tracer = Mock()
+        mock_span = Mock()
+        mock_span.is_recording.return_value = False
+        mock_tracer.start_span.return_value = mock_span
+        mock_tracer.use_span.return_value.__enter__ = mock_span
+        mock_tracer.use_span.return_value.__exit__ = True
+        with patch("opentelemetry.trace.get_tracer") as tracer:
+            tracer.return_value = mock_tracer
+            ec2 = self.session.create_client("ec2", region_name="us-west-2")
+            ec2.describe_instances()
+            self.assertFalse(mock_span.is_recording())
+            self.assertTrue(mock_span.is_recording.called)
+            self.assertFalse(mock_span.set_attribute.called)
+            self.assertFalse(mock_span.set_status.called)
+
+    @mock_ec2
+    def test_traced_client_analytics(self):
+        ec2 = self.session.create_client("ec2", region_name="us-west-2")
+        ec2.describe_instances()
+
+        spans = self.memory_exporter.get_finished_spans()
+        assert spans
+
+    @mock_s3
+    def test_s3_client(self):
+        s3 = self.session.create_client("s3", region_name="us-west-2")
+
+        s3.list_buckets()
+        s3.list_buckets()
+
+        spans = self.memory_exporter.get_finished_spans()
+        assert spans
+        span = spans[0]
+        self.assertEqual(len(spans), 2)
+        self.assertEqual(span.attributes["aws.operation"], "ListBuckets")
+        assert_span_http_status_code(span, 200)
+        self.assertEqual(
+            span.resource,
+            Resource(
+                attributes={"endpoint": "s3", "operation": "listbuckets"}
+            ),
+        )
+
+        # testing for span error
+        self.memory_exporter.get_finished_spans()
+        with self.assertRaises(ParamValidationError):
+            s3.list_objects(bucket="mybucket")
+        spans = self.memory_exporter.get_finished_spans()
+        assert spans
+        span = spans[2]
+        self.assertEqual(
+            span.resource,
+            Resource(
+                attributes={"endpoint": "s3", "operation": "listobjects"}
+            ),
+        )
+
+    # Comment test for issue 1088
+    @mock_s3
+    def test_s3_put(self):
+        params = dict(Key="foo", Bucket="mybucket", Body=b"bar")
+        s3 = self.session.create_client("s3", region_name="us-west-2")
+        location = {"LocationConstraint": "us-west-2"}
+        s3.create_bucket(Bucket="mybucket", CreateBucketConfiguration=location)
+        s3.put_object(**params)
+
+        spans = self.memory_exporter.get_finished_spans()
+        assert spans
+        span = spans[0]
+        self.assertEqual(len(spans), 2)
+        self.assertEqual(span.attributes["aws.operation"], "CreateBucket")
+        assert_span_http_status_code(span, 200)
+        self.assertEqual(
+            span.resource,
+            Resource(
+                attributes={"endpoint": "s3", "operation": "createbucket"}
+            ),
+        )
+        self.assertEqual(spans[1].attributes["aws.operation"], "PutObject")
+        self.assertEqual(
+            spans[1].resource,
+            Resource(attributes={"endpoint": "s3", "operation": "putobject"}),
+        )
+        self.assertEqual(spans[1].attributes["params.Key"], str(params["Key"]))
+        self.assertEqual(
+            spans[1].attributes["params.Bucket"], str(params["Bucket"])
+        )
+        self.assertTrue("params.Body" not in spans[1].attributes.keys())
+
+    @mock_sqs
+    def test_sqs_client(self):
+        sqs = self.session.create_client("sqs", region_name="us-east-1")
+
+        sqs.list_queues()
+
+        spans = self.memory_exporter.get_finished_spans()
+        assert spans
+        span = spans[0]
+        self.assertEqual(len(spans), 1)
+        self.assertEqual(span.attributes["aws.region"], "us-east-1")
+        self.assertEqual(span.attributes["aws.operation"], "ListQueues")
+        assert_span_http_status_code(span, 200)
+        self.assertEqual(
+            span.resource,
+            Resource(
+                attributes={"endpoint": "sqs", "operation": "listqueues"}
+            ),
+        )
+
+    @mock_kinesis
+    def test_kinesis_client(self):
+        kinesis = self.session.create_client(
+            "kinesis", region_name="us-east-1"
+        )
+
+        kinesis.list_streams()
+
+        spans = self.memory_exporter.get_finished_spans()
+        assert spans
+        span = spans[0]
+        self.assertEqual(len(spans), 1)
+        self.assertEqual(span.attributes["aws.region"], "us-east-1")
+        self.assertEqual(span.attributes["aws.operation"], "ListStreams")
+        assert_span_http_status_code(span, 200)
+        self.assertEqual(
+            span.resource,
+            Resource(
+                attributes={"endpoint": "kinesis", "operation": "liststreams"}
+            ),
+        )
+
+    @mock_kinesis
+    def test_unpatch(self):
+        kinesis = self.session.create_client(
+            "kinesis", region_name="us-east-1"
+        )
+
+        BotocoreInstrumentor().uninstrument()
+
+        kinesis.list_streams()
+        spans = self.memory_exporter.get_finished_spans()
+        assert not spans, spans
+
+    @mock_sqs
+    def test_double_patch(self):
+        sqs = self.session.create_client("sqs", region_name="us-east-1")
+
+        BotocoreInstrumentor().instrument()
+        BotocoreInstrumentor().instrument()
+
+        sqs.list_queues()
+
+        spans = self.memory_exporter.get_finished_spans()
+        assert spans
+        self.assertEqual(len(spans), 1)
+
+    @mock_lambda
+    def test_lambda_client(self):
+        lamb = self.session.create_client("lambda", region_name="us-east-1")
+
+        lamb.list_functions()
+
+        spans = self.memory_exporter.get_finished_spans()
+        assert spans
+        span = spans[0]
+        self.assertEqual(len(spans), 1)
+        self.assertEqual(span.attributes["aws.region"], "us-east-1")
+        self.assertEqual(span.attributes["aws.operation"], "ListFunctions")
+        assert_span_http_status_code(span, 200)
+        self.assertEqual(
+            span.resource,
+            Resource(
+                attributes={"endpoint": "lambda", "operation": "listfunctions"}
+            ),
+        )
+
+    @mock_kms
+    def test_kms_client(self):
+        kms = self.session.create_client("kms", region_name="us-east-1")
+
+        kms.list_keys(Limit=21)
+
+        spans = self.memory_exporter.get_finished_spans()
+        assert spans
+        span = spans[0]
+        self.assertEqual(len(spans), 1)
+        self.assertEqual(span.attributes["aws.region"], "us-east-1")
+        self.assertEqual(span.attributes["aws.operation"], "ListKeys")
+        assert_span_http_status_code(span, 200)
+        self.assertEqual(
+            span.resource,
+            Resource(attributes={"endpoint": "kms", "operation": "listkeys"}),
+        )
+
+        # checking for protection on sts against security leak
+        self.assertTrue("params" not in span.attributes.keys())