internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: DataDog/dd-trace-py

Issue:
"cannot decode v0.3 services payload" error when tracing aiopg
I have a Python application using aiopg and monitored by Datadog. When starting, it sends a bad frame to `trace-agent` and then everything goes fine.

Versions:
- datadog-agent: 6.2.1
- ddtrace-py: 0.12.1
- aiopg: 0.14.0

Here is a minimalist test case:

```python
import asyncio
import aiopg

from ddtrace import patch

patch(aiopg=True)


async def main():
    async with aiopg.connect(host=None) as db:
        pass

    while True:
        await asyncio.sleep(0.1)


loop = asyncio.get_event_loop()
loop.run_until_complete(main())
```

This logs the following error:
```
failed_to_send services to Agent: HTTP error status 400, reason Bad Request, message Content-Type: text/plain; charset=utf-8
X-Content-Type-Options: nosniff
Date: Mon, 18 Jun 2018 15:25:18 GMT
Content-Length: 59
```

And then `trace-agent` reports:
```
trace-agent[4437]: 2018-06-18 15:31:16 ERROR (receiver.go:275) - cannot decode v0.3 services payload: msgp: attempted to decode type "nil" with method for "str"
```

I believe this is related to https://github.com/DataDog/datadog-trace-agent/issues/350.


Discussion:
Hello @k4nar ! this error seems related to the services payload that is deprecated and is going to be removed in Tracer 1.0. It seems a bug in one of our integrations that is still using it but it doesn't affect tracing reporting. Did you test the latest version of our tracer? https://github.com/DataDog/dd-trace-py/releases/tag/v0.12.1

If yes, can you list the integrations you're using? Probably we need to update them as we did for others (https://github.com/DataDog/dd-trace-py/pull/468)

Thanks for reporting!
Thanks for the response :) .

In the test I've provided it happens with dd-trace-py 0.12.1 and with only the `aiopg` integration enabled.
OK I think the problem is the same as the other one. Let me check the `aiopg` integration so I can provide you a quick branch to test.

Resolving patch:
diff --git a/ddtrace/contrib/aiopg/connection.py b/ddtrace/contrib/aiopg/connection.py
--- a/ddtrace/contrib/aiopg/connection.py
+++ b/ddtrace/contrib/aiopg/connection.py
@@ -4,8 +4,8 @@
 from aiopg.utils import _ContextManager
 
 from .. import dbapi
-from ...ext import sql
 from ...pin import Pin
+from ...ext import sql, AppTypes
 
 
 class AIOTracedCursor(wrapt.ObjectProxy):
@@ -63,10 +63,11 @@ def callproc(self, proc, args):
 class AIOTracedConnection(wrapt.ObjectProxy):
     """ TracedConnection wraps a Connection with tracing code. """
 
-    def __init__(self, conn):
+    def __init__(self, conn, pin=None):
         super(AIOTracedConnection, self).__init__(conn)
         name = dbapi._get_vendor(conn)
-        Pin(service=name, app=name).onto(self)
+        db_pin = pin or Pin(service=name, app=name, app_type=AppTypes.db)
+        db_pin.onto(self)
 
     def cursor(self, *args, **kwargs):
         # unfortunately we also need to patch this method as otherwise "self"


Test patch:
Repository: DataDog/dd-trace-py

Issue:
Configuring tracer endpoint
The current tracer implementation relies on a global variable that gets initialised with the default Tracer() values. Once this is done, traces and spans can be created and published.

Now, we want to have the tracer send traces to a different endpoint. The use-case for this is that the traced program lives in a container and needs to talk to the trace agent running on the host. As a result, `localhost:8126` is not a valid endpoint, from the application point-of-view.

Configuring the tracer client is achieved by the `bootstrap` phase of the application and calls the `tracer.configure()` function to do so.

The problem is that this happens after traces have already been submitted. As a result, https://github.com/DataDog/dd-trace-py/blob/master/ddtrace/writer.py#L36 has already created an AsyncWorker that tries to send traces to localhost.

When `trace.configure()` is called, it will happily overwrite the existing worker and replace it with the new one, using the correct endpoint, **but it never stops the thread started by the previous worker**.

As a result:

* the library logs errors submitting traces, since it targets the wrong endpoint. From a user perspective, this is misleading since all traces generated after the config change are properly uploaded and the user might think the configuration step never succeeded
* all traces generated before the configuration is done are lost.





Discussion:

Resolving patch:
diff --git a/ddtrace/tracer.py b/ddtrace/tracer.py
--- a/ddtrace/tracer.py
+++ b/ddtrace/tracer.py
@@ -1,6 +1,6 @@
 import functools
 import logging
-from os import getpid
+from os import environ, getpid
 
 from .ext import system
 from .provider import DefaultContextProvider
@@ -27,7 +27,7 @@ class Tracer(object):
         from ddtrace import tracer
         trace = tracer.trace("app.request", "web-server").finish()
     """
-    DEFAULT_HOSTNAME = 'localhost'
+    DEFAULT_HOSTNAME = environ.get('DATADOG_TRACE_AGENT_HOSTNAME', 'localhost')
     DEFAULT_PORT = 8126
 
     def __init__(self):


Test patch:
diff --git a/tests/test_tracer.py b/tests/test_tracer.py
--- a/tests/test_tracer.py
+++ b/tests/test_tracer.py
@@ -4,7 +4,9 @@
 
 import time
 from os import getpid
+import sys
 
+import mock
 from nose.tools import assert_raises, eq_, ok_
 from unittest.case import SkipTest
 
@@ -517,3 +519,16 @@ def get_dummy_tracer():
     tracer = Tracer()
     tracer.writer = DummyWriter()
     return tracer
+
+
+def test_default_hostname_from_env():
+    # it should use default hostname from DATADOG_TRACE_AGENT_HOSTNAME if available
+    try:
+        with mock.patch.dict('os.environ', {'DATADOG_TRACE_AGENT_HOSTNAME': 'customhost'}):
+            del sys.modules['ddtrace.tracer']  # force reload of module
+            from ddtrace.tracer import Tracer
+            eq_('customhost', Tracer.DEFAULT_HOSTNAME)
+    finally:
+        del sys.modules['ddtrace.tracer']  # clean up our test module
+        from ddtrace.tracer import Tracer
+        eq_('localhost', Tracer.DEFAULT_HOSTNAME)
Repository: DataDog/dd-trace-py

Issue:
Leaking sockets and connections
It looks like ddtrace writer is not closing the http connections it opens. 

```
  File "../lib/python3.7/threading.py", line 885, in _bootstrap
    self._bootstrap_inner()
  File "../lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "../lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "../lib/python3.7/site-packages/ddtrace/writer.py", line 168, in _target
    result_services = None
  File "../lib/python3.7/http/client.py", line 408, in close
    self._close_conn()
  File "../lib/python3.7/http/client.py", line 401, in _close_conn
    fp.close()
  File "../lib/python3.7/socket.py", line 660, in close
    self._sock = None
  File "../lib/python3.7/warnings.py", line 99, in _showwarnmsg
    msg.file, msg.line)
  File "/app/core.py", line 30, in warn_with_traceback
    traceback.print_stack(file=log)
../lib/python3.7/socket.py:660: ResourceWarning: unclosed <socket.socket fd=14, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('0.0.0.0', 54954), raddr=('0.0.0.0', 8126)>
  self._sock = None
```


Looking at the code, the issue is in the `_put` method of the `API` object. It creates an `HTTPConnection` but doesn't close it. 


Discussion:

Resolving patch:
diff --git a/ddtrace/api.py b/ddtrace/api.py
--- a/ddtrace/api.py
+++ b/ddtrace/api.py
@@ -133,11 +133,13 @@ def send_services(self, services):
 
     def _put(self, endpoint, data, count=0):
         conn = httplib.HTTPConnection(self.hostname, self.port)
-
-        headers = self._headers
-        if count:
-            headers = dict(self._headers)
-            headers[TRACE_COUNT_HEADER] = str(count)
-
-        conn.request("PUT", endpoint, data, headers)
-        return get_connection_response(conn)
+        try:
+            headers = self._headers
+            if count:
+                headers = dict(self._headers)
+                headers[TRACE_COUNT_HEADER] = str(count)
+
+            conn.request("PUT", endpoint, data, headers)
+            return get_connection_response(conn)
+        finally:
+            conn.close()


Test patch:
Repository: DataDog/dd-trace-py

Issue:
Unable to install with opentracing extras
I was following the [OpenTracing setup instructions](https://docs.datadoghq.com/tracing/advanced_usage/?tab=python#opentracing) but got a warning about missing extras:

```sh
(blah-YneZd-6L) sam@sam-Q325UAR ~/t/blah> pip list
Package    Version
---------- -------
pip        18.0   
setuptools 40.4.1 
wheel      0.31.1 

(blah-YneZd-6L) sam@sam-Q325UAR ~/t/blah> python --version
Python 2.7.14

(blah-YneZd-6L) sam@sam-Q325UAR ~/t/blah> pip --version
pip 18.0 from /home/sam/.local/share/virtualenvs/blah-YneZd-6L/lib/python2.7/site-packages/pip (python 2.7)


(blah-YneZd-6L) sam@sam-Q325UAR ~/t/blah> pip install 'ddtrace[opentracing] == 0.14.0'
Collecting ddtrace[opentracing]==0.14.0
  ddtrace 0.14.0 does not provide the extra 'opentracing'
Collecting msgpack-python (from ddtrace[opentracing]==0.14.0)
Collecting wrapt (from ddtrace[opentracing]==0.14.0)
Installing collected packages: msgpack-python, wrapt, ddtrace
Successfully installed ddtrace-0.14.0 msgpack-python-0.5.6 wrapt-1.10.11
```

> `ddtrace 0.14.0 does not provide the extra 'opentracing'`

Happens on Python 3.6 as well.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -60,10 +60,10 @@ def run_tests(self):
         "wrapt",
         "msgpack-python",
     ],
-    extra_requires={
+    extras_require={
         # users can include opentracing by having:
         # install_requires=["ddtrace[opentracing]", ...]
-        "opentracing": ["opentracing"],
+        "opentracing": ["opentracing>=2.0.0"],
     },
     # plugin tox
     tests_require=['tox', 'flake8'],


Test patch:
Repository: DataDog/dd-trace-py

Issue:
Scrolling for the left-side menu on the API docs is broken
Chromium, Ubuntu

Go to http://pypi.datadoghq.com/trace/docs/advanced_usage.html
Scroll up and down
If your browser window is short enough, you'll notice the left-side menu doesn't scroll with the page, leaving some parts inaccessible.

Video: [vokoscreen-2019-04-25_08-21-40.zip](https://github.com/DataDog/dd-trace-py/files/3117626/vokoscreen-2019-04-25_08-21-40.zip)

Since the API docs are generated from this repo, I figured I'd report the issue here.


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -145,7 +145,6 @@
 #
 html_theme_options = {
     'description': 'Datadog\'s Python tracing client',
-    'fixed_sidebar': True,
 }
 
 # Add any paths that contain custom themes here, relative to this directory.


Test patch:
Repository: DataDog/dd-trace-py

Issue:
Django middleware not properly supported if left empty
In Django 1.10 and above, `dd-trace-py` will attempt to check first if the user is using the `MIDDLEWARE` Django setting and will fallback to `MIDDLEWARE_CLASSES` if unset. In my application, I deliberately declare an empty list for the `MIDDLEWARE` setting to tell Django that I am not personally putting any middleware into the application. Unfortunately, `dd-trace-py` simply checks if the `settings.MIDDLEWARE` attribute is falsy and since this is true it then falls back to `MIDDLEWARE_CLASSES`.

If an empty list/tuple is declared in Django settings for `MIDDLEWARE`, it should still be properly injected.


Discussion:

Resolving patch:
diff --git a/ddtrace/contrib/django/middleware.py b/ddtrace/contrib/django/middleware.py
--- a/ddtrace/contrib/django/middleware.py
+++ b/ddtrace/contrib/django/middleware.py
@@ -52,7 +52,7 @@ def get_middleware_insertion_point():
     If middleware cannot be found, returns None for the middleware collection."""
     middleware = getattr(django_settings, MIDDLEWARE, None)
     # Prioritise MIDDLEWARE over ..._CLASSES, but only in 1.10 and later.
-    if middleware and django.VERSION >= (1, 10):
+    if middleware is not None and django.VERSION >= (1, 10):
         return MIDDLEWARE, middleware
     return MIDDLEWARE_CLASSES, getattr(django_settings, MIDDLEWARE_CLASSES, None)
 


Test patch:
diff --git a/tests/contrib/django/test_autopatching.py b/tests/contrib/django/test_autopatching.py
--- a/tests/contrib/django/test_autopatching.py
+++ b/tests/contrib/django/test_autopatching.py
@@ -76,3 +76,23 @@ def test_autopatching_twice_middleware(self):
 
         found_mw = settings.MIDDLEWARE.count('ddtrace.contrib.django.TraceExceptionMiddleware')
         assert found_mw == 1
+
+
+class DjangoAutopatchCustomMiddlewareTest(DjangoTraceTestCase):
+    @skipIf(django.VERSION < (1, 10), 'skip if version is below 1.10')
+    def test_autopatching_empty_middleware(self):
+        with self.settings(MIDDLEWARE=[]):
+            patch(django=True)
+            django.setup()
+        assert django._datadog_patch
+        assert 'ddtrace.contrib.django' in settings.INSTALLED_APPS
+        assert settings.MIDDLEWARE[0] == 'ddtrace.contrib.django.TraceMiddleware'
+        # MIDDLEWARE_CLASSES gets created internally in django 1.10 & 1.11 but doesn't
+        # exist at all in 2.0.
+        assert not getattr(settings, 'MIDDLEWARE_CLASSES', None) or \
+            'ddtrace.contrib.django.TraceMiddleware' \
+            not in settings.MIDDLEWARE_CLASSES
+        assert settings.MIDDLEWARE[-1] == 'ddtrace.contrib.django.TraceExceptionMiddleware'
+        assert not getattr(settings, 'MIDDLEWARE_CLASSES', None) or \
+            'ddtrace.contrib.django.TraceExceptionMiddleware' \
+            not in settings.MIDDLEWARE_CLASSES
Repository: DataDog/dd-trace-py

Issue:
DDLogger rewrites LogRecord.msg, which causes Sentry events duplication
Sentry uses `LogRecord.msg` to identify log events. LogRecord.msg is the log message template, to be formatted on demand.

When rewriting `msg`, one should not enrich it with arbitrary values, like `logging_bucket.skipped`.

The line
```
                record.msg = '{}, {} additional messages skipped'.format(record.msg, logging_bucket.skipped)
```

should be something like:

```
                record.msg = '{}, %s additional messages skipped'.format(record.msg)
                record.args = record.args + (logging_bucket.skipped,)
```

Culprit:
https://github.com/DataDog/dd-trace-py/blob/914cbca4ba5ec53ff17cb67164cb51b7bcd91ac2/ddtrace/internal/logger.py#L113

Example of message duplication:
![image](https://user-images.githubusercontent.com/43616133/67687095-ae372580-f98f-11e9-9b81-2a0ee61f175c.png)



Discussion:

Resolving patch:
diff --git a/ddtrace/internal/logger.py b/ddtrace/internal/logger.py
--- a/ddtrace/internal/logger.py
+++ b/ddtrace/internal/logger.py
@@ -110,7 +110,8 @@ def handle(self, record):
         if logging_bucket.bucket != current_bucket:
             # Append count of skipped messages if we have skipped some since our last logging
             if logging_bucket.skipped:
-                record.msg = '{}, {} additional messages skipped'.format(record.msg, logging_bucket.skipped)
+                record.msg = '{}, %s additional messages skipped'.format(record.msg)
+                record.args = record.args + (logging_bucket.skipped, )
 
             # Reset our bucket
             self.buckets[key] = DDLogger.LoggingBucket(current_bucket, 0)


Test patch:
diff --git a/tests/internal/test_logger.py b/tests/internal/test_logger.py
--- a/tests/internal/test_logger.py
+++ b/tests/internal/test_logger.py
@@ -263,7 +263,9 @@ def test_logger_handle_bucket_skipped_msg(self, base_handle):
         log = get_logger('test.logger')
 
         # Create log record to handle
-        record = self._make_record(log)
+        original_msg = 'hello %s'
+        original_args = (1, )
+        record = self._make_record(log, msg=original_msg, args=(1, ))
 
         # Create a bucket entry for this record
         key = (record.name, record.levelno, record.pathname, record.lineno)
@@ -277,7 +279,9 @@ def test_logger_handle_bucket_skipped_msg(self, base_handle):
         # We passed to base Logger.handle
         base_handle.assert_called_once_with(record)
 
-        self.assertEqual(record.msg, 'test, 20 additional messages skipped')
+        self.assertEqual(record.msg, original_msg + ', %s additional messages skipped')
+        self.assertEqual(record.args, original_args + (20, ))
+        self.assertEqual(record.getMessage(), 'hello 1, 20 additional messages skipped')
 
     def test_logger_handle_bucket_key(self):
         """
Repository: DataDog/dd-trace-py

Issue:
ddtrace not working with new aiobotocore library

### Which version of dd-trace-py are you using?
0.34.0

### Which version of the libraries are you using?

```
aiobotocore==0.12.0
aiohttp==3.6.2
aioitertools==0.5.1
async-timeout==3.0.1
attrs==19.3.0
botocore==1.15.15
chardet==3.0.4
ddtrace==0.34.0
docutils==0.15.2
idna==2.9
jmespath==0.9.5
msgpack==1.0.0
multidict==4.7.5
python-dateutil==2.8.1
six==1.14.0
urllib3==1.25.8
wrapt==1.12.0
yarl==1.4.2
```

### How can we reproduce your problem?
```python
    from ddtrace import patch

    patch(aiobotocore=True)
```

### What is the result that you get?
```
ImportError: cannot import name 'ClientResponseContentProxy' from 'aiobotocore.endpoint' 
    raise PatchException('integration not available')
ddtrace.monkey.PatchException: integration not available
```

### What is result that you expected?
Successfully patched `aiobotocore` library


Discussion:

Resolving patch:
diff --git a/ddtrace/contrib/aiobotocore/patch.py b/ddtrace/contrib/aiobotocore/patch.py
--- a/ddtrace/contrib/aiobotocore/patch.py
+++ b/ddtrace/contrib/aiobotocore/patch.py
@@ -3,7 +3,11 @@
 from ddtrace import config
 import aiobotocore.client
 
-from aiobotocore.endpoint import ClientResponseContentProxy
+try:
+    from aiobotocore.endpoint import ClientResponseContentProxy
+except ImportError:
+    # aiobotocore>=0.11.0
+    from aiobotocore._endpoint_helpers import ClientResponseContentProxy
 
 from ...constants import ANALYTICS_SAMPLE_RATE_KEY, SPAN_MEASURED_KEY
 from ...pin import Pin


Test patch:
Repository: DataDog/dd-trace-py

Issue:
Add uds_path to ddtrace.opentracer.Tracer
### Which version of dd-trace-py are you using?
0.33.0 ?

### Which version of the libraries are you using?
0.33.0

### How can we reproduce your problem?
Try setting `uds_path` when creating a `ddtrace.opentracer.Tracer`. Oh wait.

### What is the result that you get?
This nastiness:
```
ddtracer = Tracer('appname', config={'agent_hostname': None, 'agent_port': None})
set_global_tracer(ddtracer)
tracer.configure(uds_path='/path/to/datadog.sock')
```

### What is result that you expected?
This beauty:
```
ddtracer = Tracer('appname', config={'agent_hostname': None, 'agent_port': None, 'uds_path': '/path/to/datadog.sock'})
set_global_tracer(ddtracer)
```
It is not clear from the documentation how to set up the tracer here, if using unix domain sockets but not using `ddtrace-run`.

I'm only filing to set up a PR real quick. Feel free to decline both if these are irrelevant.


Discussion:

Resolving patch:
diff --git a/ddtrace/opentracer/settings.py b/ddtrace/opentracer/settings.py
--- a/ddtrace/opentracer/settings.py
+++ b/ddtrace/opentracer/settings.py
@@ -10,6 +10,7 @@
     'GLOBAL_TAGS',
     'SAMPLER',
     'PRIORITY_SAMPLING',
+    'UDS_PATH',
     'SETTINGS',
 ]
 
@@ -25,6 +26,7 @@
     GLOBAL_TAGS='global_tags',
     SAMPLER='sampler',
     PRIORITY_SAMPLING='priority_sampling',
+    UDS_PATH='uds_path',
     SETTINGS='settings',
 )
 
diff --git a/ddtrace/opentracer/tracer.py b/ddtrace/opentracer/tracer.py
--- a/ddtrace/opentracer/tracer.py
+++ b/ddtrace/opentracer/tracer.py
@@ -27,6 +27,7 @@
     keys.GLOBAL_TAGS: {},
     keys.SAMPLER: None,
     keys.PRIORITY_SAMPLING: None,
+    keys.UDS_PATH: None,
     keys.SETTINGS: {
         FILTERS_KEY: [],
     },
@@ -89,6 +90,7 @@ def __init__(self, service_name=None, config=None, scope_manager=None, dd_tracer
                                   sampler=self._config.get(keys.SAMPLER),
                                   settings=self._config.get(keys.SETTINGS),
                                   priority_sampling=self._config.get(keys.PRIORITY_SAMPLING),
+                                  uds_path=self._config.get(keys.UDS_PATH),
                                   context_provider=dd_context_provider,
                                   )
         self._propagators = {


Test patch:
Repository: DataDog/dd-trace-py

Issue:
ddtrace.Pin() for multiple grpc channels doesn't work
Thanks for taking the time for reporting an issue!

Before reporting an issue on dd-trace-py, please be sure to provide all
necessary information.

If you're hitting a bug, make sure that you're using the latest version of this
library.

### Which version of dd-trace-py are you using?
0.38.2
I didn't find anything related to this issue in the release notes of the releases after this version.

### Which version of the libraries are you using?
datadog==0.36.0

### How can we reproduce your problem?
Approach 1:
servers is a list of grpc server addresses
```
for server in servers:
    channel = grpc.insecure_channel(server)
    Pin.override(channel, service=server)
    # Do something with the channel
```

Since `Pin.override(grpc.Channel, service=server)` worked with one server, I also tried the following to see how it looks
Approach 2:
servers is a list of grpc server addresses
```
for server in servers:
    Pin.override(grpc.Channel, service=server)
    channel = grpc.insecure_channel(server)
    # Do something with the channel
```

### What is the result that you get?
In Approach 1, Pin.override did not set the service name correctly. Everywhere in Datadog, I could see it as `grpc-client` which is the default value.
In Approach 2, since I I don't pass the channels corresponding to each server, all servers are overriden by Pin to the final server (probably because it's the last one in the loop)

### What is the result that you expected?
ddtrace.Pin() onto multiple grpc channels should work and I should be able to see the correct `service` in Datadog APM traces and Service Map


Discussion:

Resolving patch:
diff --git a/ddtrace/contrib/grpc/patch.py b/ddtrace/contrib/grpc/patch.py
--- a/ddtrace/contrib/grpc/patch.py
+++ b/ddtrace/contrib/grpc/patch.py
@@ -98,7 +98,7 @@ def _unpatch_server():
 def _client_channel_interceptor(wrapped, instance, args, kwargs):
     channel = wrapped(*args, **kwargs)
 
-    pin = Pin.get_from(constants.GRPC_PIN_MODULE_CLIENT)
+    pin = Pin.get_from(channel)
     if not pin or not pin.enabled():
         return channel
 


Test patch: