Repository: DataDog/dd-trace-py

Issue:
Pyramid instrumentation breaks Configurator object.
### Which version of dd-trace-py are you using?

ddtrace==0.54.1

### Which version of the libraries are you using?

```
attrs==21.2.0
ddtrace==0.54.1
# Editable Git install with no remote (ddtracer-test==0.0)
-e /home/ergo/workspace/ddtracer_test
hupper==1.10.3
Jinja2==3.0.2
Mako==1.1.5
MarkupSafe==2.0.1
packaging==21.0
PasteDeploy==2.1.1
plaster==1.0
plaster-pastedeploy==0.7
protobuf==3.19.0
Pygments==2.10.0
pyparsing==2.4.7
pyramid==2.0
pyramid-debugtoolbar==4.9
pyramid-jinja2==2.8
pyramid-mako==1.1.0
repoze.lru==0.7
six==1.16.0
tenacity==8.0.1
translationstring==1.4
venusian==3.0.0
waitress==2.0.0
WebOb==1.8.7
zope.deprecation==4.4.0
zope.interface==5.4.0
```

### How can we reproduce your problem?

I've prepared a minimal test case here

https://github.com/ergo/ddtrace_pyramid

### What is the result that you get?

When running:

`ddtrace-run pserve development.ini `

```
Traceback (most recent call last):
  File "/home/ergo/workspace/ddtracer_test/env/lib/python3.9/site-packages/pyramid/path.py", line 387, in _zope_dottedname_style
    found = getattr(found, n)
AttributeError: module 'paste.deploy' has no attribute 'routes'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ergo/workspace/ddtracer_test/env/bin/pserve", line 8, in <module>
    sys.exit(main())
  File "/home/ergo/workspace/ddtracer_test/env/lib/python3.9/site-packages/pyramid/scripts/pserve.py", line 30, in main
    return command.run()
  File "/home/ergo/workspace/ddtracer_test/env/lib/python3.9/site-packages/pyramid/scripts/pserve.py", line 271, in run
    app = loader.get_wsgi_app(app_name, config_vars)
  File "/home/ergo/workspace/ddtracer_test/env/lib/python3.9/site-packages/plaster_pastedeploy/__init__.py", line 125, in get_wsgi_app
    return loadapp(
  File "/home/ergo/workspace/ddtracer_test/env/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 253, in loadapp
    return loadobj(APP, uri, name=name, **kw)
  File "/home/ergo/workspace/ddtracer_test/env/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 278, in loadobj
    return context.create()
  File "/home/ergo/workspace/ddtracer_test/env/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 715, in create
    return self.object_type.invoke(self)
  File "/home/ergo/workspace/ddtracer_test/env/lib/python3.9/site-packages/paste/deploy/loadwsgi.py", line 152, in invoke
    return fix_call(context.object, context.global_conf, **context.local_conf)
  File "/home/ergo/workspace/ddtracer_test/env/lib/python3.9/site-packages/paste/deploy/util.py", line 55, in fix_call
    val = callable(*args, **kw)
  File "/home/ergo/workspace/ddtracer_test/ddtracer_test/__init__.py", line 9, in main
    config.include('.routes')
  File "/home/ergo/workspace/ddtracer_test/env/lib/python3.9/site-packages/pyramid/config/__init__.py", line 632, in include
    c = self.maybe_dotted(callable)
  File "/home/ergo/workspace/ddtracer_test/env/lib/python3.9/site-packages/pyramid/config/__init__.py", line 740, in maybe_dotted
    return self.name_resolver.maybe_resolve(dotted)
  File "/home/ergo/workspace/ddtracer_test/env/lib/python3.9/site-packages/pyramid/path.py", line 327, in maybe_resolve
    return self._resolve(dotted, package)
  File "/home/ergo/workspace/ddtracer_test/env/lib/python3.9/site-packages/pyramid/path.py", line 334, in _resolve
    return self._zope_dottedname_style(dotted, package)
  File "/home/ergo/workspace/ddtracer_test/env/lib/python3.9/site-packages/pyramid/path.py", line 389, in _zope_dottedname_style
    __import__(used)
ModuleNotFoundError: No module named 'paste.deploy.routes'
```

I've found potential problem - monkeypatching pyramid changes how package and root_package are being computed - so import calls break in our case with the exception I've posted above

https://github.com/DataDog/dd-trace-py/blob/master/ddtrace/contrib/pyramid/patch.py#L77 - I've tracked this piece of code to be problematic.

I think the `caller_package` level argument needs to be adjusted - when I set it to 2 the application start.

### What is the result that you expected?

`pserve development.ini `
```
Starting server in PID 230548.
2021-10-22 16:06:31,327 INFO  [waitress:485][MainThread] Serving on http://127.0.0.1:8081
```



Discussion:
Hi @marcin-lulek-cint, thanks for the super detailed report, repro and triage! This is much appreciated - we'll take a look.

Feel free to put up a PR with your suspected fix in the meantime!
Hi @marcin-lulek-cint! https://github.com/DataDog/dd-trace-py/releases/tag/v0.55.1 should fix this. It will be available to install via pip in the next 40 minutes or so. Could you please try upgrading and let us know?
@ZStriker19 did you run it on my test example? Because for me it is still failing same as before.
Here is my PR that fixed it for me:
https://github.com/ergo/dd-trace-py/commit/53e68ea4d6543b04af4736a5aeeef41f3fba6b50
I thought that the fix you guys did would be the equivalent but it turns out its not because, the frame and parent package is different.
Hmm yeah I'm still seeing it too @ergo 🤔. We'll take another look!

Resolving patch:
diff --git a/ddtrace/contrib/pyramid/patch.py b/ddtrace/contrib/pyramid/patch.py
--- a/ddtrace/contrib/pyramid/patch.py
+++ b/ddtrace/contrib/pyramid/patch.py
@@ -62,6 +62,15 @@ def traced_init(wrapped, instance, args, kwargs):
     # If the tweens are explicitly set with 'pyramid.tweens', we need to
     # explicitly set our tween too since `add_tween` will be ignored.
     insert_tween_if_needed(trace_settings)
+
+    # The original Configurator.__init__ looks up two levels to find the package
+    # name if it is not provided. This has to be replicated here since this patched
+    # call will occur at the same level in the call stack.
+    if not kwargs.get("package", None):
+        from pyramid.path import caller_package
+
+        kwargs["package"] = caller_package(level=2)
+
     kwargs["settings"] = trace_settings
     wrapped(*args, **kwargs)
     trace_pyramid(instance)
diff --git a/riotfile.py b/riotfile.py
--- a/riotfile.py
+++ b/riotfile.py
@@ -944,6 +944,7 @@ def select_pys(min_version=MIN_PYTHON_VERSION, max_version=MAX_PYTHON_VERSION):
                     pkgs={
                         "requests": [latest],
                         "webtest": [latest],
+                        "tests/contrib/pyramid/pserve_app": [latest],
                         "pyramid": [
                             "~=1.7",
                             "~=1.8",


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -1,4 +1,5 @@
 import contextlib
+import os
 import sys
 
 import pytest
@@ -61,7 +62,7 @@ def snapshot(request):
     assert len(marks) < 2, "Multiple snapshot marks detected"
     if marks:
         snap = marks[0]
-        token = _request_token(request)
+        token = _request_token(request).replace(" ", "_").replace(os.path.sep, "_")
         with _snapshot_context(token, *snap.args, **snap.kwargs) as snapshot:
             yield snapshot
     else:
diff --git a/tests/contrib/pyramid/pserve_app/app/__init__.py b/tests/contrib/pyramid/pserve_app/app/__init__.py
new file mode 100644
--- /dev/null
+++ b/tests/contrib/pyramid/pserve_app/app/__init__.py
@@ -0,0 +1,37 @@
+from pyramid.config import Configurator
+from pyramid.response import Response
+
+from ddtrace import tracer
+from ddtrace.filters import TraceFilter
+
+
+class PingFilter(TraceFilter):
+    def process_trace(self, trace):
+        # Filter out all traces with trace_id = 1
+        # This is done to prevent certain traces from being included in snapshots and
+        # accomplished by propagating an http trace id of 1 with the request to the webserver.
+        return None if trace and trace[0].trace_id == 1 else trace
+
+
+tracer.configure(
+    settings={
+        "FILTERS": [PingFilter()],
+    }
+)
+
+
+def tracer_shutdown(request):
+    tracer.shutdown()
+    return Response("shutdown")
+
+
+def main(global_config, **settings):
+    """This function returns a Pyramid WSGI application."""
+    with Configurator(settings=settings) as config:
+        config.add_route("tracer-shutdown", "/shutdown-tracer")
+        config.add_view(tracer_shutdown, route_name="tracer-shutdown")
+        # This will trigger usage of the package name reproing an issue
+        # reported in #2942.
+        config.include(".routes")
+        config.scan()
+    return config.make_wsgi_app()
diff --git a/tests/contrib/pyramid/pserve_app/app/routes.py b/tests/contrib/pyramid/pserve_app/app/routes.py
new file mode 100644
--- /dev/null
+++ b/tests/contrib/pyramid/pserve_app/app/routes.py
@@ -0,0 +1,10 @@
+from pyramid.response import Response
+
+
+def hello_world(request):
+    return Response("Hello World!")
+
+
+def includeme(config):
+    config.add_route("hello", "/")
+    config.add_view(hello_world, route_name="hello")
diff --git a/tests/contrib/pyramid/pserve_app/development.ini b/tests/contrib/pyramid/pserve_app/development.ini
new file mode 100644
--- /dev/null
+++ b/tests/contrib/pyramid/pserve_app/development.ini
@@ -0,0 +1,11 @@
+###
+# app configuration
+# https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/environment.html
+###
+
+[app:main]
+use = egg:pserve_test_app
+
+[server:main]
+use = egg:waitress#main
+listen = localhost:8000
diff --git a/tests/contrib/pyramid/pserve_app/setup.py b/tests/contrib/pyramid/pserve_app/setup.py
new file mode 100644
--- /dev/null
+++ b/tests/contrib/pyramid/pserve_app/setup.py
@@ -0,0 +1,13 @@
+from setuptools import setup, find_packages
+
+setup(
+    name="pserve_test_app",
+    packages=find_packages(),
+    include_package_data=True,
+    zip_safe=False,
+    entry_points={
+        "paste.app_factory": [
+            "main = app:main",
+        ],
+    },
+)
diff --git a/tests/contrib/pyramid/test_pyramid.py b/tests/contrib/pyramid/test_pyramid.py
--- a/tests/contrib/pyramid/test_pyramid.py
+++ b/tests/contrib/pyramid/test_pyramid.py
@@ -113,8 +113,13 @@ def test_distributed_tracing_disabled(self):
         assert span.get_tag(ORIGIN_KEY) != "synthetics"
 
 
+@pytest.fixture
+def pyramid_app():
+    return "ddtrace-run python tests/contrib/pyramid/app/app.py"
+
+
 @pytest.fixture(scope="function")
-def pyramid_client(snapshot):
+def pyramid_client(snapshot, pyramid_app):
     """Runs a Pyramid app in a subprocess and returns a client which can be used to query it.
 
     Traces are flushed by invoking a tracer.shutdown() using a /shutdown-tracer route
@@ -124,7 +129,7 @@ def pyramid_client(snapshot):
     env = os.environ.copy()
     env["SERVER_PORT"] = str(SERVER_PORT)
 
-    cmd = ["ddtrace-run", "python", "tests/contrib/pyramid/app/app.py"]
+    cmd = pyramid_app.split(" ")
     proc = subprocess.Popen(
         cmd,
         stdout=subprocess.PIPE,
@@ -146,6 +151,13 @@ def pyramid_client(snapshot):
         proc.terminate()
 
 
+@pytest.mark.parametrize(
+    "pyramid_app",
+    [
+        "ddtrace-run pserve tests/contrib/pyramid/pserve_app/development.ini",
+        "ddtrace-run python tests/contrib/pyramid/app/app.py",
+    ],
+)
 @pytest.mark.snapshot()
 def test_simple_pyramid_app_endpoint(pyramid_client):
     r = pyramid_client.get("/")
diff --git a/tests/snapshots/tests.contrib.pyramid.test_pyramid.test_simple_pyramid_app_endpoint[ddtrace-run_pserve_tests_contrib_pyramid_pserve_app_development.ini].json b/tests/snapshots/tests.contrib.pyramid.test_pyramid.test_simple_pyramid_app_endpoint[ddtrace-run_pserve_tests_contrib_pyramid_pserve_app_development.ini].json
new file mode 100644
--- /dev/null
+++ b/tests/snapshots/tests.contrib.pyramid.test_pyramid.test_simple_pyramid_app_endpoint[ddtrace-run_pserve_tests_contrib_pyramid_pserve_app_development.ini].json
@@ -0,0 +1,26 @@
+[[
+  {
+    "name": "pyramid.request",
+    "service": "pyramid",
+    "resource": "GET hello",
+    "trace_id": 0,
+    "span_id": 1,
+    "parent_id": 0,
+    "type": "web",
+    "meta": {
+      "http.method": "GET",
+      "http.status_code": "200",
+      "http.url": "http://localhost:8000/",
+      "pyramid.route.name": "hello",
+      "runtime-id": "3cfe7c1618be4fc4a090f2a7dab9dcc9"
+    },
+    "metrics": {
+      "_dd.agent_psr": 1.0,
+      "_dd.measured": 1,
+      "_dd.tracer_kr": 1.0,
+      "_sampling_priority_v1": 1,
+      "system.pid": 656011
+    },
+    "duration": 128359,
+    "start": 1635529254779423958
+  }]]
diff --git a/tests/snapshots/tests.contrib.pyramid.test_pyramid.test_simple_pyramid_app_endpoint[ddtrace-run_python_tests_contrib_pyramid_app_app.py].json b/tests/snapshots/tests.contrib.pyramid.test_pyramid.test_simple_pyramid_app_endpoint[ddtrace-run_python_tests_contrib_pyramid_app_app.py].json
new file mode 100644
--- /dev/null
+++ b/tests/snapshots/tests.contrib.pyramid.test_pyramid.test_simple_pyramid_app_endpoint[ddtrace-run_python_tests_contrib_pyramid_app_app.py].json
@@ -0,0 +1,26 @@
+[[
+  {
+    "name": "pyramid.request",
+    "service": "pyramid",
+    "resource": "GET hello",
+    "trace_id": 0,
+    "span_id": 1,
+    "parent_id": 0,
+    "type": "web",
+    "meta": {
+      "http.method": "GET",
+      "http.status_code": "200",
+      "http.url": "http://localhost:8000/",
+      "pyramid.route.name": "hello",
+      "runtime-id": "b1da81278b714ff5a820c0ed676af592"
+    },
+    "metrics": {
+      "_dd.agent_psr": 1.0,
+      "_dd.measured": 1,
+      "_dd.tracer_kr": 1.0,
+      "_sampling_priority_v1": 1,
+      "system.pid": 656017
+    },
+    "duration": 119727,
+    "start": 1635529255205817037
+  }]]