Repository: DataDog/dd-trace-py

Issue:
Using library injection on Kubernetes, /datadog-init/ddtrace_pkgs/bytecode is mangled
### Summary of problem

When automatically injecting ddtrace via library injection on Kubernetes, the `bytecode` package (a dependency of `ddtrace`) which is installed is a mangled combination of versions, with files from older versions taking priority.

### Which version of dd-trace-py are you using?

`v1.11.2`, also reproduces from the current head of the `v1.x` branch of the `dd-trace-py` repo

### Which version of pip are you using?

N/A

### Which libraries and their versions are you using?

N/A

### How can we reproduce your problem?

1. Make sure you have docker and git installed.
2. `git clone https://github.com/DataDog/dd-trace-py`
3. `cd dd-trace-py/lib-injection`
4. `docker build . --tag test-init-image --build-arg DDTRACE_PYTHON_VERSION=v1.11.2`
5. `docker run -it test-init-image sh`

You should now have a shell running inside the Docker image. (You can also just pull a released init image, but I thought I'd demonstrate all the steps to directly build one.)

6. `cd /datadog-init/ddtrace_pkgs/bytecode`
7. `tail -1 version.py`

For me, this shows `__version__ = "0.14.1"` -- so far so good, [this is the latest version of `bytecode`](https://pypi.org/project/bytecode/)

8. `head -1 __init__.py`

Now this shows `__version__ = "0.13.0"`. That's different! Older! Not the same!

### What is the result that you get?

Mismatched version numbers. More significantly, `__init__.py` and other files come from [version 0.13.0 of the module](https://github.com/MatthieuDartiailh/bytecode/tree/0.13.0/bytecode). However, `version.py` and `py.typed` hail from [version 0.14.1 of the module](https://github.com/MatthieuDartiailh/bytecode/tree/0.14.1/src/bytecode).

This would be a minor quirk -- `version.py` and `py.typed` aren't important in a deployment context -- but most of the files come from version 0.13.0 which is from March 2020 and doesn't support newer versions of Python, among other things because calls to `types.CodeType(...)` in `concrete.py` use the wrong constructor arguments, so the instrumentation crashes under some circumstances.

### What is the result that you expected?

A coherent, consistent, and appropriate version of the `bytecode` module, bundled with `ddtrace`.

### Why does this happen?

The `ddtrace` package depends on different versions of `bytecode` depending on the Python version:
https://github.com/DataDog/dd-trace-py/blob/ba0397a1cc5017d85a1fac3d78848a44deaf76e0/setup.py#L330

However, the `dl_megawheel.py` script used in building the init container image simply iterates through all combinations of supported Python versions, architectures, and platforms, downloads the `.whl` files for `ddtrace` and its dependencies (including `bytecode`), and unzips all those `.whl` files into a single directory:
https://github.com/DataDog/dd-trace-py/blob/ba0397a1cc5017d85a1fac3d78848a44deaf76e0/lib-injection/dl_megawheel.py#L114

Since different versions of the `bytecode` wheel will be downloaded based on the Python version, that means several different such files are downloaded and unzipped, and all of the files are splatted on top of each other willy-nilly, leading to the mishmash of versions described above.


Discussion:
hey @egnor thanks for the report! Yep, this an issue introduced in 1.11.1 with the new approach we're using for the injection image exactly as you described.

We're working on a fix for this. In the meantime if you'd like to use automatic injection v1.11.0 should not have the issue (but comes with the drawbacks of `pip install`-ing at application startup).

Resolving patch:
diff --git a/lib-injection/dl_megawheel.py b/lib-injection/dl_wheels.py
similarity index 86%
rename from lib-injection/dl_megawheel.py
rename to lib-injection/dl_wheels.py
--- a/lib-injection/dl_megawheel.py
+++ b/lib-injection/dl_wheels.py
@@ -13,7 +13,7 @@
 20.0.2.
 
 Usage:
-        ./dl_megawheel.py --help
+        ./dl_wheels.py --help
 
 
 The downloaded wheels can then be installed locally using:
@@ -21,7 +21,6 @@
 """
 import argparse
 import itertools
-import os
 import subprocess
 import sys
 
@@ -105,12 +104,3 @@
 
     if not args.dry_run:
         subprocess.run(cmd, capture_output=not args.verbose, check=True)
-
-        # Unzip all the wheels into the output directory
-        wheel_files = [f for f in os.listdir(dl_dir) if f.endswith(".whl")]
-        for whl in wheel_files:
-            wheel_file = os.path.join(dl_dir, whl)
-            # -q for quieter output, else we get all of the files being unzipped.
-            subprocess.run(["unzip", "-q", "-o", wheel_file, "-d", dl_dir])
-            # Remove the wheel as it has been unpacked
-            os.remove(wheel_file)
diff --git a/lib-injection/sitecustomize.py b/lib-injection/sitecustomize.py
--- a/lib-injection/sitecustomize.py
+++ b/lib-injection/sitecustomize.py
@@ -1,29 +1,50 @@
 """
-When included on the PYTHONPATH this module will initialize the PYTHONPATH to
-include the directory containing the ddtrace package and its dependencies. It
-then imports the ddtrace.bootstrap.sitecustomize module to automatically
-instrument the application.
+This module when included on the PYTHONPATH will install the ddtrace package
+from the locally available wheels that are included in the image.
 """
 import os
 import sys
 
 
-def _add_to_pythonpath(path):
-    # type: (str) -> None
-    """Adds a path to the start of PYTHONPATH."""
+def _configure_ddtrace():
+    # This import has the same effect as ddtrace-run for the current process.
+    import ddtrace.bootstrap.sitecustomize
+
+    bootstrap_dir = os.path.abspath(os.path.dirname(ddtrace.bootstrap.sitecustomize.__file__))
     prev_python_path = os.getenv("PYTHONPATH", "")
-    os.environ["PYTHONPATH"] = "%s%s%s" % (path, os.pathsep, prev_python_path)
-    sys.path.insert(0, path)
+    os.environ["PYTHONPATH"] = "%s%s%s" % (bootstrap_dir, os.path.pathsep, prev_python_path)
+
+    # Also insert the bootstrap dir in the path of the current python process.
+    sys.path.insert(0, bootstrap_dir)
+    print("datadog autoinstrumentation: successfully configured python package")
+
+
+# Avoid infinite loop when attempting to install ddtrace. This flag is set when
+# the subprocess is launched to perform the installation.
+if "DDTRACE_PYTHON_INSTALL_IN_PROGRESS" not in os.environ:
+    try:
+        import ddtrace  # noqa: F401
+
+    except ImportError:
+        import subprocess
 
+        print("datadog autoinstrumentation: installing python package")
 
-pkgs_path = os.path.join(os.path.dirname(__file__), "ddtrace_pkgs")
-bootstrap_dir = os.path.join(pkgs_path, "ddtrace", "bootstrap")
-_add_to_pythonpath(pkgs_path)
-_add_to_pythonpath(bootstrap_dir)
+        # Set the flag to avoid an infinite loop.
+        env = os.environ.copy()
+        env["DDTRACE_PYTHON_INSTALL_IN_PROGRESS"] = "true"
 
-try:
-    import ddtrace.bootstrap.sitecustomize  # noqa: F401
-except BaseException as e:
-    print("datadog autoinstrumentation: ddtrace failed to install:\n %s" % str(e))
-else:
-    print("datadog autoinstrumentation: ddtrace successfully installed")
+        # Execute the installation with the current interpreter
+        try:
+            pkgs_path = os.path.join(os.path.dirname(__file__), "ddtrace_pkgs")
+            subprocess.run(
+                [sys.executable, "-m", "pip", "install", "--find-links", pkgs_path, "ddtrace"], env=env, check=True
+            )
+        except BaseException as e:
+            print("datadog autoinstrumentation: failed to install python package %s" % str(e))
+        else:
+            print("datadog autoinstrumentation: successfully installed python package")
+            _configure_ddtrace()
+    else:
+        print("datadog autoinstrumentation: ddtrace already installed, skipping install")
+        _configure_ddtrace()


Test patch: