Repository: DataDog/dd-trace-py

Issue:
DeprecationWarning: the imp module is deprecated in favour of importlib
When running a Django project using ddtrace with [warnings enabled](https://docs.python.org/3/using/cmdline.html#cmdoption-w), this warning is emitted:

## Issue

> `/usr/local/lib/python3.7/dist-packages/ddtrace/bootstrap/sitecustomize.py:7`: `DeprecationWarning`: the `imp` module is deprecated in favour of `importlib`; see the [module's documentation](https://docs.python.org/3/library/imp.html) for alternative uses

## Details

The line in question:

https://github.com/DataDog/dd-trace-py/blob/94148324196eb41c1f6bef56be51bdd96c758fa7/ddtrace/bootstrap/sitecustomize.py#L7

How it's used: 

https://github.com/DataDog/dd-trace-py/blob/94148324196eb41c1f6bef56be51bdd96c758fa7/ddtrace/bootstrap/sitecustomize.py#L103-L120

Documentation note for [`imp.find_module()`](https://docs.python.org/3/library/imp.html#imp.find_module):

> Deprecated since version 3.3: Use `importlib.util.find_spec()` instead unless Python 3.3 compatibility is required, in which case use `importlib.find_loader()`. For example usage of the former case, see the Examples section of the `importlib` documentation.

Documentation note for [`imp.load_module()`](https://docs.python.org/3/library/imp.html#imp.load_module):

> Deprecated since version 3.3: If previously used in conjunction with `imp.find_module()` then consider using `importlib.import_module()`, otherwise use the loader returned by the replacement you chose for `imp.find_module()`. If you called `imp.load_module()` and related functions directly with file path arguments then use a combination of `importlib.util.spec_from_file_location()` and `importlib.util.module_from_spec()`. See the Examples section of the `importlib` documentation for details of the various approaches.

## Resolution

I suspect [this example](https://docs.python.org/3/library/importlib.html#approximating-importlib-import-module) could be worth building off of to do the necessary path customization.


Discussion:
Thanks for the report @noelleleigh! We'll target upgrading to `importlib` for version `0.45`.

Resolving patch:
diff --git a/ddtrace/bootstrap/sitecustomize.py b/ddtrace/bootstrap/sitecustomize.py
--- a/ddtrace/bootstrap/sitecustomize.py
+++ b/ddtrace/bootstrap/sitecustomize.py
@@ -4,7 +4,6 @@
 """
 import logging
 import os
-import imp
 import sys
 
 from ddtrace.utils.formats import asbool, get_env, parse_tags_str
@@ -100,24 +99,40 @@ def update_patched_modules():
         env_tags = os.getenv("DD_TRACE_GLOBAL_TAGS")
         tracer.set_tags(parse_tags_str(env_tags))
 
-    # Ensure sitecustomize.py is properly called if available in application directories:
-    # * exclude `bootstrap_dir` from the search
-    # * find a user `sitecustomize.py` module
-    # * import that module via `imp`
+    # Check for and import any sitecustomize that would have normally been used
+    # had ddtrace-run not been used.
     bootstrap_dir = os.path.dirname(__file__)
-    path = list(sys.path)
-
-    if bootstrap_dir in path:
-        path.remove(bootstrap_dir)
-
-    try:
-        (f, path, description) = imp.find_module("sitecustomize", path)
-    except ImportError:
-        pass
+    if bootstrap_dir in sys.path:
+        index = sys.path.index(bootstrap_dir)
+        del sys.path[index]
+
+        # NOTE: this reference to the module is crucial in Python 2.
+        # Without it the current module gets gc'd and all subsequent references
+        # will be `None`.
+        ddtrace_sitecustomize = sys.modules["sitecustomize"]
+        del sys.modules["sitecustomize"]
+        try:
+            import sitecustomize  # noqa
+        except ImportError:
+            # If an additional sitecustomize is not found then put the ddtrace
+            # sitecustomize back.
+            log.debug("additional sitecustomize not found")
+            sys.modules["sitecustomize"] = ddtrace_sitecustomize
+        else:
+            log.debug("additional sitecustomize found in: %s", sys.path)
+        finally:
+            # Always reinsert the ddtrace bootstrap directory to the path so
+            # that introspection and debugging the application makes sense.
+            # Note that this does not interfere with imports since a user
+            # sitecustomize, if it exists, will be imported.
+            sys.path.insert(index, bootstrap_dir)
     else:
-        # `sitecustomize.py` found, load it
-        log.debug("sitecustomize from user found in: %s", path)
-        imp.load_module("sitecustomize", f, path, description)
+        try:
+            import sitecustomize  # noqa
+        except ImportError:
+            log.debug("additional sitecustomize not found")
+        else:
+            log.debug("additional sitecustomize found in: %s", sys.path)
 
     # Loading status used in tests to detect if the `sitecustomize` has been
     # properly loaded without exceptions. This must be the last action in the module


Test patch:
diff --git a/tests/commands/test_runner.py b/tests/commands/test_runner.py
--- a/tests/commands/test_runner.py
+++ b/tests/commands/test_runner.py
@@ -21,12 +21,12 @@ def inject_sitecustomize(path):
     # required otherwise `ddtrace` scripts are not found when `env` kwarg is
     # passed
     env = os.environ.copy()
-    sitecustomize = os.path.join(root_folder, "..", path)
+    sitecustomize = os.path.abspath(os.path.join(root_folder, "..", path))
 
     # Add `bootstrap` directory to the beginning of PYTHONTPATH so we know
     # if `import sitecustomize` is run that it'll be the one we specify
     python_path = [sitecustomize] + list(sys.path)
-    env["PYTHONPATH"] = ":".join(python_path)
+    env["PYTHONPATH"] = os.pathsep.join(python_path)
     return env