Repository: DataDog/dd-trace-py

Issue:
Import errors when trying to use profiler
Thanks for taking the time for reporting an issue!

Before reporting an issue on dd-trace-py, please be sure to provide all
necessary information.

If you're hitting a bug, make sure that you're using the latest version of this
library.

### Which version of dd-trace-py are you using?
0.35.0

### Which version of the libraries are you using?

```
alembic==1.0.3
analytics-python==1.2.9
aniso8601==7.0.0
asn1crypto==1.2.0
Authlib==0.6
awscli==1.16.186
Babel==1.3
backoff==1.6.0
bcrypt==3.1.4
bleach==3.1.4
blinker==1.3
boto3==1.9.176
botocore==1.12.176
boxsdk==1.5.5
cachetools==2.0.1
certifi==2019.11.28
cffi==1.13.2
chardet==3.0.4
Click==7.0
click-log==0.3.2
colorama==0.3.9
cryptography==2.7
datadog==0.35.0
ddtrace==0.35.0
decorator==4.4.1
defusedxml==0.5.0
docutils==0.14
expiringdict==1.2.0
Flask==1.1.1
Flask-Caching==1.8.0
Flask-GraphQL==2.0.0
Flask-Login==0.4.1
Flask-Script==2.0.6
Flask-Session==0.3.1
Flask-SQLAlchemy==2.4.1
Flask-WTF==0.14
freezegun==0.3.5
google-api-python-client==1.6.5
google-auth==1.4.1
google-auth-httplib2==0.0.3
google-auth-oauthlib==0.2.0
graphene==2.1.8
graphql-core==2.2.1
graphql-relay==2.0.0
graphql-server-core==1.1.3
holidays==0.3.1
html5lib==1.0.1
httplib2==0.11.0
httpretty==0.9.5
humanize==0.5.1
idna==2.6
ijson==2.3
iso8601==0.1
isodate==0.5.0
itsdangerous==0.24
Jinja2==2.10.3
jira==2.0.0
jmespath==0.9.3
jsondiff==1.1.2
launchdarkly-server-sdk==6.9.3
lxml==4.5.0
Mako==1.1.0
MarkupSafe==1.1.1
marshmallow==3.0.0rc3
MonthDelta==0.9.1
msgpack==1.0.0
mysqlclient==1.4.6
oauth2client==4.1.2
oauthlib==2.0.7
pathspec==0.6.0
pbr==5.4.4
Pillow==6.2.0
pkgconfig==1.5.1
promise==2.3
pyasn1==0.4.2
pyasn1-modules==0.2.1
pycparser==2.19
PyJWT==1.7.1
pyRFC3339==1.1
pysnow==0.7.4
python-dateutil==2.7.5
python-editor==1.0.4
python-json-logger==0.1.10
python-slugify==1.2.4
python3-saml==1.8.0
pytoml==0.1.21
pytz==2015.2
PyYAML==5.1
redis==3.3.11
requests==2.20.0
requests-oauthlib==0.8.0
requests-toolbelt==0.8.0
rsa==3.4.2
Rx==1.6.1
s3transfer==0.2.1
semver==2.9.0
sentry-sdk==0.14.3
sh==1.11
simplejson==3.16.0
six==1.13.0
SQLAlchemy==1.3.11
sqlalchemy-easy-profile==1.0.2
sqlparse==0.3.0
tabulate==0.8.3
tlslite==0.4.6
Unidecode==1.0.22
uritemplate==3.0.0
urllib3==1.24.3
uWSGI==2.0.17.1
uwsgi-tools==1.1.1
webargs==5.1.3
webencodings==0.5.1
Werkzeug==0.16.0
wrapt==1.11.1
WTForms==2.0.2
xmlsec==1.3.3
xmltodict==0.12.0
yamllint==1.11.1
yasha==4.3
zipstreamer==0.1.3
```


### How can we reproduce your problem?

```
import ddtrace.profile.auto
```

### What is the result that you get?

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py", line 215, in wrapped_find_and_load_unlocked
    return exec_and_call_hooks(module_name, wrapped, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py", line 171, in exec_and_call_hooks
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/ddtrace/profile/auto.py", line 2, in <module>
    from ddtrace.profile.bootstrap import sitecustomize  # noqa
  File "/usr/local/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py", line 215, in wrapped_find_and_load_unlocked
    return exec_and_call_hooks(module_name, wrapped, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py", line 171, in exec_and_call_hooks
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/ddtrace/profile/bootstrap/sitecustomize.py", line 7, in <module>
    from ddtrace.profile import profiler
  File "/usr/local/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py", line 215, in wrapped_find_and_load_unlocked
    return exec_and_call_hooks(module_name, wrapped, args, kwargs)
  File "/usr/local/lib/python3.7/site-packages/ddtrace/internal/import_hooks.py", line 171, in exec_and_call_hooks
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/ddtrace/profile/profiler.py", line 10, in <module>
    from ddtrace.profile.collector import stack
ImportError: cannot import name 'stack' from 'ddtrace.profile.collector' (/usr/local/lib/python3.7/site-packages/ddtrace/profile/collector/__init__.py)

### What is the result that you expected?

Not an exception


Discussion:
Hi,

It looks like the extension has not been compiled. How did you install the library?
Following the normal instructions, 

```
pip install ddtrace[profile]==0.35.0
```

This is being tested in an Alpine Linux container
So this definitely looks like an issue with building in Alpine Linux:

In python:3.7 (Debian)

```
docker run -it --rm python:3.7 bash
root@10a961d023a2:/# pip install ddtrace[profile]==0.36.0
Collecting ddtrace[profile]==0.36.0
  Downloading ddtrace-0.36.0-cp37-cp37m-manylinux2010_x86_64.whl (891 kB)
     |████████████████████████████████| 891 kB 4.9 MB/s
Collecting msgpack>=0.5.0
  Downloading msgpack-1.0.0-cp37-cp37m-manylinux1_x86_64.whl (275 kB)
     |████████████████████████████████| 275 kB 24.1 MB/s
Collecting intervaltree; extra == "profile"
  Downloading intervaltree-3.0.2.tar.gz (30 kB)
Collecting protobuf>=3; extra == "profile"
  Downloading protobuf-3.11.3-cp37-cp37m-manylinux1_x86_64.whl (1.3 MB)
     |████████████████████████████████| 1.3 MB 25.5 MB/s
Collecting sortedcontainers<3.0,>=2.0
  Downloading sortedcontainers-2.1.0-py2.py3-none-any.whl (28 kB)
Requirement already satisfied: setuptools in /usr/local/lib/python3.7/site-packages (from protobuf>=3; extra == "profile"->ddtrace[profile]==0.36.0) (46.1.3)
Collecting six>=1.9
  Downloading six-1.14.0-py2.py3-none-any.whl (10 kB)
Building wheels for collected packages: intervaltree
  Building wheel for intervaltree (setup.py) ... done
  Created wheel for intervaltree: filename=intervaltree-3.0.2-py3-none-any.whl size=25793 sha256=a958c9799a44688ed7ce9f742c8b5343749cd1d6bb2cd87265ae88271bc34a58
  Stored in directory: /root/.cache/pip/wheels/68/2d/6f/68fdd1342f58dac32f7d2781b00e9fd43b15d51e4c1ff9b4fb
Successfully built intervaltree
Installing collected packages: msgpack, sortedcontainers, intervaltree, six, protobuf, ddtrace
Successfully installed ddtrace-0.36.0 intervaltree-3.0.2 msgpack-1.0.0 protobuf-3.11.3 six-1.14.0 sortedcontainers-2.1.0
root@10a961d023a2:/# python
Python 3.7.7 (default, Mar 31 2020, 15:46:29)
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ddtrace.profile.auto
__main__:1: RemovedInDDTrace10Warning: 'ddtrace.profile' is deprecated and will be remove in future versions. Use ddtrace.profiling instead.
No exporters are configured, no profile will be output
```

There may be a build time dependency missing, but I don't see any errors during `pip install`


There are 3 different problems:
1. We provide pre-built wheels for Linux, but Alpine is not compatible with them, so you have to install it by source
2. Alpine does not have any C compiler by default, so you can't compile our wheels
3. Our wheel building / install mechanism should but it does not

I'll fix 3. by making sure you see an error when installing. That being said, this is not going to solve your issue — you'll just get an error message.

If you want to use Alpine Linux, you need (at least) to install `gcc`.

> There may be a build time dependency missing, but I don't see any errors during pip install

Yeah this is due to 3. You can see `WARNING` being print with `pip -v`, but they should be errors, not warning. Our bad.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,8 +1,6 @@
-import copy
 import os
 import sys
 
-from distutils.errors import CCompilerError, DistutilsExecError, DistutilsPlatformError
 from setuptools import setup, find_packages
 from setuptools.command.test import test as TestCommand
 
@@ -90,101 +88,6 @@ def run_tests(self):
 [visualization docs]: https://docs.datadoghq.com/tracing/visualization/
 """
 
-# Base `setup()` kwargs without any C-extension registering
-setup_kwargs = dict(
-    name="ddtrace",
-    description="Datadog tracing code",
-    url="https://github.com/DataDog/dd-trace-py",
-    author="Datadog, Inc.",
-    author_email="dev@datadoghq.com",
-    long_description=long_description,
-    long_description_content_type="text/markdown",
-    license="BSD",
-    packages=find_packages(exclude=["tests*"]),
-    python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
-    # enum34 is an enum backport for earlier versions of python
-    # funcsigs backport required for vendored debtcollector
-    # encoding using msgpack
-    install_requires=["enum34; python_version<'3.4'", "funcsigs>=1.0.0; python_version=='2.7'", "msgpack>=0.5.0",],
-    extras_require={
-        # users can include opentracing by having:
-        # install_requires=['ddtrace[opentracing]', ...]
-        "opentracing": ["opentracing>=2.0.0"],
-        # TODO: remove me when everything is updated to `profiling`
-        "profile": ["protobuf>=3", "intervaltree",],
-        "profiling": ["protobuf>=3", "intervaltree",],
-    },
-    # plugin tox
-    tests_require=["tox", "flake8"],
-    cmdclass={"test": Tox, "build_ext": Cython.Distutils.build_ext},
-    entry_points={
-        "console_scripts": [
-            "ddtrace-run = ddtrace.commands.ddtrace_run:main",
-            "pyddprofile = ddtrace.profiling.__main__:main",
-        ]
-    },
-    classifiers=[
-        "Programming Language :: Python",
-        "Programming Language :: Python :: 2.7",
-        "Programming Language :: Python :: 3.5",
-        "Programming Language :: Python :: 3.6",
-        "Programming Language :: Python :: 3.7",
-        "Programming Language :: Python :: 3.8",
-    ],
-    use_scm_version=True,
-    setup_requires=["setuptools_scm", "cython"],
-    ext_modules=cythonize(
-        [
-            Cython.Distutils.Extension(
-                "ddtrace.profiling.collector.stack",
-                sources=["ddtrace/profiling/collector/stack.pyx"],
-                language="c",
-                extra_compile_args=["-DPy_BUILD_CORE"],
-            ),
-            Cython.Distutils.Extension(
-                "ddtrace.profiling.collector._traceback",
-                sources=["ddtrace/profiling/collector/_traceback.pyx"],
-                language="c",
-            ),
-            Cython.Distutils.Extension(
-                "ddtrace.profiling._build", sources=["ddtrace/profiling/_build.pyx"], language="c",
-            ),
-        ],
-        compile_time_env={
-            "PY_MAJOR_VERSION": sys.version_info.major,
-            "PY_MINOR_VERSION": sys.version_info.minor,
-            "PY_MICRO_VERSION": sys.version_info.micro,
-        },
-    ),
-)
-
-
-if sys.platform == "win32":
-    build_ext_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError, IOError, OSError)
-else:
-    build_ext_errors = (CCompilerError, DistutilsExecError, DistutilsPlatformError)
-
-
-class BuildExtFailed(Exception):
-    pass
-
-
-# Attempt to build a C-extension, catch exceptions so failed building skips the extension
-# DEV: This is basically what `distutils`'s' `Extension(optional=True)` does
-class optional_build_ext(Cython.Distutils.build_ext):
-    def run(self):
-        try:
-            Cython.Distutils.build_ext.run(self)
-        except DistutilsPlatformError as e:
-            extensions = [ext.name for ext in self.extensions]
-            print("WARNING: Failed to build extensions %r, skipping: %s" % (extensions, e))
-
-    def build_extension(self, ext):
-        try:
-            Cython.Distutils.build_ext.build_extension(self, ext)
-        except build_ext_errors as e:
-            print("WARNING: Failed to build extension %s, skipping: %s" % (ext.name, e))
-
 
 def get_exts_for(name):
     try:
@@ -197,24 +100,74 @@ def get_exts_for(name):
         return []
 
 
-# Try to build with C extensions first, fallback to only pure-Python if building fails
-try:
-    all_exts = []
-    for extname in ("wrapt", "psutil"):
-        exts = get_exts_for(extname)
-        if exts:
-            all_exts.extend(exts)
-
-    kwargs = copy.deepcopy(setup_kwargs)
-    kwargs["ext_modules"] += all_exts
-    # DEV: Make sure `cmdclass` exists
-    kwargs.setdefault("cmdclass", dict())
-    kwargs["cmdclass"]["build_ext"] = optional_build_ext
-    setup(**kwargs)
-except Exception as e:
-    # Set `DDTRACE_BUILD_TRACE=TRUE` in CI to raise any build errors
-    if os.environ.get("DDTRACE_BUILD_RAISE") == "TRUE":
-        raise
-
-    print("WARNING: Failed to install with ddtrace C-extensions, falling back to pure-Python only extensions: %s" % e)
-    setup(**setup_kwargs)
+# Base `setup()` kwargs without any C-extension registering
+setup(
+    **dict(
+        name="ddtrace",
+        description="Datadog tracing code",
+        url="https://github.com/DataDog/dd-trace-py",
+        author="Datadog, Inc.",
+        author_email="dev@datadoghq.com",
+        long_description=long_description,
+        long_description_content_type="text/markdown",
+        license="BSD",
+        packages=find_packages(exclude=["tests*"]),
+        python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
+        # enum34 is an enum backport for earlier versions of python
+        # funcsigs backport required for vendored debtcollector
+        # encoding using msgpack
+        install_requires=["enum34; python_version<'3.4'", "funcsigs>=1.0.0; python_version=='2.7'", "msgpack>=0.5.0",],
+        extras_require={
+            # users can include opentracing by having:
+            # install_requires=['ddtrace[opentracing]', ...]
+            "opentracing": ["opentracing>=2.0.0"],
+            # TODO: remove me when everything is updated to `profiling`
+            "profile": ["protobuf>=3", "intervaltree",],
+            "profiling": ["protobuf>=3", "intervaltree",],
+        },
+        # plugin tox
+        tests_require=["tox", "flake8"],
+        cmdclass={"test": Tox, "build_ext": Cython.Distutils.build_ext},
+        entry_points={
+            "console_scripts": [
+                "ddtrace-run = ddtrace.commands.ddtrace_run:main",
+                "pyddprofile = ddtrace.profiling.__main__:main",
+            ]
+        },
+        classifiers=[
+            "Programming Language :: Python",
+            "Programming Language :: Python :: 2.7",
+            "Programming Language :: Python :: 3.5",
+            "Programming Language :: Python :: 3.6",
+            "Programming Language :: Python :: 3.7",
+            "Programming Language :: Python :: 3.8",
+        ],
+        use_scm_version=True,
+        setup_requires=["setuptools_scm", "cython"],
+        ext_modules=cythonize(
+            [
+                Cython.Distutils.Extension(
+                    "ddtrace.profiling.collector.stack",
+                    sources=["ddtrace/profiling/collector/stack.pyx"],
+                    language="c",
+                    extra_compile_args=["-DPy_BUILD_CORE"],
+                ),
+                Cython.Distutils.Extension(
+                    "ddtrace.profiling.collector._traceback",
+                    sources=["ddtrace/profiling/collector/_traceback.pyx"],
+                    language="c",
+                ),
+                Cython.Distutils.Extension(
+                    "ddtrace.profiling._build", sources=["ddtrace/profiling/_build.pyx"], language="c",
+                ),
+            ],
+            compile_time_env={
+                "PY_MAJOR_VERSION": sys.version_info.major,
+                "PY_MINOR_VERSION": sys.version_info.minor,
+                "PY_MICRO_VERSION": sys.version_info.micro,
+            },
+        )
+        + get_exts_for("wrapt")
+        + get_exts_for("psutil"),
+    )
+)


Test patch: