Repository: numba/numba

Issue:
NumPy is not a requirement when installing numba with pip
Typically when a package is installed with `pip`, it first checks for dependencies and installs those. `numpy` is also available on pip, so numba's setup.py should list numpy as a dependency so pip can automatically install it.



Discussion:
Can you elaborate a bit on what you're doing? `numpy` is listed in `install_requires` in `setup.py` (and in `requirements.txt` as well).

@pitrou Running `pip install numba` without numpy installed means it will fail and force you to install numpy manually.

I see... Yes, the problem is that `setup.py` needs to import `numpy` even before it can build up the `setup()` call.

I can't think of a simple way to solve this, so I would recommend you explicitly install `numpy` before `numba`.
(note we don't really support installing Numba using `pip`; our official method of distribution is through `conda`)

You could try:

```
try:
    import numpy
except ImportError:
    print("Please install numpy first!")
    sys.exit()

```

​conda requires installing a whole load of packages that a lot of people
don't have space for.​

On Thu, Aug 11, 2016 at 9:38 AM, Antoine Pitrou notifications@github.com
wrote:

> I can't think of a simple way to solve this, so I would recommend you
> explicitly install numpy before numba.
> (note we don't really support installing Numba using pip; our official
> method of distribution is through conda)
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> https://github.com/numba/numba/issues/2000#issuecomment-239162985, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/ACQO4XObXL2E4CmMK_WlbbYtI-PYsIZRks5qeyXOgaJpZM4JTnZq
> .

I don't want to argue too much about that, but a Miniconda download is around 30MB and will give you a fully usable conda + Python install (see https://repo.continuum.io/miniconda/). I believe that's quite small for most disk sizes nowadays.

@pitrou oh, didn't know. makes it harder to do packaging and platforms like Heroku can only install modules from PyPI.

Does Heroku provide the LLVM headers? Otherwise you won't be able to build llvmlite either...

@pitrou you have to do a lot of manual config. It's really complicated. There's even a custom buildpack to do it too and that takes a lot of config too...

https://github.com/dbrgn/heroku-buildpack-python-sklearn

Uh, I see. Perhaps someone could suggest Heroku they provide a conda install? That would totally solve the problem.

@pitrou If a numpy core dev submitted request, Heroku would most certainly listen.

We should still fix the install from pip issue at some level, even if we can't shield users from the awkwardness of having to build LLVM if you go down that route.

But also agree that encouraging Heroku to allow conda installation is also a good thing.

Due to the circularity mentioned above, I think the best we can do is to improve the error message as @CrazyPython suggested.

Oh, hey, I created issue #2000!

Are @pitrou or @seibert core numba devs? Heroku listens to open source leaders.

How do other projects that need to link to NumPy's C API handle this problem with pip?  It looks like the circularity is because we need to pass the names of the NumPy C headers to the Extension object.

That's a good point. Scipy uses this complicated sequence:
https://github.com/scipy/scipy/blob/master/setup.py#L355-L409

We may have to borrow some of this function: https://github.com/scipy/scipy/blob/master/setup.py#L208

Pandas doesn't seem to use `numpy.distutils` directly, which is surprising...
Scikit-learn has something similar to Scipy, though cruder: https://github.com/scikit-learn/scikit-learn/blob/master/setup.py#L219-L235


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -12,9 +12,6 @@
 import sys
 import os
 
-import numpy as np
-import numpy.distutils.misc_util as np_misc
-
 import versioneer
 
 
@@ -24,6 +21,7 @@ class build_doc(build.build):
     def run(self):
         spawn(['make', '-C', 'docs', 'html'])
 
+
 versioneer.VCS = 'git'
 versioneer.versionfile_source = 'numba/_version.py'
 versioneer.versionfile_build = 'numba/_version.py'
@@ -33,9 +31,6 @@ def run(self):
 cmdclass = versioneer.get_cmdclass()
 cmdclass['build_doc'] = build_doc
 
-setup_args = {
-    'long_description': open('README.rst').read(),
-}
 
 GCCFLAGS = ["-std=c89", "-Wdeclaration-after-statement", "-Werror"]
 
@@ -44,93 +39,125 @@ def run(self):
 else:
     CFLAGS = []
 
+install_name_tool_fixer = []
+if sys.platform == 'darwin':
+    install_name_tool_fixer += ['-headerpad_max_install_names']
+
 
-cpp_link_args = []
+def is_building():
+    """
+    Parse the setup.py command and return whether a build is requested.
+    If False is returned, only an informational command is run.
+    If True is returned, information about C extensions will have to
+    be passed to the setup() function.
+    """
+    if len(sys.argv) < 2:
+        # User forgot to give an argument probably, let setuptools handle that.
+        return True
 
+    info_commands = ['--help-commands', '--name', '--version', '-V',
+                     '--fullname', '--author', '--author-email',
+                     '--maintainer', '--maintainer-email', '--contact',
+                     '--contact-email', '--url', '--license', '--description',
+                     '--long-description', '--platforms', '--classifiers',
+                     '--keywords', '--provides', '--requires', '--obsoletes']
+    # Add commands that do more than print info, but also don't need
+    # any build step.
+    info_commands.extend(['egg_info', 'install_egg_info', 'rotate'])
 
-install_name_tool_fixer = []
+    for command in info_commands:
+        if command in sys.argv[1:]:
+            return False
 
-if sys.platform == 'darwin':
-    install_name_tool_fixer += ['-headerpad_max_install_names']
+    return True
 
 
-npymath_info = np_misc.get_info('npymath')
-
-ext_dynfunc = Extension(name='numba._dynfunc',
-                        sources=['numba/_dynfuncmod.c'],
-                        extra_compile_args=CFLAGS,
-                        depends=['numba/_pymodule.h',
-                                 'numba/_dynfunc.c'])
-
-ext_dispatcher = Extension(name="numba._dispatcher",
-                           include_dirs=[np.get_include()],
-                           sources=['numba/_dispatcher.c',
-                                    'numba/_typeof.c',
-                                    'numba/_hashtable.c',
-                                    'numba/_dispatcherimpl.cpp',
-                                    'numba/typeconv/typeconv.cpp'],
-                           depends=["numba/_pymodule.h",
-                                    "numba/_dispatcher.h",
-                                    "numba/_typeof.h",
-                                    "numba/_hashtable.h"],
-                           extra_link_args=cpp_link_args)
-
-ext_helperlib = Extension(name="numba._helperlib",
-                          include_dirs=npymath_info['include_dirs'],
-                          libraries=npymath_info['libraries'],
-                          library_dirs=npymath_info['library_dirs'],
-                          define_macros=npymath_info['define_macros'],
-                          sources=["numba/_helpermod.c",
-                                   "numba/_math_c99.c"],
-                          extra_compile_args=CFLAGS,
-                          extra_link_args=install_name_tool_fixer,
-                          depends=["numba/_pymodule.h",
-                                   "numba/_math_c99.h",
-                                   "numba/_helperlib.c",
-                                   "numba/_lapack.c",
-                                   "numba/_npymath_exports.c",
-                                   "numba/_random.c",
-                                   "numba/mathnames.inc"])
-
-ext_typeconv = Extension(name="numba.typeconv._typeconv",
-                         sources=["numba/typeconv/typeconv.cpp",
-                                  "numba/typeconv/_typeconv.cpp"],
-                         depends=["numba/_pymodule.h"],
-                         extra_link_args=cpp_link_args)
-
-ext_npyufunc_ufunc = Extension(name="numba.npyufunc._internal",
-                               sources=["numba/npyufunc/_internal.c"],
-                               include_dirs=[np.get_include()],
-                               depends=["numba/npyufunc/_ufunc.c",
-                                        "numba/npyufunc/_internal.h",
-                                        "numba/_pymodule.h"])
-
-ext_npyufunc_workqueue = Extension(
-    name='numba.npyufunc.workqueue',
-    sources=['numba/npyufunc/workqueue.c'],
-    depends=['numba/npyufunc/workqueue.h'])
-
-
-ext_mviewbuf = Extension(name='numba.mviewbuf',
-                         sources=['numba/mviewbuf.c'])
-
-ext_nrt_python = Extension(name='numba.runtime._nrt_python',
-                           sources=['numba/runtime/_nrt_pythonmod.c',
-                                    'numba/runtime/nrt.c'],
-                           depends=['numba/runtime/nrt.h',
-                                    'numba/_pymodule.h',
-                                    'numba/runtime/_nrt_python.c'],
-                           include_dirs=["numba"] + npymath_info['include_dirs'])
-
-ext_jitclass_box = Extension(name='numba.jitclass._box',
-                             sources=['numba/jitclass/_box.c'],
-                             depends=['numba/_pymodule.h'],
-                             include_dirs=['numba'])
-
-ext_modules = [ext_dynfunc, ext_dispatcher,
-               ext_helperlib, ext_typeconv,
-               ext_npyufunc_ufunc, ext_npyufunc_workqueue, ext_mviewbuf,
-               ext_nrt_python, ext_jitclass_box]
+def get_ext_modules():
+    """
+    Return a list of Extension instances for the setup() call.
+    """
+    # Note we don't import Numpy at the toplevel, since setup.py
+    # should be able to run without Numpy for pip to discover the
+    # build dependencies
+    import numpy.distutils.misc_util as np_misc
+
+    # Inject required options for extensions compiled against the Numpy
+    # C API (include dirs, library dirs etc.)
+    np_compile_args = np_misc.get_info('npymath')
+
+    ext_dynfunc = Extension(name='numba._dynfunc',
+                            sources=['numba/_dynfuncmod.c'],
+                            extra_compile_args=CFLAGS,
+                            depends=['numba/_pymodule.h',
+                                     'numba/_dynfunc.c'])
+
+    ext_dispatcher = Extension(name="numba._dispatcher",
+                               sources=['numba/_dispatcher.c',
+                                        'numba/_typeof.c',
+                                        'numba/_hashtable.c',
+                                        'numba/_dispatcherimpl.cpp',
+                                        'numba/typeconv/typeconv.cpp'],
+                               depends=["numba/_pymodule.h",
+                                        "numba/_dispatcher.h",
+                                        "numba/_typeof.h",
+                                        "numba/_hashtable.h"],
+                               **np_compile_args)
+
+    ext_helperlib = Extension(name="numba._helperlib",
+                              sources=["numba/_helpermod.c",
+                                       "numba/_math_c99.c"],
+                              extra_compile_args=CFLAGS,
+                              extra_link_args=install_name_tool_fixer,
+                              depends=["numba/_pymodule.h",
+                                       "numba/_math_c99.h",
+                                       "numba/_helperlib.c",
+                                       "numba/_lapack.c",
+                                       "numba/_npymath_exports.c",
+                                       "numba/_random.c",
+                                       "numba/mathnames.inc"],
+                              **np_compile_args)
+
+    ext_typeconv = Extension(name="numba.typeconv._typeconv",
+                             sources=["numba/typeconv/typeconv.cpp",
+                                      "numba/typeconv/_typeconv.cpp"],
+                             depends=["numba/_pymodule.h"],
+                             )
+
+    ext_npyufunc_ufunc = Extension(name="numba.npyufunc._internal",
+                                   sources=["numba/npyufunc/_internal.c"],
+                                   depends=["numba/npyufunc/_ufunc.c",
+                                            "numba/npyufunc/_internal.h",
+                                            "numba/_pymodule.h"],
+                                   **np_compile_args)
+
+    ext_npyufunc_workqueue = Extension(
+        name='numba.npyufunc.workqueue',
+        sources=['numba/npyufunc/workqueue.c'],
+        depends=['numba/npyufunc/workqueue.h'])
+
+
+    ext_mviewbuf = Extension(name='numba.mviewbuf',
+                             sources=['numba/mviewbuf.c'])
+
+    ext_nrt_python = Extension(name='numba.runtime._nrt_python',
+                               sources=['numba/runtime/_nrt_pythonmod.c',
+                                        'numba/runtime/nrt.c'],
+                               depends=['numba/runtime/nrt.h',
+                                        'numba/_pymodule.h',
+                                        'numba/runtime/_nrt_python.c'],
+                               **np_compile_args)
+
+    ext_jitclass_box = Extension(name='numba.jitclass._box',
+                                 sources=['numba/jitclass/_box.c'],
+                                 depends=['numba/_pymodule.h'],
+                                 )
+
+    ext_modules = [ext_dynfunc, ext_dispatcher,
+                   ext_helperlib, ext_typeconv,
+                   ext_npyufunc_ufunc, ext_npyufunc_workqueue, ext_mviewbuf,
+                   ext_nrt_python, ext_jitclass_box]
+
+    return ext_modules
 
 
 def find_packages(root_dir, root_name):
@@ -151,17 +178,20 @@ def rec(path, pkg_name):
 packages = find_packages("numba", "numba")
 
 
+build_requires = ['numpy']
+
 install_requires = ['llvmlite', 'numpy']
 if sys.version_info < (3, 4):
     install_requires.extend(['enum34', 'singledispatch'])
 if sys.version_info < (3, 3):
     install_requires.append('funcsigs')
 
-setup(name='numba',
-      description="compiling Python code using LLVM",
-      version=versioneer.get_version(),
+metadata = dict(
+    name='numba',
+    description="compiling Python code using LLVM",
+    version=versioneer.get_version(),
 
-      classifiers=[
+    classifiers=[
         "Development Status :: 4 - Beta",
         "Intended Audience :: Developers",
         "License :: OSI Approved :: BSD License",
@@ -171,8 +201,8 @@ def rec(path, pkg_name):
         "Programming Language :: Python :: 3.4",
         "Programming Language :: Python :: 3.5",
         "Topic :: Software Development :: Compilers",
-      ],
-      package_data={
+    ],
+    package_data={
         # HTML templates for type annotations
         "numba.annotations": ["*.html"],
         # Various test data
@@ -183,14 +213,22 @@ def rec(path, pkg_name):
         "numba": ["*.c", "*.h"],
         "numba.pycc": ["*.c", "*.h"],
         "numba.runtime": ["*.c", "*.h"],
-      },
-      scripts=["numba/pycc/pycc", "bin/numba"],
-      author="Continuum Analytics, Inc.",
-      author_email="numba-users@continuum.io",
-      url="http://numba.github.com",
-      ext_modules=ext_modules,
-      packages=packages,
-      install_requires=install_requires,
-      license="BSD",
-      cmdclass=cmdclass,
-      **setup_args)
+    },
+    scripts=["numba/pycc/pycc", "bin/numba"],
+    author="Continuum Analytics, Inc.",
+    author_email="numba-users@continuum.io",
+    url="http://numba.github.com",
+    packages=packages,
+    setup_requires=build_requires,
+    install_requires=install_requires,
+    license="BSD",
+    cmdclass=cmdclass,
+    )
+
+with open('README.rst') as f:
+    metadata['long_description'] = f.read()
+
+if is_building():
+    metadata['ext_modules'] = get_ext_modules()
+
+setup(**metadata)


Test patch: