Repository: pandas-dev/pandas

Issue:
Pip build test CI failures
I've seen a couple of these. Is this intermittent? Any idea what's going on?

https://travis-ci.org/pandas-dev/pandas/jobs/340640051

```
    running build_ext
    pandas._libs.algos: -> [['pandas/_libs/algos.c']]
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-nv04tz46-build/setup.py", line 741, in <module>
        **setuptools_kwargs)
      File "/home/travis/miniconda3/envs/pandas/lib/python3.6/site-packages/setuptools/__init__.py", line 129, in setup
        return distutils.core.setup(**attrs)
      File "/home/travis/miniconda3/envs/pandas/lib/python3.6/distutils/core.py", line 148, in setup
        dist.run_commands()
      File "/home/travis/miniconda3/envs/pandas/lib/python3.6/distutils/dist.py", line 955, in run_commands
        self.run_command(cmd)
      File "/home/travis/miniconda3/envs/pandas/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/travis/miniconda3/envs/pandas/lib/python3.6/site-packages/setuptools/command/install.py", line 61, in run
        return orig.install.run(self)
      File "/home/travis/miniconda3/envs/pandas/lib/python3.6/distutils/command/install.py", line 545, in run
        self.run_command('build')
      File "/home/travis/miniconda3/envs/pandas/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/travis/miniconda3/envs/pandas/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/travis/miniconda3/envs/pandas/lib/python3.6/distutils/command/build.py", line 135, in run
        self.run_command(cmd_name)
      File "/home/travis/miniconda3/envs/pandas/lib/python3.6/distutils/cmd.py", line 313, in run_command
        self.distribution.run_command(command)
      File "/home/travis/miniconda3/envs/pandas/lib/python3.6/distutils/dist.py", line 974, in run_command
        cmd_obj.run()
      File "/home/travis/miniconda3/envs/pandas/lib/python3.6/distutils/command/build_ext.py", line 339, in run
        self.build_extensions()
      File "/tmp/pip-nv04tz46-build/setup.py", line 367, in build_extensions
        self.check_cython_extensions(self.extensions)
      File "/tmp/pip-nv04tz46-build/setup.py", line 364, in check_cython_extensions
        """.format(src=src))
    Exception: Cython-generated file 'pandas/_libs/algos.c' not found.
                    Cython is required to compile pandas from a development branch.
                    Please install Cython or download a release package of pandas.
    
```


Discussion:
a little further up
```
gcc -pthread -shared -B /home/travis/miniconda3/envs/pandas/compiler_compat -L/home/travis/miniconda3/envs/pandas/lib -Wl,-rpath=/home/travis/miniconda3/envs/pandas/lib -Wl,--no-as-needed -Wl,--sysroot=/ build/temp.linux-x86_64-3.6/pandas/_libs/tslibs/conversion.o build/temp.linux-x86_64-3.6/pandas/_libs/src/datetime/np_datetime.o build/temp.linux-x86_64-3.6/pandas/_libs/src/datetime/np_datetime_strings.o -o build/lib.linux-x86_64-3.6/pandas/_libs/tslibs/conversion.cpython-36m-x86_64-linux-gnu.so
  build/temp.linux-x86_64-3.6/pandas/_libs/tslibs/conversion.o: file not recognized: File truncated
  collect2: error: ld returned 1 exit status
  error: command 'gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for pandas
```

ccache seems like a suspect
https://stackoverflow.com/questions/5713894/file-not-recognized-file-truncated-gcc-error

It is used a little further up to build `conversion.o`, but I'm not familiar enough with it to know offhand what's actually going wrong or how to fix (blunt fix is probably to disable ccache entirely for now).  cc @jbrockmendel 

```
ccache gcc -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/home/travis/miniconda3/envs/pandas/lib/python3.6/site-packages/numpy/core/include -I/home/travis/miniconda3/envs/pandas/include/python3.6m -c pandas/_libs/tslibs/conversion.c -o build/temp.linux-x86_64-3.6/pandas/_libs/tslibs/conversion.o -Wno-unused-function
```




not don’t disable caching!!!

but u can clear the travis cache on master of u want
@hexgnu it's looking like https://github.com/pandas-dev/pandas/commit/39e7b6916b07982240bac87132848fb2665806a2#diff-2eeaed663bd0d25b7e608891384b7298 is the commit that started causing issues here (that's the first commit that touched `setup.py` that broke things).

The issue is that we want users to be able to install pandas from source without Cython. Any ideas?

To test you can run `bash scripts/build_dist_for_release.sh` from an env with Cython, and then `pip install path/to/pandas/dist/*.tar.gz` from an env without cython.
 
Think I found the issue hold on...

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -311,7 +311,6 @@ class CheckSDist(sdist_class):
                  'pandas/_libs/missing.pyx',
                  'pandas/_libs/reduction.pyx',
                  'pandas/_libs/testing.pyx',
-                 'pandas/_libs/window.pyx',
                  'pandas/_libs/skiplist.pyx',
                  'pandas/_libs/sparse.pyx',
                  'pandas/_libs/parsers.pyx',
@@ -331,6 +330,10 @@ class CheckSDist(sdist_class):
                  'pandas/_libs/writers.pyx',
                  'pandas/io/sas/sas.pyx']
 
+    _cpp_pyxfiles = ['pandas/_libs/window.pyx',
+                     'pandas/io/msgpack/_packer.pyx',
+                     'pandas/io/msgpack/_unpacker.pyx']
+
     def initialize_options(self):
         sdist_class.initialize_options(self)
 
@@ -338,12 +341,17 @@ def run(self):
         if 'cython' in cmdclass:
             self.run_command('cython')
         else:
-            for pyxfile in self._pyxfiles:
-                cfile = pyxfile[:-3] + 'c'
-                msg = ("C-source file '{source}' not found.\n"
-                       "Run 'setup.py cython' before sdist.".format(
-                           source=cfile))
-                assert os.path.isfile(cfile), msg
+            # If we are not running cython then
+            # compile the extensions correctly
+            pyx_files = [(self._pyxfiles, 'c'), (self._cpp_pyxfiles, 'cpp')]
+
+            for pyxfiles, extension in pyx_files:
+                for pyxfile in pyxfiles:
+                    sourcefile = pyxfile[:-3] + extension
+                    msg = ("{extension}-source file '{source}' not found.\n"
+                           "Run 'setup.py cython' before sdist.".format(
+                               source=sourcefile, extension=extension))
+                    assert os.path.isfile(sourcefile), msg
         sdist_class.run(self)
 
 
@@ -417,6 +425,11 @@ def get_tag(self):
     cmdclass['build_src'] = DummyBuildSrc
     cmdclass['build_ext'] = CheckingBuildExt
 
+if sys.byteorder == 'big':
+    endian_macro = [('__BIG_ENDIAN__', '1')]
+else:
+    endian_macro = [('__LITTLE_ENDIAN__', '1')]
+
 lib_depends = ['inference']
 
 
@@ -453,6 +466,7 @@ def pxd(name):
                        'pandas/_libs/src/datetime/np_datetime_strings.h']
 np_datetime_sources = ['pandas/_libs/src/datetime/np_datetime.c',
                        'pandas/_libs/src/datetime/np_datetime_strings.c']
+
 tseries_depends = np_datetime_headers + ['pandas/_libs/tslibs/np_datetime.pxd']
 
 # some linux distros require it
@@ -618,17 +632,42 @@ def pxd(name):
     '_libs.window': {
         'pyxfile': '_libs/window',
         'pxdfiles': ['_libs/skiplist', '_libs/src/util'],
-        'language': 'c++'},
+        'language': 'c++',
+        'suffix': '.cpp'},
     '_libs.writers': {
         'pyxfile': '_libs/writers',
         'pxdfiles': ['_libs/src/util']},
     'io.sas._sas': {
-        'pyxfile': 'io/sas/sas'}}
+        'pyxfile': 'io/sas/sas'},
+    'io.msgpack._packer': {
+        'macros': endian_macro,
+        'depends': ['pandas/_libs/src/msgpack/pack.h',
+                    'pandas/_libs/src/msgpack/pack_template.h'],
+        'include': ['pandas/_libs/src/msgpack'] + common_include,
+        'language': 'c++',
+        'suffix': '.cpp',
+        'pyxfile': 'io/msgpack/_packer',
+        'subdir': 'io/msgpack'},
+    'io.msgpack._unpacker': {
+        'depends': ['pandas/_libs/src/msgpack/unpack.h',
+                    'pandas/_libs/src/msgpack/unpack_define.h',
+                    'pandas/_libs/src/msgpack/unpack_template.h'],
+        'macros': endian_macro,
+        'include': ['pandas/_libs/src/msgpack'] + common_include,
+        'language': 'c++',
+        'suffix': '.cpp',
+        'pyxfile': 'io/msgpack/_unpacker',
+        'subdir': 'io/msgpack'
+    }
+}
 
 extensions = []
 
 for name, data in ext_data.items():
-    sources = [srcpath(data['pyxfile'], suffix=suffix, subdir='')]
+    source_suffix = suffix if suffix == '.pyx' else data.get('suffix', '.c')
+
+    sources = [srcpath(data['pyxfile'], suffix=source_suffix, subdir='')]
+
     pxds = [pxd(x) for x in data.get('pxdfiles', [])]
     if suffix == '.pyx' and pxds:
         sources.extend(pxds)
@@ -642,46 +681,11 @@ def pxd(name):
                     depends=data.get('depends', []),
                     include_dirs=include,
                     language=data.get('language', 'c'),
+                    define_macros=data.get('macros', []),
                     extra_compile_args=extra_compile_args)
 
     extensions.append(obj)
 
-# ----------------------------------------------------------------------
-# msgpack
-
-if sys.byteorder == 'big':
-    macros = [('__BIG_ENDIAN__', '1')]
-else:
-    macros = [('__LITTLE_ENDIAN__', '1')]
-
-msgpack_include = ['pandas/_libs/src/msgpack'] + common_include
-msgpack_suffix = suffix if suffix == '.pyx' else '.cpp'
-unpacker_depends = ['pandas/_libs/src/msgpack/unpack.h',
-                    'pandas/_libs/src/msgpack/unpack_define.h',
-                    'pandas/_libs/src/msgpack/unpack_template.h']
-
-packer_ext = Extension('pandas.io.msgpack._packer',
-                       depends=['pandas/_libs/src/msgpack/pack.h',
-                                'pandas/_libs/src/msgpack/pack_template.h'],
-                       sources=[srcpath('_packer',
-                                suffix=msgpack_suffix,
-                                subdir='io/msgpack')],
-                       language='c++',
-                       include_dirs=msgpack_include,
-                       define_macros=macros,
-                       extra_compile_args=extra_compile_args)
-unpacker_ext = Extension('pandas.io.msgpack._unpacker',
-                         depends=unpacker_depends,
-                         sources=[srcpath('_unpacker',
-                                  suffix=msgpack_suffix,
-                                  subdir='io/msgpack')],
-                         language='c++',
-                         include_dirs=msgpack_include,
-                         define_macros=macros,
-                         extra_compile_args=extra_compile_args)
-extensions.append(packer_ext)
-extensions.append(unpacker_ext)
-
 # ----------------------------------------------------------------------
 # ujson
 


Test patch: