Repository: pandas-dev/pandas

Issue:
BLD: tslib dependencies not fully specified
```
    '_libs.tslib': {'pyxfile': '_libs/tslib',
                    'pxdfiles': ['_libs/src/util'],
                    'depends': tseries_depends,
                    'sources': np_datetime_sources},
```

is not complete, in fact this directly depends on 

```
(pandas) bash-3.2$ ls pandas/_libs/tslibs/*.pxd
pandas/_libs/tslibs/conversion.pxd      pandas/_libs/tslibs/nattype.pxd         pandas/_libs/tslibs/timedeltas.pxd
pandas/_libs/tslibs/frequencies.pxd     pandas/_libs/tslibs/np_datetime.pxd     pandas/_libs/tslibs/timezones.pxd
(pandas) bash-3.2$ ls pandas/_libs/tslib*.pxd
pandas/_libs/tslib.pxd
```

not really sure of the exact rule, but I think anything you cimport from a .pxd must be listed as a dep.
Now rebuilding from clean just works, but rebuilding from a change in one part of the structure will only *sometimes* work.
BUG: Panel.fillna() ignores inplace=True
I'm not sure of `Panel` is still being supported, but `Panel.fillna()` seems to ignore `inplace=True`:

```
In [1]: import numpy as np

In [2]: import pandas as pd

In [5]: p1 = pd.Panel([[[0,1],[2,np.nan]], [[10,11],[12,np.nan]]], items=['a','b'], minor_axis=['x','y'])

In [7]: p1['a']
Out[7]:
     x    y
0  0.0  1.0
1  2.0  NaN

In [8]: p1['b']
Out[8]:
      x     y
0  10.0  11.0
1  12.0   NaN

In [9]: p1.fillna(method='ffill', inplace=True)
Out[9]:
<class 'pandas.core.panel.Panel'>
Dimensions: 2 (items) x 2 (major_axis) x 2 (minor_axis)
Items axis: a to b
Major_axis axis: 0 to 1
Minor_axis axis: x to y

In [10]: p1['a']
Out[10]:
     x    y
0  0.0  1.0
1  2.0  NaN

In [11]: p1['b']
Out[11]:
      x     y
0  10.0  11.0
1  12.0   NaN

In [12]: p2 = p1.fillna(method='ffill', inplace=True)

In [13]: p2['a']
Out[13]:
     x    y
0  0.0  1.0
1  2.0  1.0

In [14]: p2['b']
Out[14]:
      x     y
0  10.0  11.0
1  12.0  11.0

In [15]: pd.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.4.4.final.0
python-bits: 64
OS: Windows
OS-release: 10
machine: AMD64
processor: Intel64 Family 6 Model 62 Stepping 4, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None

pandas: 0.18.0
nose: 1.3.7
pip: 8.1.0
setuptools: 20.2.2
Cython: None
numpy: 1.10.4
scipy: 0.17.0
statsmodels: 0.6.1
xarray: 0.7.1
IPython: 4.1.2
sphinx: None
patsy: 0.4.1
dateutil: 2.5.0
pytz: 2016.1
blosc: None
bottleneck: None
tables: 3.2.2
numexpr: 2.5
matplotlib: 1.5.1
openpyxl: 2.3.3
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: 4.4.1
html5lib: 1.0b8
httplib2: None
apiclient: None
sqlalchemy: 1.0.12
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
jinja2: 2.8
boto: None
```



Discussion:
@jbrockmendel need to fix this before any other changes.
>  but rebuilding from a change in one part of the structure will only sometimes work.

Do you have an example for how this fails?  My understanding is that cython checks the timestamps on other cython files for determining if dependencies have changed.
try making a change on a .pxd (e.g moving something), them try to make, it will fail because it can't load the object.
nothing to do with time stamps, rather the build process doesn't know about deps. that it is depending, IOW tslib depends on tslibs/converstion.pxd, but this is not indicated anywhere (to rebuild tslib).
> try making a change on a .pxd (e.g moving something), them try to make, it will fail because it can't load the object

I noticed this yesterday when I tried to build `pandas` from source in an existing environment.  Some Cython files didn't get compiled again as needed (because dependencies were not specified), causing a statement like `import pandas` to fail.
> nothing to do with time stamps, rather the build process doesn't know about deps

OK I'll try to grok this.

Cython clearly knows _something_ about up-to-dateness:

```
$python setup.py build_ext --inplace
running build_ext
skipping 'pandas/io/sas/sas.c' Cython extension (up-to-date)
skipping 'pandas/_libs/properties.c' Cython extension (up-to-date)
skipping 'pandas/_libs/groupby.c' Cython extension (up-to-date)
[...]
```
@jbrockmendel : That's actually us :smile: .  We compare timestamps on all files and dependencies to determine whether they need updating.
> try making a change on a .pxd (e.g moving something), them try to make, it will fail because it can't load the object.

OK, I just took an existing branch, went to conversion.pxd and made a whitespace edit, then ran setup.py build_ext:


```
$python setup.py build_ext --inplace
[...]
skipping 'pandas/_libs/tslibs/frequencies.c' Cython extension (up-to-date)
skipping 'pandas/_libs/hashtable.c' Cython extension (up-to-date)
cythoning pandas/_libs/tslibs/conversion.pyx to pandas/_libs/tslibs/conversion.c
building 'pandas._libs.tslibs.conversion' extension
clang -fno-strict-aliasing -fno-common -dynamic -g -O2 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/usr/local/lib/python2.7/site-packages/numpy/core/include -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/sqlite/include -I/usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/include/python2.7 -c pandas/_libs/tslibs/conversion.c -o build/temp.macosx-10.12-x86_64-2.7/pandas/_libs/tslibs/conversion.o -Wno-unused-function
[...]
skipping 'pandas/_libs/interval.c' Cython extension (up-to-date)
[...]
$python
>>> import pandas as pd
>>>
```

So far so good.  So I guess you're referring to something less benign than a whitespace change?  If I comment out `convert_to_tsobject` in conversion.pxd then I'll get an error at import-time.  Ideally this would raise at build-time, which I guess is what you're getting at here.

The cython folks are at least [aware](https://github.com/cython/cython/issues/1428
) of this; lack of discussion suggests it isn't a high priority.

There is a cython option that would allow us to do without pxd files entirely in some cases, but I'm a bit wary of it: https://github.com/cython/cython/issues/1866

Pretty sure I've seen this on the cython mailing list, will try to track that down. 
> That's actually us 😄 . We compare timestamps on all files to determine whether they need updating.

Hah.  Good to know.
> If I comment out convert_to_tsobject in conversion.pxd then I'll get an error at import-time. Ideally this would raise at build-time, which I guess is what you're getting at here.

Yes, that's exactly what happened to me yesterday.  I wouldn't worry too much about anything Cython-issue-related, as that's beyond our control from a `pandas` perspective.  The best thing would be to try to correct the dependencies on our end so that we don't get import errors.
> Yes, that's exactly what happened to me yesterday.

OK, so IIUC you're making a change that doesn't actually break `conversion` internally, but gives a mangled error message on `import pandas as pd`:

`ImportError: C extension: pandas._libs.tslibs.conversion does not export expected C function convert_to_tsobject not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace --force' to build the C extensions first.`

That error is re-raised in `pandas/__init__.py`, which seems to incorrectly assume that the build state is binary.  If the failed import is taken outside of the try/except block, the exception message is:

`ImportError: pandas._libs.tslibs.conversion does not export expected C function convert_to_tsobject`

In this particular failure mode, this error message is pretty helpful.  In fact, it is more helpful than the error message we would get if we added conversion.pxd to tslib's `'pxdfile'` list in setup.py:

```
Error compiling Cython file:
------------------------------------------------------------
...
from numpy cimport ndarray, int64_t

from tslibs.conversion cimport convert_to_tsobject
^
------------------------------------------------------------

pandas/_libs/tslib.pxd:3:0: 'pandas/_libs/tslibs/conversion/convert_to_tsobject.pxd' not found
[...]
[26 lines of less helpful information to scroll back up past to find the real error]
```

So I can put together a PR that will exhaustively list in setup.py all of the pxdfiles relevant to every cython module.  Then we can vigilantly keep that up to date.  OR I can fix the error message in `__init__`.  The former has the benefit of raising at build-time, but the latter is less vulnerable to future leakage.  (Longer-term I'm hoping cython catches this at build-time).
Incidentally, the str-replacement in the error message in `__init__` probably needs to be updated.  Right now after `python setup.py clean` importing will raise with the message

```
ImportError: C extension: No module named tslib not built.[...]
```

which I think is intended to be

```
Import Error: C extension: tslib not built.[...]
```
Related:
Could we generate .pxd files from .pyx files https://groups.google.com/forum/#!topic/cython-users/jqWnMwETW_s
cythonize should recompile on dependency changes https://groups.google.com/forum/#!topic/cython-users/yNCMjqTijGU

> So I can put together a PR that will exhaustively list in setup.py all of the pxdfiles relevant to every cython module. Then we can vigilantly keep that up to date. OR I can fix the error message in __init__. The former has the benefit of raising at build-time, but the latter is less vulnerable to future leakage. (Longer-term I'm hoping cython catches this at build-time).

Given where things are right now, it should be our responsibility to keep track of dependencies.  The latter can be used to provide a better error message.  However, to close the issue, the former should be done.
One last-ditch argument for the "fix the import error message" solution.

Listing these all in setup.py is asking for trouble.  There is plenty of evidence that the deps listed in the `ext_data` dict do not get updated when the actual modules are changed.  Until recently the `tslib` entry had '_libs/lib' in its 'pxdfiles' entry, while the `lib` entry did not (and does not) have `tslib` in its 'pxdfiles' (or 'depends') entries; i.e. the exact opposite of what the dependencies _actually_ are.

Quick: which entries get '_libs/src/foo' and which get 'pandas/_libs/src/foo'?  Does the entry for `lib` need to explicitly list out its recursive dependencies (as it kinda-sorta does by including `tseries_depends`  in its 'depends' entry)?  Does '_libs/src/util' belong under 'depends' or 'pxdfiles' (under the status quo it is inconsistent)?  How does the inclusion of `common_include` affect any of this?

My point being: the `ext_data` dictionary is Frankensteined together and we should be working to get stuff _out_ of it rather than dump more stuff in (e.g. #18059).

For modules in `src` we _do_ need to specify in setup where to look for things.  For regular cython modules we can let imports fail _in the exact same way_ as python imports fail.
@jbrockmendel put it this way. if I have to make clean every once in a while, when I didn't have to before, no big deal. But this highlites a problem. I would like to fix this before doing anything else. 

I don't care about the error message, in fact that is a red herring. The build process is messed up.
I can offer moral support, but it seems like I don't understand the problem.

"When I change something that should cause the code to break, the code breaks" doesn't sound like a problem.  If the problem is about the error being at import-time instead of compile-time, that I could understand.  If the problem is about the error reported at import-time being unhelpful, that I could understand.

> The build process is messed up.

Per my previous comment, I more or less agree.  I just don't see how throwing more stuff at the wall makes it better.  So for the time being I'll stick to moral support.
@jbrockmendel : The whole idea is that you need to add more dependencies to specific Cython files.  You can figure out which ones are dependencies given that you know where the imports fail.
> given that you know where the imports fail.

Until something gets broken, imports don't fail.  Are you suggesting a fuzzing operation to trace these out?  I'm honestly not trying to be obstinate here, just feel like I'm missing some major part of the picture.
> Are you suggesting a fuzzing operation to trace these out? 

No, no, don't do that :smile:

You did a lot of refactoring of the Cython modules in previous PR's, right?  So you're going to have to retrace your steps and see where imports are coming from with your new refactorings.  There are a couple of ways to achieve this AFAICT:

1) Build `pandas` from source pre-PR, build again from source post-PR (but don't do a clean build), and then seeing where Python complains when you try to import `pandas`.

2) Look over your changes visually and make those modifications yourself based on whether Cython files import variables from different files.
> Build pandas from source pre-PR, build again from source post-PR (but don't do a clean build), and then seeing where Python complains when you try to import pandas.

Sounds like having a not-screwed-up exception at import would help with this.

> Look over your changes visually and make those modifications yourself based on whether Cython files import variables from different files.

OK, so a hypothetical contributor can go through all these files and find which modules import from which other modules... just like python does at import time.  So what is this accomplishing?

Supposing you convince/browbeat this contributor into doing this, 

 - Does the 'depends' entry for `tslib` get 'pandas/_libs/tslibs/conversion.pyx' or 'pandas/_libs/tslibs/conversion.pxd'?  Both for good measure?
 - Does the pxd file go into the entry for `pxdfiles`?
 - Do we care about the non-tslibs entries?  e.g. `_libs.hashing` doesn't list anything other than its own pyx file, but cimports `util`.

Does it matter that khash isn't listed in `ext_data['tslib']`?
@jbrockmendel : I understand that this work is not necessarily appealing, but given that you were the one who initiated a lot of these refactorings, you would be most familiar with the new files that you created.  AFAICT, the changes you need to make would be to add your newly created files as dependencies.

Don't worry about other files for the moment, just figure out where your newly created files should be as dependencies for the time being.
See recent PR that does this, ignores internal consistency, makes black boxes larger, as requested.

Anticipating that part of your response, the PR ignores non-tslib(s).  Good thing you didn’t address the other bullet points; wouldn’t want to accidentally give the impression getting this right matters.
> wouldn’t want to accidentally give the impression getting this right matters.

Oh, getting it completely right matters.  We just don't need to rush it too quickly :smile: 

Don't worry about the black box for now.  Dependencies are a bit clunky, but let's at least make sure that we can do builds without having do a clean one every time.
@jbrockmendel 

> Does the 'depends' entry for tslib get 'pandas/_libs/tslibs/conversion.pyx' or 'pandas/_libs/tslibs/conversion.pxd'? Both for good measure?
Does the pxd file go into the entry for pxdfiles?
Do we care about the non-tslibs entries? e.g. _libs.hashing doesn't list anything other than its own pyx file, but cimports util.

I don't really know *what* should be listed; the point is the dependency tree is broken. listing lots of things is not the answer, but I suspect we *do* need to list .pxd files.
For testing purposes I've implemented tslibs as effectively a standalone packages.  The setup.py for this package is relatively simple, the relevant bits being

```
import Cython
from Cython.Build import cythonize

exts = [
    [...]
    Extension("tslibs.conversion",
              sources=["tslibs/conversion.pyx"],
              include_dirs=[np.get_include()],
              define_macros=macros),
    [...]
]

setup(name='tslibs',
           [...],
           cmdclass={'build_ext': Cython.Distutils.build_ext},
           ext_modules=cythonize(exts, compiler_directives=directives))
```

The reason I bring this up is because when I run setup.py build_ext --inplace I see a bunch of messages:

```
Compiling tslibs/npy_dtime.pyx because it depends on tslibs/npy_dtime.pxd.
Compiling tslibs/period.pyx because it depends on tslibs/npy_dtime.pxd.
Compiling tslibs/api.pyx because it changed.
Compiling tslibs/timestamps.pyx because it depends on tslibs/npy_dtime.pxd.
```

i.e. it looks very much like cython is doing some kind of caching and dependency recognition internally.
I asked about this on the cython mailing list.  Two key points from the answer:

- Conditional on using Cython's `build_ext`, this is apparently the appropriate way to specify dependencies.

- Doing so is outdated.  Using `cythonize` will handle most of this on its own.

https://groups.google.com/forum/#!topic/cython-users/YdGpGqi6Qw4
Indeed. The whole point of `cythonize()` is that it does dependency checking for you.
and we don’t actually use cythonize; this setup.py was built with early versions of cython

would take a PR that uses cythonize to correctly infer our deps 
OK, we're very nearly ready to use cythonize.  The remaining warnings/errors that I get when doing this locally are

```$ python setup.py build_ext --inplace
Warning: Extension name 'pandas.io.sas._sas' does not match fully qualified name 'pandas.io.sas.sas' of 'pandas/io/sas/sas.pyx'
[...]
$ python -m pytest pandas
[...]
_____________________________________ TestSAS7BDAT.test_from_file ______________________________________

self = <pandas.tests.io.sas.test_sas7bdat.TestSAS7BDAT object at 0x115816a50>

    def test_from_file(self):
        for j in 0, 1:
            df0 = self.data[j]
            for k in self.test_ix[j]:
                fname = os.path.join(
                    self.dirpath, "test{k}.sas7bdat".format(k=k))
>               df = pd.read_sas(fname, encoding='utf-8')

pandas/tests/io/sas/test_sas7bdat.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/io/sas/sasreader.py:58: in read_sas
    from pandas.io.sas.sas7bdat import SAS7BDATReader
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    """
    
    import pandas as pd
    from pandas import compat
    from pandas.io.common import get_filepath_or_buffer, BaseIterator
    from pandas.errors import EmptyDataError
    import numpy as np
    import struct
    import pandas.io.sas.sas_constants as const
>   from pandas.io.sas._sas import Parser
E   ImportError: dynamic module does not define init function (init_sas)

pandas/io/sas/sas7bdat.py:24: ImportError
```
Renaming `sas.pyx` to `_sas.pyx` should fix that...
I'll give it a shot over the weekend and see how it goes.  There's a decent chance we might be able to get rid of all of my PRs in the next day or two.
@jbrockmendel anything left to do for this issue? (didn't read the full thing :)) Is it a blocker for a 0.23 release?
Nothing blocking here.  It's basically on hold until the next bump in the cythonize version requirements (I forget the exact cutoff, just remember we're one below it) at which point some nice cleanup can be done.
OK, removing the 0.23 milestone then
hmm, that should work. ok will mark it. thanks. 

closed by #12633 


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -40,9 +40,11 @@ def is_platform_windows():
 try:
     import Cython
     ver = Cython.__version__
+    from Cython.Build import cythonize
     _CYTHON_INSTALLED = ver >= LooseVersion(min_cython_ver)
 except ImportError:
     _CYTHON_INSTALLED = False
+    cythonize = lambda x, *args, **kwargs: x  # dummy func
 
 # The import of Extension must be after the import of Cython, otherwise
 # we do not get the appropriately patched class.
@@ -419,11 +421,66 @@ def get_tag(self):
     cmdclass['build_src'] = DummyBuildSrc
     cmdclass['build_ext'] = CheckingBuildExt
 
+# ----------------------------------------------------------------------
+# Preparation of compiler arguments
+
 if sys.byteorder == 'big':
     endian_macro = [('__BIG_ENDIAN__', '1')]
 else:
     endian_macro = [('__LITTLE_ENDIAN__', '1')]
 
+
+if is_platform_windows():
+    extra_compile_args = []
+else:
+    # args to ignore warnings
+    extra_compile_args = ['-Wno-unused-function']
+
+
+# enable coverage by building cython files by setting the environment variable
+# "PANDAS_CYTHON_COVERAGE" (with a Truthy value)
+linetrace = os.environ.get('PANDAS_CYTHON_COVERAGE', False)
+CYTHON_TRACE = str(int(bool(linetrace)))
+
+# Note: if not using `cythonize`, coverage can be enabled by
+# pinning `ext.cython_directives = directives` to each ext in extensions.
+# github.com/cython/cython/wiki/enhancements-compilerdirectives#in-setuppy
+directives = {'linetrace': False}
+macros = []
+if linetrace:
+    # https://pypkg.com/pypi/pytest-cython/f/tests/example-project/setup.py
+    directives['linetrace'] = True
+    macros = [('CYTHON_TRACE', '1'), ('CYTHON_TRACE_NOGIL', '1')]
+
+
+# ----------------------------------------------------------------------
+# Specification of Dependencies
+
+# TODO: Need to check to see if e.g. `linetrace` has changed and possibly
+# re-compile.
+def maybe_cythonize(extensions, *args, **kwargs):
+    """
+    Render tempita templates before calling cythonize
+    """
+    if len(sys.argv) > 1 and 'clean' in sys.argv:
+        # Avoid running cythonize on `python setup.py clean`
+        # See https://github.com/cython/cython/issues/1495
+        return extensions
+
+    numpy_incl = pkg_resources.resource_filename('numpy', 'core/include')
+    # TODO: Is this really necessary here?
+    for ext in extensions:
+        if (hasattr(ext, 'include_dirs') and
+                numpy_incl not in ext.include_dirs):
+            ext.include_dirs.append(numpy_incl)
+
+    if cython:
+        build_ext.render_templates(_pxifiles)
+        return cythonize(extensions, *args, **kwargs)
+    else:
+        return extensions
+
+
 lib_depends = ['inference']
 
 
@@ -434,23 +491,13 @@ def srcpath(name=None, suffix='.pyx', subdir='src'):
 if suffix == '.pyx':
     lib_depends = [srcpath(f, suffix='.pyx', subdir='_libs/src')
                    for f in lib_depends]
-    lib_depends.append('pandas/_libs/src/util.pxd')
+    lib_depends.append('pandas/_libs/util.pxd')
 else:
     lib_depends = []
 
 common_include = ['pandas/_libs/src/klib', 'pandas/_libs/src']
 
 
-def pxd(name):
-    return pjoin('pandas', name + '.pxd')
-
-
-if is_platform_windows():
-    extra_compile_args = []
-else:
-    # args to ignore warnings
-    extra_compile_args = ['-Wno-unused-function']
-
 lib_depends = lib_depends + ['pandas/_libs/src/numpy_helper.h',
                              'pandas/_libs/src/parse_helper.h',
                              'pandas/_libs/src/compat_helper.h']
@@ -466,22 +513,18 @@ def pxd(name):
 ext_data = {
     '_libs.algos': {
         'pyxfile': '_libs/algos',
-        'pxdfiles': ['_libs/src/util', '_libs/algos', '_libs/hashtable'],
         'depends': _pxi_dep['algos']},
     '_libs.groupby': {
         'pyxfile': '_libs/groupby',
-        'pxdfiles': ['_libs/src/util', '_libs/algos'],
         'depends': _pxi_dep['groupby']},
     '_libs.hashing': {
         'pyxfile': '_libs/hashing'},
     '_libs.hashtable': {
         'pyxfile': '_libs/hashtable',
-        'pxdfiles': ['_libs/hashtable', '_libs/missing', '_libs/khash'],
         'depends': (['pandas/_libs/src/klib/khash_python.h'] +
                     _pxi_dep['hashtable'])},
     '_libs.index': {
         'pyxfile': '_libs/index',
-        'pxdfiles': ['_libs/src/util', '_libs/hashtable'],
         'depends': _pxi_dep['index'],
         'sources': np_datetime_sources},
     '_libs.indexing': {
@@ -490,21 +533,15 @@ def pxd(name):
         'pyxfile': '_libs/internals'},
     '_libs.interval': {
         'pyxfile': '_libs/interval',
-        'pxdfiles': ['_libs/hashtable'],
         'depends': _pxi_dep['interval']},
     '_libs.join': {
         'pyxfile': '_libs/join',
-        'pxdfiles': ['_libs/src/util', '_libs/hashtable'],
         'depends': _pxi_dep['join']},
     '_libs.lib': {
         'pyxfile': '_libs/lib',
-        'pxdfiles': ['_libs/src/util',
-                     '_libs/missing',
-                     '_libs/tslibs/conversion'],
         'depends': lib_depends + tseries_depends},
     '_libs.missing': {
         'pyxfile': '_libs/missing',
-        'pxdfiles': ['_libs/src/util'],
         'depends': tseries_depends},
     '_libs.parsers': {
         'pyxfile': '_libs/parsers',
@@ -514,12 +551,9 @@ def pxd(name):
         'sources': ['pandas/_libs/src/parser/tokenizer.c',
                     'pandas/_libs/src/parser/io.c']},
     '_libs.reduction': {
-        'pyxfile': '_libs/reduction',
-        'pxdfiles': ['_libs/src/util']},
+        'pyxfile': '_libs/reduction'},
     '_libs.ops': {
-        'pyxfile': '_libs/ops',
-        'pxdfiles': ['_libs/src/util',
-                     '_libs/missing']},
+        'pyxfile': '_libs/ops'},
     '_libs.properties': {
         'pyxfile': '_libs/properties',
         'include': []},
@@ -534,113 +568,66 @@ def pxd(name):
         'depends': _pxi_dep['sparse']},
     '_libs.tslib': {
         'pyxfile': '_libs/tslib',
-        'pxdfiles': ['_libs/src/util',
-                     '_libs/tslibs/conversion',
-                     '_libs/tslibs/timedeltas',
-                     '_libs/tslibs/timestamps',
-                     '_libs/tslibs/timezones',
-                     '_libs/tslibs/nattype',
-                     '_libs/tslibs/offsets'],
         'depends': tseries_depends,
         'sources': np_datetime_sources},
     '_libs.tslibs.ccalendar': {
         'pyxfile': '_libs/tslibs/ccalendar'},
     '_libs.tslibs.conversion': {
         'pyxfile': '_libs/tslibs/conversion',
-        'pxdfiles': ['_libs/src/util',
-                     '_libs/tslibs/nattype',
-                     '_libs/tslibs/timezones',
-                     '_libs/tslibs/timedeltas'],
         'depends': tseries_depends,
         'sources': np_datetime_sources},
     '_libs.tslibs.fields': {
         'pyxfile': '_libs/tslibs/fields',
-        'pxdfiles': ['_libs/tslibs/ccalendar',
-                     '_libs/tslibs/nattype'],
         'depends': tseries_depends,
         'sources': np_datetime_sources},
     '_libs.tslibs.frequencies': {
-        'pyxfile': '_libs/tslibs/frequencies',
-        'pxdfiles': ['_libs/src/util']},
+        'pyxfile': '_libs/tslibs/frequencies'},
     '_libs.tslibs.nattype': {
-        'pyxfile': '_libs/tslibs/nattype',
-        'pxdfiles': ['_libs/src/util']},
+        'pyxfile': '_libs/tslibs/nattype'},
     '_libs.tslibs.np_datetime': {
         'pyxfile': '_libs/tslibs/np_datetime',
         'depends': np_datetime_headers,
         'sources': np_datetime_sources},
     '_libs.tslibs.offsets': {
         'pyxfile': '_libs/tslibs/offsets',
-        'pxdfiles': ['_libs/src/util',
-                     '_libs/tslibs/ccalendar',
-                     '_libs/tslibs/conversion',
-                     '_libs/tslibs/frequencies',
-                     '_libs/tslibs/nattype'],
         'depends': tseries_depends,
         'sources': np_datetime_sources},
     '_libs.tslibs.parsing': {
-        'pyxfile': '_libs/tslibs/parsing',
-        'pxdfiles': ['_libs/src/util']},
+        'pyxfile': '_libs/tslibs/parsing'},
     '_libs.tslibs.period': {
         'pyxfile': '_libs/tslibs/period',
-        'pxdfiles': ['_libs/src/util',
-                     '_libs/tslibs/ccalendar',
-                     '_libs/tslibs/timedeltas',
-                     '_libs/tslibs/timezones',
-                     '_libs/tslibs/nattype',
-                     '_libs/tslibs/offsets'],
         'depends': tseries_depends + ['pandas/_libs/src/period_helper.h'],
         'sources': np_datetime_sources + ['pandas/_libs/src/period_helper.c']},
     '_libs.tslibs.resolution': {
         'pyxfile': '_libs/tslibs/resolution',
-        'pxdfiles': ['_libs/src/util',
-                     '_libs/khash',
-                     '_libs/tslibs/ccalendar',
-                     '_libs/tslibs/frequencies',
-                     '_libs/tslibs/timezones'],
         'depends': tseries_depends,
         'sources': np_datetime_sources},
     '_libs.tslibs.strptime': {
         'pyxfile': '_libs/tslibs/strptime',
-        'pxdfiles': ['_libs/src/util',
-                     '_libs/tslibs/nattype'],
         'depends': tseries_depends,
         'sources': np_datetime_sources},
     '_libs.tslibs.timedeltas': {
         'pyxfile': '_libs/tslibs/timedeltas',
-        'pxdfiles': ['_libs/src/util',
-                     '_libs/tslibs/nattype',
-                     '_libs/tslibs/offsets'],
         'depends': np_datetime_headers,
         'sources': np_datetime_sources},
     '_libs.tslibs.timestamps': {
         'pyxfile': '_libs/tslibs/timestamps',
-        'pxdfiles': ['_libs/src/util',
-                     '_libs/tslibs/ccalendar',
-                     '_libs/tslibs/conversion',
-                     '_libs/tslibs/nattype',
-                     '_libs/tslibs/offsets',
-                     '_libs/tslibs/timedeltas',
-                     '_libs/tslibs/timezones'],
         'depends': tseries_depends,
         'sources': np_datetime_sources},
     '_libs.tslibs.timezones': {
-        'pyxfile': '_libs/tslibs/timezones',
-        'pxdfiles': ['_libs/src/util']},
+        'pyxfile': '_libs/tslibs/timezones'},
     '_libs.testing': {
         'pyxfile': '_libs/testing'},
     '_libs.window': {
         'pyxfile': '_libs/window',
-        'pxdfiles': ['_libs/skiplist', '_libs/src/util'],
         'language': 'c++',
         'suffix': '.cpp'},
     '_libs.writers': {
-        'pyxfile': '_libs/writers',
-        'pxdfiles': ['_libs/src/util']},
+        'pyxfile': '_libs/writers'},
     'io.sas._sas': {
         'pyxfile': 'io/sas/sas'},
     'io.msgpack._packer': {
-        'macros': endian_macro,
+        'macros': endian_macro + macros,
         'depends': ['pandas/_libs/src/msgpack/pack.h',
                     'pandas/_libs/src/msgpack/pack_template.h'],
         'include': ['pandas/_libs/src/msgpack'] + common_include,
@@ -652,7 +639,7 @@ def pxd(name):
         'depends': ['pandas/_libs/src/msgpack/unpack.h',
                     'pandas/_libs/src/msgpack/unpack_define.h',
                     'pandas/_libs/src/msgpack/unpack_template.h'],
-        'macros': endian_macro,
+        'macros': endian_macro + macros,
         'include': ['pandas/_libs/src/msgpack'] + common_include,
         'language': 'c++',
         'suffix': '.cpp',
@@ -668,10 +655,6 @@ def pxd(name):
 
     sources = [srcpath(data['pyxfile'], suffix=source_suffix, subdir='')]
 
-    pxds = [pxd(x) for x in data.get('pxdfiles', [])]
-    if suffix == '.pyx' and pxds:
-        sources.extend(pxds)
-
     sources.extend(data.get('sources', []))
 
     include = data.get('include', common_include)
@@ -681,7 +664,7 @@ def pxd(name):
                     depends=data.get('depends', []),
                     include_dirs=include,
                     language=data.get('language', 'c'),
-                    define_macros=data.get('macros', []),
+                    define_macros=data.get('macros', macros),
                     extra_compile_args=extra_compile_args)
 
     extensions.append(obj)
@@ -708,7 +691,8 @@ def pxd(name):
                                     'pandas/_libs/src/ujson/lib',
                                     'pandas/_libs/src/datetime'],
                       extra_compile_args=(['-D_GNU_SOURCE'] +
-                                          extra_compile_args))
+                                          extra_compile_args),
+                      define_macros=macros)
 
 
 extensions.append(ujson_ext)
@@ -718,7 +702,8 @@ def pxd(name):
 # extension for pseudo-safely moving bytes into mutable buffers
 _move_ext = Extension('pandas.util._move',
                       depends=[],
-                      sources=['pandas/util/move.c'])
+                      sources=['pandas/util/move.c'],
+                      define_macros=macros)
 extensions.append(_move_ext)
 
 # The build cache system does string matching below this point.
@@ -729,7 +714,7 @@ def pxd(name):
       version=versioneer.get_version(),
       packages=find_packages(include=['pandas', 'pandas.*']),
       package_data={'': ['templates/*', '_libs/*.dll']},
-      ext_modules=extensions,
+      ext_modules=maybe_cythonize(extensions, compiler_directives=directives),
       maintainer_email=EMAIL,
       description=DESCRIPTION,
       license=LICENSE,


Test patch: