BLD: include dll in package_data on Windows
State: closed

- [x] closes #21106
- [ ] tests added / passed
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Not sure how to test this, but I believe should remove the runtime dependency by static linking against the MSVC++ runtime.  @cgohlke any thoughts?

Labels:
name: Build
description: Library building on various platforms

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21321?src=pr&el=h1) Report
> Merging [#21321](https://codecov.io/gh/pandas-dev/pandas/pull/21321?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d79203af0552e73933e6f80f4284ac2697372eaa?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/21321/graphs/tree.svg?src=pr&token=eZ4WkYLtcO&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/21321?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #21321   +/-   ##
=======================================
  Coverage   91.89%   91.89%           
=======================================
  Files         153      153           
  Lines       49596    49596           
=======================================
  Hits        45576    45576           
  Misses       4020     4020
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.29% <ø> (ø)` | :arrow_up: |
| #single | `41.86% <ø> (ø)` | :arrow_up: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/21321?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/reshape/merge.py](https://codecov.io/gh/pandas-dev/pandas/pull/21321/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvcmVzaGFwZS9tZXJnZS5weQ==) | `94.25% <0%> (ø)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21321?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/21321?src=pr&el=footer). Last update [d79203a...7845eb2](https://codecov.io/gh/pandas-dev/pandas/pull/21321?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

If I read this correctly, this will not only static link the C++ runtime to one C++ extension, but static link the C multi-threaded runtime library to all Pandas extension modules. That is [not a good idea](https://bugs.python.org/issue25027) since Python can only load a total of ~127 of such DLLs. 

It's probably better to allow distributors to include the C++ runtime DLL,, in the wheels, e.g. 
```diff
diff --git a/setup.py b/setup.py
index 6febe674f..e161b1534 100755
--- a/setup.py
+++ b/setup.py
@@ -733,7 +733,7 @@ setup(name=DISTNAME,
       maintainer=AUTHOR,
       version=versioneer.get_version(),
       packages=find_packages(include=['pandas', 'pandas.*']),
-      package_data={'': ['data/*', 'templates/*'],
+      package_data={'': ['data/*', 'templates/*', '_libs/*.dll'],
                     'pandas.tests.io': ['data/legacy_hdf/*.h5',
                                         'data/legacy_pickle/*/*.pickle',
                                         'data/legacy_msgpack/*/*.msgpack',

```
Distributors could then copy (or not) the C++ runtime DLL to the `_libs` folder before building the wheel. 
@chris-b1 is this the fix mpl did?
No, mpl packages the whole DLL.  I thought static linking might be better in terms of limiting ending binary size, but wasn't aware of issue @cgohlke linked.

@cgohlke - wasn't clear to me looking at mpl setup, do you just manually copy DLL over into that directory?  I see the stuff adding it to `package_data` but couldn't figure out how the DLL gets there in the first place.

https://github.com/matplotlib/matplotlib/blob/6ec80eac66c4828f28540f76734cae811787d626/setupext.py#L1770


> wasn't clear to me looking at mpl setup, do you just manually copy DLL over into that directory? 

Yes, this way it is up to the distributor to copy any required DLL to the `_libs` folder before building the wheel. IMO it is not worth automating this in setup.py because there are many possible Visual Studio versions and use cases to consider. Canopy and Anaconda distributions won't need to package any C/C++ runtime DLL.
@TomAugspurger - is it correct that the Windows wheels you upload to PyPI are from @cgohlke?  If that's the case, @cgohlke does this work for you?
That's correct.

On Wed, Jun 6, 2018 at 12:43 PM, chris-b1 <notifications@github.com> wrote:

> @TomAugspurger <https://github.com/TomAugspurger> - is it correct that
> the Windows wheels you upload to PyPI are from @cgohlke
> <https://github.com/cgohlke>? If that's the case, @cgohlke
> <https://github.com/cgohlke> does this work for you?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/21321#issuecomment-395154446>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIiPF0-76XXhgbdfSUie9JWRh288Pks5t6BSzgaJpZM4UZ_5n>
> .
>

> does this work for you?

I'm sure it does for Python 3. The build script will copy `MSVCP140.DLL` and `CONCRT140.DLL` from the Visual Studio redist directory to `pandas\_libs\` before running setup.py, e.g.:
```
copy /Y /B "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\redist\x64\Microsoft.VC140.CRT\MSVCP140.DLL" pandas\_libs\
copy /Y /B "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\redist\x64\Microsoft.VC140.CRT\CONCRT140.DLL" pandas\_libs\
```

It's probably better not to copy `msvcp90.dll` for Python 2.7. IIRC the Python installer already installs the C++ runtime unless it's a per user installation. The .pyd file may or may not be missing a manifest to load `msvcp90.dll` next to it. I do not have any system for testing without the redistributable package installed. I am not aware of any other package that distributes `msvcp90.dll`, not matplotlib and not pyzmq.
does the MANIFEST.in need updating here?
@chris-b1 can you rebase. ready to go?
Just rebased, Should be good, I'm not familiar with MANIFEST.in, but seems to only involve source distributions?

https://docs.python.org/3/distutils/sourcedist.html#specifying-the-files-to-distribute
@chris-b1 ok that sounds good then.