Repository: pandas-dev/pandas

Issue:
build failure with Xcode 10 - libstdc++ not supported anymore
#### Code Sample, a copy-pastable example if possible

```python
(pandas-dev) bilbo:pandas-rb robertbuckley$ python setup.py build_ext --inplace -j 4

running build_ext
building 'pandas._libs.window' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include -arch x86_64 -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include -arch x86_64 -Ipandas/_libs -I./pandas/_libs -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/Users/robertbuckley/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/numpy/core/include -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include/python3.6m -c pandas/_libs/window.cpp -o build/temp.macosx-10.7-x86_64-3.6/pandas/_libs/window.o -Wno-unused-function
warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
pandas/_libs/window.cpp:655:10: fatal error: 'ios' file not found
#include "ios"
         ^~~~~
1 warning and 1 error generated.
error: command 'gcc' failed with exit status 1
```
#### Problem description

I setup a dev environment on Mac OS 10.13 according to https://pandas.pydata.org/pandas-docs/stable/contributing.html#creating-a-development-environment and all was well

Upgraded to 10.14 (mojave) and Xcode 10, and now i get the above failure.

[Xcode 10 release notes](https://developer.apple.com/documentation/xcode_release_notes/xcode_10_release_notes?language=objc) includes the following deprecation notice

> 
Building with libstdc++ was deprecated with Xcode 8 and is not supported in Xcode 10 when targeting iOS. C++ projects must now migrate to libc++ and are recommended to set a deployment target of macOS 10.9 or later, or iOS 7 or later. Besides changing the C++ Standard Library build setting, developers should audit hard-coded linker flags and target dependencies to remove references to libstdc++ (including -lstdc++, -lstdc++.6.0.9, libstdc++.6.0.9.tbd, and libstdc++.6.0.9.dylib). Project dependencies such as static archives that were built against libstdc++ will also need to be rebuilt against libc++. (40885260)

Some workaround suggestions are at https://stackoverflow.com/questions/52425766/stdlibc-headers-not-found-error-on-xcode-10 and the answer its linked to. 

I dont have access to Xcode 9 anymore, so i can't try the workaround of copying the old libs from there


Discussion:
Another contributor fixed that with `xcode select --install`. Does that work for  you?
Couldn't reproduce this on Mojave with Xcode 10.0 - let us know if the above suggestion doesn't fix it
@TomAugspurger i already did that, to fix another issue. Now when I enter `xcode-select --install` i get in response:
`xcode-select: error: command line tools are already installed, use "Software Update" to install updates`

I've updated to xcode 10.1, the issue is still there. Tomorrow I'll try to install Xcode  9 again.

Might there be a case for moving to libc++ anyway ?
hi, 

i managed to get it building by deleting Xcode, and dowloading + installing Xcode 9.4.1 Command Line Tools from the Apple Developer website. 

There is still something funny going on, because I then installed Xcode 10 Command Line Tools (after first renaming the old ones at `/Library/Developer/CommandLineTools`). While I could use `sudo xcode-select -s <path>` to switch between the 2 versions, and `clang --version` confirmed it was switching the compiler version at least, I am suspicious that the libstdc++ libs and headers from 9.4.1 have been left somewhere thats not affected by the switch.  So there may be some scenarios where this breaks (eg fresh install of Mojave).

I'll continue to investigate, if thats OK
Not sure if it's the same issue or if this is a good fix, but I also couldn't build after updating to mojave (I got my error at a different point though). I fixed it by adding the argument `extra_link_args=["-stdlib=libc++", "-mmacosx-version-min=10.9"]` to the initialization of `Extension` object `obj` here.

https://github.com/pandas-dev/pandas/blob/d78bd7a65f87022bae5021d36c8656c0ea5a7b76/setup.py#L672-L680

Credit for idea [here](https://stackoverflow.com/questions/52460913/compiling-cython-with-xcode10)

Here's my error log before performing the above step:

```
g++ -bundle -undefined dynamic_lookup -L/Users/admin/anaconda3/envs/pandas-dev/lib -arch x86_64 -L/Users/admin/anaconda3/envs/pandas-dev/lib -arch x86_64 -L/usr/local/opt/libarchive/lib -I/usr/local/opt/libarchive/include -arch x86_64 build/temp.macosx-10.7-x86_64-3.6/pandas/_libs/window.o -o /Users/admin/Documents/Projects/pandas-jz/pandas/_libs/window.cpython-36m-darwin.so
clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
ld: library not found for -lstdc++
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'g++' failed with exit status 1
```
https://github.com/cython/cython/issues/2694 also suggests that upgrading the command line tools was sufficient.

@JustinZhengBC, did you get the same error (without the extra link args) in a clean environment? Or was this in the same environment (with previously compiled files) from before you upgraded your command line tools?
Yes, I get the error after running `python setup.py clean` as well as in a fresh download. I have reinstalled the dependencies from `ci/requirements_dev.txt`, and I am using the latest version of XCode and command line tools as suggested in that thread.
Thanks, that's strange.

I'm just hesitant to add extra arguments to our setup script when we (I) don't really understand the root problem and why adding those arguments fixes it.
@JustinZhengBC any chance you installed some of the components via homebrew as well? Shot in the dark but I'm wondering if there could be some type of conflict there between that installation alongside the standard Xcode-select install
@WillAyd yes, I use homebrew and some c++ libraries do show up in `brew list`
Wonder if that could be it. There's also a page for homebrew dedicated to the topic:

https://docs.brew.sh/C++-Standard-Libraries

Do you have a way of either removing or upgrading those installed libraries to see if that resolves?
I tried renaming the Cellar folder and running build, but it still failed.

However, I did manage to get it building without extra arguments after running `/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg` as described [here](https://blog.driftingruby.com/updated-to-mojave/). My build is working as intended now.
Hmm that's interesting. I wonder if that is part of the "Additional Components" that you get prompted to install when opening Xcode - any chance you recall having installed that or not?
I just ran through the installer again, no such prompt appeared. I've opened XCode to look at the settings but I haven't opened any projects in it since updating, so I might've missed the prompt there
I believe if you uninstall and reinstall Xcode it should prompt you for additional components the first time you open it (no project required).

Something does seem off here; if you could help pinpoint would be great. May need some kind of doc updates
A fresh install of the latest Xcode and command line tools can build with no errors and no prompts. However, while I can get old versions of Xcode from the Apple developer's website, those seem to be standalones so I can't replicate the update process.


i managed to repro the issue on my system with the following sequence of steps:
1. uninstalling Xcode.app and command line tools (rm -rf /Library/Developer)
2. removing /usr/include. This needs ’System Integrity Protection’ to be disabled, see (https://superuser.com/questions/995360/missing-usr-include-in-os-x-el-capitan#1009364)
3. install ‘Command Line Tools (mac OS 10.14) for Xcode 10.1 from https://developer.apple.com/download/more/ (needs a [free] apple developer login)

After this I could reproduce the original failure mode i reported, and could get the extensions compiling OK using a variant of the fix proposed by @JustinZhengBC:
```
obj = Extension('pandas.{name}'.format(name=name),
                sources=sources,
                depends=data.get('depends', []),
                include_dirs=include,
                language=data.get('language', 'c'),
                define_macros=data.get('macros', macros),
                extra_compile_args=extra_compile_args + ['-stdlib=libc++'],
                extra_link_args=['-stdlib=libc++']
```

Probably these changes should be wrapped with a check for Mac OS >= 10.9 to avoid breaking other platforms. 

If i added only the part in extra_link_args, I see the compiler error i originally reported:

```
building 'pandas._libs.window' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include -arch x86_64 -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include -arch x86_64 -Ipandas/_libs -I./pandas/_libs -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/Users/robertbuckley/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/numpy/core/include -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include/python3.6m -c pandas/_libs/window.cpp -o build/temp.macosx-10.7-x86_64-3.6/pandas/_libs/window.o
warning: include path for stdlibc++ headers not found; pass '-std=libc++' on the command line to use the libc++ standard library instead
      [-Wstdlibcxx-not-found]
pandas/_libs/window.cpp:655:10: fatal error: 'ios' file not found
#include "ios"
         ^~~~~
1 warning and 1 error generated.
error: command 'gcc' failed with exit status 1
```

If i added only the extra_compile_args part, the compile succeeds but the linker would throw an error:

```
building 'pandas._libs.window' extension
gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include -arch x86_64 -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include -arch x86_64 -Ipandas/_libs -I./pandas/_libs -Ipandas/_libs/src/klib -Ipandas/_libs/src -I/Users/robertbuckley/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/numpy/core/include -I/Users/robertbuckley/anaconda3/envs/pandas-dev/include/python3.6m -c pandas/_libs/window.cpp -o build/temp.macosx-10.7-x86_64-3.6/pandas/_libs/window.o -Wno-unused-function -stdlib=libc++
In file included from pandas/_libs/window.cpp:663:
In file included from /Users/robertbuckley/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4:
In file included from /Users/robertbuckley/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18:
In file included from /Users/robertbuckley/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1823:
/Users/robertbuckley/anaconda3/envs/pandas-dev/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: 
      "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-W#warnings]
#warning "Using deprecated NumPy API, disable it by " \
 ^
pandas/_libs/window.cpp:45351:27: warning: comparison of integers of different signs: 'size_t' (aka 'unsigned long') and 'Py_ssize_t' (aka 'long')
      [-Wsign-compare]
  __pyx_t_2 = ((__pyx_t_1 != __pyx_v_N) != 0);
                ~~~~~~~~~ ^  ~~~~~~~~~
2 warnings generated.
g++ -bundle -undefined dynamic_lookup -L/Users/robertbuckley/anaconda3/envs/pandas-dev/lib -arch x86_64 -L/Users/robertbuckley/anaconda3/envs/pandas-dev/lib -arch x86_64 -arch x86_64 build/temp.macosx-10.7-x86_64-3.6/pandas/_libs/window.o -o /Users/robertbuckley/python-scripts/pandas-rb/pandas/_libs/window.cpython-36m-darwin.so
clang: warning: libstdc++ is deprecated; move to libc++ with a minimum deployment target of OS X 10.9 [-Wdeprecated]
ld: library not found for -lstdc++
clang: error: linker command failed with exit code 1 (use -v to see invocation)
error: command 'g++' failed with exit status 1

```

My working theory is that the older version of Xcode command line tools adds the headers into /usr/include and the libstdc++ binaries into somewhere like /usr/lib, and that subsequent uninstalls of Xcode / Command Line Tools don’t remove them.


possibly useful liinks
---------------------

libc++ https://libcxx.llvm.org/
LLVM (clang) changes where the warning stdlibcxx-not-found was added https://reviews.llvm.org/D48297
extensive discussion on conda-forge https://github.com/conda-forge/conda-forge.github.io/issues/183
hi @WillAyd the behaviour seems to depend if clang is able to find stdlibc++ headers/libs. This depends on the xcode install history, hence the difficulty in reproducing this.

i suppose there are 2 options: (1) require the user to workaround this by installing stdlibc++ or (2) change mac os targets to build + link libc++. 

So far i am unable to get (1) working again on my system. I've tried:
- `xcode-select--install`, this responds with `xcode-select: error: command line tools are already installed, use "Software Update" to install updates`.
- installing `/Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg` as per @JustinZhengBC 

But after each step I get the original compile error I reported. 

for option 2 i've created a PR. I didn't meant to preempt any decision on taking this - its my first PR and i didnt realise that pushing a branch to my fork would update the GH issue. CI is running now with travis, azure (passed), circle ci, appveyor (passed)

please let me know how you'd like to proceed
Hmm is our only requirement for the C++ headers/libs stemming from this?

https://github.com/pandas-dev/pandas/pull/19549#issuecomment-364257051

Not saying it's worth changing here but curious if we previously looked at the performance difference in linking against the c++ implementation vs using `collections.deque` in Cython and if the latter wouldn't simplify linkage

@chris-b1 @jreback 

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -10,6 +10,8 @@
 from os.path import join as pjoin
 
 import pkg_resources
+import platform
+from distutils.sysconfig import get_config_var
 import sys
 import shutil
 from distutils.version import LooseVersion
@@ -24,6 +26,10 @@ def is_platform_windows():
     return sys.platform == 'win32' or sys.platform == 'cygwin'
 
 
+def is_platform_mac():
+    return sys.platform == 'darwin'
+
+
 min_numpy_ver = '1.12.0'
 setuptools_kwargs = {
     'install_requires': [
@@ -434,6 +440,19 @@ def get_tag(self):
     extra_compile_args = ['-Wno-unused-function']
 
 
+# For mac, ensure extensions are built for macos 10.9 when compiling on a
+# 10.9 system or above, overriding distuitls behaviour which is to target
+# the version that python was built for. This may be overridden by setting
+# MACOSX_DEPLOYMENT_TARGET before calling setup.py
+if is_platform_mac():
+    if 'MACOSX_DEPLOYMENT_TARGET' not in os.environ:
+        current_system = LooseVersion(platform.mac_ver()[0])
+        python_target = LooseVersion(
+            get_config_var('MACOSX_DEPLOYMENT_TARGET'))
+        if python_target < '10.9' and current_system >= '10.9':
+            os.environ['MACOSX_DEPLOYMENT_TARGET'] = '10.9'
+
+
 # enable coverage by building cython files by setting the environment variable
 # "PANDAS_CYTHON_COVERAGE" (with a Truthy value) or by running build_ext
 # with `--with-cython-coverage`enabled


Test patch: