Repository: pandas-dev/pandas

Issue:
Installation problem on FreeBSD 10.0 with Python 3.4.3
Hi,

I'm attempting to install pandas in a virtualenv on a FreeBSD system with Python 3.4.3. I can't see that I'm missing any obvious steps or dependancies. I get an error in a C module when importing pandas if I install pandas from pip, from a release tarball and with the repo (same error every time) that would imply something pretty major is missing. numpy, scipy and scikit-learn all install without any issues in the same venv. Example of the error:

``` bash
(venv)hostname% pip install --upgrade pandas
Collecting pandas
Requirement already up-to-date: pytz>=2011k in ./venv/lib/python3.4/site-packages (from pandas)
Requirement already up-to-date: numpy>=1.7.0 in ./venv/lib/python3.4/site-packages (from pandas)
Requirement already up-to-date: python-dateutil>=2 in ./venv/lib/python3.4/site-packages (from pandas)
Requirement already up-to-date: six>=1.5 in ./venv/lib/python3.4/site-packages (from python-dateutil>=2->pandas)
Installing collected packages: pandas
Successfully installed pandas-0.16.2

(venv)hostname% python
Python 3.4.3 (default, Jun 18 2015, 17:00:30)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
  File "/home/someuser/venv/lib/python3.4/site-packages/pandas/__init__.py", line 7, in <module>
    from pandas import hashtable, tslib, lib
ImportError: /home/someuser/venv/lib/python3.4/site-packages/pandas/hashtable.so: Undefined symbol "get_c_string"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/someuser/venv/lib/python3.4/site-packages/pandas/__init__.py", line 13, in <module>
    "extensions first.".format(module))
ImportError: C extension: /home/someuser/venv/lib/python3.4/site-packages/pandas/hashtable.so: Undefined symbol "get_c_string" not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.
>>>
```

When installing from a tarbal or the a checkout of the repo where setup.py is accessible and following the above suggestion (`python setup.py build_ext --inplace`) it compiles without issue it does not resolve the issue.

Some system information (print_versions.py taken from a repo checkout):

``` bash
(venv)hostname% python print_versions.py

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.3.final.0
python-bits: 64
OS: FreeBSD
OS-release: 10.1-RELEASE-p10
machine: amd64
processor: amd64
byteorder: little
LC_ALL: None
LANG: None

pandas: None
nose: 1.3.7
Cython: 0.22.1
numpy: 1.9.2
scipy: 0.15.1
statsmodels: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.4.2
pytz: 2015.4
bottleneck: 1.0.0
tables: None
numexpr: 2.4.3
matplotlib: 1.4.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
```

Any pointers would be appreciated, thanks. 

Installation problem on FreeBSD 10.0 with Python 3.4.3
Hi,

I'm attempting to install pandas in a virtualenv on a FreeBSD system with Python 3.4.3. I can't see that I'm missing any obvious steps or dependancies. I get an error in a C module when importing pandas if I install pandas from pip, from a release tarball and with the repo (same error every time) that would imply something pretty major is missing. numpy, scipy and scikit-learn all install without any issues in the same venv. Example of the error:

``` bash
(venv)hostname% pip install --upgrade pandas
Collecting pandas
Requirement already up-to-date: pytz>=2011k in ./venv/lib/python3.4/site-packages (from pandas)
Requirement already up-to-date: numpy>=1.7.0 in ./venv/lib/python3.4/site-packages (from pandas)
Requirement already up-to-date: python-dateutil>=2 in ./venv/lib/python3.4/site-packages (from pandas)
Requirement already up-to-date: six>=1.5 in ./venv/lib/python3.4/site-packages (from python-dateutil>=2->pandas)
Installing collected packages: pandas
Successfully installed pandas-0.16.2

(venv)hostname% python
Python 3.4.3 (default, Jun 18 2015, 17:00:30)
[GCC 4.2.1 Compatible FreeBSD Clang 3.4.1 (tags/RELEASE_34/dot1-final 208032)] on freebsd10
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
Traceback (most recent call last):
  File "/home/someuser/venv/lib/python3.4/site-packages/pandas/__init__.py", line 7, in <module>
    from pandas import hashtable, tslib, lib
ImportError: /home/someuser/venv/lib/python3.4/site-packages/pandas/hashtable.so: Undefined symbol "get_c_string"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/someuser/venv/lib/python3.4/site-packages/pandas/__init__.py", line 13, in <module>
    "extensions first.".format(module))
ImportError: C extension: /home/someuser/venv/lib/python3.4/site-packages/pandas/hashtable.so: Undefined symbol "get_c_string" not built. If you want to import pandas from the source directory, you may need to run 'python setup.py build_ext --inplace' to build the C extensions first.
>>>
```

When installing from a tarbal or the a checkout of the repo where setup.py is accessible and following the above suggestion (`python setup.py build_ext --inplace`) it compiles without issue it does not resolve the issue.

Some system information (print_versions.py taken from a repo checkout):

``` bash
(venv)hostname% python print_versions.py

INSTALLED VERSIONS
------------------
commit: None
python: 3.4.3.final.0
python-bits: 64
OS: FreeBSD
OS-release: 10.1-RELEASE-p10
machine: amd64
processor: amd64
byteorder: little
LC_ALL: None
LANG: None

pandas: None
nose: 1.3.7
Cython: 0.22.1
numpy: 1.9.2
scipy: 0.15.1
statsmodels: None
IPython: None
sphinx: None
patsy: None
dateutil: 2.4.2
pytz: 2015.4
bottleneck: 1.0.0
tables: None
numexpr: 2.4.3
matplotlib: 1.4.3
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
httplib2: None
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: 2.6.1 (dt dec pq3 ext lo64)
```

Any pointers would be appreciated, thanks. 



Discussion:
Are there errors related to Cython in the setup log if you run 

```
python setup.py install
```

No errors from Cython. Cython was installed from pip. I have also tried removing Cython and installing/compiling it manually from the latest 0.22.1 tarball. Cython installs perfectly without issues as far as I can see.

I removed and re-installed Pandas from the repo after manually installing Cython and this had no effect. I have also been able to reproduce this error on a clean FreeBSD 10.1 vm.

Sorry, I mean when you build install pandas.  Something like:

```
cd ~/pandas
python setup.py clean
python setup.py build
```

Does the log from the last step show anything strange?

There's a load of warnings and notes, but it does seem to build all the required files. Here's the full output:

https://gist.githubusercontent.com/meeb/160fd9b409491cb9546a/raw/4dbad70b8f359a3714e82dd999571dfc0f471f28/gistfile1.txt

@meeb these are all ok

Yeah thought as much. Any other info I can provide or is this actually just something wrong in the build process on this platform?

I think you might have a partial pip install. try

```
pip uninstall pandas
pip install pandas
```

I've tried installing this in multiple completely clean venvs on different FreeBSD 10.1 installs (on different physical machines) all with the same results.

It's reproducible for Python3.4.2 on every FreeBSD 10.1 install I've tried so far. Interestingly, I've just noticed that installing pandas in a Python2.7.9 venv works fine without any errors at all on the same machines, both building from repo and installing via pip. Log:

https://gist.github.com/meeb/e68a685ad9acf038b7ae

Hi!
I have python34 on FreeBSD 10.1, and face the same problem. 
After digging around the code, I figure out that the root cause is the behavior of `inline`.
On gcc4x, it will choose `gnu89` as the C model.

... But on FreeBSD 10, we are using clang3.4, which choose standard C99 model by default.
[Ref is here](http://clang.llvm.org/compatibility.html#inline)

So... how to fix it? I will compile it with following command:

``` shell
CFLAGS='-fgnu89-inline' python setup.py build_ext -f --inplace
```

Maybe the proper way is patching the `Makefile`. 

way to do this properly is to put in options based on the platform in `setup.py`. Though AFAIK this has never been an issue before; I would make sure that cython is up to date.

Hi @jreback !
Hmm... actually the problematic code do not come from cython. Those code from cython `inline` always carry `static`.
I will make some explanation.

E.g. consider `pandas/src/numpy_helper.h`

``` c
PANDAS_INLINE npy_int64                                                            
get_nat(void) {                                                                    
  return NPY_MIN_INT64;                                                            
}
```

Note that there is a macro called `PANDAS_INLINE`.

This macro is defined in `pandas/src/helper.h`.
By default, this macro will be expanded as `__inline__`, because `clang` will define `__GUNC__` as  well.

Those code are not generated by cython.

IMO, there are two solution for this inline problem. (To be honest, many project still have this problem on FreeBSD, e.g. our depend: `ujson`)
1. Simply using the CFLAGS `-fgnu89-inline`
2. Make sure our(also our depends) macro `PANDAS_INLINE` consider following situation:
- `__GNUC_GNU_INLINE__`  (this is the behavior of `--std=gnu89`) define as `__inline__`
- `__GNUC_STDC_INLINE__` (`--std=c99` or later) define as `static inline`

ok I see

well happy to take a patch that ifdefs things and add platform based CFLAGS is ok too (would add in setup.py)

btw tha pandas.json IS ujson (with a layer on top for pandas use)

Are there errors related to Cython in the setup log if you run 

```
python setup.py install
```

No errors from Cython. Cython was installed from pip. I have also tried removing Cython and installing/compiling it manually from the latest 0.22.1 tarball. Cython installs perfectly without issues as far as I can see.

I removed and re-installed Pandas from the repo after manually installing Cython and this had no effect. I have also been able to reproduce this error on a clean FreeBSD 10.1 vm.

Sorry, I mean when you build install pandas.  Something like:

```
cd ~/pandas
python setup.py clean
python setup.py build
```

Does the log from the last step show anything strange?

There's a load of warnings and notes, but it does seem to build all the required files. Here's the full output:

https://gist.githubusercontent.com/meeb/160fd9b409491cb9546a/raw/4dbad70b8f359a3714e82dd999571dfc0f471f28/gistfile1.txt

@meeb these are all ok

Yeah thought as much. Any other info I can provide or is this actually just something wrong in the build process on this platform?

I think you might have a partial pip install. try

```
pip uninstall pandas
pip install pandas
```

I've tried installing this in multiple completely clean venvs on different FreeBSD 10.1 installs (on different physical machines) all with the same results.

It's reproducible for Python3.4.2 on every FreeBSD 10.1 install I've tried so far. Interestingly, I've just noticed that installing pandas in a Python2.7.9 venv works fine without any errors at all on the same machines, both building from repo and installing via pip. Log:

https://gist.github.com/meeb/e68a685ad9acf038b7ae

Hi!
I have python34 on FreeBSD 10.1, and face the same problem. 
After digging around the code, I figure out that the root cause is the behavior of `inline`.
On gcc4x, it will choose `gnu89` as the C model.

... But on FreeBSD 10, we are using clang3.4, which choose standard C99 model by default.
[Ref is here](http://clang.llvm.org/compatibility.html#inline)

So... how to fix it? I will compile it with following command:

``` shell
CFLAGS='-fgnu89-inline' python setup.py build_ext -f --inplace
```

Maybe the proper way is patching the `Makefile`. 

way to do this properly is to put in options based on the platform in `setup.py`. Though AFAIK this has never been an issue before; I would make sure that cython is up to date.

Hi @jreback !
Hmm... actually the problematic code do not come from cython. Those code from cython `inline` always carry `static`.
I will make some explanation.

E.g. consider `pandas/src/numpy_helper.h`

``` c
PANDAS_INLINE npy_int64                                                            
get_nat(void) {                                                                    
  return NPY_MIN_INT64;                                                            
}
```

Note that there is a macro called `PANDAS_INLINE`.

This macro is defined in `pandas/src/helper.h`.
By default, this macro will be expanded as `__inline__`, because `clang` will define `__GUNC__` as  well.

Those code are not generated by cython.

IMO, there are two solution for this inline problem. (To be honest, many project still have this problem on FreeBSD, e.g. our depend: `ujson`)
1. Simply using the CFLAGS `-fgnu89-inline`
2. Make sure our(also our depends) macro `PANDAS_INLINE` consider following situation:
- `__GNUC_GNU_INLINE__`  (this is the behavior of `--std=gnu89`) define as `__inline__`
- `__GNUC_STDC_INLINE__` (`--std=c99` or later) define as `static inline`

ok I see

well happy to take a patch that ifdefs things and add platform based CFLAGS is ok too (would add in setup.py)

btw tha pandas.json IS ujson (with a layer on top for pandas use)


Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -11,6 +11,7 @@
 import shutil
 import warnings
 import re
+import platform
 from distutils.version import LooseVersion
 
 # versioning
@@ -289,7 +290,10 @@ def run(self):
 
 
 class CheckingBuildExt(build_ext):
-    """Subclass build_ext to get clearer report if Cython is necessary."""
+    """
+    Subclass build_ext to get clearer report if Cython is necessary.
+    Also, add some platform based compiler flags.
+    """
 
     def check_cython_extensions(self, extensions):
         for ext in extensions:
@@ -302,8 +306,27 @@ def check_cython_extensions(self, extensions):
 
     def build_extensions(self):
         self.check_cython_extensions(self.extensions)
+        self.add_gnu_inline_flag(self.extensions)
         build_ext.build_extensions(self)
 
+    def add_gnu_inline_flag(self, extensions):
+        '''
+        Add CFLAGS `-fgnu89-inline` for clang on FreeBSD 10+
+        '''
+        if not platform.system() == 'FreeBSD':
+            return
+
+        try:
+            bsd_release = float(platform.release().split('-')[0])
+        except ValueError:  # unknow freebsd version
+            return
+
+        if bsd_release < 10:  # 9 or earlier still using gcc42
+            return
+
+        for ext in extensions:
+            ext.extra_compile_args += ['-fgnu89-inline']
+
 
 class CythonCommand(build_ext):
     """Custom distutils command subclassed from Cython.Distutils.build_ext


Test patch: