BUG: Raise TypeError when serializing 0d ndarray
State: closed

Previously pandas ujson Segfault. see #9576 

This is an initial fix. I'll try to make sens out of ujson and see if 0d array can be handle in a better way. However, At the moment raising an exception is still better than a Segfault.


Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Another approach would be to simply substitute a scalar value for 0d arrays. NumPy sometimes plays a little loose with the distinction between 0d arrays and scalars so this might be more usable.

@shoyer I agree with you and is what I described in #9576 and what I meant by "initial fix". I'll try to implement this solution in another PR. However the fix is not that trivial due to the nature of a 0d array (can contain anything even an ndarray) and the way ujson work.

I would still appreciate if this PR could be merged in the mid-time (once I fixed the python3 issue, of course). An exception is infinitely better than segfault. An exception can be catched log, notification can send and is easier to debug. In addition, segfault are security hazard.

Certainly, this is a vast improvement over a segfault! But it looks like your patch is currently failing on Python 3 -- see the logs from Travis-CI: https://travis-ci.org/pydata/pandas/builds/52924267

This will also need a release note in `0.16.0.txt`.

needs testing on windows 

you also realize that putting an ndarray inside series elements is not supported in any ways
and is completely inefficient 

@shoyer Yeah, I saw it. I'm looking at the best way around it.

@jreback Once I found a windows VM for testing, is there an easy way to build all the dependencies. I didn't use windows for years and in my memory installing a compiler with all necessary library was a nightmare.

The fact that I was putting array in a Series was a "bug" in my app, I didn't realize that function build with `scipy.interpolate.interp1d` are returning 0d arrays when "feed" with scalar.

The issue seems to be in python now http://bugs.python.org/issue7608. It's fixed in 2.7 but not 2.6. I'll use pre processor instead to support both python 2 and 3.

@jreback I found this, https://github.com/pydata/pandas/wiki/Building-On-Windows... still seems to be a nightmare. I'll try.

you can just do

`conda create -n pandas_dev pandas cython libpython` (you specify pandas to get all of the deps)
and you can easily build on windows in that env.

@jreback is there a good reason why we don't have windows CI testing with appveyor hooked up?

I did configure it but it is the slowest thing on earth 

I sm going to hook up binstar soon 

does 32/64 bit Linux,windows,Mac (it's not free but continuum will sponsor)

@jreback I tested on windows, pandas/io/tests/test_json/test_ujson.py are passing. However when trying all the tests I'm getting a segfault on the master branch. I'm not sure what I'm doing wrong.

@gpoulin did might need to clean first, then build again.

@jreback I was running the build on a fresh pull of master. I'm a complete noob on building on windows, I might be doing something wrong. Particularly, the `C:\Program Files\Microsoft SDKs\Windows\v7.0\Bin` folder doesn't seem to have `SetEnv.cmd` on my VM. I'm currently fighting against PowerShell.

I'm +10000...00 for a windows CI. Or maybe a pre configured windows VM. That way it would only require to checkout your branch and to launch a script that would build and launch the unit tests. However, I'm not sure it's possible with the windows license.

just install using the `conda` command above.

@jreback I already tried that. Now I start to be really confused. Here is the commands that I ran

``` bash
conda create -n pandas_test cython pandas libpython
activate pandas_test
git clone https://github.com/pydata/pandas.git pandas2
cd pandas2
python setup.py build_ext --inplace
nosetests -v
# segfault at pandas/io/tests/test_cparser.py:TestCParser.test_header_not_enough_lines
nosetests -v pandas\io\tests\test_cparser.py
# 22 tests ran with success (including pandas/io/tests/test_cparser.py:TestCParser.test_header_not_enough_lines)
```

Is it possible that my locale break stuff? Or is there something wrong in what I do?

can you rebase this when you have a chance

@gpoulin can you rebase this?

@jreback done! Sorry for the delay. I was pretty busy recently.

@gpoulin can you rebase this?

@gpoulin can you rebase

@gpoulin status on this?

merged via https://github.com/pydata/pandas/commit/a06816eb193762d58ffecadec2b08607cb955397

thanks!