BLD: windows builds failing in pandas.json (#4764)
State: closed


Labels:

Comments:
now compiles ok, but (and last test causes it to crash python in both cases)

```
 H:\jreback\pandas> C:\python27\Scripts\nosetests.exe .\pandas\io\tests\test_json -v
test_axis_dates (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_convert_dates (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_date_format_frame (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_date_format_series (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_date_unit (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_doc_example (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_frame_double_encoded_labels (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_frame_from_json_bad_data (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ok
test_frame_from_json_nones (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_frame_from_json_precise_float (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_frame_from_json_to_json (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_frame_non_unique_columns (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_frame_non_unique_index (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_frame_to_json_except (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_misc_example (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ok
test_path (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_reconstruction_index (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_round_trip_exception_ (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ok
test_series_from_json_precise_float (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_series_from_json_to_json (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_series_non_unique_index (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_series_to_json_except (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_typ (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ERROR
test_url (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... SKIP
test_v12_compat (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... Exception OverflowError: 'long too big to convert' in 'pandas.tslib.cast_from_unit' ignored
Exception OverflowError: 'long too big to convert' in 'pandas.tslib.cast_from_unit' ignored
Exception OverflowError: 'long too big to convert' in 'pandas.tslib.cast_from_unit' ignored
Exception OverflowError: 'long too big to convert' in 'pandas.tslib.cast_from_unit' ignored
Exception OverflowError: 'long too big to convert' in 'pandas.tslib.cast_from_unit' ignored
Exception OverflowError: 'long too big to convert' in 'pandas.tslib.cast_from_unit' ignored
Exception OverflowError: 'long too big to convert' in 'pandas.tslib.cast_from_unit' ignored
Exception OverflowError: 'long too big to convert' in 'pandas.tslib.cast_from_unit' ignored
FAIL
test_weird_nested_json (pandas.io.tests.test_json.test_pandas.TestPandasContainer) ... ok
testArrayNumpyExcept (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testArrayNumpyLabelled (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testArrays (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testBool (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testBoolArray (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testFloat (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testFloatArray (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testFloatMax (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testInt (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testIntArray (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testIntMax (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testDataFrame (pandas.io.tests.test_json.test_ujson.PandasJSONTests) ... ok
testDataFrameNested (pandas.io.tests.test_json.test_ujson.PandasJSONTests) ... ok
testDataFrameNumpy (pandas.io.tests.test_json.test_ujson.PandasJSONTests) ... ok
testDataFrameNumpyLabelled (pandas.io.tests.test_json.test_ujson.PandasJSONTests) ... ok
testIndex (pandas.io.tests.test_json.test_ujson.PandasJSONTests) ... ok
testSeries (pandas.io.tests.test_json.test_ujson.PandasJSONTests) ... PS H:\jreback\pandas>
```

```
PS H:\jreback\pandas> C:\python27\Scripts\nosetests.exe .\pandas\io\tests\test_json\test_ujson.py -v
testArrayNumpyExcept (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testArrayNumpyLabelled (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testArrays (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testBool (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testBoolArray (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testFloat (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testFloatArray (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testFloatMax (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testInt (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testIntArray (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testIntMax (pandas.io.tests.test_json.test_ujson.NumpyJSONTests) ... ok
testDataFrame (pandas.io.tests.test_json.test_ujson.PandasJSONTests) ... ok
testDataFrameNested (pandas.io.tests.test_json.test_ujson.PandasJSONTests) ... ok
testDataFrameNumpy (pandas.io.tests.test_json.test_ujson.PandasJSONTests) ... ok
testDataFrameNumpyLabelled (pandas.io.tests.test_json.test_ujson.PandasJSONTests) ... ok
testIndex (pandas.io.tests.test_json.test_ujson.PandasJSONTests) ... ok
testSeries (pandas.io.tests.test_json.test_ujson.PandasJSONTests) ... PS H:\jreback\pandas>
```

You might be able to create a virtual windows box with vagrant (maybe just do 32-bit as its easier to compile).

http://kamalim.github.io/blogs/how-to-create-you-own-vagrant-base-boxes/

This works really well on linux type boxes as you can then get the exact configuration for say a travis build (e.g. I just debugged a python 3.2 with numpy 1.6.1 problem!)

Ick, ok I'll try to get my hands on a windows setup and take a closer look. Thanks!

OK just tested on a windows 7, 32-bit setup and was unable to reproduce the failures / crashing. Worked for me with both mingw (with an additional build fix, see commit) and visual express 2010. 

```
C:\Users\kieran\Documents\GitHub\pandas>nosetests pandas\io\tests\test_json
.......................S................................................................................................
..................................
----------------------------------------------------------------------
Ran 154 tests in 38.860s

OK (SKIP=1)
```

In the VS2010 setup I did see a segmentation fault in the unrelated cparser tests but I suspect that's due to mixing MinGW compiled Python(x,y) / Cython with VSC compiled pandas. 

@jreback could you give more info on the test failures / errors, and the setup you are encountering them on, is it numpy 1.6? I'll see if I can reproduce on 64 bit...

numpy 1.7
I compile on 64-bit

FYI my laptop decided to give up the ghost this morning so this is going to delay me a bit more :-(

To get the windows build fixed right now, either the json datetime changes could be factored out or the two commits above could be applied. I assume the tests will still fail but at least things will build.

I'll try and get these test issues sorted once I get myself back up and running.

@jreback FWIW (and as a note for others who go through windows pain) I tested a 32-bit install on Win7 64-bit box, with enthought canopy (32-bit) as a base.

My first attempt was with VS2010 (I couldn't find 2008). Same deal as before, JSON tests pass for me but the cparser tests crash out. Proceeding with the assumption that this is because of mixing compilers (VS2008 compiled Python, numpy etc and VS2010 compiled pandas+cython) I set out to get my hands on VS2008.

Found VS2008 thanks to [stackoverflow](http://stackoverflow.com/questions/15318560/visual-c-2008-express-download-link-dead). When compiled all tests run ok, no python crashing, hurrah! I did see a few minor test failures though.

Out of interest what is your setup on windows: enthought, compiler etc? I'm still a bit suspicious of the tests failing so spectacularly.

Test failures for ref (are these known windows issues?):

```
======================================================================
FAIL: test_round_trip_frame (pandas.io.tests.test_clipboard.TestClipboard)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\io\tests\test_clipboard.py", line 50, in test
_round_trip_frame
    self.check_round_trip_frame(dt)
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\io\tests\test_clipboard.py", line 46, in chec
k_round_trip_frame
    tm.assert_frame_equal(data, result)
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 295, in assert_frame_e
qual
    check_less_precise=check_less_precise)
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 244, in assert_series_
equal
    assert_attr_equal('dtype', left, right)
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 161, in assert_attr_eq
ual
    assert_equal(left_attr,right_attr,"attr is not equal [{0}]" .format(attr))
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 146, in assert_equal
    assert a == b, "%s: %r != %r" % (msg.format(a,b), a, b)
AssertionError: attr is not equal [dtype]: dtype('int32') != dtype('int64')

======================================================================
FAIL: test_constructor_generator (pandas.tests.test_frame.TestDataFrame)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\tests\test_frame.py", line 2727, in test_cons
tructor_generator
    assert_frame_equal(result, expected)
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 295, in assert_frame_e
qual
    check_less_precise=check_less_precise)
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 244, in assert_series_
equal
    assert_attr_equal('dtype', left, right)
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 161, in assert_attr_eq
ual
    assert_equal(left_attr,right_attr,"attr is not equal [{0}]" .format(attr))
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 146, in assert_equal
    assert a == b, "%s: %r != %r" % (msg.format(a,b), a, b)
AssertionError: attr is not equal [dtype]: dtype('int64') != dtype('int32')

======================================================================
FAIL: test_plot_xy (pandas.tests.test_graphics.TestDataFramePlots)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\tests\test_graphics.py", line 454, in test_pl
ot_xy
    == np.array((16., 8.))).all())
AssertionError: False is not true

======================================================================
FAIL: test_unsorted_index (pandas.tests.test_graphics.TestDataFramePlots)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\tests\test_graphics.py", line 500, in test_un
sorted_index
    tm.assert_series_equal(rs, df.y)
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 244, in assert_series_
equal
    assert_attr_equal('dtype', left, right)
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 161, in assert_attr_eq
ual
    assert_equal(left_attr,right_attr,"attr is not equal [{0}]" .format(attr))
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 146, in assert_equal
    assert a == b, "%s: %r != %r" % (msg.format(a,b), a, b)
AssertionError: attr is not equal [dtype]: dtype('int64') != dtype('int32')

======================================================================
FAIL: test_plot (pandas.tests.test_graphics.TestSeriesPlots)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\tests\test_graphics.py", line 80, in test_plo
t
    == np.array((16., 8.))).all())
AssertionError: False is not true

======================================================================
FAIL: test_partial_setting (pandas.tests.test_indexing.TestIndexing)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\tests\test_indexing.py", line 1380, in test_p
artial_setting
    assert_frame_equal(df,expected)
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 295, in assert_frame_e
qual
    check_less_precise=check_less_precise)
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 244, in assert_series_
equal
    assert_attr_equal('dtype', left, right)
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 161, in assert_attr_eq
ual
    assert_equal(left_attr,right_attr,"attr is not equal [{0}]" .format(attr))
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 146, in assert_equal
    assert a == b, "%s: %r != %r" % (msg.format(a,b), a, b)
AssertionError: attr is not equal [dtype]: dtype('int32') != dtype('int64')

======================================================================
FAIL: test_constructor_iterator (pandas.tests.test_series.TestSeries)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\tests\test_series.py", line 360, in test_cons
tructor_iterator
    assert_series_equal(result, expected)
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 244, in assert_series_
equal
    assert_attr_equal('dtype', left, right)
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 161, in assert_attr_eq
ual
    assert_equal(left_attr,right_attr,"attr is not equal [{0}]" .format(attr))
  File "C:\Users\kieran\Documents\GitHub\pandas\pandas\util\testing.py", line 146, in assert_equal
    assert a == b, "%s: %r != %r" % (msg.format(a,b), a, b)
AssertionError: attr is not equal [dtype]: dtype('int32') != dtype('int64')

----------------------------------------------------------------------
Ran 3705 tests in 1360.689s

FAILED (SKIP=168, failures=7)
```

I have vs2010 so some of the parser tests crash for me as well
so not sure of these are real failures or related to that
I guess will have to wait for @wesm to get daily builds going to see

some of those I fixed today
I'll fix up the non-plotting ones 

Can you merge in the mingw build fix aswell? Then this PR can be closed.

sure
I think u need to rebase 
as the other fix is in here?

whoops yeah will do, sry

Should be good to go now

can u repost the test failures that u see now after rebuilding? (I did fix some of them earlier today)
but only some showed up on 32-bit Linux and not at all on travis

Thanks @jreback, submitted issue here #4866

perfect

I know what most of these are except for the plotting ones
can use go into nosetests debug and see if u can tell why they r failing?