Datetime tweaks to improve numpy 1.6/1.7 datetime64 compatibility
State: closed

I've tried to isolate pandas usage of the datetime64 from the aspects of the numpy
API which would be nice to evolve. Pandas wasn't directly using this API for
functionality or interoperability, but more as a consequence of borrowing some
of the functional datetime operations from the numpy codebase.

As a result of doing this, some datetime64-related bugs were exposed in numpy
master, and I've submitted pull requests to fix them.


Labels:

Comments:
This is great, thanks. I'll merge it in today or tomorrow. I figure that, unfortunately, I'll probably need to support 1.6.1 users for at least a year

Awesome, thanks. The pandas tests found a few bugs in numpy 1.7, and there are still a few failing pandas tests that need investigating. Is there a command to run all the tests at once? I tried test.sh, but it didn't work (also would be able to use it only on linux not win).

Just `nosetests pandas` from the command line should be the way to go. the .sh files are just conveniences for me (like test.sh runs nose with coverage)

This unfortunately doesn't work for me. The only way that has worked so far is directly running the test_*.py files.

```
 ~/Develop/pandas $ nosetests pandas
E
======================================================================
ERROR: Failure: ImportError (C extensions not built: if you installed already verify that you are not importing from the source directory)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/pymodules/python2.7/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/pymodules/python2.7/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/mwiebe/Develop/pandas/pandas/__init__.py", line 15, in <module>
    raise ImportError('C extensions not built: if you installed already '
ImportError: C extensions not built: if you installed already verify that you are not importing from the source directory

----------------------------------------------------------------------
Ran 1 test in 0.001s

FAILED (errors=1)
```

Oh have you built the C extensions in place?

`python setup.py build_ext --inplace`

No, I installed them into a virtual-env. I will try in-place

Still fails, looks like nosetests conflicts with the virtual-env somehow. Running the test_*.py directly still works.

```
 ~/Develop/pandas $ nosetests pandas
E
======================================================================
ERROR: Failure: ValueError (numpy.dtype has the wrong size, try recompiling)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/usr/lib/pymodules/python2.7/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/usr/lib/pymodules/python2.7/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/mwiebe/Develop/pandas/pandas/__init__.py", line 10, in <module>
    import pandas._tseries as lib
  File "numpy.pxd", line 149, in init pandas._tseries (pandas/src/tseries.c:147609)
ValueError: numpy.dtype has the wrong size, try recompiling
```

I don't use virtualenv myself. Running nosetests pandas outside of the development directory inside the virtualenv doesn't work? I am running with `python setup.py develop` pretty much all the time and no virtualenvs

there are failures because I'm working on something right now

```
11:22 ~/code/pandas  (master)$ cd ~
11:22 ~ $ nosetests pandas
................................................................................................................................................................................................................................................................................S.........................................................................................S............................................................................F....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................S.................................S...................................F.....................................................................................................................................................................................................................................................................................................................................................
======================================================================
FAIL: test_pivot_no_level_overlap (pandas.tools.tests.test_pivot.TestPivotTable)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/wesm/code/pandas/pandas/tools/tests/test_pivot.py", line 172, in test_pivot_no_level_overlap
    tm.assert_frame_equal(table, expected)
  File "/home/wesm/code/pandas/pandas/util/testing.py", line 146, in assert_frame_equal
    assert(col in right)
AssertionError

======================================================================
FAIL: test_unstack_no_empty_columns (pandas.tests.test_multilevel.TestMultiLevel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/wesm/code/pandas/pandas/tests/test_multilevel.py", line 621, in test_unstack_no_empty_columns
    assert_frame_equal(unstacked, expected)
  File "/home/wesm/code/pandas/pandas/util/testing.py", line 146, in assert_frame_equal
    assert(col in right)
AssertionError

----------------------------------------------------------------------
Ran 1816 tests in 104.256s

FAILED (SKIP=4, failures=2)
```

Here it suggests a special command is required:

http://stackoverflow.com/questions/864956/problems-using-nose-in-a-virtualenv

and I get this if I try it:

```
~ $ ~/VE/numpy-dev/bin/python /usr/bin/nosetests pandas
/home/mwiebe/VE/numpy-dev/local/lib/python2.7/site-packages/pandas-0.8.0.dev_53112e9-py2.7-linux-x86_64.egg/pandas/__init__.py:10: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  import pandas._tseries as lib
/home/mwiebe/VE/numpy-dev/local/lib/python2.7/site-packages/pandas-0.8.0.dev_53112e9-py2.7-linux-x86_64.egg/pandas/sparse/array.py:15: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility
  from pandas._sparse import BlockIndex, IntIndex

----------------------------------------------------------------------
Ran 0 tests in 0.001s

OK
```

I haven't got any suggestions for you. Add another bullet point why Python packaging is a ghetto

I rebased/merged this and reverted the `NaT` values so the test suite passes, for now 

For testing purposes, should I be using NumPy git master or a different branch? Will have to make sure the test suite passes cleanly on 1.6 and 1.7+

Git master is still the place to test, since 1.7 hasn't branched yet.