BrandonFors/pandas-github-issues huggingface dataset:
BUG: bug in to_msgpack for timezone aware datetime index
State: closed

BUG: bug in to_msgpack for no freqstr datetime index (GH5506)

closes #5506


Labels:

Comments:
The following is now supported (similar to json)

```
In [1]: pd.read_msgpack(DataFrame(np.random.randn(10,2)).to_msgpack())
Out[1]: 
          0         1
0 -1.755286 -0.528638
1  1.033457  1.211977
2  0.084770  2.017154
3 -0.326114  0.041828
4  2.002210 -0.620182
5  1.035762 -0.485474
6 -0.354471 -1.429922
7 -0.826269  1.331761
8 -1.893332  0.590906
```

This works too!

string concatetation of msgpacks reconstructs into a list of them (IOW each msgpacked object is a complete string in an of itself)

```
In [5]: pd.read_msgpack(DataFrame(np.random.randn(10,2)).to_msgpack() + DataFrame(np.random.randn(10,2),columns=list('AB')).to_msgpack())
Out[5]: 
[          0         1
0  0.907361 -1.070519
1  0.224163  0.231305
2 -0.307924  1.151119
3 -0.986642 -1.197128
4 -1.985134 -0.995994
5  0.706065 -0.640924
6 -0.035256  0.327645
7  0.358768  0.366619
8 -0.212340 -2.026615
9  0.472154 -1.007920,
           A         B
0 -0.139536 -2.702822
1 -0.101613  1.716294
2 -2.268146 -0.725065
3 -0.275433 -0.696792
4  1.085239 -1.267965
5  1.243207 -2.666420
6 -0.552880  1.164095
7 -0.318267  0.586724
8 -1.285713  1.753806
9 -0.680881 -0.861096]
```
ENH: accept mutliple series for FRED DataReader
State: closed

closes #3413

Let me know if you want this for .13 or .14 and I'll fill in the release notes appropriately.


Labels:

Comments:
Travis build error. What are the magic commands that will retrigger a build? I think this is good for .13 if there are no objections.

EDIT: all good here.

git commit --amend -C HEAD
git push --force

On Sunday, November 24, 2013, Tom Augspurger wrote:

> Travis build error. What are the magic commands that will retrigger a
> build? I think this is good for .13 if there are no objections.
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5492#issuecomment-29164051
> .

## 

## 

Best,
Phillip Cloud

thanks @TomAugspurger
print_versions.py fixes
State: closed

#4760
#4901
1. Path resolution was broken, so running the script outside the pandas root directory didn't work.
2. The use of `subprocess` to invoke the script from the source tree relied on system python,
   ignoring the interpreter version actually used to invoke the scripts. Travis was unaffected since the system python there
   always matches the python version of the env.
3. Using subprocess meant Ctrl-C didn't interrupt the script (meh).

cc @cpcloud 


Labels:

Comments:
> 1. Path resolution was broken, so running the script outside the pandas root directory didn't work.

I tried this. It worked just fine on my end.

> Using subprocess meant Ctrl-C didn't interrupt the script (meh).

C'mon, _really_?

No. 2 makes sense.

``` python
python: can't open file 'pandas/util/print_versions.py': [Errno 2] No such file or directory
Traceback (most recent call last):
  File "./print_versions.py", line 16, in <module>
    show_versions()
  File "./print_versions.py", line 12, in show_versions
    return subprocess.check_call(['python', sv_path])
  File "/usr/lib/python2.7/subprocess.py", line 540, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['python', 'pandas/util/print_versions.py']' returned non-zero exit status 2
```

didn't work on my end.

` `
I know, that's what the _meh_ was for.
PERF: perf regression with mixed-type ops using numexpr (GH5481)
State: closed

closes #5481

BUG: non-unique ops not aligning correctly

these are bascially a trivial op in numpy, so numexpr is slightly slower (but the
the dtype inference issue is fixed). Essentially the recreation of an int64 ndarray had to check if its a datetime-like. In this case just passing in the dtype on the reconstructed series fixes it.

Also handles non-unique columns now (no tests before, and it would fail).

```
In [1]: df = pd.DataFrame({"A": np.arange(1000000), "B": np.arange(1000000, 0, -1), "C": np.random.randn(1000000)})

In [2]: pd.computation.expressions.set_use_numexpr(False)

In [3]: %timeit df*df
100 loops, best of 3: 11 ms per loop

In [4]: pd.computation.expressions.set_use_numexpr(True)

In [5]: %timeit df*df
100 loops, best of 3: 15.7 ms per loop

In [6]: df = df.astype(float)

In [7]: pd.computation.expressions.set_use_numexpr(False)

In [8]: %timeit df*df
100 loops, best of 3: 5.16 ms per loop

In [9]: pd.computation.expressions.set_use_numexpr(True)

In [10]: %timeit df*df
100 loops, best of 3: 5.37 ms per loop

```


Labels:

Comments:
@jtratner , cc @dsm054

pls give a try with this...should fix the perf issue

Works for me; in fact `numexpr` is somewhat faster--

In [4]: pd.computation.expressions.set_use_numexpr(False)

In [5]: %timeit df*df
10 loops, best of 3: 28.3 ms per loop

In [6]: pd.computation.expressions.set_use_numexpr(True)

In [7]: %timeit df*df
10 loops, best of 3: 26.9 ms per loop

Why is this a non-unique index issue?  Do you mean that we're special casing unique ops?

no the non unique issue is separate
but was convient  to fix

So the actual fix was just passing the dtype explicitly?

well had to create the series with an explicit dtype (as opposed to passing a dict of ndarrays)

Got it - makes sense.
TST: win32 fixes
State: closed

TST: test_indexing dtype comps, closes #5466

TST don't compare invalid indices in eval operations, closes #5467

TST: invalid assignment for np.timedelta64 closes #5465

TST: skip locale checks on windows cloes #5464

TST: dtype comp issue on windows in test_indexing (GH5466)

TST: skip test_pow on windows (GH5467)


Labels:

Comments:
so do you have any sense of _why_ these issues occur? Just 32 bit issues? specific to windows?

Just want to be able to keep in mind in the future.

@jtratner these are all because the tests on a 32-bit windows default with int32 for numpy arange (so the tests are wrong), e.g. in this case numpy creates these as 32-bit integers but they are really 64-bit in the implementation

Can we add a 32bit Travis build?  Would that help?

I think that travis was using 32-bit builds...now seem 64-bit....

it DOES help catch stuff (though this is really windows specific oddities)
BUG: bool(NaT) was True
State: closed

bool(NaT) evaluates to True. I am not sure if this is by design or if it is a bug, but I think it makes more sense to evaluate it to False.


Labels:

Comments:
`bool(nan)` is also `True`, and I think it should be consistent with that. Here's the python docs on what's considered False: http://docs.python.org/2/library/stdtypes.html#truth-value-testing

This type of testing is for compat with numpy and python as @TomAugspurger points out.
use `pd.notnull/pd.isnull` for reliable/consistent testing on all structures / values

```
In [1]: bool(np.nan)
Out[1]: True

In [2]: bool(np.datetime64('NaT'))
Out[2]: True

In [3]: bool(pd.NaT)
Out[3]: True
```

```
In [8]: pd.notnull(np.nan)
Out[8]: False

In [9]: pd.notnull(pd.NaT)
Out[9]: False

In [10]: pd.notnull(np.datetime64('NaT'))
Out[10]: False
```

Thanks for setting me right :)
I'll close the pull request.
DOC: small error in cookbook (doc build warning)
State: closed


Labels:

Comments:
All doc build warnings should now be gone again :-)

(at least, except of one, which I can't get rid off: `pandas\doc\source\_templates\autosummary\class.rst:: WARNING: document isn't included in any toctree`, but normally Sphinx should ignore the files in _template)

yeh...that last one is odd.....thanks!

Last one is necessary for redirects from nested (pandas.core.Series) to top
level (pandas.Series)

@jtratner What do you mean exactly? I think I don't understand it.

For a while, all the pandas docs were nested (I think) so the autosummary
for Series was under core rather than at the toplevel. The point of that
file is to make sure that previous links to the auto-generated docs for
pandas.core.Series still work, but they aren't added to the public TOC
PERF/ENH: Use xlsxwriter by default (and then fall back on openpyxl).
State: closed

Since xlsxwriter has a smaller install base (because it's newer),
reasonable to assume that if you have it installed, you want xlsxwriter.
Plus, it has better performance characteristics in general (esp. in
terms of memory usage)


Labels:

Comments:
Any objections to this? If you don't have xlsxwriter installed doesn't matter anyways. Going to merge soon...
ENH: Always do true division on Python 2.X
State: closed

Close #5356.

Note: this version actually forces `a / b` to always do truediv as well. It's
less complicated to maintain if we do it that way, but it's also possible to
have this `a / b` use div and `a.div(b)` use truediv, just might be strange.
This is definitely different from numpy behavior.

We also may need to add the division future import to more places.


Labels:

Comments:
you can't just put the `from __future__ import division``in``pandas/__init.__py`` which AFAIK is always imported.
or is it module level somehow?

It's module level :-/

Just want to make it clear that this is the end result on Python 2.X without the future import (working interactively):

``` python
In [1]: import pandas as pd

In [2]: import numpy as np

In [3]: arr = np.array([1, 2, 3, 4])

In [4]: arr2 = np.array([5, 3, 2, 1])

In [5]: arr / arr2
Out[5]: array([0, 0, 1, 4])

In [6]: pd.Series(arr) / pd.Series(arr2)
Out[6]:
0    0.200000
1    0.666667
2    1.500000
3    4.000000
dtype: float64

In [7]:
```

Whereas before this PR:

``` python
In [2]: pd.Series(arr) / pd.Series(arr2)
Out[2]:
0    0
1    0
2    1
3    4
dtype: int64
```

maybe make a separate section in v0.13 to highlite this (with an example like this)

yep, just did that.

On Tue, Nov 5, 2013 at 7:16 AM, jreback notifications@github.com wrote:

> maybe make a separate section in v0.13 to highlite this (with an example
> like this)
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5439#issuecomment-27768032
> .

looks ok to me
TST: Use tempfiles in all tests.
State: closed

related #5419

Includes @jreback's commits from #5422 and hdf_temp:
- TST: make pytables tests go thru a temporary dir and file
- TST/BUG: incorrect way of testing for r+ modes

TST: fix temporary files by using mktemp (rather than mkstemp) which opens them


Labels:

Comments:
Why don't we just not use ensure_clean for hdf? Already has a custom
function anyways, could just create temp dir with mktemp in setUp then
remove in tearDown (module level)

don't actually use the ensure_clean very much, just a couple of time, it DOES have a custom one (that generates stores).....anyway fixed already

@jtratner ok now?

@jtratner ok?

Yes, nice and principled :)
TST: make pytables tests go thru a temporary dir and file
State: closed

TST/BUG: incorrect way of testing for r+ modes

related #5419


Labels:

Comments:
I just fixed the merge conflict on the related PR. Once it passes I'll merge it and then you can do this one. Sound good?

except don't put in any of the hdf stuff (or I can merge this first)

ok let me move this pr to your branch u can incorporate and merge all together

closing in favor of #5419
BUG: to_timedelta of a scalar returns a scalar, closes #5410.
State: closed

closes #5410


Labels:

Comments:
@danielballan I think this is just a trivial test fix...can you have a look...want to do this for 0.13

Yep, the key was `if not _np_version_under1p7`. If the build passes, we should be good to go.

Note to selves: Testing on to_timedelta is pretty light.

Hrm. This passes on my box with numpy 1.7.1. Weird. Will take another looks later.

you are comparnig that result with assert_series_equal, but you should be using assert_almost_equal (as they are arrays and not series)

this looks fine....pls give 1 more rebase and can merge it

Done.

feel free to add more tests to to_timedelta (its not _so_ light...but more tests are better!)

Yeah, that was my mistake. I was looking at the smattering of tests under pandas/tests, completely missing pandas/tseries/tests!

np...in theory they should be consolidated..but no big deal really

thanks@!
ENH/BUG: pass formatting params thru to `to_csv`
State: closed

Add support for passing remaining `csv.writer` formatting parameters thru to `DataFrame.to_csv()`. 

Maybe write tests for this? That much is over my head currently. 


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: API Design
name: IO CSV
description: read_csv, to_csv

Comments:
Can't just add this...needs to be supported by C parser too (and therefore needs tests, etc)

Yea, I didn't expect it to be that easy. Is it better etiquette to close this PR and reopen later or just leave it hanging?

oh this is to_csv -- just need to put together some tests for this.

does this close an issue?

need tests for this

I realized this commit was doing two things so I refactored into a separate PR (#6034). I'm still a little lost on what kind of test to make though since these params are just passed internally to `com.UnicodeWriter` or `csv.writer`. 

The only thing that occurs to me is to write an output file using the new parameters then read it back in and make sure the write occurred correctly. Is that overkill? Any tips or hints what to model the test(s) on?

This PR seems to cover #4528 too

I tried using `self.assertRaises` as a context manager which doesn't work for python2.6. Any tips on compatibility?

Specifically it fails like this:

``` python
Traceback (most recent call last):
  File "/home/travis/virtualenv/python2.6_with_system_site_packages/lib/python2.6/site-packages/pandas/tests/test_format.py", line 1729, in test_to_csv_quotechar
    df.to_csv(path, quoting=1, quotechar=None)
  File "/usr/lib/python2.6/contextlib.py", line 34, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/travis/virtualenv/python2.6_with_system_site_packages/lib/python2.6/site-packages/pandas/util/testing.py", line 372, in ensure_clean
    yield filename
  File "/home/travis/virtualenv/python2.6_with_system_site_packages/lib/python2.6/site-packages/pandas/tests/test_format.py", line 1728, in test_to_csv_quotechar
    with self.assertRaises(TypeError):
TypeError: failUnlessRaises() takes at least 3 arguments (2 given)
```

Use `with tm.assertRaises(TypeError)`:

https://github.com/pydata/pandas/wiki/Testing#wiki-testing-for-exceptions

Thanks @TomAugspurger 

@patricktokeeffe pls rebase, and move the release notes to 0.14 section. thanks

which issues does this cover?

@jreback I sent this PR without creating an issue. It does cover #4528 though the idea mentioned there of adding a csv dialect argument is a good one not included in this PR. 

thanks nice little fix!
DOC/BUG: fix details for 'quoting' parser parameter
State: closed

``` python
Python 2.7.5 (default, May 15 2013, 22:44:16) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

Imported NumPy 1.7.1, SciPy 0.12.0, Matplotlib 1.3.0 + guidata 1.6.1, guiqwt 2.3.1
Type "scientific" for more details.
>>> from csv import QUOTE_MINIMAL, QUOTE_ALL, QUOTE_NONNUMERIC, QUOTE_NONE
>>> QUOTE_MINIMAL
0
>>> QUOTE_ALL
1
>>> QUOTE_NONNUMERIC
2
>>> QUOTE_NONE
3
>>> 
```


Labels:

Comments:
How's it look now?

can you squash this down to a single commit?

Yeah, sure. Will I need to rebase on master too?

does this close an issue?

pls rebase this and can consider for 0.13.1

@patricktokeeffe can you rebase and make that small change?

OK @jreback I think I've finally succeeded without mucking up the repo. Let me know if it needs anything else. 

@jorisvandenbossche ok on formatting?

Just realized the issue # was missing from the commit log. It's been added & rebased over the last few commits. 

OK!

thanks
Fix Issues with FY5253 and nearest w/ year end in Dec
State: closed

There are currently some issues with FY5253 working with `variation="nearest"` and `startingMonth=12`.

This PR fixes those issues and add some more tests.


Labels:

Comments:
@jtratner Other than that, any other changes? I would like to get this fix merged in for v13

Need to rebase.

Of course

it just was not mergeable. If you make that change for clarity and pinky-swear that it's doing what it's supposed to, I'm sure it's fine (you're sort of the expert on the retail year). :smile:  I'll give it another once over after you change the naming there.

I am becoming the expert. The problem was I was previously using the "year" of the dt as the canonical FY year. Unfortunately there are circumstances where two years end in the same calendar year.

@jtratner Okay rebased, etc.

I know we don't always ask this, but can you fix up your formatting? Just run a pep8 linter and then make changes to your portion of the code. Especially spaces around operators. 

@jtratner Should be PEP8 compliant. I also moved the new classes around as mentioned previously.

@jtratner What do you think of the code?

I hate to ask this (and it's okay if you can't), but is there any way you could split this into two commits? (one that does the code move through the file and one that makes the changes). It's hard to tell which changes are what.

I'm sure a little copy and paste action will solve this; the move was relatively simple. Let me give it a shot . One day diff tools will handle situations like this. 

or maybe even save you from overzealous devs asking you to squash your commits :stuck_out_tongue_closed_eyes:

@jtratner Commit split

Thanks!
BUG: support for GA new segment format (GH5398)
State: closed

Adds support for underscores to Google's new segment format. The issue referenced is here:
closes https://github.com/pydata/pandas/issues/5398


Labels:

Comments:
please merge these into one commit and then I'll merge.

Ok. All set!

ready to merge?

Looks good to me.

@jtratner ?

fine except for the two whitespace changes. I'd prefer to make it a global, but I'm okay to merge even if you don't make that change.

I probably won't be able to make any changes through the weekend. Lemme know if any are critical. Thanks!

Well, if you could just fix the whitespace changes, I'm okay to merge this.

Normally I would but I'm literally on a train to the airport for a T Day extravaganza right now. 

Sent from my iPhone

> On Nov 27, 2013, at 3:26 PM, Jeff Tratner notifications@github.com wrote:
> 
> Well, if you could just fix the whitespace changes, I'm okay to merge this.
> —
> Reply to this email directly or view it on GitHub.

@jcbozonier can you make the changes suggested by @jtratner ?

@jcbozonier can you make those changes?

Not currently.

On Thu, Dec 5, 2013 at 10:54 AM, jreback notifications@github.com wrote:

> @jcbozonier https://github.com/jcbozonier can you make those changes?
> 
> —
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/5400#issuecomment-29914999
> .

merged via 4f9fefc2bea834690e5200978a9a32141f01bf77

thanks!

pls give master a test run to verify
Display the correct error message when calling a binary moment without appropriate parameters.
State: closed

This pr fixes a typo in the type checking of  binary moments. It makes the error messages below consistent:

> > > import pandas
> > > a = pandas.DataFrame([1,2,3,4,5,6])
> > > pandas.rolling_cov(list(), a, 3)
> > > ValueError: arguments to moment function must be of type ndarray/DataFrame
> > > 
> > > pandas.rolling_cov(a, list(), 3)
> > > TypeError: unsupported type: <type 'list'>


Labels:

Comments:
I'm not sure you have this right. Also should be TypeError not ValueError.
Please change and I'll review.

This looks good. I'll push the exception type fix seperately.

e82c1f7
Update offset prefix map
State: closed

This should address issue #5382 


Labels:

Comments:
I tried setting up Travis but ~~can't tell if it's working.~~ didn't turn on the service hook before pushing. Locally I ran `nosetests pandas` but I don't know what to think of the results:

Base case - `master` (d0342aa79d2) 2013-10-27 20:26:02

```
Ran 4331 tests in 384.605s

FAILED (SKIP=318, failures=1)
```

Test case - `update-offset-prefix-map` branch (a2cc2303681)

```
Ran 2977 tests in 190.863s

FAILED (SKIP=156, errors=1064, failures=1)
```

Weird that master has a failing test case. It appears that you introduced
some errors such that 1300 tests didn't run and another 1000 tests failed
in some way. Many tests create date indexes as part of their setup, which
may be the problem.

Also, Nano is conditionally added to the offset list based on numpy version.

@patricktokeeffe and generally doing something based on the import, e.g. `__all__` is not a nice way way of doing things

Ok. Looks like the only missing one then is `LastWeekOfMonth` and since it appears to be WIP (#4637) this PR is irrelevant.
ENH: Add mode method to Series and DataFrame
State: closed

Closes #5367 - generally as fast or faster than value_counts (which makes sense because it has to construct Series), so should be relatively good performance-wise. Also, doesn't get stuck on value_counts' pathological case (huge array with # uniques close to/at the size of the array).

Not using result of hashtable.value_count() under the hood and instead iterating over klib table directly gives a _huge_ speedup. I just moved the value_count table creation method to a separate function (zero perf hit). For performance breakouts check out this gist:

https://gist.github.com/jtratner/7225878

DataFrame version delegates to Series' version at each level.


Labels:

Comments:
@jreback should I put this on hold until after 0.13?

@jtratner seems reasonably tested.....i think its ok, need a doc mention? (in api, but near value_counts somewhere?)

Yeah, I'll add it.

Can you also add entries to api.rst?

I missed putting it there. I'll do that--thanks for noticing that!

added doc mention for Series and DataFrame