BrandonFors/pandas-github-issues huggingface dataset:
ERR: more informative error message on bool arith op failures
State: closed

closes #7011 


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
can you repush, to make it rebuild travis 

also can you update the release note with this issue as well , #6762

maybe add to API changes as well on v0.14.0, and an example if you think its worth it 

looks fine...

don't merge just yet...trying to fix the caching issue on master

sure no prob

anything i can do to help on that front?

all see
u can merge 

@jreback all see? don't understand...still failing i think there's some floating point exception ... let me check it ou

meant all set when u r ready
caching fixed in master 

hmm...can you rebase/repush...see if their error repeats

having some trouble repro-ing this. need a bit more time to track it down ... might have to spin up an ubuntu machine

hows this coming?

ahhh github search was useful: turned up #3819 :)

what's strange is that it can only be reproduced on python 3.2 modding by 0 works fine on my local python 2.7.6

@jreback i'm okay with leaving `mod` as is for now. let me check to see if my other addition to the ops module screwed anything else up ... i don't think so but better safe than sorry

3.2 runs numpy 1.6.2 iirc

yep this is fine ... the `_combine_*` handle the `r*` ops that i've given `op_str`s to.

these are also all hit in `test_arith_flex_frame`

@jreback any last words?

g2g
BLD/TST: Python3.4 testing via travis
State: closed

closes #6619


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
:+1: Awesome!

...**crazy** that nose 1.3.1 fails on py3.

seems 1.3.2 is coming out soon, but 1.3.1 breaks ALL py3, 1.3.1 breaks 3.4, so have to source from master for now.
BLD: fix failing vbench cases
State: closed

There were about 30 vbench tests that failed due to multiple recent deprecations and copy-pasting.

This PR fixes all but 8 of those that correspond to constructing frame with "exotic" time offsets (`WeekOfYear`, `LastWeekOfYear`, `FY5253` and `FY5253Quarter`) that apparently require some kwargs for construction.

This also fixes the "different name/same test" situation with `eval_frame_and` benchmarks where for some reason single-thread and multi-thread cases share setup code.


Labels:

Comments:
ok lmk when you are ready to merge

Cool, sorry if kept you waiting.
BUG: fixing tseries plot cursor display, resolves #5453
State: closed

Fixes #5453

I am not sure how to test a mouse-over, and the original commit 922c6102eebb7347cea587fffc4795a3ca3e73b2 introducing this functionality did not have tests either, which is sadly probably how this got broken. @willfurnass do you happen to have any ideas for how to test this functionality, maybe from when you first looked at this?

The new code doesn't display timezone information like the old code once did (before it was broken) as Periods don't seem to have timezone info, and all `tseries/plotting.py` plots are Period-based. One could perhaps do some minor acrobatics to convert the tz-naive start time of the Period into a tz-aware Timestamp, I can research that if people feel strongly about it.


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Visualization
description: plotting

Comments:
Hmm, actually I kind of like leaving the display as a Period - works nicely for Weeks, Quarters, etc.

Maybe you could try to test it by making a plot, and then feeding manually data to `ax.format_coord` and test if this is formatted well. Something like:

```
s = pd.Series([1,2,3], index=pd.date_range('2014-01-01', periods=3, freq='H'))
ax = s.plot()
string = ax.format_coord(some_data)
assertEqual(string, expected)
```

and that for different frequencies. Just an idea, don't know if this will really work.

Yup that works - I've added a test along those lines that would have caught the original issue.

@jorisvandenbossche @TomAugspurger ok with this...

@rosnfeld ping when green

@jreback ping

@TomAugspurger if this is ok
go ahead and merge

Thanks @rosnfeld
ENH: Implement core/strings/wrap method
State: closed

This patch implements the str.wrap function within core/strings.

Example:

```
    >>> s = pd.Series(['line to be wrapped', 'another line to be wrapped'])
    >>> s.str.wrap(12)
    0             line to be\nwrapped
    1    another line\nto be\nwrapped
```

This is a cleaned branch; the original is at #6705.


Labels:
name: Enhancement
name: Strings
description: String extension data type and string data

Comments:
@jorisvandenbossche ok?

@jeffreystarr looks good, i will rebase/squash you on merge

yep, looking good! (I think all comments were already adressed)

merged via https://github.com/pydata/pandas/commit/5b838ab4438435838a09aa0472ef0fd66ec5541a

thanks!
CLN: drop internals._invert_reordering in favour of lib.get_reverse_indexer
State: closed

I've implemented the former while refactoring BlockManager (#6745) because I didn't find the latter in lib module. This PR will remove the duplication (and also add missing changelog entry).


Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Speaking of missing changelog entry, I'm not sure about how do I mention an _internal_ class in Enhancements section...

see the release notes / whatsnew for 0.13.0 when I refactored series. But don't think you need a whole section, just a line or two referring to the issue (and a warning/note to inform the user in v0.14.0 that hey did some refactoring internally, nothing in the API changed...bu wanted to let u know)

looks fine, ping when ready

Bumped the changelog. Is this what you had in mind?

yep...that's good

Travis failed with a weird error in io test, apparently, due to github serving `salary.table` with an extra empty line at the end.

no prob...thanks
COMPAT: remove deprecated optinos from printing in test suite (GH6984)
State: closed

closes #6984


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
@jorisvandenbossche 

this https://github.com/pydata/pandas/issues/6856 might partially fix this as their was a bug in `pd.option_context` that was causing a deprecation warning (in some cases)

after I merge give a try
PERF: improved performance of compatible pickles (GH6899)
State: closed

closes #6899

though seems that cPickle can on a small file/dataset be slower

so waiting for results on #6899 tests


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
packers_read_pickle                          | 446.7856 | 2399.6547 |   0.1862 |
packers_write_pickle                         | 784.3113 | 800.6363 |   0.9796 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [0e90ef4] : PERF: improved performance of compatible pickles (GH6899)
Base   [951f055] : Merge pull request #6986 from jreback/warnings2
```

COMPAT: remove deprecated optinos from printing in test suite (GH6984)
ERR: Add check for iterators when creating DataFrame, fixes #5357
State: closed

This is a fix for #5357, and adds a more helpful error message when trying to use an iterator as the `data` argument for a `DataFrame`. I've added the type check for iterators in the final else clause where all the valid options have already been exhausted, so it shouldn't interfere with any valid cases like passing a list of iterators.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Thanks for the PR! One key thing this needs is a test case demonstrating the error. Also, stylistically, it would be nicer/clearer to move this up a level (ie as an elif rather than a nested if). Would you mind making that change?

When the above is addressed we can mergw

Alright, I've added a (simple) test case to `test_frame.py` and moved the type check up two levels, since that's where the rest of the explicit type checks are happening.

@onesandzeroes looks great - can you repush your branch so Travis can retry the tests? The build failed because of unrelated network issues.

@onesandzeroes pls add a release notes in the API section referecing the issue, other looks good

Alright, should be all set now, thanks for walking me through the steps that needed to be done!

gr8!

if you want to try your git-fu, squash down to 1 commit (if you can't I will do it...but good practice)

see here: https://github.com/pydata/pandas/wiki/Using-Git

ping me when squashed / green

OK, should be squashed into a single commit now.

@onesandzeroes thanks for the PR!
Fix for GH 6885 - get_dummies chokes on unicode values
State: closed

closes #6885

Please be gentle - this is the first time I've tried to contribute either to someone else's python project or another project on github, so if my python or git foo is lacking, I apologise in advance!


Labels:
name: Bug
name: Unicode
description: Unicode strings
name: Strings
description: String extension data type and string data

Comments:
Note that the failed test wasn't related to my changes:

```
pandas.io.data.RemoteDataError: Parsed URL 'http://finance.yahoo.com/q/op?s=AAPL&m=2014-05' has no rootelement
```

can you add a release note in Bug Fixes referecing this issue, otherwise looks good

@hayd @TomAugspurger  ok by you?

The travis failure looked like an unrelated yahoo test. I restarted it on Travis (I think that's a new feature). We'll see if Github notices and updates with the results.

Apologies for the stray print - I'll remove that and then add the edit in bugfixes

@maxgrenderjones did you force push? (your old commit is still here)

ping me when green

Last code-changing build is green - don't know if travis bothers building if you just change docs.

Apologies that a 4 character change took four commits!

merged via https://github.com/pydata/pandas/commit/7918d6000d3ccd5d19713698a47b429cecfe1c63

thanks!
Cookbook additions
State: closed

As suggested by Jeff on this [StackOverflow answer](http://stackoverflow.com/a/23233196/1222578), I'm adding an example of a boxplot that uses quartiles of a second variable as the `by` variable.

Jeff also suggested adding the example in-line, rather than just providing a link to StackOverflow, so that's what I've done for now. The resulting plot currently looks like this, I'm happy to tweak it to better get the idea across:

![Boxplot example](http://i.imgur.com/yQZMlGH.png)


Labels:
name: Docs
name: Visualization
description: plotting

Comments:
merged via https://github.com/pydata/pandas/commit/3b13646701951cb82c83d19a26743732e501abbb

thanks!

http://pandas-docs.github.io/pandas-docs-travis/cookbook.html#plotting

thanks!

any other cookbook recipies you want to expand inline - go for it!
COMPAT: fix numpy 1.9-dev deprecation warnings in test suite
State: closed

COMPAT: change neg of boolean to inv (deprecation in numpy 1.9)
COMPAT: remove deprecation warnings on **new** (in computation/pytables)
COMPAT: fix array_to_datetime on invalid comparison to NaT

related #6958

add ability to set an environmental variable to actually show the DeprecationWarnings

just

`setenv PANDAS_TESTING_MODE 'numpy_deprecate'`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
for an example https://travis-ci.org/jreback/pandas/jobs/23758981
MAINT: Numpy deprecated boolean unary '-' operator
State: closed

I'm sure there are more of these, but I just caught this one when I was raising on all warnings.


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
ahh..prob a fair number of these....

Might be trivial to write a log only on this deprecation warning to wrap the test suite in. I have no idea though.  

this is under numpy 1.9-dev right?

Yeah `1.9.0.dev-3e61f86`

we test with numpy master in every build: https://travis-ci.org/jreback/pandas/jobs/23688029

is their a way to make nose print out the warnings (i'll have to turn off the deprecation stuff) because we hide that (or maybe that's all I need to do?)

Deprecation warnings are filtered in python by default. I'm not really sure how to show them using nosetest. This is the code to show them

```
import warnings
warnings.simplefilter('always', DeprecationWarning)
```

But I'm not sure how to get nosetest to respect that. Maybe you could put it in `.pystartup`. Or put it at the beginning of a `pandas.test()` function, but I don't think pandas test suite can be run that way.

I had a filter to 'error', because I wanted to debug another warning I had, but my code raised on this first.

ok...easy enough...to run it and look for these

going to close in favor of #6960 their are like a ton (think I got them though)

@jseabold this turned into a project!

numpy really turned on the spigot with depr warnings! (of course the conversions that we were doing were a bit suspect anyhow)..

e.g. you can't compare `np.datetime('NaT')` with `tslib.NaT` (e.g. its a datetime64 cmp with an integer (but `np.datetime('NaT').astype('int64') == tslib.NaT` works just fine)
BUG: fix handling of color argument for variety of plotting functions
State: closed

parallel_coordinates
- fix reordering of class column (from set) causing possible color/class
  mismatch
- deprecated use of argument colors in favor of color

radviz
- fix reordering of class column (from set) causing possible color/class
  mismatch
- added explicit color keyword argument (avoids multiple values 'color' being
  passed to plotting method)

andrews_curves
- added explicit color keyword argument (avoids multiple values 'color' being
  passed to plotting method)

To recreate the bug:

``` python

import pandas as pd
from pandas.tools.plotting import parallel_coordinates, radviz, andrews_curves

x = pd.DataFrame([[1,2,3,'b'],
                  [2,2,1,'g'],
                  [3,3,1,'r']], columns=[0,1,2,'name'])

parallel_coordinates(x, 'name', colors=['b','g','r'])

# TypeError from scatter
radviz(x, 'name', color=['b','g','r'])

# TypeError from plot
andrews_curves(x, 'name', color=['b','g','r'])

```

parallel_coordinates before:
![ppl_old](https://cloud.githubusercontent.com/assets/3686522/2791463/28c84c9a-cbc7-11e3-9099-9e064d31e995.png)

parallel_coordinates after:
![ppl_new](https://cloud.githubusercontent.com/assets/3686522/2791462/28c73e0e-cbc7-11e3-804e-73dc43cbd6ca.png)


Labels:
name: Visualization
description: plotting

Comments:
Thanks for the PR. I had those couple comments inline. Could you also add the note you have in `doc/source/release.rst` to `v0.14` under the API section. And you can add a note to the bug section about the other bugs you fixed.

Does this close a particular Github issue?

I'll add this deprecation to https://github.com/pydata/pandas/issues/6581 when we merge.

Also it doesn't look like Travis ran. You may need to hook it up to your Github account: http://about.travis-ci.org/docs/user/getting-started/ You should just have to sign in.

I made the changes you suggested.

A Travis build seemed to get launched once I pushed my commit but there we errors that seem unrelated to my changes. Do you have any suggestions on what might be the issue? Here is the build https://travis-ci.org/anomrake/pandas/builds/24206970

pls rebase to master. travis made some changes over the last couple of days, all fixed now.

travis will automatically build on each commit.

I tried to rebase but I'm new to this and I'm a bit surprised to see these other commits show up. Any help would be great.

see here: https://github.com/pydata/pandas/wiki/Using-Git

```
git rebase -i origin/master
```

then

```
git push yourfork yourbranch -f
```

Looks to have worked now. Thanks for the help.

Also, as far as I know, there was no Github issue for the bugs. 

I made the change you requested

If you could rebase one last time and change those `irows` to `icols`, I'll get this merged. Thanks.

OK done

Great, thanks!
ENH: Quantiles accepts an array
State: closed

Doesn't quite finish #4196, but it should be easy not that `quantile` takes arrays.

I did this on top of #6953, so that should go in first.


Labels:
name: Enhancement
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
Thanks, updated.

Added a test on empty frames

Also fixed a bug, I wasn't passing the axis argument correctly when quantiles was an array.

One more thing, thinking ahead to fixing #4196; Are we ok with returning a `FloatIndex`? When you do `df.describe()` the quantiles are formatted as (e.g.) `50%` instead of `.50`. I chose to use floats instead of formatting as strs since I see `quantile` being used in a pipeline, but `describe` is more useful as a final presentation thing.

for `df,quantiles()` a `FloatIndex` is just fine (that's what your example was). I agree `describe` is more pretty output

OK. Good to merge?

can you add a test with datetimes both for Series and Frame for completeness?

Do you mean with a DatetimeIndex or Datetime values? It doesn't look like we support quantile on `datetime` columns.

this is off master:

``` python
In [15]: df
Out[15]: 
           a  b
0 2010-01-01  1
1 2011-01-01  2
2 2012-01-01  3

[3 rows x 2 columns]

In [16]: df.quantile(.5)
Out[16]: 
b    2
dtype: float64

In [17]: df.set_index('a').quantile(.5)
Out[17]: 
b    2
dtype: float64
```

have to pass `numeric_only=False`

I think it will blow up actually
datetime64[ns] need a view('i8') (and not cast to float

iin docstring can you change ref from scipy.score..... to np.percentile  (as tht is what it uses now)

yeah it blows up pretty badly. My simple fix didn't work. Separate issue?

sure

OK. I'll make an issue for that. This PR doesn't officially close any.
BUG: Pass args and kwargs to empty
State: closed

Closes #6952

I wasn't sure what the best way to test this was. I went with calling `_apply_empty_result` directly and checking for a SystemExit to get around the try: except Exception block in `_apply_empty_result`.

I still don't think I've 100% ensured that `_apply_empty_result` will always be _called with_ _args and *_kwargs (if there are any), but if they are passed things will be handled correctly.


Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Hmm py3 failures. I can't do `rase arg` in py3 even if `arg=ValueError`. You have to wrap in an `Exception(arg)`, which gets swallowed up by the `try: except` block that I'm trying to test...

suggestions?

not sure if it really needs a test; if it doesn't break anything else its ok

do you have an example that failed before? and works now?

e.g. a function on an empty frame, that could be a test

(don't have to test the generic reduction test though)

My tests in #6955 fail without this, so that should be good right?

specifically 

``` python
        # test degenerate case
        q = DataFrame({'x': [], 'y': []}).quantile(0.1, axis=0)
        assert(np.isnan(q['x']) and np.isnan(q['y']))
```

in `pandas/tests/test_frame:TestDataFrame.test_quantile`

oh sorry..didn't see that (that's ok then)

maybe in the other PR reference the PR number just in case
BUG: enabling subplots works unexpectedly
State: closed

Found 2 minor issues related to plotting with `subplots=True`.
1. If `DataFrame` only has single column, plot raises `TypeError`. It is because `_subplots` returns `Axes` instance, not array.
2. If `kind=scatter` or `kind=hexbin` is specified with `subplots=True`, additional blank axes is drawn.


Labels:
name: Visualization
description: plotting

Comments:
I found 3rd minor issue and fixed:
- Calling `Series.plot` with `subplots=True` raises `AttributeError`

seems reasonable, @TomAugspurger can you validate?

This is good. @sinhrks do you want this one merged before your #6656?

Yes, please. Later, I'll finish #6656.

@jreback  this is OK, just a conflict in `release.rst`

Can I merge this from the command line with

```
git checkout -b sinhrks-scattersp2 master
git pull git://github.com/sinhrks/pandas.git scattersp2
```

fixing the conflict in `release.rst`, rebasing onto master, checking out master and merging this PR in, and pushing to upstream/master? I shouldn't force push to master right?

correct, don't force push to master

exactly correct on your git-fu!

OK thanks.

merged via 97c4a2e5e47bf00d1da5d56b7fbf862b2fc52819

Thank @sinhrks 

@TomAugspurger maybe add this to the dev wiki somewhere? (how to merge a PR this way)

https://github.com/pydata/pandas/wiki/Merging-Commits started there. 

nice!
BUG: Fix to read decimal seconds from Excel.
State: closed

Fix to allow decimal seconds to be read from Excel dates and times
into datetime objects. #5945.

This required a fix to the `xlrd` module to return milliseconds from Excel dates and times. That fix was recently released to PyPI in xlrd version 0.9.3.

Tests, version updates and release note included.


Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel
name: Timedelta
description: Timedelta data type

Comments:
this needs to be able to raise on an older version of `xlrd` (and a test that will correctly skip) on the same.

@jreback 

Patch updated with your comments, squashed and re-pushed:
- I made the `ci/requirements-3.2.txt` change to `xlrd` version 0.9.2 for testing (although I'd prefer to change that back to 0.9.3 before merging since it is bound to cause confusion because of inconsistency).
- I made the `LooseVersion()` changes in `pandas/io/excel.py` and `pandas/io/tests/test_excel.py`.
- I tested with an older version of `xlrd`. The tests all skip and the module throws an exception:

``` python
>>> import pandas as pd
>>>
>>> pd.read_excel('decimal_seconds_1900.xls', 'Sheet1')

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pandas/io/excel.py", line 108, in read_excel
    return ExcelFile(io, engine=engine).parse(sheetname=sheetname, **kwds)
  File "pandas/io/excel.py", line 131, in __init__
    "support, current version " + xlrd.__VERSION__)
ImportError: pandas requires xlrd >= 0.9.3 for excel support, current version 0.9.2
```

@jmcnamara see my comment above. The ci tests multiple versions of every library so that back compat can be maintained. 0.9.2 should work unless I need to the new capability. Its not realistic to force a user to upgrade `xlrd` unless they actually need to use the new capabilities (because even though its the current pypi release, what's going to happen is people may not upgrade to it at the same time as they upgrade pandas or vice-versa).

@jrebeck

Ok. Fair enough.

I'll revert most of the changes and just put a conditional in the date conversion to use the millisecond method if available or use the 0.9.2 method if not.

I've reworked the PR to handle millisecond times when `xlrd >= 0.9.3` is available or to use the previous method if not.

I've added a test case for both conditions and also for the twin Excel epochs for good measure.

hmm...it is skipping on 3.2 for a different reason (xlwt is not installed). so not running your test. https://travis-ci.org/jmcnamara/pandas/jobs/23636210

so really only testing excel in 2.7...(the locale build is not even running the tests)...

let me add these deps to 2.6

hold a sec

ok...rebase on master...I added the deps to the 2.6 build; so should skip correctly there (the 3.2 build doesn't really matter) because almost all tests that need to do writing are skipping anyhow (because xlwt is not installed)

Rebase done so let's see what Travis says.

I tested locally with xlrd 0.9.2 and 0.9.3 and both passed.

looks good...thanks