BrandonFors/pandas-github-issues huggingface dataset:
BUG: fix variable overwriting in radviz
State: closed

The variable `n` is set to number of rows in dataframe, and then it overwritten with number of columns, but after that there is loop that should iterate over rows and it uses new value in `n` that holds number of columns already.


Labels:
name: Visualization
description: plotting

Comments:
can u add a test for this
(that validates the returned axes / graph)

@TomAugspurger ?

Yep, good catch. @stas-sl do you mind rebasing and adding a note in `docs/source/v0.15.0.txt`?

And if you want, a test would be great. radviz doesn't seem to be that thoroughly tested.

Did rebase and added the note, but sorry my knowledge of python, pandas and running tests in python is not sufficient, so currently can't help you with tests.

Yeah I looked at it a bit and didn't see an obvious way to check.

Thanks @stas-sl!

The docs are built automatically during the Travis run, would you mind checking back [here](http://pandas-docs.github.io/pandas-docs-travis/visualization.html#radviz) later today to see if the radviz plot looks correct?

It looks much better now )

Before:
![before](https://cloud.githubusercontent.com/assets/4602302/4327623/03f06614-3f7b-11e4-9fad-0081a13b06a9.png)

After:
![after](https://cloud.githubusercontent.com/assets/4602302/4327622/03ee48a2-3f7b-11e4-9d76-2e194d53ffff.png)

Thanks!
ENH: Add prefix parameters for join method.
State: closed

Addresses #8139. Not sure how people feel about this one, in terms of adding clutter to the api, but it _was_ pretty easy to get working, so I thought I'd put this together to show what changes would need to be made. I might still need to go over `DataFrame.merge()` to see if the same arguments can be added for consistency. 


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
Answering in #8139 (to first discuss general idea there)

closing pls reopen if/when updated
API: kdeplot fails with NaNs.
State: closed

I think the most sensible thing to do here is to drop NaN's, like histograms do, rather than filling 0's. The table in #8177 should be updated to reflect that NaN's are dropped, can do that myself after 8177 gets merged.


Labels:

Comments:
Fixes #8182.

You beat me to this! Thanks. Could you squash these down to a single commit? I'll update the table in #8177 when I merge that one.

@TomAugspurger Squashed down to a single commit, should be good.

Thanks.
ENH: more verbosity to PyTables import failures
State: closed


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO HDF5
description: read_hdf, HDFStore

Comments:
what kind of messages show up here? (ehat does a sample exception look like)

this doesn't build properly. pls update and lmk.

@jreback -- thanks for looking at this. Previously you get the exception "HDFStore requires PyTables" when there is a problem with loading the underlying HDF5 shared objects / libraries referenced by (py)tables. I had to spend a bit of extra time tracking down the error, where now the exception will say something more specific like "HDFStore requires PyTables, "could not find libhdf5.so.6" problem importing". Just a tweak for diagnosing.

ok, looks fine. pls squash to a single commit. ping when green.

@jreback -- ping.

thanks!
BUG: scatter with errorbar raises IndexError
State: closed

Closes #8081.

Also fixes a color problem with `c` kw found in #7780. (But should work with single color. It looks `errorbar` doesn't accept multiple colors).


Labels:
name: Visualization
description: plotting
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Will review later today.

---

Only slightly related, but what are your thoughts on on `Series.plot(kind='scatter')`? Right now it raises a `ValueError`, but I expected it to work with `x=series.index, y=series.values`.
BUG: barplot with NaNs
State: closed

Closes https://github.com/pydata/pandas/issues/8175

This makes barplot more consistent with how NaN's are handled in AreaPlots and PiePlots.


Labels:
name: Bug
name: Visualization
description: plotting
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
I'll add this argument to `df.plot(..., fillna=None)` and `series.plot(..., fillna=None)`. The default is `None`, which means follow these defaults:
- Line Plot: Don't fill anything, leaves gaps in the line at the missing values
- (Stacked) Bar Plot: Fill with zeros before plotting (This PR fixes the bug on stacked bar plots not drawing past a NaN).
- Hist: `df.plot(kind='hist')` <==> `df.dropna().plot(kind='hist')` (column-wise I think)
- Boxplot: `df.plot(kind='box')` <==> `df.dropna().plot(kind='box')` (column-wise I think)
- Areaplot `df.plot(kind='area')` <==> `df.fillna(0).plot(kind='area')`
- Hexbin: `df.plot(kind='hexbin')` <==> `df.dropna().plot(kind='area')`
- Pie: `df.plot(kind='pie')` <==> `df.fillna(0).plot(kind='pie')`

All look good? The only end-user change from this PR is barplot not bugging out with NaNs.
The reason that I don't see a need for a kwarg is that I don't really think there's anything to turn off here. I think that for all of these, the default is the _only_ sensible thing to do. Any counterexamples where our defaults do the wrong thing?

Complement:
- Line Plot(unstacked): Don't fill anything, leaves gaps in the line at the missing values
- Line Plot(stacked): `df.fillna(0).plot(stacked=True)`
- Kde: `df.dropna().plot(kind='kde')`
- Scatter: `df.dropna().plot(kind='scatter')`

+1 for _not_ a new keyword, let's just implement sane default behaviour (which is already the case I think apart from the bug addressed iin this PR), and let people use `fillna` if they want something else.

One remark, the default for `pie` is fill with 0, would it not more logical to drop?

Re the default for pie, I think you're correct.

``` python
In [11]: series = Series(3 * np.random.rand(8),
    index=['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h'], name='series')
In [15]: series.iloc[:4] = np.nan

In [16]: series.plot(kind='pie')
Out[16]: <matplotlib.axes._subplots.AxesSubplot at 0x11183e128>
```

![nana](https://cloud.githubusercontent.com/assets/1312546/4175194/c8b246d6-35c4-11e4-9513-b1539f956d72.png)

A bit hard to see, but all the NaNs are piled on top of each other (middle right). New issue?
BUG/API: Previously an enlargement with a mixed-dtype frame would act unlike append (related GH2578)
State: closed

related #2578

from [SO](http://stackoverflow.com/questions/25670760/copying-a-row-screws-up-pandas-column-data-types-how-to-avoid)


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: API Design

Comments:
:thumbsup: The indexing code gets even more fun!
counting sort instead of np.argsort in left outer join
State: closed

Previously I had submitted https://github.com/pydata/pandas/pull/7853. Given that we are sorting indexers, for performance reasons I should have used `groupsort_indexer` instead of `np.argsort`.

on master:

```
In [7]: np.random.seed(2718281)

In [8]: n = 50000

In [9]: left = DataFrame(np.random.randint(1, n/500, (n, 2)),
   ...:         columns=['jim', 'joe'])

In [10]: right = DataFrame(np.random.randint(1, n/500, (n, 2)),
   ....:         columns=['jolie', 'jolia']).set_index('jolie')

In [11]: %timeit left.join(right, on='jim')
1 loops, best of 3: 3.53 s per loop
```

on branch:

```
In [13]: %timeit left.join(right, on='jim')
1 loops, best of 3: 2.46 s per loop
```


Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
do you have any idea on the breakeven point between `groupsort_indexer` and `np.argsort`?

(and shouldn't these be `kind='mergesort'` (for the `argsort`, though not using that anymore...)

do we have a vbench on this ?

`groupsort_indexer` is always faster except for the case of _un-compressed_ multi-key groupby, if the number of unique values on each key is large enough.

`kind=mergesort` shouldn't matter here, because these are indexers, and the orders are indistinguishable if the indexers are the same.

I will add a vbench soon.

vbench added

looks good

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
left_outer_join_index                        | 2037.3669 | 2640.8920 |   0.7715 |
join_dataframe_index_single_key_bigger       |  13.9186 |  15.0377 |   0.9256 |
join_dataframe_integer_key                   |   1.6544 |   1.7044 |   0.9707 |
join_dataframe_index_single_key_bigger       |  13.9370 |  13.9953 |   0.9958 |
join_dataframe_integer_2key                  |   4.7170 |   4.7353 |   0.9961 |
strings_join_split                           |  31.3876 |  31.3776 |   1.0003 |
join_dataframe_index_multi                   |  18.4283 |  18.4180 |   1.0006 |
join_dataframe_index_single_key_small        |  11.5133 |  11.4816 |   1.0028 |
join_non_unique_equal                        |   1.0237 |   0.9787 |   1.0460 |
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------

Ratio < 1.0 means the target commit is faster then the baseline.
Seed used: 1234

Target [00656bf] : Merge branch 'lj-cnt-sort' of https://github.com/behzadnouri/pandas into behzadnouri-lj-cnt-sort
Base   [3ecb760] : Merge pull request #8171 from behzadnouri/count-f32

BUG: GroupBy.count() with float32 data type does not exclude nan
```
BUG: GroupBy.count() with float32 data type does not exclude nan
State: closed

closes https://github.com/pydata/pandas/issues/8169


Labels:
name: Bug
name: Groupby
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
tests!!!!!

@behzadnouri where `lib.row_bool_subset_object` is called, should be `com._ensure_object(result)` rather than passing in result (to avoid this happening on other dtypes); keep your other changes as is

test and `com._ensure_object` added

thanks!
BUG: not raising error on Newey-West corrections when clustering
State: closed

closes #5884, OLS does not correctly apply Newey-West corrections when clustering
is being used. Add an assertion to check for this.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
- need a test for this as well
- a release note in v0.15.0 reffing the issue

The assertion I added (which I'll change to a ValueError and add a test for, as requested), has caused several other tests that run OLS with clustering and Newey-West adjustments to fail. What's the recommended course of action to get all the tests to pass again? Raise nose.SkipTest() on the problematic tests? Have them test for the ValueError instead of whatever they were originally testing for? 

if other tests look for the AssertionError then change them. Generally need to be very careful about changing tests as could indicate an actual problem)

Sorry, my comment was unclear. What I meant was that some pre-existing tests, such as pandas.stats.tests.test_ols.TestPanelOLS.testRollingWithNeweyWestAndEntityCluster, fail because they try to run OLS with Newey-West and clustering, so they fail due to the new error check that I'm adding in this pull request. Because my change causes OLS to error anytime Newey-West and clustering are both used, any tests that do this will now fail. This is the issue that I'm unsure how to resolve.

@jessefarnham 

are these valid ways of using it? if not then the tests _should_ fail (so change the test to to assertRaises)

No, using OLS with clustering and Newey-West does not work properly, per GH5884. I'll change the affected tests to use assertRaises. Thanks for the help!

Changed the AssertionError to a ValueError, and changed the two broken tests to assert that the ValueError is raised when OLS is called with Newey-West and clustering.

excellent!

pls add a release note in v0.15.0.txt (you can put it in Bug Fixes)
pls squash to a single commit

ping when green

Unfortunately, I was away for a week and removed my changes from this pull request, planning to sync to head and then reapply when I got back. As a result, when this pull request was closed and merged, it appears that my changes did not actually make it into the codebase. I'll start another pull request; since I'm familiar with the procedure this time, I'll make sure all the changes are in one commit and all the tests pass, so it should be a simple, quick merge this time. Sorry for the trouble; this was my first attempt at a pull request on Github.

@jessefarnham hmm, this is very odd. Looks like an old github bug that does this. Ok pls resubmit and then will get it in.
BUG: map builtins min/max to numpy/cython versions
State: closed

closes https://github.com/pydata/pandas/issues/7722

```
-------------------------------------------------------------------------------
Test name                                    | head[ms] | base[ms] |  ratio   |
-------------------------------------------------------------------------------
groupby_agg_builtins2                        |  43.8553 | 2667.0587 |   0.0164 |
groupby_agg_builtins1                        |  11.9447 | 118.6290 |   0.1007 |
```

on master:

```
In [4]: np.random.seed(27182)

In [5]: n = 100000

In [6]: df = DataFrame(np.random.randint(1, n / 100, (n, 3)),
   ...:         columns=['jim', 'joe', 'jolie'])

In [7]: %timeit df.groupby('jim').agg([sum, min, max])
1 loops, best of 3: 204 ms per loop

In [8]: %timeit df.groupby(['jim', 'joe']).agg([sum, min, max])
1 loops, best of 3: 5.1 s per loop

In [9]: df.groupby(['jim']).apply(max).head()  # this is broken
Out[9]: 
jim
1      jolie
2      jolie
3      jolie
4      jolie
5      jolie
dtype: object
```

on branch:

```
In [9]: %timeit df.groupby('jim').agg([sum, min, max])
10 loops, best of 3: 20 ms per loop

In [10]: %timeit df.groupby(['jim', 'joe']).agg([sum, min, max])
10 loops, best of 3: 74.6 ms per loop

In [11]: df.groupby(['jim']).apply(max).head()
Out[11]: 
     jim  joe  jolie
jim                 
1      1  999    989
2      2  997    986
3      3  981    973
4      4  989    992
5      5  995    977
```


Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance

Comments:
pls add a vbench (your example is good)

added the benchs but cannot run them (i am on python 3)

can u add a test that confirm 9 above as well

merged via https://github.com/pydata/pandas/commit/6516c537dfa5db4b3cbe6dfc9a0a13c99ed21d82

thanks!
BUG: consistency between logical ops & type casts
State: closed

Closes [#6528](https://github.com/pydata/pandas/issues/6528).
Currently series logical operations are bound to return a series with `dtype='bool'`. That being a good decision or not, it is good to have some internal consistency:

```
>>> a = pd.Series([nan, nan, nan])
>>> b = pd.Series([False, True, nan])
>>> a | b == b | a  # not symmetric
0     True
1    False
2     True
dtype: bool
>>> a & b == a.astype(bool) & b.astype(bool)  # not consistent w/ type casts
0     True
1    False
2    False
dtype: bool
```

With this patch, [all these tests pass.](https://github.com/pydata/pandas/pull/8151/files#diff-933180870dd152b775952fc54a99dba7R2371)


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
need a release note

@behzadnouri can you rebase this. I believe there was another pr with modified this code.

this pr is based on the assumption that the element-wise logical operations are bound to return a series with `dtype=bool`. https://github.com/pydata/pandas/pull/9338 has changed this; so this way may not be the best way to go forward.

i feel like following python's mechanism of [boolean operations](https://docs.python.org/3/library/stdtypes.html#boolean-operations-and-or-not) is a better path. 

can u provide an example of what you think is wrong / what needs changing?

I mean element-wise and/or etc should actually do element-wise logical operations, so then `a & b` and `a | b` should be equivalent of 

```
>>> Series(map(lambda x, y: x and y, a, b))  # element-wise and
>>> Series(map(lambda x, y: x or y, a, b))  # element-wise or
```

for example:

```
>>> a = pd.Series(['a', '', np.nan])
>>> b = pd.Series([0, 1, 2])
>>> pd.Series(map(lambda x, y: x and y, a, b))
0    0
1
2    2
dtype: object
>>> pd.Series(map(lambda x, y: x or y, a, b))
0      a
1      1
2    NaN
dtype: object
```

https://github.com/pydata/pandas/pull/9338 is _sort of_ doing this for integers, but same can be done for all types as in python `"abc" and 3.14` is a valid expression.

@behzadnouri we ought to reexamine this for 0.17.0. pls rebase when you have a chance.

@jreback I close this one per my comment earlier that i no longer feel this is a good approach. I will do a new pr when i get a chance and if i could make it work.

great thanks!
BUG: use stable sort for group_index in groupby
State: closed

Following on https://github.com/pydata/pandas/pull/8128, for certain `groupby` operations such as, 

```
df.groupby(key)[col].transfrom('first')
```

it is necessary to use stable sort for group indices. As a result, `numpy.argsort` should be called with argument `kind='mergesort'`.


Labels:
name: Performance
description: Memory or execution speed performance

Comments:
can u put in a test to validate this (eg that fails w/o this fix)

I made two _dummy_ commits (will revert them later). Both _force_ the code to always use `argsort` by changing [this line](https://github.com/behzadnouri/pandas/commit/6d278bdaa89332c1ba55db9f4af5c3553d805310#diff-720d374f1a709d0075a1f0a02445cd65R3572) to `if False:`. The one which uses quick-sort fails the Travis build. The other one which uses a stable sort (merge-sort), passes the tests.

However, I was not able to make a test for the original commit, which goes through `argsort` path (i.e. multi-key with large number of keys combinations) and it generates wrong result if we do not specify `kind='mergesort'`. Whenever it _does_ use `argsort`, even with a quick-sort kind, the results looks valid to me.

My best guess is that it has something to do with the group indexer being compressed or not.

ok, need to come up with a test that fails with the original method (if you can). if you can't I don't have a problem changing this as `mergesort` is stable. but lmk.

I tried a lot but I could not make a test which fails with the original method. I am _almost_ convinced it would not fail, but it is safer to specify `mergesort`. Later if I have time, I would dig more into this.

ok, lmk when you are ready then (and rebase away those dummy commits). no hurry.

removed the dummy commits; travis links for the record:
[quick sort travid build](https://travis-ci.org/pydata/pandas/builds/33990455)
[merge sort travis build](https://travis-ci.org/pydata/pandas/builds/33991596)
GH8142
State: closed

closes #8142 


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design

Comments:
- needs tests (put in `tests/test_index.py`)
- release note in `v0.15.0.txt`
- don't merge in master, see git help here: https://github.com/pydata/pandas/wiki/Using-Git

@jasonzutty ?

let's close for now (pending #8226)
BUG: Fix bad axis limits when boxplotting with 'by' (#7528)
State: closed

closes #5517

The original bug involved the y-limits for boxplots being wrong when using `'by': turns out it has to do with the dummy axis that gets created when the plots don't fill the axis layout, e.g. when we're plotting 3 columns in a 2 by 2 layout.

The fix for this basically involves not setting the `sharey`/`sharex` argument for blank/dummy axes.

I've tried to test this in a couple of different ways, but some feedback on better ways to test would be good. In particular, I'm not sure if it's okay to check the `_sharey` attribute of the axes, given that it's private.


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
Example of a plot that previously failed:

``` python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

np.random.seed(100)

n_rows = 200
example_df = pd.DataFrame({
    'count1': np.random.exponential(50, n_rows).astype(int),
    'count2': np.random.exponential(100, n_rows).astype(int),
    'count3': np.random.exponential(20, n_rows).astype(int),
    'season': np.random.randint(1, 5, n_rows)
})
example_df['totalcount'] = example_df['count1'] + example_df['count2']

# Bad limits - grid not filled out
example_df.boxplot(['count1', 'count2', 'totalcount'], by='season')
plt.show()
```

![boxplot_axes](https://cloud.githubusercontent.com/assets/1460294/4068707/b14abeaa-2e3b-11e4-94ce-8ebb226d7adb.png)

What do you mean when you say it previously failed? This is what I get on master:

![master](https://cloud.githubusercontent.com/assets/1312546/4070242/d2843fee-2e57-11e4-9f67-baf018105408.png)

Oh wow, sorry, it took me a while to get around to this, I had the fix sitting there but had to do up the test cases. It's quite possible it was fixed somewhere else in the meantime but previously it was acting like was described in #7528 (specifically the second image there). If master is now giving the correct axis limits, maybe #7528 can just be closed.

Could also keep the test cases here in case it ever pops up again?

Current master still outputs problematic result using matplotlib 1.3.1. Thus it is nice to have a fix and explicit test for this.

This should close #5517 also. 

@onesandzeroes pls add a release not that closing #5517 as well. rebase / squash and good 2 go.

@jreback Added the release note, rebased and squashed. Should be all good.

@TomAugspurger merge when u r ok with this

Great, thanks!
DOC: Link to Python standard encodings from read_csv documentation
State: closed

Trying to make it easier to access the list of encodings from the documentation. Adding this was suggested on this [SO question](http://stackoverflow.com/questions/25530142/list-of-pandas-read-csv-encoding-list/25530279#25530279).


Labels:
name: Docs
name: Unicode
description: Unicode strings
name: IO CSV
description: read_csv, to_csv

Comments:
Thanks. Could you add it to the html documentation. That's under `doc/source/io.rst`.
I saw Jeff's comment on SO. I don't mind whether it's in the function docstring or not, but it would be a nice addition to the html docs.

@dxshank can you only have 1 link (the python 3 is good). pls rebase and ping when green

@jreback Sorry got sidetracked by some other stuff. Will rebase soon.

@jreback @TomAugspurger : PR updated to link to only one list of encodings.

ok, pls squash into a single commit and I think good 2 go

@jreback : Changes squashed into a single commit

gr8 thanks!
Made line comments work with delim_whitespace and custom line terminator
State: closed

This extends the functionality in #7582 to whitespace-delimited and custom line terminated parsing in the C reader (fixes #8115).


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
pls add a release note, then looks good

Ok, just added a release note

ok ping when green 

@jreback - Travis is passing.
BUG: broadcasting with numpy scalars (GH8116)
State: closed

closes https://github.com/pydata/pandas/issues/8116


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
always need tests and hasattr(obj,'ndim') is safer 

@jreback not sure what is the point of `hasattr(obj, 'ndim')`?

that line of code already verifies it is a `np.ndarray` and as a result it does have `ndim` attribute. it just verifies that `other.ndim != 0`.

Is there a case that we would need `hasattr(obj, 'ndim')` here?

looks fine, pls add a release note

added

ok thanks