BrandonFors/pandas-github-issues huggingface dataset:
BUG: fix HDFStore iterator to handle a where properly (GH8014)
State: closed

closes #8014


Labels:
name: Bug
name: IO HDF5
description: read_hdf, HDFStore

Comments:
cc @bboerner 
give this a try

@jreback, the fix looks good, thanks!

@bboerner ok this is fixed for a partial selection case.

note when you put up a fix, it should be on top of the current PR and only show the changes. e.g. just create whatever the changes are; these are local to you and don't modify anything else (unless they are pulled in).

@jreback, thanks I verified the fix. And for the explanation about submitting fixes.

Several thoughts:
1. I think the `current = current + self.chunksize` should revert back to above the check for `if value is None` else if None the loop continues w/o updating `current`.
2.  `current = current + self.chunksize` could be  `current = stop` - same effect and IMHO clearer.
3. As currently (and previously in the PR) implemented this can return one (or possibly many more) empty DataFrames through the generator. `concat([df for df in it])` handles that case but it doesn't seem natural to get back a large number of empty DataFrames from an iterator. That could be handled by modifying the check to be `if value is None or value.empty:` Thoughts? I can put together a test case if you want.

Regards.

2) easy

1) and 3) do you have something that breaks this?

`None` should never actually be returned (it will raise if something bad happens or return an empty frame), never `None`  (I may have had that for some reason but don't remember).

I don't think this _can_ return a list of empty frames, but just a list of a single empty frame

e.g. your selection is completely invalid, then you will get a single empty frame. (but should test this).

updated

actually I think its correct to yield empties if the query doesn't select anything (you have to remember the columns are returned correct). Otherwise this is odd I think. You should get back something for each iteration and empty is valid.

Thanks.
1. I encountered the empty DataFrames a while back selecting ~10K rows from a file containing 1.74e7 rows and got back 1 non-empty DataFrame and 173 emptys. That was unexpected and not intuitive.
2. More troubling is that it breaks with Python idiom e.g.

```
for e in []:
    print True
```

never prints True so I suggest that for a where clause which selects no rows that:

```
for df in store.select('df',where=where,iterator=True):
    print True
```

should also not print True.

Commit bboerner@7fedf5ba06511c20eafd356a5478922bcb91048e contains the test cases illustrating.

If you'd like me to open this as a seperate issue let me know.

Regards.

ok, updated. (changed 1 item in your test which seemed wrong, see my comment).

Thanks, look good. Is there a performance preference for `not len(value):` vs. `value.empty`?

Regards.

doesn't make any difference
this is all dwarfed by the actual operation of selection

Thanks.
BUG: Area plot legend has incorrect color
State: closed

Derived from #7636. Area plot sets incorrect `alpha` values to legend when `stacked=True`.
#### NG: Each areas are drawn with `alpha=1.0`, but legend has `alpha=0.5`

![figure_ng](https://cloud.githubusercontent.com/assets/1696302/3919509/643924e0-23a9-11e4-983d-23a16e9c01bc.png)
#### After fix

```
df = pd.DataFrame(np.random.rand(20, 5), columns=['A', 'B', 'C', 'D', 'E'])
df.plot(kind='area')
```

![figure_1](https://cloud.githubusercontent.com/assets/1696302/3919515/8b4ecf4e-23a9-11e4-926b-85101760882a.png)

```
# When alpha is specified
df.plot(kind='area', alpha=0.2)
```

![figure_2](https://cloud.githubusercontent.com/assets/1696302/3919516/8fc2c166-23a9-11e4-8ea6-b14479d6a72b.png)


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
this is a partial close of #7636? 

No. This is separate from original issue described in #7636 (actually #7636 is ambiguous for me), but the one which @szeitlin is pointed in #7636.

Thus I put this issue's no in the release note.
Fixed incorrect datatype conversion on multi-indexes
State: closed

When creating a new table from a `pandas` dataframe with a multi-index, the code would iterate over the index columns backwards but use their types as if counting forwards.  This meant that the created table had incorrect indexes for multi-indexes.

This code factors out a function to pull column names and column types for both SQLAlchemy and legacy table objects.  This code now iterates over the indices properly.

And additional test has been written.  All test pass on my local machine.

Closes https://github.com/pydata/pandas/issues/8021 .


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@artemyk Looks good, thanks a lot! Only some comments on the test.

You are totally right --- I didn't realize that while the write/read test would fail currently, I thought it would return both columns as having `float` dtype both pre- and post-patch.  But in fact, post-patch the dtypes are correct, while pre-patch they are both `float`.  So the test is simpler and now applies both to SQLAlchemy and legacy APIs (it is under `_TestSQLApi`).  Let me know if there is anything else to fix.

I think this may be ready to go.

@artemyk needs:
-  rebase on master (so you will pick up the changed test functions)
- a release note in v0.15.0 (bug fix?)
- a squash

@jreback Sorry, this is actually my first pull request ever so I'm not exactly sure what I need to do now.
I inserted a line in release notes.  Regarding rebasing on master, I actually cherry-picked the commit with the changed test functions, so that change is included.  Do I still need to rebase & squash?

The change in the test function is already merged in master, so if you fetch pandas master (`git fetch upstream`) it will already be included, and you certainly don't need to cherry-pick it. So try:

```
git fetch upstream
git rebase upstream/master
```

And then, with squashing we mean to combine the different update commits all into one for this PR, you can do this with `git rebase -i HEAD~6`, and then squash the necessary commits.

Just ask if something is not clear!

see here: https://github.com/pydata/pandas/wiki/Using-Git

@jorisvandenbossche instructions are good

@jreback Thanks greatly for the instructions.  I followed them however, when I did `git push` I got a "Updates were rejected because the tip of your current branch is behind its remote counterpart" error.  So I did a `git push -f`.  Does this commit look correct?
I notice that the changes include those I made to `assert_frame_equal` -- that probably doesn't belong, right?

yes, you need to use `-f`.

pls squash as well

so

```
git rebase -i origin/master
```

then 's' all of the commits (ecept the first)

then repush with the '-f'

@jreback Thanks for the help.  
This should be squashed now.

looks good to me, @jorisvandenbossche ball in your court

Tests passed, so merging. 
Thanks a lot @artemyk, excellent first (or second now) PR! 
(and more is always welcome if you want :-))
fix issue #8000 - interpolation extrapolates over trailing missing values
State: closed

This pull request is in response to [issue #8000](https://github.com/pydata/pandas/issues/8000).

Changes to core/common.py add np.nan as the default value for missing values to the left and right non-missing values during interpolation. This prevents DataFrame.interpolate() from extrapolating the last non-missing value over all trailing missing values (the default).

Changes to tests/test_generic.py add test coverage to the above change. A passing test is where an interpolated series with a trailing missing value maintains that trailing missing value after interpolation.


Labels:
name: Bug
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
@grahamjeffries see my comment https://github.com/pydata/pandas/issues/8000#issuecomment-55135768

Any interest in doing a fix? I don't think we need to add `left` and `right` kwds to `.interpolate()` like `np.interp` has. We already have `.fillna` which is better for these things. I'd say we can add a keyword like `extrapolate` or `extend` or something like that to control whether things get filled on.

Right now we can just keep it as `extrapolate=True` (for compatibility), but warn that it may change in the future and recommend that people set it to false and `fillna` after interpolating.

I can take this if you aren't able/interested.

@TomAugspurger, I'd like to try my hand at this fix. It's my first though so I'll ask for your patience and close review. I'll get to this sometime next week, I suspect

No rush at all. This has been "broken" for at least a year :)

@grahamjeffries want to revist this?

@grahamjeffries can you revisit this?

I won't be able to get to this for at least a month. If there's someone else willing and able to make the fix, I'd encourage them to do so. Otherwise, I'll make an effort at that point.

closing as stale, but if you would like to reopen and fixup pls do.

@jreback What is missing from this PR to get it merged? It seems like a bare minimum:
- add `extrapolate` argument that defaults to `False`. Potentially warn if the user does not set.

you could add an `extrapolate` kw, and set it to `None` for now (which will do nothing ATM, and not warn). Then in 0.19 could do a warning if its not set. So not much really to fix this.
WIP: generalize categorical to N-dimensions
State: closed


Labels:

Comments:
closing until I make more progress

Maybe put in the `_finalize...` thingies which are used in pandas/numpy? Each method would need a change to call finalize then on new categoricals...

@shoyer so you want an n-dim codes array (rather than 1-d). How would this be used?
conceptually this is the 'same' as a DataFrame of categorical series (that maybe share levels, for 2-d). What is your usecase for this (aside from xray)?

@jreback I expect that allowing for n-dimensional categoricals could lead to much higher performance for a DataFrame with multiple columns of the same type, in the same way that multi-dimensional arrays are positive for performance for other dtypes.

e.g., you could call `unstack` on a Categorical series essentially for free (since it's just a numpy reshape under the covers). (though I'm not sure enough about the pandas Block system to be sure about this)

@shoyer ok, that is reasonable. And in fact right now `Categoricals` are kept in completely separate blocks regardless of their internal structure. It _is_ possible to combine them, so your soln would make sense.
fix issue #8000
State: closed

This pull request is in response to [issue #8000](https://github.com/pydata/pandas/issues/8000). 

Changes to core/common.py add np.nan as the default value for missing values to the left and right non-missing values during interpolation. This prevents DataFrame.interpolate() from extrapolating the last non-missing value over all trailing missing values (the default).  

Changes to tests/test_common.py add test coverage to the above change. A passing test is where an interpolated series with a trailing missing value maintains that trailing missing value after interpolation.


Labels:

Comments:
FYI, in general to simply rebase the PR rather than submit a new one
Add sharex and sharey to grouped boxplot
State: closed

sharex and sharey should be settable and by default False in a grouped boxplot. E.g. 

```
n [97]: df = DataFrame(np.random.rand(10,2), columns=['Col1', 'Col2'] )

In [98]: df['X'] =['A','A','A','A','A','B','B','B','B','B']

In [99]: df.iloc[0:5, :2] += 10000

In [100]: df.groupby('X').boxplot()  # this is unreadable
```


Labels:
name: Visualization
description: plotting

Comments:
@danfrankj this needs tests. ok with changing to match `DataFrame.hist`

@jreback  not exactly sure what a unit test for this kind of option would look like. 

take a look here: https://github.com/pydata/pandas/blob/master/pandas/tests/test_graphics.py

you need to test for the correctness, e.g. you say sharey=..., the returned axes then should be shared, etc.

closing pls reopen if/when updated

`boxplot` is being replaced to `plot(kind='box')`, and hopefully #8018 can cover grouped cases.
CLN/DOC/TST: categorical fixups (GH7768)
State: closed

replaces #7768 

closes #3678

Categorical([1, np.nan]) would end up with a single `1.` float level.
This commit ensures that if `values` is a list of ints and contains np.nan,
the float conversation does not take place.

Categorical: fix describe with np.nan

Categorical: ensure that one can assign np.nan

Categorical: fix assigning NaN if NaN in levels

API: change default Categorical.from_codes() to ordered=False

In the normal constructor `ordered=True` is only assumed if the levels
are given or the values are sortable (which is most of the cases), but
in `from_codes(...)` we can't asssume this so the default should be
`False`.

Categorical: add some links to Categorical in the other docs

Categorical: use s.values when calling private methods

s.values is the underlying Categorical object, s.cat will be changed
to only expose the API methods/properties.

Categorical: Change series.cat to only expose the API

Categorical: Fix order and na_position

Categorical: Fix comparison of Categoricals and Series|Categorical|np.array

Categorical can only be comapred to another Categorical with the same levels
and the same ordering or to a scalar value.

If the Categorical has no order defined (cat.ordered == False), only equal
(and not equal) are defined.

DOC: use okexcept in docs rather than try: except:

CLN: revised isnull treatement

TST: test for tab completion


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Categorical
description: Categorical Data Type

Comments:
closing in favor of new one from @JanSchulz
API: consistency in .loc indexing when no values are found in a list-like indexer GH7999)
State: closed

closes #7999


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design
name: MultiIndex

Comments:
@jorisvandenbossche this also fixes the case we talked about (forgot where), in a multiindex when no values are found it will now raise KeyError (I think it would return the original previously)

@jreback Did you also fix the inconsistency between `df.loc[[1, 3]]` (works as one label is in the index) and `df.loc[[1, 3],:]` (fails with KeyError)? (as I don't see it mentioned in the whatsnew entry, and they should both work)

Secondly, you now solve the inconsistency between `df.loc[[3]]` and `df.loc[[3], :]` in favor of the behaviour of  `df.loc[[3], :]`. But is there a good reason? We could also opt for the behaviour of the single axis (`df.loc[[3]]`), which is maybe(?) more used ànd is in line with `reindex` (+ see above, this is rather a bug in the `df.loc[list, :]` case as it definitely fails with if there is at least 1 label found in the index)

yes, this fixed `df.loc[[1,3]]` and `df.loc[[1,3],:]` issue as well (it was the same issue):

In [1]: df = DataFrame([['a'],['b']],index=[1,2])

In [2]: df
Out[2]: 
   0
1  a
2  b

```
In [3]: df.loc[[1, 3]]
Out[3]: 
     0
1    a
3  NaN

In [4]: df.loc[[1, 3],:]
Out[4]: 
     0
1    a
3  NaN
```

The reason I think that the indexer works liek this is: say you have a big list of labels you are getting. It is better to have it work then raise if 1 is off (and just reindex).

Conversely if nothing is found user prob screwed up so `KeyError`.

This PR doesn't really change much, just makes things consistent (wether you specify all axes or not).

@jreback yes, that makes sense. The only 'pity' is that it is a deviation from reindex, because if you say "`df.loc` with a list is just like reindex" you have to add "apart from the case if is of length 1 ..."

For `df.loc[[1,3],:]` do you add that do the whatsnew entry?

cc @ruidc 

going to do a do modification? (to include the un-documented prior behavior)

@jorisvandenbossche yeah I know this is different than `reindex`, but not sure what to do about it.

So, for the whatsnew: just something like (after the other one): `There was also a difference between ``df.loc[[1,3]]`` (returns a frame reindexed by [1, 3]) and ``df.loc[[1, 3],:]``(would raise ``KeyError`` prior to 0.15.0). Both will now return a reindexed frame.`

ok, sure

ok, updated

just a typo: line 1254 of indexing.py:
raise KeyError("Not of [%s] are in the [%s]" %
should be "Not ALL of [%s]"...
ENH/CLN: add BoxPlot class inheriting MPLPlot
State: closed

Like #7809, added `DataFrame.plot(kind='box')` to plot boxplot under the common plotting method.

**NOTE:** `DataFrame.boxplot` remains untouched (few lines are modified to share some definitions).
**NOTE:** This is based on #7736.

```
import pandas as pd
import numpy as np
pd.options.display.mpl_style = 'default'
df = pd.DataFrame(np.random.randn(100, 5))
ax = df.plot(kind='box')
```

![figure_1](https://cloud.githubusercontent.com/assets/1696302/3890938/2cc60f44-2225-11e4-8d79-4633c5ac7b40.png)

One enhancement is easier colorization by passing `color` keyword. You can pass `dict` with keys are `boxes`, `whiskers`, `medians` and `caps`. If some keys are missing in the `dict`, default colors are used for the corresponding artists. Also, boxplot has `sym` keyword to specify fliers style (This is mpl keyword and simply forward to mpl).

```
color = dict(boxes='DarkGreen', whiskers='DarkOrange',
             medians='DarkBlue', caps='Gray')
ax = df.plot(kind='box', color=color)
```

![figure_2](https://cloud.githubusercontent.com/assets/1696302/3890940/310b00e6-2225-11e4-8120-4c7fd7ffa697.png)


Labels:
name: Visualization
description: plotting

Comments:
hmm, any way to get rid of `DataFrame.boxplot` and replace with this? (I think you did that with hist yes?)

`DataFrame.boxplot` and `hist` still remain. This is because `DataFrame.plot` doesn't cover  `GroupBy` cases yet.

My idea to replace `boxplot` is enhance `DataFrame.plot` to support `GroupBy` in consistent way. Then, `plotting.scatter`, `DataFrame.hist` and `DataFrame.boxplot` can be deprecated (hopefully by 0.15). 

@TomAugspurger can you have a look

@sinhrks want to rebase now that I merged #7736 

Or actually you may not need to since they're separate commits. I'll look.

@jreback Do we really want to get rid of `DataFrame.boxplot()` et al? Because this is really used a lot .. But, for this PR a bit a seperate discussion, as the questions doesn't hinder to improve the `.plot(kind='box')` 

@jorisvandenbossche I don't think this touches `DataFrame.boxplot()`; it should really just (not sure if this PR does it), make it an alias for `df.plot(kind='box')` that's it really.

I think @sinhrks changes `df.plot(kind='hist')` to do basically the same thing (but not touch the API)

@jreback yes, but I interpreted your "get rid of DataFrame.boxplot" as deprecating it in the API, not as just making this an alias for plt(kind='box')

@sinhrks What are, as for now with this PR, still the differences between `df.boxplot()` and `df.plot(kind='box')`? (because looking at the code, `df.boxplot()` is not yet simply a call to `df.plot(kind='box')`)

Unable to replace existing boxplot now, reason is described above.

I think there are some options to "get rid of" `boxplot`. Making alias or completely remove after some span... It should be discussed separately including `hist` and `scatter` also.

OK, thanks @sinhrks.

FYI, build failed: https://travis-ci.org/pydata/pandas/builds/33461054

Looking into it. Doesn't seem related to this one.
COMPAT: change pytables to use 3.0.0 API (GH7990)
State: closed

closes #7990 


Labels:
name: IO HDF5
description: read_hdf, HDFStore
name: CI
description: Continuous Integration
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
nice: - is bigger than +

that's a first!

@cpcloud @jorisvandenbossche comments?

minor comments. otherwise looks good

Would this be annoying then? look at the tests

https://travis-ci.org/jreback/pandas/jobs/32325575

I think it is a useful and informative message, unless you don't bother to use numexpr (and it is eg installed in your distribution) ... then it can become annoying (but that is always like this with warnings no?). 
I just looked, I have eg still numexpr 2.0.1 (installed a time ago with anaconda), and I suppose I am not alone (but with conda of course easy to upgrade).

What is reason for upgrading the required versions? Is it only testing? (that we have to test less versions?) Because both PyTables 3 and Numexpr 2.1 are only 1 to 1.5 year old.

@jorisvandenbossche yes < 2.1 is pretty old at this point.

The reason for the changes is motiviated by using conda for ci, as it doesn't support these (pretty old) versions.

And we have to do this at some point before 3.2 PyTables is out (maybe end of year). As the api will start showing a different kind of warning. (it shows DeprecationWarning on all tests atm, but we are suppressing this).

I don't think this is really a big deal. 99% of users dumped 2.4 as soon as 3.0.0 came out because it supports py3 (and had a lot of bug fixes). (and its backward compatible).

@jreback OK, sounds reasonable

@jreback I got the warning :-) 
But I now got: `"UserWarning: The installed version of numexpr 2.0.1 is not supported in pandas and will be not be used"` -> maybe add what version it should be at least?

hmm, was 2.0.0, now its 2.1, ok easy to fix

https://github.com/pydata/pandas/commit/2594bea462dcfa55eaaf76ed1b3e922443a81f82

Thanks!
Fix DataFrame.to_latex() midrule positioning with MultiIndex columns
State: closed

Currently, the positioning of \midrule is determined by the number of index
levels, not columns levels:

```
>>> print pd.DataFrame({('x', 'y'): ['a']}).to_latex()
\begin{tabular}{ll}
\toprule
{} &  x \\
\midrule
{} &  y \\
0 &  a \\
\bottomrule
\end{tabular}
```

The fix is simple: use the number of column levels instead of the number
of index levels.


Labels:
name: Bug
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
looks fine to me.

@jseabold @jorisvandenbossche any objections?

Seems simple and good fix!

BTW, I noticed in your test that a multi-index in the rows is just put in one column with a space. Wouldn't this be better put in two seperate columns? (but that is for another issue/pr)

@jorisvandenbossche Yes, I agree. Multiple columns (instead of a space) for a multi-index index would be better (but not in this PR).

With MultiIndex index should there be & separating the levels? http://stackoverflow.com/q/25734454/1240268

Update: I see @jorisvandenbossche commented on that very thing!

The linked PR fixes this for MI index.