BrandonFors/pandas-github-issues huggingface dataset:
ENH: NaTType can be unpickled properly
State: closed

Closes #4606. 

I'm not sure this also be a solution for #5388, but seems to be a workaround.


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
@sinhrks ok...this is ok for #4606 (though do you really need that separate function `_nat_upickle`?

This 'solves' #5388 but not for the right reason. This is of course object type, but _can_ represented if transposed into 'correct' blocks.

I am not sure what to do about this. Multi-dtypes have to be inserted into a Panel on only certain axes otherwise it forces `object`.

So pls pull out the tests for #5388. Let's keep that as a separate issue.

```
In [36]: wp.to_frame()
Out[36]: 
                              i1                   i2
major minor                                          
0     A                        1                    1
      B      2010-01-01 00:00:00  2010-01-01 00:00:00
1     A                        2                    2

[3 rows x 2 columns]

In [37]: wp.to_frame().dtypes
Out[37]: 
i1    object
i2    object
dtype: object

In [38]: wp.transpose(2,0,1).to_frame().dtypes
Out[38]: 
A            object
B    datetime64[ns]
dtype: object

In [39]: wp.transpose(2,0,1).to_frame()
Out[39]: 
             A          B
major minor              
i1    0      1 2010-01-01
i2    0      1 2010-01-01

[2 rows x 2 columns]
```

merged via https://github.com/pydata/pandas/commit/2ada05408ea10050b0da825b716f6ba5c02f061b

thanks!

Thanks to merge.

> though do you really need that separate function `_nat_upickle`?

I believe so, because `pickle` module search the function under the module namespace and raise `PicklingError` if it cannot find it.

got it...thanks
BUG/API Raise ValueError for non-unique stack by name
State: closed

Closes https://github.com/pydata/pandas/issues/6729

We should raise a ValueError when (un)stacking and referring to a non-unique index _name_. You can still stack and unstack by position like normal (no ambiguity there, unless your names are integers and that integer is duplicated, in which case we'll raise a ValueError).

Right now I'm assuming that index names are always hashable, but that's not necessarily true, is it? Any good ways to check the uniqueness of the names? I didn't see anything on MultiIndex that did this.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@TomAugspurger my expectation would be that index names should be hashable. They're supposed to be key-esque, right? (plus tuples are still hashable). If you needed to, I'd catch the TypeError from trying to hash non-hashable object and just warn in that case.

@TomAugspurger pls rebase....seems ok otherwise

@TomAugspurger this closes #6319 as well?

This only closes #6319 if we don't want to support pivots/crosstabs where the names are the same. But we should be able to support them. I can do a proper fix for #6319 for .14 probably. 

I haven't merged this because my pushes to the remote branch aren't showing up here. All I need to do is rebase, so I may just do that as I'm merging. Should just be release notes / v0.14 that have conflicts. Sound OK?

how so you push to the remote?
do u have it set as a tracking branch?

Somewhere along the way I created two branches: `origin:unstack-nonunique` and `origin:origin/unstack-nonunique`. This PR is form `origin:origin/unstack-nonunique`, but when I push to that the changes don't appear.

Making a new PR at #6849
ENH: drop function now has errors keyword for non-existing column handling
State: closed

Closes #5300.

Currently `drop` raises `ValueError` when non-existing label is passed. I think it is useful if `drop` has an option to suppress error and drop existing labels only.

For example, I sometimes process lots of files which has slightly different columns, and want to `drop` if data has unnecessary columns. Previously, I have to prepare different `drop` arguments checking columns existing each data. 


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: API Design

Comments:
this could be added on after #6599 
@hayd 

This all looks fine to me - thanks for the patch.

We might want to reduce the number of test cases (just because we don't necessarily need to duplicate), but aside from that, quite good. - somebody else want to take a look too? I may be a bit rusty :P

Looks all good to me. But let's wait for #6599. :)

yep....so @hayd will revisit after that #6599 merge

@sinhrks this idea is ok, can you rebase

@jreback Yes, rebased.

have to think about this

@sinhrks can you rebase this.

Sure, rebased.

@sinhrks thanks!
ENH: SQL multiindex support
State: closed

Further work on #6292.

This adds:
- fixes the `columns` argument (wasn't passed) + added test for that
- adds multi-index support in `to_sql`, by just using `reset_index` on the frame (+ tests)
- adds multi-index support in read_table (+ tests)


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
I think you should raise if the multi index doesn't have named levels
I do that in hdfstore - it can work but it's relying in the level naming scheme and so a bit unexpected
better to force the user to name levels

@jreback Only for multi-index when it has no names? And for a single index leave it? (so 'index' is used in that case)

@jreback the tests fail on `assertListEqual` not known on Travis, but this did work on my local machine. Any idea?

I use assertEqual for lists - though assertLisstEqual sounds better

And the strange thing is, it only broke on Travis for one of the builds, while the sql tests are at least also tested in one of the other builds.

Aha, assertListEqual is new in Python 2.7. Just use assertEqual then?

look at HDFStore.Table.validate_mulitindex

I do allow none levels (but then convert them back when reading)

only problem was levels overlapping with column names

Pushed new version with slightly other approach.

But some questions on the API:
- should we allow a MultiIndex without level names? (suggested by @jreback to raise and force user to provide `index_labels`). Now they get the default names `level_0`, `level_1`, ... like in `reset_index`.
- if we detect columns with names like `level_0` or `index`, should we set them as index when reading the data? (personally I would say we leave this to the user, as we can never be fully sure)

@mangecoeur @hayd 

@jorisvandenbossche 

since SQL cannot really store meta data, its a bit non-trivial to automatically figure out the `index_col` whether it is `index` or `level_n`. Bigger problem is that a user will not know how to specify the index columns in the first place.

Maybe best not to allow non-named levels at all?

@mangecoeur Would you have some time to review this? (as you wrote most of this code)

Since there isn't more feedback, I am going to merge this to not hold up further PRs. But off course, if there still feedback, glad to hear it! It can always be adapted afterwards.

@jreback I am going to take the same way as `reset_index`, so allowing non-named index levels. That seems the most consistent with rest of pandas. And also not trying to guess what the index was when reading the data back in from sql.

The only problem I still have is the legacy mode. I didn't add multi-index support to legacy `to_sql`, so the support to write the index is only half baked. It is also new functionality since 0.13.1, so actually an API change for the legacy mode. But I will open a seperate issue for that.

@jorisvandenbossche all fine

FYI if u reset and get an error I think in HDFStore I raise
that - meaning reset dailies because of duplicate index/columns

@jreback Ah, yes, good idea, I catched the exception and expanded the error message.
BUG: MPLPlot cannot make loglog keyword worked
State: closed

Changes made in #5638 make `loglog` keyword not worked, and it fixes the problem.


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
looks good to me - thanks for submitting this! Doesn't need a doc note b/c both are within 0.14

@sinhrks could you add a note to the docstrings that marks the existence of `loglog`? e.g. around line 1881 in `pandas/tool/plotting.py` after the comment descibing `logy`. That would help other users actually know about this.

Thank you for your confirmation. Docstring added, and I removed descriptions specifying line plot because now log-related keyword can be used in all plots thanks to #5638.

@sinhrks thanks!
BUG: Bug in consistency of groupby aggregation when passing a custom function (GH6715)
State: closed

closes #6715


Labels:
name: Bug
name: Groupby

Comments:
@jorisvandenbossche works but is a big hacky

Thanks! Will test it this evening.

For `g.agg(P1)` it does work, but `g.apply(P1)` not yet (or this way maybe not meant to fix that?). It just seems intuitive if you can do `df.apply(P1`, that you also can do `g.apply(P1)`

`g.apply(P1)` is correct as it is passed as a full dataframe (and not series by series). I don't see a way to avoid doing that. The paths groupby takes are pretty complicated. So not going to be able to fix. If you want to create a separeate issue maybe can address in 0.15.
DEPR: Deprecate cols in several functions
State: closed

closes #6680 -- Deprecate `cols` in `drop_duplicates()` and `duplicated()` (rename to `subset`)
closes #6645 -- Deprecate `cols` in `to_csv()` and `to_excel()` (rename to `columns`)
releated to #6686 
Related #5505 -- Deprecate `cols` and `rows` in `pivot_table()` and `crosstab()` (rename to `columns` and `index`)

Related #6581 -- Reminder to come back in 0.16, remove deprecation warnings, and use new arguments


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
maybe should refactor the code that does this cols deprecation check to a function (and also need to change the ones u did already, eg pivot and such)

maybe stick it in pandas.util.misc or something (their is a decorator deprecate already in pandas.util.decorators but not sure if that would work or maybe it would)

Made a new decorator that deprecates a keyword argument of a function and implemented on the 6 functions in question.

Thanks for catching the bone-headed overlook of not using the decorator on to_csv and to_excel

looks good

@jorisvandenbossche ?

There should be some wrapping of the function in order to keep its name and docstring.

wouldn't it make sense to add a version to the decorator, so that this information can be printed as well and maybe make this fail in case some env variable is present (TRAVIS) and pandas version is higher (automatic failures on version update :-) )

@JanSchulz why would you need a version in this - its already versioned by definition as its in master. 

Even though we have created an issue to take this out for good (say in 0.16), that might get changed until its actually removed. Their are some very old deprecations thatreally do need to get taken out. 

I mean something like this: `deprecate_kwarg(old_arg_name="old", new_arg_name="new", version="0.5")` will output "old is deprecated, use new instead (old will be removed in version 0.5)" and tests using old will fail on travis if pd.**version** is higher than 0.5.

can't do it because what happens when that 0.5 changes to 0.6 (eg we didn't remove it when we said but later)

it's not always clear how long to deprecate and when to remove

in theory you are right but practically it's a problem

Added wraps decorator

merged via https://github.com/pydata/pandas/commit/599156fe06ca7f4bc7f87fca982f561f0ec675a9

thanks!
ENH/VIS: Dataframe bar plot can now handle align keyword properly
State: closed

Closes #4525.

I modified the `BarPlot` internal alignment to meet line coordinates. Previously, `BarPlot` doesn't pass `align` keyword to matplotlib (thus matplotlib uses `align='edge'`), but it adjusts bar locations to looks like `align='center'`.

Now `BarPlot` pass `align=center` to matplotlib by default and ticks are being locates on integer value starts from 0 (0.0, 1.0 ...). Drawing bar and line on the same axes looks like below.

_Output using current master:_
![align_incorrect](https://f.cloud.github.com/assets/1696302/2492234/0df22b26-b21c-11e3-8a96-95cffa20a7c6.png)

_Output after fix_
![align_correct](https://f.cloud.github.com/assets/1696302/2492238/5f5bc26a-b21c-11e3-9b70-76892cea9eca.png)


Labels:
name: Enhancement
name: Visualization
description: plotting

Comments:
Thanks for this.

We may want a little warning in the release notes that this will change the look of the default output. I'm onto sure how guarantee plots being identical from one version to another.

You have a few [magic numbers](http://en.wikipedia.org/wiki/Magic_number_%28programming%29#Unnamed_numerical_constants) in the `_check_bar_alignment` function, like in `tickoffset = width * (position - 0.5) + p.get_width() * 1.5`. Are the .5 and 1.5 just constants that matplotlib uses?

I had a couple of notes inline, but overall this looks good. If you look at those and rebase I'll get it merged. Thanks.

Thanks for your confirmation. I added decorators / warning description in what's new txt, and rebased.

Fixed numbers in `_check_bar_alignment` is derived from `BarPlot` class logic (line 1690 of plotting.py). 
- `width * (position - 0.5)` derived from `position` 's default (0.5). Because `BarPlot` now specifies the center-bottom coordinate of bar, no adjustment should be done when `position` is 0.5. Thus, it should be substituted.
- `p.get_width() * 1.5` is derived from previous version's logic. `BarPlot` has draw non-stacked plot starting from `(i +1) * width` location (line 1766 of plotting.py). Now we have to specify the bar location, it should be changed to `(i + 1.5) * width`. Thus, to get the same offset as the previous logic, I used 1.5 to get the offset.

OK. I have understood to write a warning about backward-compatibility in release note, but it should be in plotting.rst, or simply removed? I think usage is not necessary because the `align` keyword works as the same as matplotlib.

I'll correct my typo once confirmed where the description should be (or removed).

ok so move both the release note and the v0.14.0 note to the API section - change from an actual warning block to just text in the note (no need for a separate warning block) 

Sure. Descriptions are under API Changes section, and corrected typo and warning block.

ok by me... @TomAugspurger if ok, then go ahead and merge

Looks good. Thanks
DEPR: deprecate cols / change to columns in to_csv / to_excel
State: closed

Fix #6645


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
I think there are also some occurences in the docs:

```
>>>-------------------------------------------------------------------------
Warning in /tmp/doc/source/comparison_with_r.rst at block ending on line 175
Specify :okwarning: as an option in the ipython:: block to suppress this message
----------------------------------------------------------------------------
/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pandas/tools/pivot.py:82: FutureWarning: cols is deprecated, use columns

warnings.warn("cols is deprecated, use columns", FutureWarning)
<<<-------------------------------------------------------------------------

>>>-------------------------------------------------------------------------
Warning in /tmp/doc/source/comparison_with_r.rst at block ending on line 407
Specify :okwarning: as an option in the ipython:: block to suppress this message
----------------------------------------------------------------------------
/home/travis/virtualenv/python2.7_with_system_site_packages/local/lib/python2.7/site-packages/pandas/tools/pivot.py:73: FutureWarning: rows is deprecated, use index

warnings.warn("rows is deprecated, use index", FutureWarning)
<<<-------------------------------------------------------------------------
```

Otherwise looking OK!

ah, it is still from the pivot/pivot_table PR, but still needs to be updated.

Btw - you can also explicitly set the warnings module run to raise on
warnings...
On Mar 22, 2014 1:04 AM, "Joris Van den Bossche" notifications@github.com
wrote:

> ah, it is still from the pivot/pivot_table PR, but still needs to be
> updated.
> 
> ## 
> 
> Reply to this email directly or view it on GitHubhttps://github.com/pydata/pandas/pull/6686#issuecomment-38345685
> .

Fixed two noted issues in docs.  @jorisvandenbossche I couldn't find the result you pasted in any of the test jobs; how did you find that result?

You would get the same if you build the docs yourself, but Travis also builds the docs, very handy! In the log of Travis, almost at the bottom, you have `$ if [ -f /tmp/doc.log ]; then cat /tmp/doc.log; fi`, where you get the output of the docs builds (you have to look at travis job 3, with `DOC_BUILD=true`)

@jsexauer And you still missed 1 occurence of  `cols` and 2 of `rows` in comparison_with_r.rst (the docs only give one warning, because always only the first one is warned for)

@jsexauer looks fine....pls create an issue to implement the removal of this code (reference this issue); i'll tag and put in 0.16.

@jorisvandenbossche @jtratner ok with this?

@jsexauer #6680 has your name on it too!

Just amended #6581, figured best keep it all in one ticket when we have to remove it all :)

Closing issue.  Going to fix in #6693 instead.
BUG: NaN values not converted to Stata missing values (GH6684)
State: closed

closes #6684

Stata does not correctly handle NaNs, and so these must be replaced with Stata
missing values (. by default).  The fix checks floating point columns for nan
and replaces these with the Stata numeric code for (.).

The write_index option was also being ignored by omission. This has been fixed and
numerous tests which were not correct have been fixed.


Labels:
name: Bug
name: IO Data
description: IO issues that don't fit into a more specific label
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
Fix for #6684 

is the missing value indicator 'standard' or something the user would want to set? (e.g. you should it be a paramater with a default?)

There are many missing value indicators ranging from . (dot), .a, .b, ..., .z.    In principle these allow the reason for the missing value to be encoded (e.g. .a for non-response, .b for negative value, etc),  In theory these could be set, and I thought about how to do this once, but I think it probably isn't a feature that would have much use since pandas doesn't really have a good method to handle different types of missing values.  

The idea I had at the time would require a passing a second DataFrame with some additional information about the missing data codes to use.  Not worth the effort IMO.

@bashtage totally fine..

looks good
can u add a release note?

@jreback Assuming this passes, then this should do it.  I looked at coverage and added a few more tests for some corner cases and removed a small amount of unreachable/unnecessary code.

@bashtage can you rebase and push again

Rebased.  

Fixed a dictionary comprehension, so I think this is finished.

thank you sir!

@bashtage see issue #5781 if you have a chance.

I think these reading/writing needs to handle endianess properly (most machines are little, but big does exist!)

I don't have reasonable access to a big endian machine.   I would rate this as pretty low priority since Stata does not support big endian platforms (any more).

turns out was pretty trivial, closed by #7272
PERF: Quick Shift Implementation (GH5609)
State: closed

closes #5609 
## Test code

```
import pandas as pd
import numpy as np
import timeit

xdim = 10000
ydim = 500
repeats = 10
df = pd.DataFrame(np.random.rand(xdim,ydim))

# axis 0 shift
s = timeit.default_timer()
for i in range(repeats):
    df1 = df.shift(1,axis=0)
e = timeit.default_timer()
print "Axis 0 shift",e-s

# axis 1 shift
s = timeit.default_timer()
for i in range(repeats):
    df1 = df.shift(1,axis=1)
e = timeit.default_timer()
print "Axis 1 shift",e-s
```
## Results

```
V0.13.1

Axis 0 shift 0.958936203491
Axis 1 shift: IndexError: index 500 is out of bounds for size 500

----------------------------------------------------------------------------------
V0.13.1-iss5609 branch

Axis 0 shift 0.1622
Axis 1 shift 0.3466 

```


Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
looks good....

if you feel comfortable squashing go ahead

otherwise you can ping me when its green

I have squashed my commits. 

looks good...ping me when green

Good to go!

@gouthambs thanks!

next time will be easier!
BUG: Seed RNG in test_partially_invalid_plot_data
State: closed

Came up upon merging another PR. See comment https://github.com/pydata/pandas/pull/6644#issuecomment-38057195

The test relies on drawing at least 1 draw out of 10 uniform draw to be larger than 0.5; even if this didn't cause the failure originally, it should still be fixed. I set a seed that's known to produce the desired values.


Labels:

Comments:
gr8! nice to remove the randomness..

I think this might set some global state (which is prob ok), maybe do at the top of the file?
(I think you can also do this in a `with`)

I didn't see a context manager for the seed, so I defined one in `pandas/utils/testing`.

Do we need a release note? This is mostly internal.

no release note needed...go ahead when green

OK. Does anything need to be done on https://github.com/pydata/pandas/pull/6644? Retrigger the build to check that this actually was the bug?

EDIT: I guess this build will include everything from #6644, so it should be OK?

yep you started with master so if this passes then ok
Add layout support in plot_frame similar to the hist_frame layout
State: closed

Add an optional layout keyword argument in order to set the subplots layout. 
If none provided, falls back to default behavior.


Labels:
name: Enhancement
name: Visualization
description: plotting

Comments:
- This needs a couple of tests which test passing an invalid layout.
- Pls add a mention in release notes (refereing this PR #)

You will also have to expose this in `plot_frame` (https://github.com/pascal-schetelat/pandas/blob/master/pandas/tools/plotting.py#L1827) to make it usable in `df.plot()`

@jreback I duplicated the same tests on the layout as in hist_frame in _get_layout. Do you think additional tests are required?

@jorisvandenbossche you mean adding an entry in the doc or explicitely passing a layout argument in plot_frame? 

@pascal-schetelat Yes, at least adding it to the docs. 
I also meant to explicitly pass it as a argument, but I see that there are also others that are passed through the `*kwds`, so I am not really sure about this what is best.

@pascal-schetelat this will close #6306, yes?

@pascal-schetelat pls rebase and push again, your changes are not showing up here

I think the shape of returned axes should be cared. `hist` returns ndarray which has the same shape as the specified layout, not flattened. Personally whichever possible if there is a policy.