BrandonFors/pandas-github-issues huggingface dataset:
BUG: Categorical.remove_categories(np.nan) fails when underlying dtype is float
State: closed

Fixes GH #10156. This also makes different null values indistinguishable inside of remove_categories, but they're already indistinguishable in most other contexts:

``` .python
>>> pd.Categorical([], categories=[np.nan, None])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "pandas/core/categorical.py", line 289, in __init__
    categories = self._validate_categories(categories)
  File "pandas/core/categorical.py", line 447, in _validate_categories
    raise ValueError('Categorical categories must be unique')
ValueError: Categorical categories must be unique
```


Labels:
name: Bug
name: Categorical
description: Categorical Data Type

Comments:
can you update

I think these tests are what you were asking for. Let me know if otherwise.

yep, ping when green.

It turns out that this already works for datetimelike categoricals. I added some tests and squashed.

awesome @evanpw ! keep em coming

Thanks very much for fixing this. I am not sure if I got this right but would not it make sense to reorder the code slightly to avoid trying removing categories twice when `nan` is present? So use this code:

```
    # GH 10156
    if any(isnull(removals)):
        not_included = [x for x in not_included if notnull(x)]
        new_categories = [x for x in new_categories if notnull(x)]
    else:
        removal_set = set(list(removals))
        not_included = removal_set - set(self._categories)
        new_categories = [ c for c in self._categories if c not in removal_set ]
```

Instead of this:

```
    removal_set = set(list(removals))
    not_included = removal_set - set(self._categories)
    new_categories = [ c for c in self._categories if c not in removal_set ]

    # GH 10156
    if any(isnull(removals)):
        not_included = [x for x in not_included if notnull(x)]
        new_categories = [x for x in new_categories if notnull(x)]
```

In your if branch, what's the original definition of not_included and new_categories? It looks like they're defined in terms of themselves.

Fair point ;)
Thanks again for the fix.
ENH: added rsplit to StringMethods
State: closed

as a part of https://github.com/pydata/pandas/issues/9111 

cc @sinhrks 


Labels:
name: API Design
name: Strings
description: String extension data type and string data

Comments:
Thanks! I think this is OK for 0.16.2 as a follow-up of string enh if there is no objections. Could you update:
- Release note
- `text.rst` and `api.rst`

@sinhrks it's updated, please take a look 

thanks!
DOC: #7828 Add json_normalize to api docs.
State: closed

closes #7828 

Is this all we need?


Labels:
name: Docs

Comments:
yep

thanks!

@lexual Can you put the `read_json` and `json_normalize` in two different `autosummary` directives? As I want that `read_json` is still displayed as being top level pandas pd.read_json (instead of pd.io.json.read_json)
BUG: plotting grouped_hist with a single row frame #10214
State: closed

closes #10214 


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
Thanks! I think it can be in 0.16.2 (being released before 0.17.0), if you move release note description from "0.17.0.txt" to "0.16.2.txt" locates on the same directory. If you have any trouble, please feel free to ask.

I did it https://github.com/scls19fr/pandas/tree/hist-ndim

see https://github.com/pydata/pandas/compare/master...scls19fr:hist-ndim

How should I submit my changes in same PR ? I wonder if committing in my branch also update this PR ?

I should fix https://github.com/pydata/pandas/issues/10214

pls see contribution guidelines [here](http://pandas.pydata.org/pandas-docs/stable/contributing.html)

to update, you simply push a new commit. don't open and close different PR's that is confusing.

if you squash/rebase, then `git push user-name branch-name -f`

pls squash.  @TomAugspurger ok with you?

Just gave a quick look. I think it's ok. 

I'm quite busy. Could you squash ? Thanks

merged via https://github.com/pydata/pandas/commit/a8b5798b31f04705bd421a2902c523f927eb6b7e

thanks!
ENH: #3335 Pivot table support for setting name of margins column.
State: closed

ref #3335.

Adds margin_column parameter to pivot_table so that user can set it to
something other than 'All'.
Raises ValueError exception if there is a conflict between the value of
margin_column and one of the other values appearing in the indices of
the pivot table.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
this is ok. in general, try to avoid reformatting other text unless its germane as its hard to see what you are changing. If you want to reformat, pls do that in a separate PR.

can you rebase / update

@lexual can you rebase / update

closing, but if you'd like to update, pls reopen

Hi, I've got some availability again to try and help out again.

I'll see if I can rebase this and try again.

Doesn't appear I can re-open though.

Maybe I have to open a brand new one ??
ENH: #2679 - DataFrame.to_html() urls_as_links parameter.
State: closed

New urls_as_links boolean paramater that will output urls as href html
links. ref #2679 

Thanks for @tdas14 for initial code.


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap

Comments:
@TomAugspurger (I recall you doing some work on conditional html formatting, though orthogonal to this affect API)

@lexual can you rebase

@lexual can you rebase / update

pls see comments on #9880 

IOW this needs to have a global option (that defaults to `False`)

closing, but if you would like to continue working on this pls reopen

I have rebased, but don't have permissions to re-open I believe.

Please advise next steps to get it merged.

Apologies for long absence ;)

https://github.com/lexual/pandas/commit/49992e9c5b5edfe9880d93466df4d448f3add5db

rebase @jreback 

@lexual this is more appropriate as a method in `df.style` for #10250 

e.g.

`df.style.show_urls_as_links()` or something. You want to do a PR against @TomAugspurger branch for that?

And the idea is that a style can be created that can be `.set` as a style (or a default), so you can simply 'use' a style as the default and build on it.

@lexual let me know if you want to do that PR agains my branch in #10250, it will actually be quite simple thanks to jinja.

Right now in the Jinja template I've got [a check](https://github.com/pydata/pandas/pull/10250/files#diff-871722e2eeea86e2eb7c981fa3748d4fR113) for if the value is a number or not. Jinja has a [urlize](http://jinja.pocoo.org/docs/dev/templates/#urlize) filter that you can apply to the else block.

We will need a more general way of applying arbitrary tags / modify the template, but that could work for now.

@TomAugspurger yep, I'll try to take a look when I get a chance.
BUG: TimedeltaIndex slicing may reset freq
State: closed

`DatetimeIndex` preserve freq after slicing.

```
di = pd.date_range('2001', '2005', freq='D')
di[1:5:2]
# DatetimeIndex(['2001-01-02', '2001-01-04'], dtype='datetime64[ns]', freq='2D', tz=None)
```

But `TimedeltaIndex` doesn't.

```
tdi = pd.timedelta_range('1day', '5day', freq='D')
tdi[1:5:2]
# TimedeltaIndex(['2 days', '4 days'], dtype='timedelta64[ns]', freq=None)
```


Labels:
name: Bug
name: Timedelta
description: Timedelta data type
name: Frequency
description: DateOffsets

Comments:
thanks you sir!
PERF: improved performance of multiindex slicing
State: closed

closes #10287

master 

```
In [22]: %timeit mdt2.loc[idx[test_A-eps_A:test_A+eps_A],:].loc[idx[:,test_B-eps_B:test_B+eps_B],:].loc[idx[:,:,test_C-eps_C:test_C+eps_C],:].loc[idx[:,:,:,test_D-eps_D:test_D+eps_D],:]
10 loops, best of 3: 141 ms per loop

In [23]: %timeit mdt2.loc[idx[test_A-eps_A:test_A+eps_A,test_B-eps_B:test_B+eps_B,test_C-eps_C:test_C+eps_C,test_D-eps_D:test_D+eps_D],:]
1 loops, best of 3: 4.23 s per loop
```

PR

```
# this actually is a bit faster in master. but this repeated chain indexing is frowned upon anyhow
In [22]: %timeit mdt2.loc[idx[test_A-eps_A:test_A+eps_A],:].loc[idx[:,test_B-eps_B:test_B+eps_B],:].loc[idx[:,:,test_C-eps_C:test_C+eps_C],:].loc[idx[:,:,:,test_D-eps_D:test_D+eps_D],:]
1 loops, best of 3: 210 ms per loop

# this is the prefered method (as you can set with this and such), and such a huge diff.
In [23]: %timeit mdt2.loc[idx[test_A-eps_A:test_A+eps_A,test_B-eps_B:test_B+eps_B,test_C-eps_C:test_C+eps_C,test_D-eps_D:test_D+eps_D],:]
1 loops, best of 3: 425 ms per loop
```

master

```
In [10]: %timeit s3.isin([1,2])
100 loops, best of 3: 8.83 ms per loop
```

PR

```
In [2]: s3 = Series(np.random.randint(1, 10, 100000)).astype('int64')

In [5]: %timeit s3.isin([1,2])
100 loops, best of 3: 2.47 ms per loop

```


Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
@sinhrks IIRC we had some discussion of the specialized `ismember_int64`. added here.

going to defer to 0.17.0 as 3.2 fails for some reason (and not going thru the trouble of creating a build environment); we are going to drop in #9118 . Its pretty hard to install 3.2.
Update v0.16.2.txt
State: closed

Added description of Pull Request 9544


Labels:

Comments:
Can you add this to the PR itself? Just commit this on the branch you are using there and push again.

Ok, because I had a old fork and I didn't know if I could push correctly.

No problem! If you have questions with git, just ask! (here or in our dev chat room https://gitter.im/pydata/pandas)

Thanks :)
TST: Check series names
State: closed

Related to #9972. Made `assert_series_equal` to check name attribute.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
lgtm. any things that we need to put to watch out for in the future? e.g. where you needed to do `check_names=False` but couldn't check the name?

As far as I know:
- `eval` using `numexpr` (#10239)
- `value_counts` (#10150)

But we may find further when adding `name` attribute to all `Series` in test cases.

@sinhrks but you are testing this no? (e.g. you added it and fixed what was necessary)

Ah, I meant the `name` attributes of existing test cases are all tested. But I didn't change test cases which uses `Series(..., name=None)` yet, and adding explicit name may reveal further problem. 

I'll continue to work on `Index` and `Series` metadata issues, hopefully change all test cases to have explicit name assignment like below (intends to be a fix for #10150):
- https://github.com/pydata/pandas/compare/master...sinhrks:base_name_handling

ahh, I see. cool. ok let's merge this. thanks!
TST: fix for bottleneck >= 1.0 nansum behavior, xref #9422
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
`bottleneck 1.0` just added to conda, so this is now failing master. This doesn not actually fix for the non-bottleneck installed case #9422, rather the tests for it has changed

@shoyer
BUG: GH10160 in  DataFrame construction from dict with datetime64 index
State: closed

closes #10160
closes #9456
 where `DataFrame/Series` construction from nested `dict` with `datetime64` index returns a `DataFrame\Series` of `NaN`s.


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
pls rebase on master and repush had an issue in the builds

ok, this overlaps with the issue in #9456 , currently with a PR of #9924 to fix

lets define a function someting like:

put in `pandas.core.common`

```
def _dict_compat(d):
    return { _maybe_box_datetimelike(key) : value for key, value in compat.iteritems(d) } 
```

then can use in both places.

@kawochen looks good so far. can you add in tests/fix for #9456 ?
i'd like to fix both in this PR (this PR can then supersede #9924)

Sure I will do that.

@kawochen any progress on this? (e.g. incorporating the Series fix as well)

Yes I did that a few days ago. I squashed the commits when I rebased (I'm new to github, maybe I shouldn't have?). 

merged via https://github.com/pydata/pandas/commit/821542fb1dfd55b0e0d25e55eb4fee6555d01cbd

thanks!

FYI, when you push again, I don't know you updated (e.g. there is no notiifcation on that, only on a new PR). So if you want me to have a look, ping!
ENH: allow as_blocks to take a copy argument (#9607)
State: closed

closes #9607

Added copy argument in as_blocks. Defaults to True as per the original code.

Wrote test cases to check the copy=True and copy=False cases work.


Labels:
name: API Design

Comments:
pls rebase on master and repush had an issue with some builds

sure. pls add a release note. I don't recall an issue related to this. so can use this PR number. Pls squash. ping when green.

merged via https://github.com/pydata/pandas/commit/8eabc4edc6beb4d4addff5dcb72eea476b3f8d40

thanks!
BUG: Should allow numeric mysql table/column names
State: closed

Closes #10255 , alternative to #10262 .


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Looks good! 
Only, seems it cannot be merged, can you rebase on the latest master? 
Will then merge when travis is green.

@jorisvandenbossche conflict in whatsnew doc... should be fixed now.

pls rebase on master and repush had an issue with some builds

@artemyk Thanks!
TST: test_sql: properly drop tables with names that need to be quoted
State: closed

Closes #10255 .

Rewrote some of the code in `test_sql.py` so that it now properly drops tables with names that need to be quoted.


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@artemyk Thanks! 

Can you add a small note in the 0.16.2 whatsnew bugfix section?

And for the rewriting of the tests, I was thinking it would maybe be better to keep this for a separate PR if you don't mind? As this would give rahter some conflicts with the connection PR, and I am already slow with reviewing that one. So I would like to first get that one in

@jorisvandenbossche OK, I will submit another PR.

@jorisvandenbossche Ping.  This makes the testing code a bit more robust by making sure all tables are dropped after every test.

Whoops, forgot this one. Looking at it now

Thanks!
BUG: SparseSeries constructor ignores input data name
State: closed

When constructing `Series` from `Series`, name attribute is preserved otherwise specified.

```
import pandas as pd
s = pd.Series([1, 2, 3], name='a')
pd.Series(s)
#0    1
#1    2
#2    3
# Name: a, dtype: int64

pd.Series(s, name='x')
#0    1
#1    2
#2    3
# Name: x, dtype: int64
```

But `SparseSeries` doesn't preserve its name.

```
s = pd.SparseSeries([1, 2, 3], name='a')
pd.SparseSeries(s)
#0    1
#1    2
#2    3
# dtype: int64
# BlockIndex
# Block locations: array([0], dtype=int32)
# Block lengths: array([3], dtype=int32)
```


Labels:
name: Bug
name: Sparse
description: Sparse Data Type

Comments:
lgtm, merge when green.

Thanks for review.
add numba example to enhancingperf.rst
State: closed

![enhancing performance pandas 0 16 1-81-g676cb95 documentation 1](https://cloud.githubusercontent.com/assets/953992/7959452/33d0364a-09c7-11e5-867c-b71bb27d085d.png)


Labels:
name: Docs
name: Performance
description: Memory or execution speed performance

Comments:
+1 !

With the latest numba, I think you can do the empty array allocation inside the jitted function, instead of passing it as an argument. I find this a bit easier to follow, but of course in this way it also works with older versions.

@jorisvandenbossche for some reason, sphinx wouldn't render the decorators in the ipython blocks (e.g. the `@jit`. very odd. (it would work with verbatim, but then you have to repeat the entire code in a suppress block to actually have it executed).

@jorisvandenbossche I put the allocation back. I think its ok. In the next version of numba, this will just be 'faster' (e.g. its all done in nopython mode). But seems to work ok.

Note that the version of Numba that will be able to do the memory allocation in nopython mode will be coming out today.

@seibert hmm, this is seeming to execute in nopython mode already though? (I guess except for the allocation, which prob doesn't matter in perf as its a single event), right?

It does not execute in nopython mode with current stable version of numba, but because of the loop lifting of the main part of the function, it does not make that much of a difference I think.

@seibert yes, I was referring to that version, but forgot that it was only a beta release announcement that I saw

Right, loop-lifting means that allocating memory in nopython mode is not as exciting as you might initially think.  :)

@jreback Could you replace the cython timings with a `:verbatim:`, too? Otherwise the numbers will get out of sync.

Generally this looks good to me

@shoyer updated to use all verbatims..
ENH: make sure return dtypes for nan funcs are consistent
State: closed

this is a follow-up to https://github.com/pydata/pandas/pull/10172

In addition to `mean()`, this PR also makes sure the returned dtype is consistent for `std()`, `var()`, `skew()`, and `kurt()`


Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
also included tests for `min()` and `max()`

thanks!
Datetime resolution coercion
State: closed

Fixes #10245 

Without this fix, if the `date_parse` function passed to e.g. `read_csv` outputs a `np.datetime64` array, then it must be of `ns` resolution. With this fix, all time resolutions may be used.


Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: IO CSV
description: read_csv, to_csv

Comments:
For some reason 706b8df fails Travis on my fork. I can't figure out why. The error doesn't seem to be related to my testing function (which was the only thing I changed, apart from importing `MultiIndex` at the top).

pull from master again and rebase. i fixed the issue.

Still failed Travis in my fork

oh, that was a generic message I sent out. You might have a specific issue then.

Then I really don't understand what's going on.
- 1549094 [passes](https://travis-ci.org/cmeeren/pandas/builds/65068780) (after fixing the issue and making the test)
- 706b8df [fails 1/5](https://travis-ci.org/cmeeren/pandas/builds/65374905) (after I updated my test) fails in a test I haven't touched 
- 3c348dd [fails 4/5](https://travis-ci.org/cmeeren/pandas/builds/65395818) (after rebasing) fails with another error

I'm at a loss here.

try `git rebase -i origin/master`
and see https://git-scm.com/book/en/v2/Git-Branching-Rebasing

There we are. Ended up adding pydata/pandas as a remote and rebasing this PR branch on top of that. Diff seems correct now.

Build passed, will this be merged? Or is anything else needed?

@cmeeren thanks for the nudge. I think this looks pretty much good now. All we need now is a release note in what's new in `0.16.2.txt`. You can put this under bug fixes.

Once you've done that, please squash to one commit.

Oh no, again the thing with the wrong commits being included on rebase. I'll rebase onto `upstream/master` and try again.

There we are.

thanks!

Oh, I thought you wanted the issue number in the "whatsnew" document. I hadn't pushed that yet. But you're the boss. :)

I forgot I asked you, and just did it. https://github.com/pydata/pandas/commit/0a428e7cc9915717eaf0b99c365561c847ea4ef0

thanks!

:+1:
BUG: GH10209 fix bug where date_format in to_csv is sometimes ignored
State: closed

closes #10209

When datetimes are not dates only i.e. have fractional day values, date_format passed to to_csv is ignored.  


Labels:
name: Output-Formatting
description: __repr__ of pandas objects, to_string

Comments:
This might be a silly question: Would I need to rebase after the PR passed the build test but has become stale? 

you need to move the release note to v0.16.2.txt

Ah. I see. Will do. Thanks!

merged via https://github.com/pydata/pandas/commit/282746ae3c05edf7eac5c9583a17aa53ea12b7da

thanks!
ENH: GH9746 DataFrame.unstack and Series.unstack now take fill_value …
State: closed

…kw for filling NaN when unstack results in a sparse DataFrame

closes #9746 


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: API Design

Comments:
This may interact with #9023

I can resolve conflicts where necessary, let me know when and how to proceed.

Reworked and simplified, and now just pass fill_value to _maybe_promote, which I believe is more correct.  As an effect of this I had to handle a None fill_value passed to _maybe_promote as this is the default of unstack().  As an alternative the default of unstack() could be fill_value=np.nan.

can you rebase/update according to comments.

@amcpherson only a couple of doc changes. pls ping when ready & green.

thanks for the patience.....i periodically cycle thru PR's and sometimes don't make it all the way thru....(before I start again) :)

can you run: `git diff master | flake8 --diff` and fix anything. going to shortly be enforcing PEP8.

can you rebase / update according to comments

Apologies for the delay, let me know how it looks now.

lmk when you have a chance to update

Some tests failing, checking now.

Nevermind, tests were failing due to out of date .so, all looks good.

@amcpherson thanks!
BUG: SparseSeries.abs() resets name
State: closed

```
# OK: name is preserved
s = pd.Series([1, 2, -3], name='a')
s.abs()
#0    1
#1    2
#2    3
# Name: a, dtype: int64

# NG: name is reset
s = pd.SparseSeries([1, 0, -3], name='a')
s.name
# 'a'

s.abs()
#0    1
#1    0
#2    3
# dtype: int64
# BlockIndex
# Block locations: array([0], dtype=int32)
# Block lengths: array([3], dtype=int32)
```


Labels:
name: Bug

Comments:
this might be more general in sparse (I mean that it's not propagating meta data ) 

Looked through other methods, and found others call `__finalize__` after `_constractor`. Thus hopefully OK. I'm working on `assert_series_equal` to check `Series` name. If any problem found, I'll prepare a separate fix. 

Changed `abs` to use `__finalize__` also.

lgtm

thanks!
BUG: plotting grouped_hist with a single row frame #10214
State: closed

Should closes https://github.com/pydata/pandas/issues/10214


Labels:
name: Bug
name: Visualization
description: plotting

Comments:
We had a couple stray commits slip in. Will cleanup later.

@scls19fr can you rebase on master. ping when green.

Sorry @jreback I don't feel very confortable with Git.

I did:

```
$ git rebase master

$ git pull

$ git push
```

What should I do ?

I would like to have my master branch sync with upstream.

Maybe this PR https://github.com/pydata/pandas/pull/10298 is better.
I deleted my repository. Fork again master. Create branch. Change 3 files and send PR.
Close mysql connection in TestXMySQL to prevent tests freezing
State: closed

The unit tests freeze on my machine without this change because you can't drop a table until other connections that are using that table disconnect.


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Thanks! 
Can confirm this fixes it (have to admit I almost never ran those mysql tests locally ..)

Will just wait on travis to merge when green.

Do you have the same issue locally before this change?

yes, it specifically hangs on the `test_execute_closed_connection` test for me. But strange that it runs on travis OK. I almost never ran those old tests (`TestXMySQL`) locally, but relied on travis for these.

(BTW, we should clean this up once, remove that entire class, but check if there is still something tested there that is not tested in the other tests)