BrandonFors/pandas-github-issues huggingface dataset:
Handle construction of string ExtensionArray from lists
State: closed

I had to add a string-based Arrow Extension array to trigger the bug but did not run the same test suite as we do on the boolean array as I don't see it adding value but just runtime at the moment.

- [x] closes #27673
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks.

I'm wondering a bit more general fix. What about something like

```python
# in sanitize_array. Somewhere before `data = extract_array(...)`

if not is_array_like(data) and is_extension_array_dtype(dtype):
    data = dtype.construct_array_type()._from_sequence(data, dtype=dtype, copy=copy)
```

Would that work? Then we get to the EA handling earlier on, and don't require special casing later on.
I moved the check slightly up but didn't move it significantly up as the above code could actually be taken by ExtensionArray data. If I run into other issues, I would revisit this if I run into other problems while workin on `fletcher` but moving it to a more broad scope at the moment seems to me a bit risky as `sanitize_array` is not directly tested but only indirectly via `Series` constructions.
OK, thanks.

@jbrockmendel can you take a look at this?
small comment, otherwise looks good
@jbrockmendel Adapted that but Travis keeps faling for unrelated reasons.
Rebased on master and everything is green now.
Thanks @xhochy !
PERF: Improve performance of cut with IntervalIndex bins
State: closed

- [X] closes #27668
- [X] benchmarks added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [X] whatsnew entry

ASV output:
```
       before           after         ratio
     [143bc34a]       [d531e7b0]
     <master>         <perf-cut-ii>
-            198M             122M     0.62  reshape.Cut.peakmem_cut_interval(1000)
-            197M             122M     0.62  reshape.Cut.peakmem_cut_interval(4)
-            197M             122M     0.62  reshape.Cut.peakmem_cut_interval(10)
-      2.11±0.02s          910±2ms     0.43  reshape.Cut.time_cut_interval(1000)
-      1.95±0.01s          761±2ms     0.39  reshape.Cut.time_cut_interval(4)
-      1.95±0.03s         763±10ms     0.39  reshape.Cut.time_cut_interval(10)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Interval
description: Interval data type

Comments:
very nice @jschendel
BUG: fix nested meta path bug (GH 27220)
State: closed

- [x] closes #27220 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

This PR attempts to solve the missing value problems when using nested meta with nested record path. 

The assumption is that we want to get the value of nested meta field even it is not on the record path. For example, in the example of GH 27220, ['info', 'governor'] and ['counties', 'name'] are not on the same path. The current behavior is to raise key error. I change the behavior to return the value of the key if we can find it in the json.

Let me know how we want to move forward with this. Thanks~

Labels:
name: Bug
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
@yanglinlee since we are dropping Python 3.5 very shortly, it may be best to skip the test on Python 3.5 by adding `@pytest.mark.skipif(not PY36, reason="...")`
Do I need to make more changes to this PR? Thanks for the reviews!
Fixed the merge conflict. Ping on green.
@TomAugspurger The checks have passed. Now it is green. 
@yanglinlee looks like a merge conflict and CI failure - mind taking a look?
@yanglinlee i think the CI failure for the docbuild is unrelated.  can you re-push and we'll see if its fixed
@yanglinlee we would take a patch like this if you want to update. closing as stale for now.
BUG: pd.crosstab not working when margin and normalize are set together
State: closed

- [ ] closes #27500 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
ok this looks good, can you merge master and fix the conflic. ping on green.
Thanks for your follow-up review @jreback !! I merged the master and resolved the conflict.
Yeah, this time it passes all checks. @jreback
DOC: add anonymizeIp for Google analytics in docs
State: closed

Using a more recent snippet (following their current docs), and adding anonymization of the IP (according to their docs, this should only "slightly reduce the accuracy of geolocation"). 

If we do this, should do the same for main website as well.

Labels:
name: Docs

Comments:
lgtm. if anyone @pandas-dev/pandas-core can give this a look see.
thanks @jorisvandenbossche
BUG: pd.crosstab not working when margin and normalize are set together
State: closed

- [ ] closes #27500 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Hello @charlesdong1991! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-07-30 18:51:43 UTC
BUG: Fix dir(interval_index)
State: closed

- [x] closes #27571
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
> I think the proper fix is to infer_dtype for letting it handle EAs correctly

Yah, the fix I'm working on locally makes infer_dtype handle interval correctly.  Getting all EAs right is out of scope.
thanks @jbrockmendel
improve warnings for Series.{real,imag}
State: closed

- [x] closes #27610
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Series
description: Series data structure

Comments:
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 0.25.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 143bc34aa8f068b18e7137df7ca91b9929cc1389
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #27651: improve warnings for Series.{real,imag}'
```

4. Push to a named branch :

```
git push YOURFORK 0.25.x:auto-backport-of-pr-27651-on-0.25.x
```

5. Create a PR against branch 0.25.x, I would have named this PR:

> "Backport PR #27651 on branch 0.25.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: Add mapping for pyqt for successful package installation
State: closed

- [ ] closes #26838 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
Merged master to fix the CI issue.

Ping on green.
ehh, how could I merge and resumbit the PR, the request of merge got rejected because there is a remote branch? shall i just do `git pull` and push to master? or do `git pull remote_branch_name`, but i dont know the remote branch name? I am not sure how to do it probably, and really would like to learn how ^^ @TomAugspurger 
Git pull, and then git merge with this remote branch. 

> On Aug 5, 2019, at 03:46, Kaiqi Dong <notifications@github.com> wrote:
> 
> ehh, how could I merge and resumbit the PR, the request of merge got rejected because there is a remote branch? shall i just do git pull and push to master? or do git pull remote_branch_name, but i dont know the remote branch name? I am not sure how to do it probably, and really would like to learn how ^^ @TomAugspurger
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or mute the thread.

thanks @TomAugspurger will try later!
this passed checks too ^^ @jreback 
any follow up?
Thanks @charlesdong1991!
DOC: Add Pandas-Bokeh to pandas ecosystem page
State: closed

This PR adds Pandas-Bokeh reference to Pandas ecosystem page. 

Best, Patrik

Labels:
name: Docs
name: Dependencies
description: Required and optional dependencies

Comments:
Docs look good.  I would merge / rebase with `master` to re-trigger the CI.
> Docs look good. I would merge / rebase with master to re-trigger the CI.

The docs and checks CI builds are passing, so a re-trigger is not needed I think.
Thanks @PatrikHlobil ! (and also thanks for working on the pandas-bokeh package!)
@jorisvandenbossche : Thanks for reviewing this PR so fast and also thanks for working on Pandas ;)
Backport PR #27631 on branch 0.25.x (BUG: raise when wrong level name is passed to "unstack")
State: closed

Backport PR #27631: BUG: raise when wrong level name is passed to "unstack"

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Error Reporting
description: Incorrect or improved errors from pandas
name: MultiIndex

Comments:
@jreback @TomAugspurger @toobaz: Is there a reason for back-porting this?  The bug that was being addressed was more than two years old.
The only reason I can think of is "this is a very simple patch". But definitely not urgent.
(Oh, and: over time, backporting patches might be easier if we do it routinely, so the code is more similar. But I have no idea of whether we have any general rule)
Yeah, simple bug fixes are (IMO) fine to backport. It's relatively painless
these days.

On Tue, Jul 30, 2019 at 5:57 AM Pietro Battiston <notifications@github.com>
wrote:

> (Oh, and: over time, backporting patches might be easier if we do it
> routinely, so the code is more similar. But I have no idea of whether we
> have any general rule)
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/27640?email_source=notifications&email_token=AAKAOIUPDATECJJK7YF2IH3QCANALA5CNFSM4IHULDA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3DS7SA#issuecomment-516370376>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIXN2K2QQYIV2L7EX43QCANALANCNFSM4IHULDAQ>
> .
>

yeah this was fine (and note already was for 0.25.1)
> Yeah, simple bug fixes are (IMO) fine to backport. It's relatively painless
these days.

But note that we should still be careful, and not just backport any bug fix. It happens regularly that a bug fix causes a new bug.
> But note that we should still be careful, and not just backport any bug fix. It happens regularly that a bug fix causes a new bug.

I agree, but in general should this be a stopper? I mean: do we consider regressions in 0.25.* more problematic than regressions in 1.*?
> do we consider regressions in 0.25.* more problematic than regressions in 1.*?

Yes, we do (IMO). People should more or less be able to carelessly update 0.25.0 to 0.25.1 while updating a x.y to x.y+1 always should be done carefully (as there can always be API breakages anyway).
> Yes, we do (IMO). People should more or less be able to carelessly update 0.25.0 to 0.25.1 while updating a x.y to x.y+1

Right, I missed that, I had just in mind the comparison between 0.25.0->0.25.1 and, say, 1.0.0->1.0.1.
DOC: Validate docstring directives
State: closed

- [X] closes #27629
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
For the moment, I have hardcoded the directives: 
- deprecated
- versionadded
- versionchanged

Nevertheless, I have noted there exist other directives used in pandas:
- code-block
- note
- plot
- warning

May I also include these?
Any reviewer? ;) @gfyoung @datapythonista Thanks.
All in GREEN (I had already fixed the docstrings #27621).
Thanks @albertvillanova !
Make interpolate_2d handle datetime64 correctly
State: closed

Broken off from #27626 because I decided that is poorly scoped.

Labels:
name: Datetime
description: Datetime data dtype
name: Algos
description: Non-arithmetic algos: value_counts, factorize, sorting, isin, clip, shift, diff

Comments:
Test added, unnecessary piece removed.  No release note needed since no user-facing changes.
> datetime64[ns]-dtype. datetime with TZ seems fine
> DataFrame.interpolate. Series[datetime64[ns]].interpolate seems fine

I'm not sure I understand your example.  Is it relevant that you passed tz to date_range?  You subsequently astype-d it away.

Best guess: DatetimeTZBlock.fillna dispatches to DatetimeArray.fillna, so doesn't go directly through here.
@jreback gentle ping.  Between this and the quantile PR, we're about to be about to be able to rip out a bunch of _try_coerce_result code
thanks
CLN: de-kludge Block.quantile
State: closed

Broken off from #27626 since I decided that is poorly scoped.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
@TomAugspurger i will look soonish. but I have comments on nearly all of these PRs so pls wait.
@jreback there's nothing urgent here, but you're gonna like the follow-up that comes after this and #27628.
CLN: de-kludge quantile, make interpolate_with_fill understand datetime64
State: closed

2 related-but-separate things here

-quantile: de-kludging the base class method and putting the datetimetz-specific stuff in a subclass method.  This will allow us to implement the refactor suggested in #14562.  If/when DTA supports 2D, we'll be able to get rid of the subclass override kludge altogether.

- interpolate_2d: have the core.missing function understand datetime64 dtypes so the blocks don't have to implement special handling.  There are only a couple other places where this is needed before we can get rid of the datetimetz-specific try_coerce_result altogether.

Labels:

Comments:
Closing in favor of #27628, #27627.
remove undesired values kwarg
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
thanks @jbrockmendel
BUG: cells are missing in the excel file when exporting excel using xlsxwriter with option constant_memory set to True (#15392)
State: closed

- [x] closes #15392
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
Hello @KenTsui! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-07-28 19:03:37 UTC
I think all of this logic should be contained within the `write_cells` method of pandas.io.excel._xlsxwriter.py - does moving it there make things easier?
> I think all of this logic should be contained within the `write_cells` method of pandas.io.excel._xlsxwriter.py - does moving it there make things easier?

It seems not quite possible as the sequence of generating cells, which is the crux of this bug, is not controlled by `write_cells`.

BTW, I am stuck on how to debug using azure pipelines.  Does anyone have some good resources to learn about that?
@KenTsui can you merge master? As far as log failures go if you click on next to failing logs you should be able to click through and find test failure output. Should match what you would see locally in a lot of cases
@KenTsui is this still active?
Closing as stale but ping if you'd like to pick back up
Fix AttributeError in scripts/validate_docstrings for doc.returns[0].name
State: closed

- [X] closes #27622
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
can you add some tests for this
What version of numpydoc is this? IIRC they have been using named tuples in more recent versions which is preferable from a readability perspective. Does upgrading resolve your issue?
@WillAyd, yes you are right: since numpydoc version 0.9, they use a Parameter named tuple (https://github.com/numpy/numpydoc/commit/136d4dd75207f529078c5c47dc4db53d0f38b4bf); I have v0.8.0 in my pandas dev environment.
I’d prefer to just bump our min version here for readability. Still pretty new (0.9 came out in April) but I think OK for a dev dependency.

> On Jul 27, 2019, at 11:51 AM, Albert Villanova del Moral <notifications@github.com> wrote:
> 
> @WillAyd <https://github.com/WillAyd>, yes you are right: since numpydoc version 0.9, they use a Parameter named tuple (numpy/numpydoc@136d4dd <https://github.com/numpy/numpydoc/commit/136d4dd75207f529078c5c47dc4db53d0f38b4bf>); I have v0.8.0 in my pandas dev environment.
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub <https://github.com/pandas-dev/pandas/pull/27623?email_source=notifications&email_token=AAEU4UM33HN2YFO5B7QNOGDQBSKK3A5CNFSM4IHJA6T2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD26QV4Y#issuecomment-515705587>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAEU4UMM73AGIPD7L2V5KK3QBSKK3ANCNFSM4IHJA6TQ>.
> 


> Still pretty new (0.9 came out in April) but I think OK for a dev dependency.

I'm in agreement on this.  We can be a little more aggressive for development dependencies.
We actually already do require this in environment.yaml per #26188 so I don't think anything to do here (unless we care to document further)
> I don't think anything to do here 

I'm agreement with this.

@albertvillanova : I'm closing this because the fix is just to upgrade to `numpydoc` per the new requirements on `master`.  Thanks for contributing though!  You are more than welcome to tackle any of our other open issues in our issue tracker.
Yes, I had noticed the version requirement in environment.yaml.

@gfyoung @WillAyd What about documenting how to update the dev environment in Contributing section?
When you say document, what would you add about upgrading?  It’s just doing a `pip` or `conda` upgrade, no?
You have thoroughly documented the creation of the dev environment using the environment.yaml file. However nothing is said about updating it (using the environment.yaml, which might have changed since the last time, with new packages or new version requirements). Nevertheless, if you think it is OK as it is... Nevermind!
> Nevertheless, if you think it is OK as it is... Nevermind!

Not saying that I think it's OK as is.  I'm just trying to understand what you wanted to do.

Why don't you submit a PR, and we can better evaluate.
Fix docstrings
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

This fixes some docstring formatting issues: e.g. some deprecated directives do not appear in current docs because the two colons are missing `.. deprecated 0.25.0`.

Labels:
name: Docs

Comments:
odd that the doc string checks don’t catch this
don’t mind fixing these - but should put in place in validate_docstrings.py

@datapythonista 
@datapythonista 
> It feels like sphinx should be raising warnings for those, but I guess both sphinx and our script consider those as just part of a paragraph.

2 dots followed by text (and no 2 colons) are valid reST _comments_, thus ignored by the parser: http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#comments

Thanks @albertvillanova, that's good to know. I think then we should make sure in `scripts/validate_docstrings.py` that we don't find "comments" that start with one of the keywords that we know won't be comments (deprecated, versionadded,...).

If you don't mind creating an issue for that, and referencing this PR in it, so we can see where it came from, that would be great.

And thanks for the PR!
Continue simplifying indexing code
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Clean

Comments:
thanks @jbrockmendel
DEPR: Deprecate NDFrame.filter
State: closed

- [ ] xref #26642
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

I propose to deprecate (DataFrame|Series).filter. The method's functionality is already in the ``.loc``method, except the regex functionality, for which I've made a proposal adding regex functionality  to ``.loc`` in #27363.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
@topper-123 can you rebase
can you rebase again.
I've rebased. As you noted, this depends on #27363.

@topper-123 im trying to clear the PR backlog; can this be closed pending #27363?
Knowing the outstanding issue is around regex capability, maybe we should just deprecate all keywords from filter for now that aren't regex?
That sounds like a stepping stone to deprecating. Wouldn't it make more sense then to just deprecate this method before 1.0 is released?
I don't *think* we'd want to outright deprecate without a replacement for the regex unless we instruct users to do `df[(re.search(x) for x in df.columns)]`
Oh yes, I meant choose another way to select by regex at the same time, sorry. 
can you merge master and will look again
I've rebased.

I do think an alternative should be settled on for the regex use case and I've had two proposals that were not accepted. So I'm not sure where this is floating currently?
Thanks for the PR! But I think agreement on approach here is still outstanding. If something you want to drive forward can revisit discussion in issue first