BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fix bug, where BooleanDtype columns are converted to Int64
State: closed

- [x] closes #32287
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
@AnnaDaglis do you think you'll have time to update this soon? We're hoping to get 1.0.2 out in the next couple days. If not, no problem, we'll just move it to the next release.
> Should this entire check be inside a big if not is_extension_array_dtype(input_array.dtype)? There aren't any cases where we want to convert from extension type to extension type, right?

Yes, right now, if the dtype is already an extension dtype, we don't want to change it (in the future, if there is eg a new nullable timestamp dtype, we might want to do this, but that is not yet relevant right now)
Ha @AnnaDaglis I was just pushing the same changes here. Looks like we ran into the same issue with the Series.astype failing to copy.

Mind if I push some commits here? I think I tracked it down.
I might end up just force pushing to your branch if that's OK. It'll mess some things up but will be easiest I think. 
@TomAugspurger No problem!
OK thanks! If you want to sync up, you can `git fetch origin` and then `git reset --hard origin/fix_convert_dtypes_bool` when you have that branch checked out locally. Hopefully we're good to go.

@jbrockmendel on master `Series.astype()` wasn't always copying for datetime64 dtypes. Those changes are in 934147b492d265cfd798c5288821f665c82a7a7c.

nice catch
thanks @AnnaDaglis very nice!
@meeseksdev backport 1.0.x
@meeseeksdev backport 1.0.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.0.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 a2acd1b2c662e6ae87e923989df7c0c95444f6d7
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #32490: BUG: Fix bug, where BooleanDtype columns are converted to Int64'
```

4. Push to a named branch :

```
git push YOURFORK 1.0.x:auto-backport-of-pr-32490-on-1.0.x
```

5. Create a PR against branch 1.0.x, I would have named this PR:

> "Backport PR #32490 on branch 1.0.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Huh, this seemingly wasn't backported, but the release notes are there... I'll do something manually.
CLN: trying isort-dev
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

---

Trying ```isort``` 5.0.0 which is still under development.

Labels:
name: Clean

Comments:
-0.5 on the cython edits here.  I'm fine with the non-cython edits.

Detecting unused imports in cython would be nice.
Closing this as this meant to be a pilot, to see if we will have problems with ```isort 5.0.0```, I will open a similar PR trying isort master, after they will do some changes.
CLN: Remove redundant tests for .duplicated and .drop_duplicates in tests/base
State: closed

part of #23877
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

duplicated and drop_duplicates are already thoroughly tested in `tests/indexes` and `tests/series`. I added comments to highlight the redundancy and extended the existing test cases where needed.

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Same comment as #32484 are you sure this doesn't lose any coverage?
@WillAyd I added comments highlighting and linking the redundant tests.

Where needed I extended the existing tests in `tests/indexes/test_common.py` or `tests/series/methods/test_drop_duplicates.py`
~~can you rebase, ping on green.~~
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>
@jreback CI is green
thanks @SaturnFromTitan 

there might be other opportunities to use the `keep` fixture in the codebase as well.
CLN: Replace isinstance(foo, Class) with isinstance(foo, ABCClass)
State: closed

- [x] ref #27353
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Nice idea but I don't think worth changing this. Looks to be compat code which I assume we will get rid of soon anyway - @TomAugspurger may know when
Agreed with @WillAyd.  I expect the failing tests are because this puts an unexpected name into the top-level namespace.
Closing for now since I don't think there is a desire for this one, but thanks anyway @MomIsBestFriend
CLN: Remove redundant index test from tests/base/test_ops.py
State: closed

part of #23877
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Boolean indexing is already thoroughly tested in `test_series_mask_boolean` in `tests/indexes/test_na_indexing.py` - [see here](https://github.com/pandas-dev/pandas/blob/master/pandas/tests/indexing/test_na_indexing.py#L7-L61)

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
Good point @WillAyd, I'll check that later. I was assuming that if we lose coverage, then I'd get a codecov warning here in the PR. Isn't there a bot for these sort of things?
We do have codecov but it’s not generally very insightful (at least for us)

Sent from my iPhone

> On Mar 7, 2020, at 1:24 AM, Martin Winkel <notifications@github.com> wrote:
> 
> ﻿
> Good point @WillAyd, I'll check that later. I was assuming that if we lose coverage, then I'd get a codecov warning here in the PR. Isn't there a bot for these sort of things?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub, or unsubscribe.

@WillAyd I think this is ready for another review now. I added an `indexer_class` parametrization to `test_series_mask_boolean` and refactored the test a little bit (which I mostly had to do to get it passing)

looks ok, will have to look again after @WillAyd comments are addressed.
@jreback I addressed all comments of @WillAyd (added the lost test cases to the parametrization of `test_series_mask_boolean`)
thanks @SaturnFromTitan
CLN: Split and fixturized test_fillna in tests/base/test_ops.py
State: closed

part of #23877
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@MomIsBestFriend Seems like that didn't work for the `isinstance(obj, ABCIndex)` cases because...
```
isinstance(pd.Float64Index([]), ABCIndex) is False
isinstance(pd.IntervalIndex([]), ABCIndex) is False
isinstance(pd.CategoricalIndex([]), ABCIndex) is False
...
```

I'll revert it to a simple `isinstance(obj, Index)`
thanks @SaturnFromTitan 

ideally if you can add the nulls fixture would be great :->
CLN: imports in pandas/io/excel/_base.py
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Clean

Comments:
lgtm - not sure on doc failure. Try merging master?
Thanks @MomIsBestFriend
BUG: iloc.__setitem__ with duplicate columns
State: closed

- [x] closes #15686
- [x] closes #22036
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
updated+green
thanks!
CLN: use _values_for_argsort for join_non_unique, join_monotonic
State: closed

With the `.copy()` removed from `Categorical._values_for_argsort`, `ea_backed_index._data._values_for_argsort()` matches `ea_backed_index._ndarray_values` in all extant cases.

cc @jorisvandenbossche @TomAugspurger need to confirm 

a) this is an intended-adjacent use of _values_for_argsort, and not just a coincidence that it matches extant behavior
b) the `.copy()` this removes from `Categorical._values_for_argsort` is not important for some un-tested reason

xref #32452, #32426

Labels:
name: Clean
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
I believe the only requirement on values_for_argsort is that it's a monotonic transformation (it preserves the ordering).

I'm not sure why that copy was there.
> I believe the only requirement on values_for_argsort is that it's a monotonic transformation (it preserves the ordering).

It isn't clear to me what distinguishes this from values_for_factorize, as the docstring there also says `An array suitable for factorization. This should maintain order and be a supported dtype`.  (the "supported dtype" bit isnt in the values_for_argsort docstring, but seems implied since it returns an ndarray)

This also came up in #30673 (attempt to implement value_counts in terms of EA methods).  See also #32412.

Do we know of any 3rd-party EAs where _ndarray_values, values_for_argsort() and _values_for_factorize()[0] are meaningfully distinct?
are there any user facing things that this now allows? e.g. joins on EA?
> are there any user facing things that this now allows? e.g. joins on EA?

behavior is unchanged
thanks, pls followon with consoilidations when you can
My *feeling* says that this should use `_values_for_factorize`, as joining is a factorize-based algo?
> My feeling says that this should use _values_for_factorize, as joining is a factorize-based algo?

I think you're right, but ATM _ndarray_values matches values_for_argsort for all our Index-backing EAs, but values_for_factorize()[0] is slightly different for DTA/TDA (_data vs asi8).  If we determine that we can change DTA/TDA _values_for_factorize (possibly as part of the discussion in #32586) then ill switch over these usages.
TST: Fixed xfail for tests in pandas/tests/tseries/offsets/test_offsets_properties.py
State: closed

- [x] ref https://github.com/pandas-dev/pandas/pull/32459#issuecomment-595277125
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
```mypy``` is failing?
looks like the build logs have expired; can you rebase and we'll see if this is ready
Couple of comments, looks nearly there
```
=================================== FAILURES ===================================
_______________________ test_apply_index_implementations _______________________
[gw1] linux -- Python 3.6.10 /home/vsts/miniconda3/envs/pandas-dev/bin/python

    @given(gen_yqm_offset, gen_date_range)
>   def test_apply_index_implementations(offset, rng):

pandas/tests/tseries/offsets/test_offsets_properties.py:99: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/core/indexes/datetimes.py:974: in date_range
    **kwargs,
pandas/core/arrays/datetimes.py:412: in _generate_range
    index.asi8, tz, ambiguous=ambiguous, nonexistent=nonexistent
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   raise pytz.NonExistentTimeError(stamp)
E   pytz.exceptions.NonExistentTimeError: 1991-03-31 00:04:11.080387

pandas/_libs/tslibs/tzconversion.pyx:276: NonExistentTimeError
```
Is this still active? 
closing as stale; if you can continue pls ping / open a new PR.
Disallow .__call__() as workaround for non-named functions
State: closed

Currently this script:
```python
import pandas as pd
funcs = [lambda: 1]
pd.eval("funcs[0]()")
```

Fails with:

    TypeError: Only named functions are supported

however this can easily be worked around by adding `.__call__`:

```python
pd.eval("funcs[0].__call__()")
```

I'm assuming we don't want to allow this workaround. This PR ensures that it will fail with the same error.

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
It's not about lambdas. In this context "named" means a variable name or an attribute - pandas doesn't allow calling arbitrary expressions. For example `funcs[0]` is a subscript expression so it can't be called. 
@jreback ping!
Hello @alexmojaki! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-05-10 17:52:29 UTC
@jreback pong!
thanks @alexmojaki
CLN: Removed unused variable
State: closed

I can't see the reason for ```res_view``` to exist, the CI should tell me if I am wrong.

Also this is removing one warning when compiling ```pandas/_libs/internals.pyx```

This is the warning that is removing:

```
pandas/_libs/internals.c:8952:36: warning: ‘__pyx_t_23’ may be used uninitialized in this function [-Wmaybe-uninitialized]
 8952 |         __pyx_cur_scope->__pyx_t_9 = __pyx_t_23;
```

Labels:

Comments:
@MomIsBestFriend we have a rather large queue atm so closing this one since I don't think would go forward with it anyway, but if you see a way to get the warnings out of this function would love a separate PR for that
VIZ: Fixing Matplotlib depecation warnings for ax.rowNum and ax.colNum
State: closed

- [X] xref #32444

Fixing the deprecation warning in the docs (testing if we really need to be compatible with the old syntax)

Labels:
name: Docs
name: Visualization
description: plotting
name: CI
description: Continuous Integration

Comments:
Compatibility is indeed needed (`get_suplotspec` exist in old versions, but not its attributes `rowspan`...).
DEPS: Removing snappy from local/docs dependencies
State: closed

- [X] xref #32417

The docs build should fail if snappy is required to build the docs, and should point out where it's being used. If it's not failed, I think this can be merged and we can remove snappy from the local and docs dependencies.


Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
The parquet examples require snappy (so "the docs build should fail if snappy is required to build the docs" -> this is happening).

Snappy is not a hard requirement, but it would need some code edits to make the examples pass (specify a compression option manually). Personally I would just keep snappy in the env.
(sorry, missed the discussion in the issue)
Removing snappy from the local dependencies make the docs build fail (when calling `to_parquet` by default snappy is used, and we have that in the docs, including to show benchmarks). Closing this, as the main point was to test that, More details, and the discussion about options in the issue.
TST/VIZ: add test for legend colors for DataFrame with duplicate column labels #11136
State: closed

- [X] closes #11136
- [X] tests added / passed
- [X] passes `black pandas`
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Visualization
description: plotting

Comments:
I have just made the changes. Can you check if is everything right now? Really thanks!
Fix were made, the for syntax is better now and now im using methods from matplotlib like get_color() and get_legend().
Great thanks @gabrielvf1
CI: Fix flaky test_value_counts_null
State: closed

As mentioned by @simonjayhawkins in #32438

Trace of the failing test:
https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=29965&view=logs&j=077026cf-93c0-54aa-45e0-9996ba75f6f7&t=e95cf409-86ae-5b4d-6c5f-79395ef75e8f

Labels:
name: CI
description: Continuous Integration

Comments:
@WillAyd @simonjayhawkins I added a follow-up issue and will come back to it later. Otherwise I think I addressed all comments, so this should be good to merge
Thanks @SaturnFromTitan 
Generally fixes for flaky tests should be backported. Does the 1.0.x branch suffer from the original flakiness?
no. #32281 was not backported.
Great, thanks.
CI: ax.rowNum and ax.colNum attributes deprecated in Matplotlib 3.2
State: closed

https://github.com/pandas-dev/pandas/pull/32442/checks?check_run_id=485569958

Labels:
name: CI
description: Continuous Integration

Comments:
@WillAyd not sure about this. should we just be silencing the deprecation warnings instead. `The rowNum attribute was deprecated in Matplotlib 3.2 and will be removed two minor releases later. Use ax.get_subplotspec().rowspan.start instead.`
On a phone so haven’t looked deeply but does swapping out the attribute access for what the message says not work?
> On a phone so haven’t looked deeply but does swapping out the attribute access for what the message says not work?

it doesn't work on older versions. hopefully latest commit fixes but is more complex (may not be relevant).

The alternative is silence the warnings and fix at a later date when the attributes have been deprecated.
> The alternative is silence the warnings and fix at a later date when the attributes have been deprecated.

assuming our minimum supported version has 'SubplotSpec' object has attribute 'rowspan' by then.
hmm, appears new matplotlib related errors on Linux py37_locale.
right. fixed warnings in web and docs, but for some reason have two new test failures. have skipped so we can get to ci to green, raise an issue, and resolve independently
@WillAyd I'm not happy with this but I think ok to merge. I think silencing the warnings may be a better approach, but sphinx is not my forte, so I leave that for others to investigate.
@simonjayhawkins how about just capping matplotlib for now to get CI green then can come back to this at a deeper level (?)
> There is a typo.

nice catch. Thanks @datapythonista
> And probably worth adding a test

ok as a follow-up if this is green?
> ok as a follow-up if this is green?

Sure, the docs are being our tests already. Just thought that would be good to see this warning failing the tests. But happy to see the CI fixed asap.
> But happy to see the CI fixed asap.

yeah, if this fails, will go with @WillAyd suggestion, https://github.com/pandas-dev/pandas/pull/32444#issuecomment-594967595
CI: fix test_matplotlib_scatter_datetime64
State: closed

- [ ] closes #32440
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration

Comments:
@WillAyd This should be good to go. the web and docs is in a separate PR.
Thanks @simonjayhawkins
DOC: Fix capitalization of the word pandas in the docs
State: closed

- [x] closes #32316 
- [x] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

As stated in the referred issue, _pandas_ reference in the docs should be standardised to pandas, not *pandas* or Pandas.

I've only changed the references in `doc/source/development/*.rst` for now, to see if the changes are okay or not

Labels:
name: Docs
name: Clean

Comments:
@datapythonista , sure I'll add the checks and update the required changes, I should add the checks in a separate PR right? 
> @datapythonista , sure I'll add the checks and update the required changes, I should add the checks in a separate PR right?

You can add them here or in a separate PR, as you prefer. Not sure how many cases will still fail, if there are lots, probably better to open a new PR, so the PR is not so big.
@joybhallaa do you have time to add the checks and fix the CI?
Yes!
Thanks @joybhallaa
CI: mypy fixup for #32261
State: closed

https://github.com/pandas-dev/pandas/runs/485121000

Labels:
name: CI
description: Continuous Integration
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Thanks for looking at the CI failures. Looks like you have a few PRs - can you advise on the priority of the,?
> Thanks for looking at the CI failures. Looks like you have a few PRs - can you advise on the priority of the,?

hold on this for a minute. 
need to check Windows py36_np15 
> need to check Windows py36_np15

looks like test_value_counts_null is still flakey, xref #32281 cc @SaturnFromTitan 
darn.. I'll have a look
Can you point me to a stacktrace for further investigation @simonjayhawkins?
@SaturnFromTitan https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=29965&view=logs&j=077026cf-93c0-54aa-45e0-9996ba75f6f7&t=e95cf409-86ae-5b4d-6c5f-79395ef75e8f
@WillAyd I think this is ok to merge now.
Thanks @simonjayhawkins
DOC: Fix EX02 in pandas.Index.get_loc
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Related to #27977. 

```
################################################################################
################################## Validation ##################################
################################################################################

5 Errors found:
        No extended summary found
        Parameter "key" has no description
        The first line of the Returns section should contain only the type, unless multiple values are being returned
        Return value has no description
        See Also section not found
```

Labels:
name: Docs

Comments:
Also, adding 
```
xref #27977
```
to the pull request description would be good
> Thanks @farhanreynaldo
> 
> In the future I think it would help to also post the output of
> 
> ```
> python scripts/validate_docstrings.py pandas.Index.get_loc
> ```
> 
> which here is
> 
> ```
> ################################################################################
> ################################## Validation ##################################
> ################################################################################
> 
> 5 Errors found:
>         No extended summary found
>         Parameter "key" has no description
>         The first line of the Returns section should contain only the type, unless multiple values are being returned
>         Return value has no description
>         See Also section not found
> ```
> 
> to show that the doctests now pass

Thank you for the reminder, @MarcoGorelli, I will add it next time.
Great thanks @farhanreynaldo for PR and @MarcoGorelli for review
WIP: ENH: Add engine keyword argument to groupby.apply to leverage Numba
State: closed

- [x] closes #31845
- [ ] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Enhancement
name: Groupby

Comments:
Hello @mroeschke! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-03-04 08:40:09 UTC
Closing as I am going to try to address this in another fashion