BrandonFors/pandas-github-issues huggingface dataset:
DOC: Improve DataFrame.dropna subset example
State: closed

In the examples for [DataFrame.dropna](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.dropna.html), which begin like so:

```pycon
>>> df = pd.DataFrame({"name": ['Alfred', 'Batman', 'Catwoman'],
...                    "toy": [np.nan, 'Batmobile', 'Bullwhip'],
...                    "born": [pd.NaT, pd.Timestamp("1940-04-25"),
...                             pd.NaT]})
>>> df
       name        toy       born
0    Alfred        NaN        NaT
1    Batman  Batmobile 1940-04-25
2  Catwoman   Bullwhip        NaT
```

the example that uses the `subset` parameter,

```pycon
>>> df.dropna(subset=['name', 'born'])
       name        toy       born
1    Batman  Batmobile 1940-04-25
```

yields the same output as the example that doesn't pass in any arguments,

```pycon
>>> df.dropna()
     name        toy       born
1  Batman  Batmobile 1940-04-25
```

I think changing the `subset` example to

```pycon
>>> df.dropna(subset=['name', 'toy'])
       name        toy       born
1    Batman  Batmobile 1940-04-25
2  Catwoman   Bullwhip        NaT
```

would better illustrate the concept, as its result contains a row with one missing value, just in a column that wasn't specified in `subset`.

-----

I'm not able to test the docstring locally at the moment, but I ran the code in my console up to the new example, and it worked as expected.

```pycon
>>> import pandas as pd
>>> import numpy as np
>>> df = pd.DataFrame({"name": ['Alfred', 'Batman', 'Catwoman'],
...                    "toy": [np.nan, 'Batmobile', 'Bullwhip'],
...                    "born": [pd.NaT, pd.Timestamp("1940-04-25"),
...                             pd.NaT]})
>>> df
       name        toy       born
0    Alfred        NaN        NaT
1    Batman  Batmobile 1940-04-25
2  Catwoman   Bullwhip        NaT
>>> df.dropna()
     name        toy       born
1  Batman  Batmobile 1940-04-25
>>> df.dropna(axis='columns')
       name
0    Alfred
1    Batman
2  Catwoman
>>> df.dropna(how='all')
       name        toy       born
0    Alfred        NaN        NaT
1    Batman  Batmobile 1940-04-25
2  Catwoman   Bullwhip        NaT
>>> df.dropna(thresh=2)
       name        toy       born
1    Batman  Batmobile 1940-04-25
2  Catwoman   Bullwhip        NaT
>>> df.dropna(subset=['name', 'toy'])
       name        toy       born
1    Batman  Batmobile 1940-04-25
2  Catwoman   Bullwhip        NaT
```

Labels:
name: Docs

Comments:
Thanks @tywmick
Fix MPL dt conversion (epoch2num deprecation) #34850
State: closed

Fix datetime conversion after matplotlib (3.3.0 update) deprecating epoch2num.

- [ ] closes #34850
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
Thanks @misantroop. Seems to be some test failures. Can you check?

```
____________________ TestDateTimeConverter.test_conversion _____________________
[gw2] darwin -- Python 3.6.10 /Users/runner/miniconda3/envs/pandas-dev/bin/python

self = <pandas.tests.plotting.test_converter.TestDateTimeConverter object at 0x161aaac18>

    def test_conversion(self):
        rs = self.dtc.convert(["2012-1-1"], None, None)[0]
        xp = datetime(2012, 1, 1).toordinal()
>       assert rs == xp
E       assert 15340.0 == 734503

pandas/tests/plotting/test_converter.py:150: AssertionError
```

Can you post images of the outputs with / without your changes for matplotlib 3.3 and an earlier version?
Please see an alternative circumvent that takes into account the same problem here:
https://github.com/pandas-dev/pandas/issues/34850#issuecomment-661249836
Issue fixed in MPL: https://github.com/matplotlib/matplotlib/pull/17983
REF: remove special casing from Index.equals (always dispatch to subclass)
State: closed


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
small request, otherwise LGTM
yep lgtm.
ok to merge?
do we have a test for this?
we have tests in pandas\tests\indexes\test_base.py and pandas\tests\indexes\multi\test_equivalence.py and 
thanks @simonjayhawkins
Optimize array_equivalent for NDFrame.equals
State: closed

Closes #35249

```python
In [1]: import numpy as np
In [2]: import pandas as pd
In [3]: N = 10**3
In [4]: float_df = pd.DataFrame(np.random.randn(N, N))
# 1.0.5
1.52 ms ± 94 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

# master
83.6 ms ± 1.15 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

# branch
31.2 ms ± 2.06 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

And proof of this being better for short-circuiting:

```python
In [4]: N = 10**3

In [5]: a = pd.DataFrame(np.random.randn(N, N))

In [6]: b = pd.DataFrame(np.random.randn(N, N))

In [7]: %timeit a.equals(b)
# 1.0.5
1.97 ms ± 127 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [10]: %timeit a.equals(b)
56.4 µs ± 1.67 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)
```

Discussion: the key thing here is that `BlockManager.equals` requires that the dtypes of `left` and `right` match. If they don't then we've already returned False. That lets us do a few things

1. Avoid many `is_foo_dtype` checks.
2. Use optimized checks like `np.isnan`
3. Avoid the slow case of considering `np.array([1, 2])` equivalent to `np.array([1.0, 2.0])`.

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
restarting failed travis job 

```
=========================== short test summary info ============================
FAILED pandas/tests/io/parser/test_common.py::test_chunks_have_consistent_numerical_type[python]
= 1 failed, 69943 passed, 4255 skipped, 1020 xfailed, 13 warnings in 808.93s (0:13:28) =
```
Travis finished. I'm going to selfishly ignore https://github.com/pandas-dev/pandas/pull/35328#discussion_r456589372 so that I can get the RC process rolling, but happy to engage more on followups if I missed something.
Fix AttributeError when groupby as_index=False on empty DataFrame
State: closed

- [x] closes #35246
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Groupby
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Thanks @salem3358 for the PR. Can you add a release note to 1.1.0. even if this doesn't make the RC, since this fixes a regression we will be backporting if necessary. cc @TomAugspurger 
Added a releases note. Merging on green.
Remove deprecated warn kwarg for matplotlib use
State: closed

The warn param has been deprecated in 3.2.1: https://matplotlib.org/3.2.1/api/prev_api_changes/api_changes_3.2.0.html#matplotlib-use

On the most recent matplotlib, this is failing for me with 

```
TypeError: use() got an unexpected keyword argument 'warn'
```


Labels:
name: Visualization
description: plotting

Comments:
restarting failed Windows py37_np18 job

```
=========================== short test summary info ===========================
FAILED pandas/tests/io/parser/test_common.py::test_chunks_have_consistent_numerical_type[python]
= 1 failed, 72698 passed, 1743 skipped, 1053 xfailed, 58 warnings in 719.92s (0:11:59) =
```
Thanks!
Thanks!
Update multi.py
State: closed

fix bugs be metioned on issue #35301

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:

Comments:
> We need a test for this
> 
> Also - are you sure this is enough to fix the original problem? I tried this quickly and it seems like the error persists



> Cool, I think that does fix the original issue!
> 
> Unfortunately as is it seems to break some of the other tests so that's something to look into (in particular a few go wrong in `pandas/tests/multi/test_indexing.py`)

thanks , I will fix that
shall I close this pr? xs is intended for scalar lookups
as indicted in the issue would take a better error message
add a test from the OP
DOC: Rst Formatting, make sure continuation prompt are used.
State: closed

Without the `...` for some tools it may look like
`with ExcelWriter('path_to_file.xlsx',` is an input and
`date...` is the output.

I initially thought that I could use the hanging indent to separate
code continuation from output but `pandas/core/indexing.py`
proved me wrong as the example just after the one I fixed have
the first line of the output be indented.



Labels:
name: Docs

Comments:
Can you build these docstrings to check how these render? I think OK but good to check
Unfortunately I didn't manage to build pandas nor the docs. I came across this via another tools on my prebuilt pandas install from conda. 
Thanks @Carreau
DOC: extra closing parens make example invalid.
State: closed

No attached issues or need for what's new / test... unless you want to actually get a linter the parse the docstrings and make the the examples are syntactically correct ?

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
we do run some but not all doc-tests which run the doc-strings

happy to have a follow on which expands that check (see code_check.sh)
pin numpy<1.19 in doc build
State: closed

Doc build is failing on master with a warning from NumPy via matplotlib. Waiting on a release with https://github.com/matplotlib/matplotlib/pull/17289 (3.3.0), so pinning numpy<1.19 for now.

Labels:
name: Build
description: Library building on various platforms
name: Docs

Comments:
cool maybe make an issue to revert this
TYP: Add MyPy Error Codes
State: closed

- [ ] closes #29197
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Code Style
description: Code style, linting, code_checks
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@WillAyd have added a code-check to enforce this but may not be able to enable until we bump mypy due a bug in mypy not suppressing some additional notes, so have just commented out the code check for now
@WillAyd gentle ping!
Thanks @simonjayhawkins
CLN: consistent EA._reduce signatures
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Clean
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @simonjayhawkins
TST: xfail 32-bit test
State: closed

ref #35294

Labels:

Comments:
Does that fail for you on a 32-bit system? The second passes for me locally

```
pandas/tests/window/test_api.py::test_multiple_agg_funcs[rolling-2-expected_vals0] XFAIL                                                              [ 50%]
pandas/tests/window/test_api.py::test_multiple_agg_funcs[expanding-None-expected_vals1] PASSED                                                        [100%]
```

Also this wouldn't close #35294. To close that we would need to make the code compatible with 32-bit systems.
> Does that fail for you on a 32-bit system? The second passes for me locally

Not sure how to test it locally, I thought I push it and see if it passes the tests.

> Also this wouldn't close #35294. To close that we would need to make the code compatible with 32-bit systems.

Yes, my bad, removed the linked issue. 
Our CI isn't set up to handle this yet. You're welcome to give that a shot, but it's not clear how best to do that since 32-bit conda packages aren't readily available.
@TomAugspurger so should I close this PR for now, then later maybe open an issue + PR to handle 32-bit failures?
Yeah, we'll close this.

I don't think we need any new issues though I may be wrong.
Fixed reindexing arith with duplicates
State: closed

Closes https://github.com/pandas-dev/pandas/issues/35194

Still need to run ASV on this. This was a regression from 0.25.x to 1.0. It doesn't have to go in 1.1, but probably better for 1.1.0.rc0 than a 1.1.1 release.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
cc @jbrockmendel.
Having trouble with ASV right now. Things seem OK in this small benchmark though

```python
In [15]: df1 = pd.DataFrame(np.random.randn(1000, 500))

In [16]: df2 = pd.DataFrame(np.random.randn(1000, 501))

In [17]: %timeit df1 + df2
# master
3.8 ms ± 115 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

# PR
3.77 ms ± 104 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
LGTM
thanks @TomAugspurger
BUG: date_range doesn't propagate ambigous=False to tz_localize
State: closed

- [x] closes #35297 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Bug
name: Timezones
description: Timezone data dtype

Comments:
@mroeschke Getting an error in the doctests that I think is unrelated to this PR:
```
=================================== FAILURES ===================================
__________________ [doctest] pandas.core.frame.DataFrame.info __________________
2593             Data columns (total 3 columns):
2594              #   Column    Non-Null Count    Dtype
2595             ---  ------    --------------    -----
2596              0   column_1  1000000 non-null  object
2597              1   column_2  1000000 non-null  object
2598              2   column_3  1000000 non-null  object
2599             dtypes: object(3)
2600             memory usage: 22.9+ MB
2601 
2602             >>> df.info(memory_usage='deep')
Differences (unified diff with -expected +actual):
    @@ -2,9 +2,9 @@
     RangeIndex: 1000000 entries, 0 to 999999
     Data columns (total 3 columns):
    - #   Column    Non-Null Count    Dtype
    ----  ------    --------------    -----
    + #   Column    Non-Null Count    Dtype 
    +---  ------    --------------    ----- 
      0   column_1  1000000 non-null  object
      1   column_2  1000000 non-null  object
      2   column_3  1000000 non-null  object
     dtypes: object(3)
    -memory usage: 188.8 MB
    +memory usage: 165.9 MB
```
I merged with master just now so I don't think it's because this branch is out-of-date
No worries, looks unrelated to this PR

Actually sorry, could you move the creation of the 1.2.0 whatsnew doc to a separate PR? Would make this PR cleaner.
Thanks @arw2019
ENH: add percentage threshold to dropna
State: closed

- [x] closes #35299 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Needs Discussion
description: Requires discussion from core team before further action

Comments:
@erfannariman are you still working on this?
@simonjayhawkins yes, just waiting for input in the issue discussion. Would love to make more speed on this one, we just need to decide how we move forward.
Fix indexing, reindex on all-sparse SparseArray.
State: closed

Closes https://github.com/pandas-dev/pandas/issues/35286.

Also added a regression tests for the issue reported there.

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Sparse
description: Sparse Data Type

Comments:
Looks like https://github.com/pandas-dev/pandas/pull/34908 built on the buggy functionality. I may need to revert it as well.
Thanks, I think we're good here.
To latex position
State: closed

- [ ] closes #35281 
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: IO LaTeX
description: to_latex

Comments:
Hello @WillAyd , thanks for the review ! 

I did the changes you asked for, but one test seems to have failed, I'm not sure it's related to what I've written

Would you also want tests when the caption, the label and the position is given ?
Just restarted the failure let's see
@toobaz
@toobaz I tried to match the two functions `_write_table_begin` and `_write_longtable_begin`, but I don't think there is a need for the `_table_float` here, because:
- There is no change in the begin line: whether or not a caption, label or position is given, it will always be `\begin{longtable}...`
- The only difference would be for the line `buf.write("\\\\\n")` but here again the `_table_float` argument is not relevant because this behaviour is needed when a caption or a label is provided, but if only the position would be given it shouldn't be needed
> I tried to match the two functions

My suggestion was actually to entirely suppress ``_write_longtable_begin``, given that the only implications of ``longtable=True`` are that the float must be produced, and that its environment name must be ``longtable`` rather than ``table``: all the rest is the same as ``_write_tabular_begin`` (same for ``*_end``).

But this is not a strict necessity, at least in this PR. So if you prefer we can merge this PR after solving my last comment above.
> My suggestion was actually to entirely suppress `_write_longtable_begin`

OK @toobaz I hadn't understood ! I tried doing something to resolve this, but I suggest doing it on a separate PR because I think it will get a bit messy.

This work is currently on a [branch of mine](https://github.com/SylvainLan/pandas/blob/merge_table_longtable/pandas/io/formats/latex.py) and we can look at it once this PR is merged

Looks ready to merge to me. @WillAyd any idea on the two checks still failing?
> Looks ready to merge to me. @WillAyd any idea on the two checks still failing?

see #35481. there are 20 fails here, should be only 4 now with lastest numpy-dev
Thanks @SylvainLan
Revert "BUG: fix union_indexes not supporting sort=False for Index subclasses"
State: closed

Reverts pandas-dev/pandas#35098

Closes https://github.com/pandas-dev/pandas/issues/35238 I'll also push a test from https://github.com/pandas-dev/pandas/issues/35238 here, and add the xfailing tests from #35098.

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
thanks @TomAugspurger 
Tests still need to be added?
CI: Unpin pytest
State: closed

- [ ] closes #35261

have made a start working through the failures. we should be able to get mypy to green here and then when pytest 6.0.0 is released the upstream fixes should cause ci to go red since we have `warn_unused_ignores = True` in setup.cfg.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
cc @bluetech @asottile 
Hello @simonjayhawkins! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-07-29 14:30:49 UTC
Nice work. Are you planning to drop some of the type ignores on the next RC? I see some of the upstream issues you've opened have been closed
> Nice work. Are you planning to drop some of the type ignores on the next RC? I see some of the upstream issues you've opened have been closed

we could release the pin on `CI / Checks` only with this PR and get immediate benefit from the type checking. or happy to wait for the upstream fixes and reduce the diff here. (the type:ignores for the fixed issues will give mypy errors since we have `warn_unused_ignores = True` in the config)

@bluetech has added tests for the fixed issues so shouldn't be an issue, but could also change the `CI /Checks` environment here to use pytest master to make sure.



> I see some of the upstream issues you've opened have been closed

There are basically three issues. two are fixed. I still need to look into the other. https://github.com/pytest-dev/pytest/issues/7495
@TomAugspurger I've labelled this 1.2 for now. Is there a benefit to unpinning on 1.1.x?
test failures unrelated. The ci environments are resolving to pytest 6.0.0 in Linux py37_locale and Windows py36_np15 on azure, CI / Checks on GitHub actions so I think this is good to go. @TomAugspurger 
Actually, I'm second-guessing not backporting this. I think we'll want a consistent pytest version for both branches being actively developed. What do you think?
> Actually, I'm second-guessing not backporting this. I think we'll want a consistent pytest version for both branches being actively developed. What do you think?

sure. happy to backport. should be a benefit to users of 1.1.1 onwards (will be interesting to see if we get any issues on 1.1.0 regarding the pin, now that pytest 6.0.0 is released.)
Great, thanks.
@MeeseeksDev backport to 1.1.x
CI: Ignore setuptools distutils warning
State: closed

xref https://github.com/pandas-dev/pandas/issues/35252. Just ignoring in the test. Can discuss a proper solution in the issue.

Labels:
name: CI
description: Continuous Integration

Comments:
Last failure was a crashed worker (seems to be happening more often recently?) I restarted that job.
Travis finished, didn't report back.

Merging once the windows test finishes to get CI passing again.
Add keep parameter for duplicates values in idxmax, idxmin, argmax, and argmin
State: closed

- [x] closes #34205
- [ ] tests added / passed
- [x] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Enhancement
name: API Design

Comments:
Hello @MatthewMoye! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:

















There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-10-05 23:05:21 UTC
Should I move everything to nanargmax/min and revert the other files? I was unsure if I should have put the logic there when I originally starting working on this. I should have asked about this before submitting a PR, my apologies.
@MatthewMoye doctests are failing. can you also move release note to 1.2
Sorry for the delay @jreback , I updated the asvs.
@jreback The results:

![Capture](https://user-images.githubusercontent.com/31222951/90586513-60e9a600-e1a5-11ea-9df2-eda6dc9a58ff.PNG)

<details>
<summary>System Info</summary>
<p>
os: Windows 10 <br/>
cpu: AMD Ryzen 7 2700X Eight-Core Processor <br/>
ram: 32GB <br/>
</p>
</details>  
@MatthewMoye can you merge master to fix conflicts?
closing this as stale. great idea and was pretty close.
PERF: array_equivalent
State: closed

xref #35249, this gives about a 3x speedup in the asv reported there

cc @jorisvandenbossche EA.equals doesn't support a strict_nan arg, is there anything we need to do to account for that here?

Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
> EA.equals doesn't support a strict_nan arg, is there anything we need to do to account for that here?

Indeed, EA.equals doesn't support that (and not sure it should). But see also my inline comment: is it actually needed to handle EAs here?
maybe merge #35231 first since that PR adds additional tests?
> maybe merge #35231 first since that PR adds additional tests?

good idea