BrandonFors/pandas-github-issues huggingface dataset:
PERF: Allow groupby transform with numba engine to be fully parallelizable 
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

New performance comparison with 10k groups
```
In [1]: df_g = pd.DataFrame({'a': range(10**4), 'b': range(10**4), 'c': range(10**4)})

In [2]: df_g.groupby('a').transform(lambda x: x + 1)
Out[2]:
          b      c
0         1      1
1         2      2
2         3      3
3         4      4
4         5      5
...     ...    ...
9995   9996   9996
9996   9997   9997
9997   9998   9998
9998   9999   9999
9999  10000  10000

[10000 rows x 2 columns]

In [3]: %timeit df_g.groupby('a').transform(lambda x: x + 1)
9.48 s ± 885 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [4]: def f(values, index):
   ...:     return values + 1
   ...:

In [5]: df_g.groupby('a').transform(f, engine='numba', engine_kwargs={'parallel': True})
Out[5]:
            b        c
0         1.0      1.0
1         2.0      2.0
2         3.0      3.0
3         4.0      4.0
4         5.0      5.0
...       ...      ...
9995   9996.0   9996.0
9996   9997.0   9997.0
9997   9998.0   9998.0
9998   9999.0   9999.0
9999  10000.0  10000.0

[10000 rows x 2 columns]

In [6]: %timeit df_g.groupby('a').transform(f, engine='numba', engine_kwargs={'parallel': True})
4.41 ms ± 34.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Labels:
name: Groupby
name: Performance
description: Memory or execution speed performance
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
How does this compare to non-parallel numba?
> How does this compare to non-parallel numba?


```
In [1]: df_g = pd.DataFrame({'a': range(10**4), 'b': range(10**4), 'c': range(10**4)})

In [2]: In [4]: def f(values, index):
   ...:    ...:     return values + 1
   ...:

In [3]: df_g.groupby('a').transform(f, engine='numba')
Out[3]:
            b        c
0         1.0      1.0
1         2.0      2.0
2         3.0      3.0
3         4.0      4.0
4         5.0      5.0
...       ...      ...
9995   9996.0   9996.0
9996   9997.0   9997.0
9997   9998.0   9998.0
9998   9999.0   9999.0
9999  10000.0  10000.0

[10000 rows x 2 columns]

In [4]: %timeit df_g.groupby('a').transform(f, engine='numba')
5.46 ms ± 38.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
so a little better :)
thanks @mroeschke
DOC: Improve Index docstrings
State: closed

Index is now described as List instead of Set

- [x] closes #36170
- [ ] tests added / passed
- [ ] passes `black pandas`
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Docs

Comments:
Lgtm
> Lgtm

Sorry for stepping in, I think it's not accurate enough according to https://github.com/pandas-dev/pandas/issues/36170#issuecomment-688531089

Some tests are failing... I don't recall changing anything else accept the documentation, Why is this even happening?
any idea how to fix this?
> Some tests are failing... I don't recall changing anything else accept the documentation, Why is this even happening?
> any idea how to fix this?

The CI failures are unrelated to this change
In my opinion a set implies that values are unique, which isn’t true for indexes. Maybe “sequence” is better?
> In my opinion a set implies that values are unique, which isn’t true for indexes. Maybe “sequence” is better?

@topper-123 That makes sense. I also think perhaps the whole thing could be rewritten to something like "Immutable sequence used for indexing and alignment." (remove references to ndarrays, sets, or being ordered since none apply). What do you think?
Looks more appropriate to me as well
Shall I improve it?
@dsaxton @topper-123 
Rather than opening a separate PR can you make both changes here?
Sure
@dsaxton I did the required changes. This should do... Right?
Thanks @Nikhil1O1
DOC: Update groupby.rst
State: closed

The previous description still had a conflict between the column name and index level

- [x] closes #16870 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Docs

Comments:
Thanks @Nikhil1O1
TST: add test case for sort_index on multiindexed Frame with sparse cols
State: closed

- [ ] closes #29735 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] add test case for sort_index on multiindexed Frame with sparse cols


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Sparse
description: Sparse Data Type

Comments:
thanks @ylin00
TYP: @final for some Index methods
State: closed

cc @simonjayhawkins 

Tangential question: when I run flake8 it passes, but the pre-commit hook often gives me a bunch of messages like `pandas/core/indexes/base.py:956:51: F821 undefined name 'ignore'`.  Is this something I can fix by pinning a version or something?

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> Tangential question: when I run flake8 it passes, but the pre-commit hook often gives me a bunch of messages like `pandas/core/indexes/base.py:956:51: F821 undefined name 'ignore'`. Is this something I can fix by pinning a version or something?

not sure. I don't have pre-comit setup. (One thing that maybe relevant is that pyflakes is currently installed in the dev environment separately to allow mypy error codes #35311 since we have flake8 pinned <3.8.0 #34150)
before we have a dependence on pandas._vendored.typing_extensions, I think we need some sort of tests, see https://github.com/pandas-dev/pandas/pull/36000#issuecomment-684976573

When trying to use Literal and Final types, i'm getting errors like
```
error: Variable "pandas._vendored.typing_extensions.Final" is not valid as a type  [valid-type]
```
not sure what i'm doing wrong.
Lgtm - I think this is nice to do
Adding `@final` incorrectly to `get_loc` and running mypy, it isn't flagging it as incorrect.  It gets flagged correctly if I import `final` from `typing` directly` instead of `pandas._vendored.typing_extensions`.  Also if I import from a non-vendored typing_extensions it seems to work.

It also isn't flagging private methods with incorrect final tag, regardless of where i get `final` from
since this doesnt appear to have the desired effects, closing to clear the queue
@WillAyd @simonjayhawkins do the problems this had have any impact on the original decision to vendor vs depend?
BUG: Fix unintentionally added suffix in DataFrame.apply/agg and Series.apply/agg
State: closed

- [x] closes #36189
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Bug
name: Regression
description: Functionality that used to work in a prior pandas version
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
ci restarted

=========================== short test summary info ===========================
FAILED pandas/tests/plotting/test_groupby.py::TestDataFrameGroupByPlots::test_series_groupby_plotting_nominally_works
= 1 failed, 75150 passed, 1789 skipped, 1180 xfailed, 6 xpassed, 69 warnings in 1028.46s (0:17:08) =

Thanks @charlesdong1991 
@meeseeksdev backport 1.1.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.1.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 752cd42ee5b5a3909c99de1ac35077d1446fce5b
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #36231: BUG: Fix unintentionally added suffix in DataFrame.apply/agg and Series.apply/agg'
```

4. Push to a named branch :

```
git push YOURFORK 1.1.x:auto-backport-of-pr-36231-on-1.1.x
```

5. Create a PR against branch 1.1.x, I would have named this PR:

> "Backport PR #36231 on branch 1.1.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
thank you @simonjayhawkins for pushing this through yesterday! 🙇 

> =========================== short test summary info ===========================
FAILED pandas/tests/plotting/test_groupby.py::TestDataFrameGroupByPlots::test_series_groupby_plotting_nominally_works
= 1 failed, 75150 passed, 1789 skipped, 1180 xfailed, 6 xpassed, 69 warnings in 1028.46s (0:17:08) =

May I ask where was this failure popping up? is it something I should take a look for a potential issue? @simonjayhawkins 
not sure. I just add the test failure summary to increase visibility and help track. in this case has occurred b4 https://github.com/pandas-dev/pandas/pull/36147#issuecomment-688883534
failing case is here https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=42259&view=logs&jobId=a3a13ea8-7cf0-5bdb-71bb-6ac8830ae35c&j=4b4d380c-92b5-58cf-5b80-2c6c9b1d5805&t=8eab1181-1335-58ed-7ed5-a271d9d86005&s=d654deb9-056d-50a2-1717-90c08683d50a
TST GH 29699 multilevel column is called after multiple appends
State: closed



- [ ] closes #29699
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Not sure where the best place to put the test is. 

Also, this code generates a warning which doesn't seem to disappear when following the guidance of the warning i.e: set `sort=False`. As the default behaviour is meant to return a warning this is tested for.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: MultiIndex
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@TAJD can you merge master and ping when green-ish
@TAJD can you merge master again & resolve conflicts? Based on comments this looks good to go
For some reason it doesn't quite work out when I merge and then run tests. The warning still appears. I've not got much time at the moment to work this one out - I hope I've not missed anything obvious.
> For some reason it doesn't quite work out when I merge and then run tests. The warning still appears. I've not got much time at the moment to work this one out - I hope I've not missed anything obvious.

No problem! Thanks for looking into this anyways. Closing the PR for now - let us know whenever you'd like to pick this back up and we'll reopen
Change default of float_precision for read_csv and read_table to "high"
State: closed

- [x] closes #17154
- [x] tests added / passed
   - modified `tests/io/parser/test_c_parser.py` to make sure all 4 options are tested
   - added `tests/io/parser/test_c_parser.py:test_high_is_default`
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry
  - for version 1.2

See discussion at bottom of #36149 for the performance tests.  Added `float_precision="legacy"` so people can pick up the old parser.  Can't change default to "high" because of incompatibility with python parser


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
Hello @Dr-Irv! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-11 22:09:32 UTC
If there is no performance difference should we not just get rid of the legacy parsing altogether?
> If there is no performance difference should we not just get rid of the legacy parsing altogether?

My only concern here is that maybe someone has code that inadvertently depends on it, so we have to keep it in there for some form of compatibility.  I do think we could deprecate the legacy parsing

yep that sounds fine to leave th option

just update the doc string to indicate 
@jreback added the check for an invalid `float_precision` option and now all green
BUG: GH36212 DataFrame agg() raises error when DataFrame column name is `name`
State: closed

- [x] closes #36212 
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

Labels:
name: Bug
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
Hello @leonarduschen! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2020-09-12 21:56:55 UTC
@jreback @rhshadrach done!
thanks @leonarduschen
CLN: w3 formatting
State: closed

- [x] closes #35706
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Cleans a very minor formatting error to be consistent with w3schools specification.

`colspan=2   --->  colspan="2"`

Note this generally has no effect in browsers parsing HTML.

Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
Thanks @attack68 - is it possible to write a test for this?
seemed like overkill, but included..
thanks @attack68 

reason for the tests is to lock down the behavior; otherwise someone can accidently revert this and we wouldn't know :->
CLN remove trailing commas
State: closed

#35925

From this set of 10 files, only the first 3 needed changes to comply if the new version of black:
- [x] pandas/tests/io/pytables/test_timezones.py
- [x] pandas/tests/io/test_feather.py
- [x] pandas/tests/io/test_s3.py
- [ ] pandas/tests/io/test_sql.py
- [ ] pandas/tests/io/test_stata.py
- [ ] pandas/tests/plotting/test_frame.py
- [ ] pandas/tests/resample/test_datetime_index.py
- [ ] pandas/tests/reshape/merge/test_merge_index_as_string.py
- [ ] pandas/tests/reshape/test_crosstab.py
- [ ] pandas/tests/reshape/test_get_dummies.py

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @tiagohonorato
TST: skip test_groupby_rolling_index_changed on 32bit
State: closed

this is against 1.1.x directly. see https://github.com/pandas-dev/pandas/issues/35831#issuecomment-688862692

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: 32bit
description: 32-bit systems

Comments:
test_groupby_rolling_empty_frame just added in #36213 is not failing on 32 bit, yet all other tests nearby have the skip.

https://dev.azure.com/pandas-dev/pandas-wheels/_build/results?buildId=41860&view=results
restarted https://github.com/MacPython/pandas-wheels/pull/97 to check.
REF: eliminate method _write() in json writers
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

Attempted to figure out what is wrong with the issue https://github.com/pandas-dev/pandas/issues/36211.
I noticed that the private methods ``_write`` are defined in each child of class ``Writer``, which simply re-define ``obj`` and call the parent ``_write`` method.
It turns out that the source of the recursion does not lie there, but I refactored the module slightly by removing the mediator ``_write`` method. Instead I put the differences between the writers into their ``__init__`` method.

It seems that source of the problem with the recursion mentioned in the bug report is in ``pandas/_libs/src/ujson/python/objToJSON.c``.

Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
If there is a problem with updating ``self.obj`` in ``__init__`` of child classes, then we probably should abandon this PR. Maybe other opinion?
Yea I don't think this is an improvement as it is better for the subclasses to override the implementation where appropriate rather than modifying the object that gets stored during initialization.

Let's keep open for a few days as someone may disagree
Would you consider another implementation? I created a new property ``obj_to_write`` (we can call it a ``table_obj``, or something like this). This allowed us to avoid re-definition of ``self.obj``, reduce ``__init__`` boilerplate and also (as initially) get rid of ``_write`` method.
nitpick about annotations, otherwise LGTM
All checks are good after merging with master, except Windows:
```
##[error]We stopped hearing from agent Azure Pipelines 11. 
```

@jbrockmendel, unfortunately, I cannot fully resolve the issue with typing. Please see my comment above in relation to ``NDFrame`` and ``FrameOrSeries``.
Thanks @ivanovmg
BUG: GroupbyRolling with an empty frame
State: closed

- [x] closes #36197
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Groupby
name: Window
description: rolling, ewma, expanding

Comments:
Thanks @mroeschke 
@meeseeksdev backport 1.1.x
Fixed pandas.json_normalize doctests errors
State: closed

- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] Changed `pandas.` to `pd.` in the docstring of `pandas.json_normalize`. Fixed the expected results.


Labels:
name: Docs
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
thanks
REF: implement Categorical._box_func, make _box_func a method
State: closed

Defining `_box_func` on `Categorical` is to facilitate more code sharing with datetimelike.  Also there are a bunch of places that do something _similar_ but slightly different that I want to standardize within `Categorical`.

Changing `_box_func` from a property-lambda to a method on the datetimelike arrays gives a slight perf boost:

```
In [1]: import pandas as pd                                                     
In [2]: dti = pd.date_range("2016-01-01", periods=3)                 
           
In [3]: %timeit dti[1]                                                          
8.73 µs ± 50.8 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <-- master
8.08 µs ± 119 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)   # <-- PR

```


Labels:
name: Refactor
description: Internal refactoring of code
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
annotated+green
TST: DataFrame.replace: TypeError: Cannot compare types 'ndarray(dtype=int64)' and 'unicode'
State: closed

- [x] closes #16784
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`

Thanks @jbrockmendel for fixing my related issue

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Strings
description: String extension data type and string data

Comments:
Test failure looks unrelated?
thanks @phofl
BUG: Don't raise for NDFrame.mask with nullable boolean
State: closed

- [x] closes https://github.com/pandas-dev/pandas/issues/35429
- [x] closes https://github.com/pandas-dev/pandas/issues/36395
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry

~(Need to wait until the v1.1.3 whatsnew gets merged before adding a release note.)~

This is taking the approach of treating NA as False within NDFrame.mask and NDFrame.where when dealing with a nullable boolean mask.

Labels:
name: Bug
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
> (Need to wait until the v1.1.3 whatsnew gets merged before adding a release note.)

ok to target 1.1.2 if ready before tagged
> ok to target 1.1.2 if ready before tagged

I can add a v1.1.2 note just in case
@simonjayhawkins I think having a GitHub Action for merge conflicts like you mentioned would be super useful, do you know by chance how that could be set up?
i've added the 1.1.3 milestone back while the change note is still in 1.1.3.rst
seems like this is a workaround because BooleanArray doesn't have a `__inv__`.  would defining that solve this issue?
this is a good point @jbrockmendel 
> seems like this is a workaround because BooleanArray doesn't have a `__inv__`. would defining that solve this issue?

Do you mean the issue with - vs ~? I might be misunderstanding but inverting a BooleanArray seems to work okay:

```python
[ins] In [3]: arr
Out[3]: 
<BooleanArray>
[True, False, <NA>]
Length: 3, dtype: boolean

[ins] In [4]: ~arr
Out[4]: 
<BooleanArray>
[False, True, <NA>]
Length: 3, dtype: boolean
```

This PR mostly became complicated because of the existing logic surrounding inplace inside _where, and when the inversion should happen in the presence of missing values. 
> this is an intricate change and not appropriate for 1.1.2

As this PR evolves, can we review the milestone when this PR is ready but before this is merged.



Closing for now since it seems it isn't clear if / how we want to do this