BrandonFors/pandas-github-issues huggingface dataset:
ENH: `Styler.text_gradient`: easy extension alternative to `.background_gradient`
State: closed

This is minor code alteration to the `_background_gradient` function that permits a text-only coloring version, i.e. `Styler.text_gradient`.

This could alternatively be performed with a `text_only=True` keyword addition to `Styler.background_gradient` existing method, but it seemed dissimilar enough to merit its own method (and avoid adding more kwargs).

![Screen Shot 2021-04-22 at 14 24 45](https://user-images.githubusercontent.com/24256554/115713844-bef30c80-a376-11eb-94cf-62b4c63aa4de.png)
![Screen Shot 2021-04-22 at 14 24 57](https://user-images.githubusercontent.com/24256554/115713854-c1edfd00-a376-11eb-8819-ab1b6b918564.png)



Labels:
name: Enhancement
name: Styler
description: conditional formatting using DataFrame.style

Comments:
lgtm. can you rebase one more time and ping on green. also pls update  the styler docs as a followon.
@jreback pinging green now.
Thanks @attack68
TYP: Fix typehints for ExtensionDtype
State: closed

This is a fix for type hints of `ExtensionDtype`. `register_extension_dtype` was confusing pyright autocomplete. It was fixed by parameterizing its return value on its input value.

Example:

```python
import pandas as pd

pd.CategoricalDtype(categories=['a', 'b'])
```

Before this change:
```console
$ pyright --version
pyright 1.1.133
$ pyright --lib pd.py
No configuration file found.
No pyproject.toml file found.
stubPath /home/omer/src/ext/typings is not a valid directory.
Assuming Python platform Linux
Searching for source files
Found 1 source file
/home/omer/src/ext/pd.py
  /home/omer/src/ext/pd.py:3:1 - error: Expected no arguments to "ExtensionDtype" constructor (reportGeneralTypeIssues)
1 error, 0 warnings, 0 infos
Completed in 0.72sec
```

After:
```console
$ pyright --lib ../pd.py
No configuration file found.
pyproject.toml file found at /home/omer/src/ext/pandas.
Loading pyproject.toml file at /home/omer/src/ext/pandas/pyproject.toml
Pyproject file "/home/omer/src/ext/pandas/pyproject.toml" is missing "[tool.pyright] section.
stubPath /home/omer/src/ext/pandas/typings is not a valid directory.
Assuming Python platform Linux
Searching for source files
Found 1 source file
0 errors, 0 warnings, 0 infos
Completed in 0.777sec
```

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
 pyright was yelling at me that `return dtype_type.construct_from_string(dtype)` actually returns `ExtensionDtype` from `find`. I tried fixing it while I was at it but reverted back later since it broke things elsewhere with mypy. This part should be irrelevant to the rest.

https://github.com/pandas-dev/pandas/blob/ac2e3e6570d19c31041b75244f6b3a9853d15f9e/pandas/core/dtypes/base.py#L427

This should be ready for review if it passes CI. This is a very minor change but it helps removing `# type: ignore` in some cases with extension dtypes in the user code, keeping type checker and autocompleter happy.
cc @simonjayhawkins @Dr-Irv 
lgtm @Dr-Irv register_extension_dtype was not giving pyright warnings in #39501?
> lgtm @Dr-Irv register_extension_dtype was not giving pyright warnings in #39501?

@simonjayhawkins 
I may have been using an earlier version of pyright.  But I think it would be better to merge in #40421 and then see if this PR is needed, because #40421 handles many `ExtensionDType` things.
#40421 has conflicts and needs a merge master anyway. let's get this one in.
Thanks @ozars small targeted PRs for adding/refining type annotations always welcome. 
Thank you.
CI/DOC: temporary pin environment to python 3.8
State: closed

The doc build is failing on master (`/home/runner/work/pandas/pandas/doc/source/reference/api/pandas.Series.dt.rst: WARNING: document isn't included in any toctree`), and the build that starts failing seems to have switched from python 3.8 to 3.9 (I don't see any other relevant change in the environment). I suppose the default python changed at the conda(-forge) side.

Labels:
name: CI
description: Continuous Integration

Comments:
Thanks @jorisvandenbossche 

I guess this may need  to be backported.

can you open an issue to unpin.

@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 fd67546153ac6a5685d1c7c4d8582ed1a4c9120f
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #41092: CI/DOC: temporary pin environment to python 3.8'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-41092-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #41092 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Opened https://github.com/pandas-dev/pandas/issues/41096
REF: simplify ohlc
State: closed

Or more accurately, simplify BaseGroupBy._cython_agg_general by not having ohlc go through it.  This will in turn allow us to simplify SeriesGroupBy._wrap_aggregated_output.

This will have a merge conflict with #41066.  Doesn't matter which order they are merged in.

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby

Comments:
merged the other one, pls rebase
rebased + greenish (unrelated pyarrow failure thats failing on everything)
REF: hide ArrayManager implementation details from GroupBy
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby

Comments:
nice to remove this, any objections @jorisvandenbossche 
can you rebase
Rebased + green
[ArrowStringArray] PERF: implement ArrowStringArray._str_split
State: closed

marked as draft as performance issues with conversion from pyarrow result to numpy object array of lists

`str.rsplit` can probably also be added to this PR (the tests are parameterised, one failing for StringArray.. maybe precursor needed)

also issue with empty string

failing tests are xfailed.. maybe fix later.

Labels:
name: Strings
description: String extension data type and string data

Comments:
Hello @simonjayhawkins! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-05-19 13:41:18 UTC
will close for now to help clear the queue. no reason to merge a PR that is slower than object fallback (other than to exercise the pyarrow functions)
BUG: GH27721 - test groupby rank with multiindex
State: closed

- [x] closes #27721
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby

Comments:
thanks @arredond
BUG: GH35865 - int cast to str with colon setitem
State: closed

- [x] closes #35865 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry: not sure if it's needed but happy to provide one given some guidelines


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
Don't think we need a whatsnew. Thx for adding the tests
thanks @arredond
BUG: GH41044 - test loc dict assig
State: closed

- [x] closes #41044 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry: not sure if it's needed but happy to provide one given some guidelines

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
@phofl Any idea why the CI checks are failing? It doesn't seem related to the test I added but maybe I'm missing something
Did they fail because of a ResourceWarning? 

We have a flaky test, which fails sometimes
Yep, that seemed to be the cause. Just pulling in changes from `master` and pushing back to the PR and retriggering the tests seems to have fixed it
This test fails now and then, is more or less arbitrary 
thanks @arredond very nice!
[ArrayManager] TST: enable JSON tests (except for table schema)
State: closed

xref https://github.com/pandas-dev/pandas/issues/39146

This already enables most of the JSON tests (it are only the table schema tests that are segfaulting). Those are mostly working because the JSON writer falls back to using `df.values`. This only gives some errors with datetime formatting, for which one test is skipped (and this will then be fixed by letting the JSON code work column-by-column)

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Internals
description: Related to non-user accessible pandas implementation
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
The failure is a flaky ResourceWarning error
REF: BaseGroupBy methods belong in GroupBy
State: closed

BaseGroupBy is also subclassed by Resampler; if Resampler ever accessed these methods they'd get an AttributeError.

Once non-direct move is _cython_agg_general, where the implementation is moved to SeriesGroupBy instead of GroupBy, bc DataFrameGroupBy overrides it.

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby

Comments:
looks good, can ou rebase
rebased + greenish
REF: make maybe_cast_result_dtype a WrappedCythonOp method
State: closed

Abstraction-wise, this is a much better place for it.

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby

Comments:
can you rebase
rebased + greenish
can you merge master (just because merge lots of groupby things)
hah i was just about to comment "no merge conflict".  will do.
rebase + greenish (just the ResourceWarning that apparently we _still_ arent suppressing and the pyarrow stuff thats failing everywhere)
CLN: Clean-up numeric index dtype validation
State: closed

The use of a dict with `_typ` attributes as keys in `NumericIndex._validate_dtype` makes it complicated to subclass numeric indexes and change the subclass' `_typ` attribute.

This PR makes the dtype validation in `NumericIndex._validate_dtype` not dependent on the value of the `_typ` attribute, making subclassing more robust.

Labels:
name: Refactor
description: Internal refactoring of code
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Index
description: Related to the Index class or subclasses
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @topper-123
TST: GH39443 - categorical datetime concat
State: closed

- [x] closes https://github.com/pandas-dev/pandas/issues/39443
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry: Not added since I'm not sure whether it's necessary (there's an unanswered question in the issue regarding that) but am happy to do so provided some guidelines. Would it be simply modifying the latest file in `pandas/doc/source/whatsnew/`, i.e., `v1.3.0.rst`?

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type
name: Stale

Comments:
> plus would want input from others on if a whatsnew is necessary here

not needed
@arredond can you merge master and ping on green
@simonjayhawkins merged but no green. Apart from a couple of conda build fails I'm back to my original error:

```
>       tm.assert_equal(result, expected)
E       AssertionError: Attributes of DataFrame.iloc[:, 0] (column name="x") are different
E       
E       Attribute "dtype" are different
E       [left]:  CategoricalDtype(categories=['2021-01-01', '2021-01-02'], ordered=False)
E       [right]: datetime64[ns]
```
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
Thanks for the PR, but appears this PR has gone stale and looks like it needs a little further investigation to solve the bug. Closing, but please let us know if you'd be interested in continuing and we'd be happy to reopen.
TYP: libinterval
State: closed

This one is pretty ugly.


Labels:
name: Interval
description: Interval data type
name: Typing
description: type annotations, mypy/pyright type checking
name: Mothballed
description: Temporarily-closed PR the author plans to return to

Comments:
> This one is pretty ugly.

is something awry with mypy or do you know what changes here cause issues with the doc decorator.

> is something awry with mypy or do you know what changes here cause issues with the doc decorator.

no idea whats making mypy complain
> maybe break off the io changes for now

IIRC i did only what was necessary to get mypy to stop complaining about interval.pyi
> we can't merge this with all these error: Cannot determine type of ignores that have appeared. not sure what else to suggest.

IIUC mypy's complaint is bc e.g. `Index.memory_usage` is decorated with `@doc(IndexOpsMixin._memory_usage)`.  doc is annotated with `-> Callable[[F], F]` which i would expect would prevent this problem
@simonjayhawkins i _think_ the reason why so many new mypy complaints are popping up is because `Scalar` no longer resolves to Any.  will try to chip away at these
> this is a lot of change, not sure if easier to do in pieces

Haven't found a good way to do this.  bc this is the last thing that mypy doesnt recognize in pd._typing.Scalar, it means a bunch of things that currently resolve to Any are now recognized.  I'll keep trying this periodically, but for the time being this is on the backburner.
can pandas/_libs/tslibs/nattype.pyi or pandas/_libs/interval.pyi be added in seperate PRs. 

also it is possible to partially type a stub so that could be get some of this in. 

https://github.com/python/typeshed/blob/master/CONTRIBUTING.md#incomplete-stubs
Mothballing to clear the queue, as there is no clear path forward here.
DOC: more accurate wording in roadmap
State: closed

This came up the other day in https://github.com/pandas-dev/pandas/pull/40651#issuecomment-820671958.  As I [originally](https://github.com/pandas-dev/pandas/pull/35208#issuecomment-656879885) said in the PR that put this inaccurate wording in, this overstates the degree to which there is consensus.

Labels:
name: Docs

Comments:
sure
[ArrowStringArray] Use utf8_upper and utf8_lower functions from Apache Arrow
State: closed

```
data = ["Mouse", "dog", "house and parrot", "23", np.NaN] * 100_000
s = pd.Series(data, dtype="string")
s1 = pd.Series(data, dtype="arrow_string")

%timeit s.str.upper()
# 63.8 ms ± 259 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit s1.str.upper()
# 92 ms ± 152 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  <-- master
# 4.44 ms ± 35.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <-- PR

%timeit s.str.lower()
# 49.1 ms ± 273 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit s1.str.lower()
# 78.2 ms ± 814 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  <-- master
# 4.34 ms ± 32.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  <-- PR

```

Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
great, do we have sufficient asv's for these? 
> great, do we have sufficient asv's for these?

have parameterised existing benchmarks for these in #41041

```
[ 25.00%] ··· strings.Methods.time_lower                                                                                                                  ok
[ 25.00%] ··· ============== ==========
                  dtype                
              -------------- ----------
                   str        16.2±0ms 
                  string      12.6±0ms 
               arrow_string   2.67±0ms 
              ============== ==========

[ 50.00%] ··· strings.Methods.time_upper                                                                                                                  ok
[ 50.00%] ··· ============== ==========
                  dtype                
              -------------- ----------
                   str        19.4±0ms 
                  string      16.4±0ms 
               arrow_string   2.71±0ms 
              ============== ==========
```
@simonjayhawkins anyidea why this is not timing well?

```
In [39]: s = pd.concat([pd.Series(list('abc'))] * 100_000)

In [40]: s1 = pd.concat([pd.Series(list('abc'), dtype='string[pyarrow]')] * 100_000)

In [41]: %timeit s.str.upper()
48.5 ms ± 492 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [42]: %timeit s1.str.upper()
299 ms ± 14.9 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

In [43]: pd.__version__
Out[43]: '1.4.0.dev0+143.g5675cd8ab2'

In [44]: pyarrow.__version__
Out[44]: '4.0.1'
```
does it create a chunked array with 100_000 chunks?
ahh yes it does
```
s1 = s.astype('string[pyarrow]'

%timeit s_pyarrow_large.str.uppe()
1.53 ms ± 29.8 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
```
so actually this raises an issue if we should be rechunking.
let me open an issue
BUG: Handle zero-chunked pyarrow.ChunkedArray in StringArray
State: closed

- [x] closes #41040
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Strings
description: String extension data type and string data
name: IO Parquet
description: parquet, feather

Comments:
The test failure on Windows is unrelated and also happening on master.
This happens sometimes, ``ResourceWarning`` yes
> Period and Interval might have the same problem (but also fine to leave it as follow-up)

I'll fix that here, too.
Thanks @xhochy !
PERF: optimize conversion from boolean Arrow array to masked BooleanArray
State: closed

For arrays without missing values it doesn't change, but gives a decent speed-up when having missing values:

```
In [1]: arr1 = pa.array([True, False, True, True, False]*100_000)

In [2]: arr2 = pa.array([True, False, None, True, False]*100_000)

In [3]: %timeit pd.BooleanDtype().__from_arrow__(arr1)
9.34 ms ± 27.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)   <-- master
9.31 ms ± 23.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)   <-- PR

In [4]: %timeit pd.BooleanDtype().__from_arrow__(arr2)
60.6 ms ± 678 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)   <-- master
18.8 ms ± 99.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)   <-- PR
```

cc @simonjayhawkins 

Labels:
name: Performance
description: Memory or execution speed performance
name: IO Parquet
description: parquet, feather
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
@jorisvandenbossche i'm getting significant different timings and showing a much bigger improvement (30x with missing values) and same with `* 1_000_000`. can you double check you timings.

```
%timeit pd.BooleanDtype().__from_arrow__(arr1)
# 572 µs ± 4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  <-- master
# 553 µs ± 2.91 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)   <-- PR

%timeit pd.BooleanDtype().__from_arrow__(arr2)
# 34.4 ms ± 373 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)   <-- master
# 1.06 ms ± 2.35 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)   <-- PR
```
> i'm getting significant different timings and showing a much bigger improvement (30x with missing values) 

Whoops, I was using my debug build of arrow, which makes things generally much slower, that should explain it. 
But so all the better that the speed-up is larger! ;)
@simonjayhawkins any further comments?
Will add a small note
DOC: Improve to_datetime() documentation
State: closed

Returns Object Type Index when Mixed Timezones are used in list-like input to pandas.to_datetime()

- [x] closes #40808 
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: Datetime
description: Datetime data dtype

Comments:
Hello @chinmayrane! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-04-21 13:33:33 UTC
Test Failure seems unrelated. This is my first PR, so wanted to make sure, if I understand it correctly.

Seeing Error on - `TestSeriesReductions.test_empty_multi`

`AssertionError: Caused unexpected warning(s): [('ResourceWarning', ResourceWarning('unclosed transport <asyncio.sslproto._SSLProtocolTransport object at 0x000001B4D3B9EE20>'), 'C:\\Miniconda\\envs\\pandas-dev\\lib\\asyncio\\sslproto.py', 321)]`

Also got a pop up on the Azure Pipeline that this test has failed `124` times in the past `14` days.


[Test Failure](https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=58344&view=logs&j=404760ec-14d3-5d48-e580-13034792878f&t=f81e4cc8-d61a-5fb8-36be-36768e5c561a&l=113)
@mroeschke  Thanks!
thanks @chinmayrane
BUG: Slice Arrow buffer before passing it to numpy (#40896)
State: closed

Add Arrow buffer slicing before handing it over to numpy which is
needed in case the Arrow buffer contains padding or offset.

- [x] closes #40896
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Points I'm not sure about:
- I created a new test file. Would it be better to place the test somewhere else?
- The test only tests int32. Should I add more dtypes?

I'm still new to the internals of pandas and always trying to improve so feedback of any kind is highly appreciated!

Labels:
name: IO Parquet
description: parquet, feather
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Thanks for the PR!

> I created a new test file. Would it be better to place the test somewhere else?

There is a `pandas/tests/arrays/masked/test_arrow_compat.py`. I think you can put it there.

> The test only tests int32. Should I add more types?

Yes, I think it would be good to parameterize the test across different dtypes.

I moved the test and added parameterization for all numeric types.
thanks @ThomasBlauthQC 

i think the checks are running, but can merge on green.
Looks green @jreback. Merge?
@ThomasBlauthQC Thanks a lot for the fix!