BrandonFors/pandas-github-issues huggingface dataset:
future annotations
State: closed

- [x] closes #40533 
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
sure, can you merge master once more and ping on green (of course in draft status)
thanks @MarcoGorelli 
@jreback @MarcoGorelli `from __future__ import annotations` is a Python 3.7+ feature*. Can I please check whether `pandas` no longer supports Python 3.6 and earlier versions of Python?

<sup>* See here: https://docs.python.org/3/whatsnew/3.7.html#pep-563-postponed-evaluation-of-annotations</sup>
@rs2 that's right, Python3.6 is no longer supported
CI, STYLE sync flake8 versions
State: closed

There's been a couple of recent CI failures due to differences in flake8 versions: #40481, #40457

CI failures due to style checks are really annoying, so I think it's worth adding a check to ensure they're in-sync. There's already a hook with `pyyaml` as additional dependency, so that environment will be reused for this check.

Labels:
name: CI
description: Continuous Integration
name: Code Style
description: Code style, linting, code_checks

Comments:
can you merge master and merge on green.
thanks @MarcoGorelli
CI, STYLE Consolidate pygrep checks
State: closed

These checks can be consolidated using multi-line expressions and verbose regex

I think this should make it easier to extend them, so it should be straightforward to port over the remaining ones in code_checks.sh without having to keep increasing the number of hooks

It also reduces the total number of distinct checks (26 now, down from 31 on master), leading to less visual clutter every time we make a commit

The output will still be clear, as it'll just show the line in which an offending pattern was found. E.g. inserting `import conftest` into `pandas/tests/io/formats/test_info.py` results in:
```console
$ pre-commit run unwanted-patterns-in-tests -a
Unwanted patterns in tests...............................................Failed
- hook id: unwanted-patterns-in-tests
- exit code: 1

pandas/tests/io/formats/test_info.py:2:import conftest
```

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
thanks @MarcoGorelli
CLN: ensure_int64->ensure_platform_int
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Clean

Comments:
looks fine. can you rebase. ping on green.
Ping
Fix legend yerr
State: closed

- [x] closes #40044
- [x] xref #39522
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

I spent several hours on this, and I don't think there's a simple solution at the moment (though I'd love to be proved wrong!)

Background: 

in Nov 2017, #18222 was opened, in which it was noted that marker info isn't preserved in legends. This is because we were getting legend handles using `leg.legendHandles`:

https://github.com/pandas-dev/pandas/blob/b623a9df3cf3d69fbdddab0f02a03ad69281c8b0/pandas/plotting/_matplotlib/core.py#L574

Due to [an open issue in matplotlib](https://github.com/matplotlib/matplotlib/issues/11357), `leg.legendHandles` doesn't preserve marker info. So, in #27808, the code was updated to use `ax.get_legend_handles_labels`:

https://github.com/pandas-dev/pandas/blob/5d3b492743e199e82e3018bb943545181ed0018f/pandas/plotting/_matplotlib/core.py#L604

This preserves marker info _but_ has the problem that it may contain additional objects which aren't visible in the legend. The result of this is the permutations observed in #39522 . 

I think that having wrong colours in the legend, as in #39522 and #40044, is _far_ worse than the legend not preserving marker info. Solving everything together doesn't currently seem doable, at least not until https://github.com/matplotlib/matplotlib/issues/11357 is solved in matplotlib.

So my suggestion is to revert some of the changes in #27808, so that at least the colors are right, and when the matplotlib issue is addressed, then the marker info can be preserved.

----

#39522 on master:

![image](https://user-images.githubusercontent.com/33491632/113511085-fdef2880-9555-11eb-9aa1-afb1f4850a7c.png)

on this branch:

![image](https://user-images.githubusercontent.com/33491632/113511072-ed3eb280-9555-11eb-8616-f04771aa9ceb.png)

---

#40044 on master:

![image](https://user-images.githubusercontent.com/33491632/113511142-3b53b600-9556-11eb-8494-2943f070d862.png)

on this branch:

![image](https://user-images.githubusercontent.com/33491632/113511157-4c042c00-9556-11eb-944f-f11b5ce07da5.png)


Labels:
name: Visualization
description: plotting

Comments:
is  #39522 closed? (or partial)
@jreback it's only a partial fix of #39522 , as the marker info isn't preserved

> can you move other legend tests here? (if we have them)

Sure, though for ease of review, should I do that in a follow-up PR?
> Sure, though for ease of review, should I do that in a follow-up PR?

either way
OK, have moved those tests over in this pr @jreback 
thanks @MarcoGorelli very nice!
TYP/CLN: factorize_from_iterable(s)
State: closed

`factorize_from_iterable` didn't return an `CategoricalIndex`, if its input was a categorical-like. Instead it returned a plain `Categorical`.  This PR fixes that + adds return types.

Also makes the return value for `factorize_from_iterables` clearer and easier to read.


Labels:
name: Categorical
description: Categorical Data Type
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @topper-123
EHN: multi-column explode
State: closed

- [x] closes #39240
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Hello @stphnlyd! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-06-20 15:27:09 UTC
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@jreback @phofl could you please take a look at my changes and let me know if there's still something to update?
small comments, otherwise lgtm
@phofl I've pushed to address your comments
@stphnlyd pls merge master
@phofl any comments?
@stphnlyd can you add a release note. doc/source/whatsnew/v1.4.0.rst
@simonjayhawkins thanks I've squashed the PR and now target 1.4.0, and I've added release note. The PR checks failed but that's not caused by my changes. 
@phofl good here?
Yes lgtm
@jreback targeting 1.3 now
thanks @stphnlyd very nice
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
DOC/CI: temp pin of decorator (IPython dependency)
State: closed

The doc builds are failing since yesterday (in enhancingperf.rst, where using the cython ipython magic). By comparing the env from the last working / first failing build on master, one of the differences is a version bump of the `decorator` package, which is a dependency of IPython.

So checking if pinning this package fixes the issue. 

Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
That fixes it, so going to merge this then to get CI green again.
Opened an issue about it upstream: https://github.com/micheles/decorator/issues/105
Indeed, this fixes it, so merging, thanks @jorisvandenbossche 
@meeseeksdev backport 1.2.x
@simonjayhawkins this doesn't need to be backported, as it's already reverted on master
BUG: incorrect rounding in groupby.cummin near int64 implementation bounds
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

xref #40719


Labels:
name: Bug
name: Groupby

Comments:
why would we need to cast? can we not simply use a mask and leave the values as int? 
> why would we need to cast? can we not simply use a mask and leave the values as int?

that was option c in my comment, return an IntegerArray.  For that we'd need IntegerArray to be 1) not opt-in and 2a) support 2D (#38992) or 2b) go full ArrayManager
maybe u misunderstand 

in the cython code we use the incoming dtype and a mask

no casting required and you can always return the original type (except of course you may have to cast to floats if make nans but that would be rare)
> maybe u misunderstand

suggested phrasing: "maybe we're talking past each other" or "maybe there's a miscommunication"

> no casting required and you can always return the original type (except of course you may have to cast to floats if make nans but that would be rare)

the casting under discussion is on the _back end_ after the cython call.

this is a very slight user facing change right? can you add a whatsnew note (ref this PR)
thanks @jbrockmendel
TYP: timedeltas.pyi
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

@simonjayhawkins mypy is unhappy about `Timedelta.__new__` potentially returning `NaT`.  Thoughts on how to handle this?  (it'll come up again in Timestamp and Period)

```
pandas/_libs/tslibs/timedeltas.pyi:51: error: "__new__" must return a class instance (got "Union[Timedelta, NaTType]")  [misc]
```

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@simonjayhawkins im pretty sure this is non-actionable based on what I read in python/mypy#1020, but would like your say-so before mothballing this
yes. looks tricky. I would add ignores for now since imo the benefit of the added types outweighs the extra ignores.
updated per comments
Bug in loc not ordering rhs correctly for mixed indexer
State: closed

- [x] closes #40480
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them



Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
looks fine. is this a user facing change or only a patch on AM?

also merge master 
This is a regression on master, so not user facing
is the array manager failure related?
Sorry, did not scroll down up enough the last time.

This fixes part of the array manager bug, but not the overwriting, so adjusted the test accordingly
great thanks @phofl
DOC: collapse subpages in sidebar for API reference docs
State: closed

The latest version of the theme added the capability to collapse/expand subsections in the sidebar (without directly navigating to the page), see eg the new dropdown arrows on https://pandas.pydata.org/docs/dev/getting_started/index.html

But, an unintended consequence is that the build takes*much* longer (this the reason that the Web / Doc CI build is now always slow on the current open PRs), and this is especially due to our API reference page, which has more than 1000 subpages, and all those pages now have the full navigation structure to all other 1000 pages. 

In this PR I am trying to override the base theme template for the sidebar navigation, to not do this for the reference section of our docs.



xref https://github.com/pydata/pydata-sphinx-theme/issues/364

Labels:
name: Docs

Comments:
cc @datapythonista 
Looks like this speeds up to build from 75m to 35m. Having the dropdowns is nice, but I think this makes sense.
thanks @jorisvandenbossche 
@meeseeksdev backport 1.2.x
Something went wrong ... Please have  a look at my logs.
TYP: libalgos.pyi
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
CI failure unrelated
TST: [ArrowStringArray] more parameterised testing - part 2
State: closed

only a few tests updated in this PR to ensure this approach is accepted.

if we try to add directly to the parametrization of the existing tests we get `ImportError` on environments without pyarrow installed. https://github.com/pandas-dev/pandas/pull/40679#discussion_r603345758

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Strings
description: String extension data type and string data

Comments:
The failure looks like a flaky ResourceWarning related one.
yes. but I restarted anyway.
DOC: [ArrowStringArray] release note and other documentation
State: closed

extremely draft status. placeholder for discussion.

we may want to combine this with #39908, and merge #40708 before #39908

#39908 makes changes to the user facing api that affects the existing StringDtype so that may need an separate section in the release notes.

other sections of the documentation may also need updating. e.g. doc/source/user_guide/text.rst, doc/source/development/roadmap.rst and doc/source/user_guide/basics.rst

Labels:
name: Docs
name: Strings
description: String extension data type and string data

Comments:
not sure if we want to do anything with `df.convert_dtypes()`, maybe `convert_string` parameter could take an additional type to allow conversion to pyarrow backed nullable string array 
will combine this with #39908 as that PR makes the dtype public
ENH: Set column to work for datetimes for Xlsxwriter
State: closed

- [x] closes #30275
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Enhancement for Xlsxwriter to enable the usage of `set_column` for datetime columns for setting the format , just like how it can be done for numbers.
Exception includes the datetime columns which already have a formatting, for eg - `DatetimeIndex`


Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
Closing this in favor of another approach.
CLN: create core Styler rendering functions and subclass conditional styling applications
State: closed

Here is a first draft at splitting up `style.py`, ahead of adding new LaTeX functionality

### Key Points

 - All rendering and formatting methods moved to `style_render.py`:

```
class StylerRenderer:
    jinja2_loader
    def render()
    def format()
    def _translate()
    def _compute()

+ their module level helper functions
```

- `style.py` maintains all the conditional application methods and the utility methods and is subclass:

```
class Styler(StylerRenderer):
    ...
```

All methods copied, no changes made, other than deprivatizing shared method names.

Technically this now works:
```
df = pd.DataFrame([['a   b', 'c  v']])
from pandas.io.formats.style_render import StylerRenderer
StylerRenderer(df).format(str.upper).render()
```

Labels:
name: Refactor
description: Internal refactoring of code
name: Code Style
description: Code style, linting, code_checks

Comments:
cool I think need to rebase (do this pretty much every time when you push)
@jreback this is green and ready to go if you approve.
thanks @attack68
TYP: indexes
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
gentle ping; code checks are starting to touch nearby code
rebased + green (and approved)
lgtm can you rebase as merged some future annotations stuff earlier, ping on green
> pandas/core/groupby/generic.py:181: error: Cannot override final attribute "_obj_with_exclusions" (previously declared in base class "SelectionMixin")  [misc]
pandas/core/groupby/generic.py:934: error: Cannot override final attribute "_obj_with_exclusions" (previously declared in base class "SelectionMixin")  [misc]
That’s on master, orthogonal. Easy fix, PR coming in a few minutes

On Tue, Apr 13, 2021 at 8:36 AM Jeff Reback ***@***.***>
wrote:

> pandas/core/groupby/generic.py:181: error: Cannot override final attribute
> "_obj_with_exclusions" (previously declared in base class "SelectionMixin")
> [misc]
> pandas/core/groupby/generic.py:934: error: Cannot override final attribute
> "_obj_with_exclusions" (previously declared in base class "SelectionMixin")
> [misc]
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/40744#issuecomment-818833403>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AB5UM6B7SUN2OKJ3TQGK7M3TIRQIBANCNFSM42IFXYZQ>
> .
>

rebased + greenish (mypy fixed by #40930, the other is an unrelated ResourceWarning)
Backport PR #40718 on branch 1.2.x (COMPAT: matplotlib 3.4.0)
State: closed

Backport PR #40718: COMPAT: matplotlib 3.4.0

Labels:
name: Visualization
description: plotting
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
hmm didn't completely fix things, cc @simonjayhawkins 
> hmm didn't completely fix things, cc @simonjayhawkins

will look at this (and other backport PR) later today/tomorrow (currently running benchmarks https://github.com/pandas-dev/pandas/pull/40725#issuecomment-812468736)
other warnings should be fixed in #40801
REF: implement groupby.ops.WrappedCythonFunc
State: closed

I think this is much closer to "the correct" abstraction here (i find the layered abstractions with similar names in core.groupby hard to follow).  Contains \approx 0 groupby logic (takes ngroups as an argument in some methods) and is only about how to wrap/call/unwrap the libgroupby functions.

There are a few more parts of _cython_operation I'd like to move, but I'm holding off on that pending some non-refactor work that touches those pieces.

No specific plans, but I'm hopeful that something like this can be re-used to de-duplicate logic in nanops and maybe algorithms.

This uses classmethods/staticmethods (i like the explicit statelessness) and passes how+kind as arguments, but an alternative would be to instantiate with how+kind passed to `__init__`.

The only potentially-logical change is that when defining `out_dtype`, this returns "float64" instead of "float".  I haven't checked if that makes a difference on 32bit builds, but after weeks of dealing with np.intp much prefer the explicitness.


Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby

Comments:
@jbrockmendel can you hold off a bit with such refactoring changes to groupby/ops.py until https://github.com/pandas-dev/pandas/pull/40672#issuecomment-811906908 is resolved?
> can you hold off a bit with such refactoring

Sure.  The two things I have queued up in this area are 1) a refactor aimed specifically at ameliorating the linked concern and 2) a bugfix where we currently cast int64s to float64s in a lossy way.  When I post them ill put them in "draft" mode and cc you on them.
ENH: `Styler.to_latex` conversion from CSS
State: closed

This is an extension PR to #40422 (most of this code is included there), and must be committed after it.
This PR only adds the `_parse_latex_css_conversion` function and associated `convert_css` argument.

Below is the relevant section added to the user guide:

![Screen Shot 2021-04-01 at 16 37 12](https://user-images.githubusercontent.com/24256554/113310637-bf186300-9308-11eb-8f9a-2ea5e2c67bb0.png)



Labels:
name: Enhancement
name: IO LaTeX
description: to_latex
name: Styler
description: conditional formatting using DataFrame.style

Comments:
@attack68 there are some mypy errors.
> @attack68 there are some mypy errors.

thanks didn't spot, fixed now.
thanks @attack68 
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
REF: only fallback to masked op for object dtype
State: closed

While investigating https://github.com/pandas-dev/pandas/pull/40727, I noticed that we are often trying `_masked_arith_op`, while it will end up giving the exact same error (for example when using ints, which don't have NaNs). 
From what I understand, I *think* that, in theory, we only need to fallback to a masked version of the op when having object dtype (I might be wrong here, though). So this PR tries that.

For numeric dtypes, all operations should simply work with NaNs? 
For object dtype, we can have NaN or None, which doesn't necessarily work out of the box, so here we need to fallback to a masked operation. 
ExtensionArrays don't end up here, and datetimelike dtypes get wrapped into EAs before getting here.


Labels:
name: Refactor
description: Internal refactoring of code
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
just ran all the tests asserting that we only get here with object dtype, and found that the failures were floordiv/rfloordiv (handled by #40727) and operator.pow (note: not the same object as `pow`) with integer dtype.  does this handle the latter case?
See my inline comment at https://github.com/pandas-dev/pandas/pull/40728/files#r605632277, that's handling the integer `pow` case (by catching that error within expressions.py, so it falls back to `_evaluate_standard` (the plain op) within expressions.py, and not to `_masked_arith_op` in array_ops.py)
Now, there are still some failures in `test_expressions.py` that I need to look at (but since those are testing directly expressions.py, it might also be that I need to update tests)
sounds good.  also looks like with this we can get rid of the last usage of maybe_upcast_putmask
It was actually a bug in my change in the reversing of the operands
thanks @jorisvandenbossche