BrandonFors/pandas-github-issues huggingface dataset:
REF: _try_cast; go through fastpath more often
State: closed

- [x] closes #28145
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance
name: Series
description: Series data structure
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
can you post benchmark results
with `-b ctors`

```
       before           after         ratio
     [6dff995e]       [29848ac6]
     <master>         <ref-cast>
+     2.67±0.05ms       2.74±0.1ms     1.03  ctors.SeriesConstructors.time_series_constructor(<function list_of_tuples at 0x11c088670>, True, 'int')
+     2.66±0.05ms      2.69±0.09ms     1.01  ctors.SeriesConstructors.time_series_constructor(<function list_of_tuples_with_none at 0x11c088820>, True, 'float')
-     2.63±0.05ms      2.60±0.04ms     0.99  ctors.SeriesConstructors.time_series_constructor(<function list_of_lists_with_none at 0x11c0888b0>, True, 'float')
-     2.62±0.05ms      2.59±0.05ms     0.99  ctors.SeriesConstructors.time_series_constructor(<function list_of_lists_with_none at 0x11c0888b0>, False, 'float')
-     2.63±0.03ms      2.59±0.04ms     0.99  ctors.SeriesConstructors.time_series_constructor(<function list_of_lists at 0x11c088790>, False, 'float')
-     2.64±0.05ms      2.60±0.04ms     0.98  ctors.SeriesConstructors.time_series_constructor(<function list_of_lists_with_none at 0x11c0888b0>, False, 'int')
-     2.66±0.07ms      2.61±0.04ms     0.98  ctors.SeriesConstructors.time_series_constructor(<function list_of_lists_with_none at 0x11c0888b0>, True, 'int')
-     2.65±0.07ms      2.59±0.04ms     0.98  ctors.SeriesConstructors.time_series_constructor(<function list_of_lists at 0x11c088790>, True, 'float')
-      3.51±0.1ms      3.38±0.08ms     0.96  ctors.SeriesConstructors.time_series_constructor(<function gen_of_str at 0x11c088550>, False, 'int')
-     2.73±0.08ms      2.62±0.04ms     0.96  ctors.SeriesConstructors.time_series_constructor(<function list_of_tuples_with_none at 0x11c088820>, True, 'int')
-        591±30μs         565±20μs     0.96  ctors.SeriesConstructors.time_series_constructor(<function list_of_str at 0x11c0884c0>, True, 'float')
-      2.77±0.1ms      2.65±0.06ms     0.96  ctors.SeriesConstructors.time_series_constructor(<function list_of_tuples at 0x11c088670>, False, 'int')
-      50.5±0.8μs         45.4±2μs     0.90  ctors.SeriesConstructors.time_series_constructor(<function no_change at 0x11c088430>, True, 'float')
-        43.0±2μs         37.4±2μs     0.87  ctors.SeriesConstructors.time_series_constructor(<function no_change at 0x11c088430>, False, 'float')
-      52.9±0.6μs         44.5±2μs     0.84  ctors.SeriesConstructors.time_series_constructor(<function no_change at 0x11c088430>, True, 'int')
-        45.5±1μs       36.6±0.2μs     0.81  ctors.SeriesConstructors.time_series_constructor(<function no_change at 0x11c088430>, False, 'int')

```
CI: Xfails on Python 3.10
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

~~The Python 3.10 CI is actually red, but the continue-on-error flag makes it so that the job is not failed. WIP because tests actually have to be fixed.~~
Added Xfails for some tests. Pretty much green except for #41935

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration
name: Python 3.10

Comments:
right this was done on purpose so that we have time to fix the tests

so happy to merge this once all green otherwise
@jreback I think we should xfail these instead of doing this for now, and look into fixing them. The current setup means that its hard to tell if something breaks more tests on Python 3.10 in addition to the ones currently failing.
> @jreback I think we should xfail these instead of doing this for now, and look into fixing them. The current setup means that its hard to tell if something breaks more tests on Python 3.10 in addition to the ones currently failing.

sure happy to do that as well. I agree that's better.
Hello @lithomas1! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:













There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-06-28 15:35:29 UTC
Still WIP becaue there's a ton of flaky tests and idk what's wrong with sparse tests.
thanks @lithomas1 very nice!
@meeseeksdev backport 1.3.x
Something went wrong ... Please have  a look at my logs.
TYP and DOC: pd.merge accepts Series
State: closed

I noticed that the docs for `pd.merge()` and `pd.merge_asof()` were requiring the first argument to be a `DataFrame`, but in fact, they can be a `Series`.  This PR updates the docs and the types to fix that issue.

Didn't think a `whatsnew` entry was needed, and I skipped creating an issue, but will do so if required.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
> I think you should use DataFrame | Series instead of FrameOrSeriesUnion. (See #40876 for more context)

Thanks for pointing that out.  Have now changed it.
cc @simonjayhawkins if any comments.
Thanks @Dr-Irv
[ArrowStringArray] PERF: use pyarrow.compute.replace_substring(_regex) if available
State: closed

```
[ 50.00%] ··· strings.Methods.time_replace                                                                                                                ok
[ 50.00%] ··· ============== ==========
                  dtype                
              -------------- ----------
                   str        20.6±0ms 
                  string      16.3±0ms 
               arrow_string   1.83±0ms 
              ============== ==========
```

happy to break up if easier to review.

Labels:
name: Performance
description: Memory or execution speed performance
name: Strings
description: String extension data type and string data

Comments:
> happy to break up if easier to review.

will do that and mark this as draft for now.
closes #41602 , will break off once other pre-cursors merged
> closes #41602 , will break off once other pre-cursors merged

opened #41606, after that is merged, this PR is just the addition of the method on ArrowStringArray
> opened #41606, after that is merged, this PR is just the addition of the method on ArrowStringArray

will close till then to help clear the queue.
@simonjayhawkins at some point should have a whatsnew sub-section about the new arrow perf string functions.
sure. I think should probably draw a line under the str accessor methods for now (this is perf only as str accessor methods all fallback if not implemented on the array, we have parameterised test for most) and concentrate on dtype/docs/skipped tests. Will do that in parallel with preparing for the rc. 
> @simonjayhawkins at some point should have a whatsnew sub-section about the new arrow perf string functions.

added to #39908
ENH: Deprecate non-keyword arguments for MultiIndex.set_levels
State: closed

- [X] xref  #41486
- [X] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [X] whatsnew entry


Labels:
name: MultiIndex
name: Deprecate
description: Functionality to remove in pandas

Comments:
Looks like you need to tinker with stacklevels in some of the warnings
Made all the changes.
failure is unrelated; thats the build that tests against numpy's dev branch and raises if we encounter and FutureWarnings, which we now do all over
Thanks @01-vyom
DEPR: Timestamp.freq
State: closed

- [x] closes #15146
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

This in turn will allow us to remove DatetimeArray.freq and TimedeltaArray.freq, which is needed for #31218

Labels:
name: Frequency
description: DateOffsets
name: Deprecate
description: Functionality to remove in pandas

Comments:
Changed your post to avoid closing the other issue too
this would be good to get in the rc
> this would be good to get in the rc

what's the soft-deadline for this?
> > this would be good to get in the rc
> 
> what's the soft-deadline for this?

soonish

cc @simonjayhawkins 
Repeating my non-inline review comment again here to ensure it was noticed:

> Accessing the `.freq` attribute of a Timestamp instance itself is not yet deprecated? That should also be deprecated to be able to remove it?
> Repeating my non-inline review comment again here to ensure it was noticed:

Thanks, will get to that shortly.
> > > this would be good to get in the rc
> > 
> > 
> > what's the soft-deadline for this?
> 
> soonish

I'll put the blocker on this also for now.

will release 1.2.5 first. so still got a few days for this.
merging, @jbrockmendel can you followup with respect to any open comments (i think just the whatsnew but maybe more)
This resulted in a performance regression https://pandas.pydata.org/speed/pandas/#tslibs.timestamp.TimestampProperties.time_freqstr?Cython=0.29.21&jinja2=&python=3.8

however, we will be ignoring this https://github.com/pandas-dev/pandas/pull/40008#issuecomment-872368316

> BTW looking at the regressions listed, I think we can hit "ignore" for the TimestampProperties.time_freqstr ones, since those are driven by a deprecation of freqstr (and are still measured in ns)
Deprecate non-keyword arguments in mask
State: closed

- [x] xref #41485
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
hmm other is prob ok as a positional as well here
> hmm other is prob ok as a positional as well here

OK, thanks for stepping in - I'll ask for it to be positional in the `where` PR too then
@ShreyDixit can u rebase
thanks @ShreyDixit 
Thank you everyone for you guidance
BUG: Series(range_obj_outside_i8_bounds)
State: closed

- [x] closes #30173
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
Thanks @jbrockmendel
DOC: fixup docstring of deprecate_nonkeyword_arguments
State: closed

- [ ] closes #41553
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @marty-rudolf
ENH: Deprecate positional arguments for DataFrame.fillna and Series.fillna (GH41485)
State: closed

- [x] xref #41485
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry




Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
@MarcoGorelli

I was able to find a few hundreds of `DataFrame.fillna` or `Series.fillna` usages, mostly in tests, and didn't see any usages like `srs.fillna(0, None)`  where `method` and other args are passed in directly as positional. Those usages include a few function calls in string format like [here](https://github.com/pandas-dev/pandas/blob/master/pandas/tests/groupby/test_groupby.py#L2066-L2067) and [here](https://github.com/pandas-dev/pandas/blob/master/pandas/tests/apply/test_frame_transform.py#L42-L52). So please let me know if I missed anywhere.

On a separate note, should positional args for `Block.fillna` also be deprecated?
Nice, thanks!

> On a separate note, should positional args for Block.fillna also be deprecated?

The value of `inplace` doesn't change the return type there, so it's not so crucial for overloads, and I think it's only used internally, so I'm pretty sure it doesn't need changing
Gotcha! 
thanks @jackzyliu
DEPR: DataFrame([categorical, ...]) special casing
State: closed

- [x] closes #38845
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Categorical
description: Categorical Data Type
name: Deprecate
description: Functionality to remove in pandas

Comments:
thanks @jbrockmendel
DEPR: Series/DataFrame with tzaware data and tznaive dtype
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Timezones
description: Timezone data dtype
name: Deprecate
description: Functionality to remove in pandas

Comments:
@jbrockmendel can you resolve conflicts
and add a release note. and a link to where this has been discussed.
whatsnew added + green
ENH: Deprecate non-keyword arguments for Index.set_names.
State: closed

- [ ] xref #41485
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Index
description: Related to the Index class or subclasses

Comments:
can you merge master
CLN: Replace FrameOrSeriesUnion annotation by DataFrame | Series
State: closed

- [ ] closes #40876 
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


The FrameOrSeriesUnion check was already removed. I have removed the FrameOrSeriesUnion alias, and replaced all instances of FrameOrSeriesUnion with DataFrame | Series

Labels:
name: Clean
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
HI, I have created a new commit. Please review it. Thank you.
> HI, I have created a new commit. Please review it. Thank you.

Thanks - can you fetch and merge upstream/master please?
Hi @MarcoGorelli, I seem to be stuck. I updated the forked repo on my github page by fetch and merge upstream, and tried to push changes. However, I cannot see the Compare and Pull Request button that usually appears. Initially, I tried force pushing as git would not let me push changes. Thanks. 
> I cannot see the Compare and Pull Request button that usually appears

If you've already opened a pull request (as you have), there's no need to open a new one, you can just add a commit and push it
Looks like there's a merge conflict, could you resolve it please?

My workflow for doing this would be
```
git fetch upstream
git merge upstream/master
# resolve any conflicts
git commit -a
git push origin
```
Hello @Swanand01! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:















































There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-06-11 17:30:30 UTC
REF: Grouper.grouper -> Grouper._gpr_index
State: closed

ATM we set `self.grouper` in both _get_grouper and in _set_grouper, but to different types.  This changes _set_grouper to set _gpr_index instead, which we can type as `Index | None`

Labels:
name: Refactor
description: Internal refactoring of code
name: Groupby

Comments:
looks fine, pls rebase
rebased + green
[ArrowStringArray] REF: str.extract - precusor to move from accessor to array
State: closed

perf neutral refactor

combine  `_str_extract_expand`, `_str_extract_noexpand` and `_groups_or_na_fun` into a single function `_str_extract` ready to move to array method.

Labels:
name: Refactor
description: Internal refactoring of code
name: Strings
description: String extension data type and string data

Comments:
great
BUG: lib.infer_dtype with mixed-freq Periods
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Also adds a convert_period option to lib.maybe_infer_objects


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Period
description: Period data type

Comments:
> does this have any user facing changes?

`replace` handles periods better 
> > does this have any user facing changes?
> 
> `replace` handles periods better

kk can you add a note.
whatsnew added + green
thanks
REF: make array_to_datetime single-pass
State: closed

Or more specifically, remove the need for falling back to datetime_to_datetime64 which does a second pass.

Labels:
name: Datetime
description: Datetime data dtype
name: Timezones
description: Timezone data dtype
name: Stale

Comments:
cc @mroeschke 
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
can you rebase
> can you rebase

yah will circle back before too long, need to address some other comments above.  pretty focused on MultiIndex bugs ATM.
mothballing to clear the queue, will revisit
Allow deprecate_nonkeyword_arguments to not specify version, show class name
State: closed

xref https://github.com/pandas-dev/pandas/pull/41511#discussion_r633386611

as request, ping @jreback @simonjayhawkins 

I've removed the ability to pass an integer to `allowed_arguments` as it added complexity, was only used once in the codebase anyway, and wasn't even covered by tests

Now, if `version` is None, then it'll just print "In a future version of pandas", it'll show the class name (e.g. `Series.drop_duplicates` instead of just `drop_duplicates`) and won't print `'self'` in the warning message

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
> Now, if version is None, then it'll just print "In a future version of pandas"

Personally, I think it is good practice to mention the version when it will be removed/changed, so forcing us to do it here is maybe actually fine.
> > Now, if version is None, then it'll just print "In a future version of pandas"
> 
> Personally, I think it is good practice to mention the version when it will be removed/changed, so forcing us to do it here is maybe actually fine.

disagree this locks us in, and we don't do this anywhere now. this should be a deliberate decision. 
> disagree this locks us in, and we don't do this anywhere now. this should be a deliberate decision.

1\) we are already doing this for all other cases of deprecated positional keyword arguments, and 2) this has been decided in our version policy
Hi @jorisvandenbossche - deprecated positional keyword arguments are quite rare in pandas, there's not been many so far, and most other FutureWarnings don't mention specific versions, e.g.
```python
            warn(
                "The `inplace` parameter in pandas.Categorical."
                "set_categories is deprecated and will be removed in "
                "a future version. Removing unused categories will always "
                "return a new Categorical object.",
                FutureWarning,
                stacklevel=2,
            )
```

It's true that the version policy says

> Deprecations will only be enforced in **major** releases. For example, if a
behavior is deprecated in pandas 1.2.0, it will continue to work, with a
warning, for all releases in the 1.x series. The behavior will change and the
deprecation removed in the next major release (2.0.0).

though it doesn't say that warnings should specify versions.

I have no preference here though, am happy to go with whatever you all want
CLN: remove try/except in _prep_ndarray
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Recent-ish edits to the `convert` function make the fallback unnecessary.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: Clean

Comments:
nice ping on green
ping
ENH: add `styler` option context for sparsification of columns and index separately
State: closed

This PR closes #41142.

It adds a new `pandas.options` context for Styler (see #41395), and create the `sparse.index` and `sparse.columns` options, defaulting to True.

It also allows a keyword application for sparsification via the `.render` method, (this might not really be necessary: opinion welcome)

- [x] tests added for kwarg application and option context 


Labels:
name: Enhancement
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: IO LaTeX
description: to_latex

Comments:
great thanks!
Deprecate passing args as positional in DataFrame/Series.clip
State: closed

- [ ] xref #41485
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
can you rebase
@MarcoGorelli it looks like deprecate_nonkeyword_arguments is not preserving the type annotations. maybe should fix before too many of these deprecations are merged.
Hey @simonjayhawkins - unfortunately, I don't think that's possible. I think we need to choose between:

- making these arguments keyword-only without a warning
- not making these arguments keyword-only and putting up with tons of overloads
- making these arguments keyword-only, but temporarily (until 1.3 is out - we can then remove the `deprecate_nonkeyword_arguments` decorator on master) lose some internal type checking

and I'd like to make the case that the third one is the least undesirable
not tried but cannot do `-> Callable[[_F], _F]`?
I'll try again, hopefully I spoke prematurely
> I'll try again, hopefully I spoke prematurely

Indeed - #41608 - thanks for having pushed me on this :pray:
Deprecate passing args as positional in DataFrame/Series.interpolate
State: closed

- [ ] xref #41485
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
Maybe allow `method` to be positional?
can you rebase
thanks @MarcoGorelli 
@MarcoGorelli if you wouldn't mind adding these to the deprecations list see #30228 (I added this one).

also after these are 'done', it may make sense to make a separate deprecation section and/or normalize the language for these in the whatsnew.
will do - and yes, once they're in I'll sort out the whatsnew entry