BrandonFors/pandas-github-issues huggingface dataset:
CLN: use getvalue() instead of seek(0)+read()
State: closed

`BytesIO`'s `getvalue()` is the same as `seek(0)` followed by `read()` if the buffer is not used in any other place (`getvalue` doesn't change the position within the buffer). getvalue is slightly faster then seek+read.

Labels:
name: IO Stata
description: read_stata, to_stata
name: Clean

Comments:
thanks @twoertwein
BUG: DataFrame.apply axis=1 for str ops with no axis argument
State: closed

- [x] closes #39211
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry



Labels:
name: Bug
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @rhshadrach pls create an issue (or PR!) for re-orging the apply tests :->
DOC: add new page 'how to read these doc' in get started section
State: closed

This page explains the code block are run in notebook

- [x] closes #39130
- [x] ~tests added / passed~
- [x] ~Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them~
- [x] ~whatsnew entry~


Labels:
name: Docs

Comments:
Here is a screenshot of the new page: 

![image](https://user-images.githubusercontent.com/15191323/104813161-b46d2800-5807-11eb-93d9-7f174515cd46.png)

instead of a separate page how about a note at the top of the getting started docs.
Thanks for the PR @rhuille but it appears this PR has gotten stale. Please let us know if you have time to merge master and to implement this as suggested in https://github.com/pandas-dev/pandas/pull/39207#issuecomment-779469318 and we'd be happy to reopen. Closing for now.
Hi @mroeschke, I've picked up where @rhuille left off and I've added the changes to the top of the Getting Started page.
Does this look better? If so, I'll submit a new PR for my branch and reference it here.

![PandasChanges](https://user-images.githubusercontent.com/20362314/155195157-87be8f83-1e29-4406-8ff5-818bcf419fdc.png)
Thanks @ehallam this looks pretty good. One suggestion: Instead of putting this in `Getting Started`, I think it's would be more appropriate in https://pandas.pydata.org/docs/user_guide/index.html since that's where most of the code snippets are.
Does this look good @mroeschke? I've also added a "Guides" heading to separate the "How to read these guides" section and the list of guides below.

![PandasChanges4](https://user-images.githubusercontent.com/20362314/155207328-f6714470-d392-4c87-81bf-3952b017997e.png)
Looks great! Looking forward to the PR 👍
BUG: assert_frame_equal raising TypeError with check_like and mixed dtype in Index or columns
State: closed

- [x] closes #39168
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Worked on 1.1.5 for assert_frame_equal and was not supported for assert_index_equal

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
@jreback green 
thanks @phofl 
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 c5b4272ed1e7d71266e06660ce9970527711fd55
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #39204:  BUG: assert_frame_equal raising TypeError with check_like and mixed dtype in Index or columns'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-39204-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #39204 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
REGR: NaT.__richmp__(dateobj)
State: closed

- [x] closes #39151
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Datetime
description: Datetime data dtype
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
thanks @jbrockmendel 
@meeseeksdev backport 1.2.x
Something went wrong ... Please have  a look at my logs.
REGR: fillna on datetime64[ns, UTC] column hits RecursionError
State: closed

- [ ] closes #38851
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

@jbrockmendel this reverts #37040 in it's entirety for backport

feel free to push to this branch if we **need** any of the ancillary cleanup on 1.2.x otherwise can open a new PR to add those cleanups back to master.

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
> if we need any of the ancillary cleanup

nope

I'll make a PR with a longer-term fix before long, but this seems like the right immediate solution
thanks @simonjayhawkins 
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 fe92f93e16b6d24f1b8917ee6d2b79952562a0c9
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #39194: REGR: fillna on datetime64[ns, UTC] column hits RecursionError'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-39194-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #39194 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 fe92f93e16b6d24f1b8917ee6d2b79952562a0c9
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #39194: REGR: fillna on datetime64[ns, UTC] column hits RecursionError'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-39194-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #39194 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
cc @simonjayhawkins for backport
CI: Set xfail to strict=False for network tests
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

This one is ugly. Botocore and boto3 were upgraded tonight to 1.19.55 and 1.16.55. Since aiobotocore requires them to be .52 this is no longer installed, but aiobotocore is an required dependency from s3fs 0.5.*. So this get's downgraded to 0.4.2 were only botocore was a required dependency (no version specification). As soon as the requirements of aiobotocore are updated again this will switch back to s3fs=0.5.2 which will cause the tests to fail again. Hence setting them to strict=False

sorry for the long explanation :)

Labels:
name: CI
description: Continuous Integration
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues
name: Dependencies
description: Required and optional dependencies

Comments:
Other option is just to pin s3fs in CI files? @phofl 
Could do this currently, but do not know if we risk conflicts there in the future
Would prefer a version check of s3fs over pinning, but this might be overkill
this is fine for 1.2.1

i think pinning is fine for 1.3 (and then we can revert that when this is fixed).
@meeseeksdev backport 1.2.x
Revert "BUG/REG: RollingGroupby MultiIndex levels dropped (#38737)"
State: closed

This reverts commit a37f1a45e83d7f803d7fcab7d384da28e9c1e714.

reverts PR #38737

I think we ended up deciding to revert on master and backport to keep master and 1.2.x in sync and not release this change in 1.2.1 to allow for further discussion.

once this revert PR merged:
- [x] backport
- [x] the blocker tag can then be removed from #38787
- [x] #38523 will be reopen temporarily
- [ ] @mroeschke will open a new PR with these changes (in draft? milestoned 1.2.2?)

if not these changes, the mutliindex with a single level may still need resolution (and backport) to close #38523

cc @mroeschke @jorisvandenbossche 

add 1.2.1 milestone for this revert for backport.



Labels:
name: Window
description: rolling, ewma, expanding

Comments:
thanks @simonjayhawkins 
@meeseeksdev backport 1.2.x
REGR: Different results from DataFrame.apply and str accessor
State: closed

- [ ] closes #38979
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


@jbrockmendel despite my comment https://github.com/pandas-dev/pandas/issues/38979#issuecomment-760470446, I had no joy trying to locate the caching issue.


Labels:
name: Strings
description: String extension data type and string data
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thanks @simonjayhawkins 
@meeseeksdev backport 1.2.x
REF: extract classes in pandas/core/describe.py
State: closed

- [ ] xref #36833
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Extract classes ``SeriesDescriber`` and ``DataFrameDescriber``.
In the next steps I am going to enable strategy pattern and move corresponding describe functions to the concrete strategy classes (for each datatype).

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
thanks @ivanovmg
DOC: set the documentation language
State: closed

Ensures the language is specified in the generated HTML. [Documentation on the configuration option.](https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-language)

Before:

```
$ curl -s https://pandas.pydata.org/docs/dev/getting_started/comparison/comparison_with_spreadsheets.html | grep "<html"
<html>
```

After:

```
$ python make.py --single getting_started/comparison/comparison_with_spreadsheets.rst
$ grep "<html" build/html/getting_started/comparison/comparison_with_spreadsheets.html 
<html lang="en">
```

- [ ] ~~closes #xxxx~~
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] ~~whatsnew entry~~


Labels:
name: Docs

Comments:
seems fine to me. cc @datapythonista
CI/DOC: Add option to build whatsnew
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Adds the option to compile the whatsnew. It also builds the API (to resolve links) and opens the whatsnew page in a browser when complete. This is the most frequent reason why I build the docs, and cuts down build time by 40% on my machine.

Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
@rhshadrach if you can merge master
@jreback Merged and green.
Thanks!
CI: Pin nbformat to 5.0.8
State: closed

- [x] xref #39176

cc @jreback nbformat should be enoguh I hope.

Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
excellent.

cc @simonjayhawkins backporting as well.
thanks @phofl 
@meeseeksdev backport 1.2.x
Something went wrong ... Please have  a look at my logs.
CLN,TYP Remove string return annotations
State: closed


- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

this was just done with
```python
    parser = argparse.ArgumentParser()
    parser.add_argument('paths', nargs='*', type=Path)
    args = parser.parse_args()
    for path in args.paths:
        text = path.read_text()
        text = re.sub(r'-> "(\w+)"', '-> \\1', text)
        path.write_text(text)
```

There's other to do (which I'll get to in other PRs), but this already gets rid of a big load of them

Labels:
name: Clean
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Not a huge deal, but I like the string versions since they make the dependency structure clearer, i.e. I’m not going to incorrectly infer that Series is in the namespace.
Sure, but you can't click and go to a definition if it's a string version (this is my main reason for wanting to make this kind of change)

I think it only makes the dependency structure clearer if string ones are used consistently (currently there's a mix) - up to you anyway, my preference here isn't too strong
> Sure, but you can't click and go to a definition if it's a string version (this is my main reason for wanting to make this kind of change)

some discussion in https://github.com/pandas-dev/pandas/pull/36034#issuecomment-684972993


> I think it only makes the dependency structure clearer if string ones are used consistently (currently there's a mix)

agreed. You can (and people have) quote types even if the type is imported at runtine, or quote the whole type expression such as `"List[grouper.Grouping]"` which the PR resolves.

"we" agreed (https://github.com/pandas-dev/pandas/pull/36034#issuecomment-685186742) on going this route. Do we want another discussion?
> Do we want another discussion?

Nope, I'm pretty happy with my "defer to Simon when it comes to annotations" policy.
OK, merging following https://github.com/pandas-dev/pandas/pull/39174#pullrequestreview-568442160 and https://github.com/pandas-dev/pandas/pull/39174#issuecomment-760992916
Thanks @MarcoGorelli
REF: extract more functions in pandas/core/describe.py
State: closed

- [ ] xref #36833
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Extracted ``describe_series``, ``describe_frame`` and ``select_column`` in ``pandas/core/describe.py``.
The next time I think I will introduce the classes ``SeriesDescriber`` and ``DataFrameDescriber`` and will start moving the functions into the methods incrementally, if that is OK.

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
yep looks good @ivanovmg
REF/TYP: extract function/type args in describe.py
State: closed

- [ ] xref #36833
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Extracted function ``_refine_percentiles`` and typed function arguments in ``pandas/core/describe.py``.

Labels:
name: Refactor
description: Internal refactoring of code

Comments:
xref #37715
Fix regression in loc setitem raising KeyError when enlarging df with multiindex
State: closed

- [x] closes #39147
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

The previous fix only viewed row indexers not column indexers

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version
name: MultiIndex

Comments:
thanks @phofl 
@meeseeksdev backport 1.2.x
ERR: Fix missing space in warning message
State: closed

- [x] closes #39158 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
> Thanks @MicaelJarniac for the PR.
> 
> We normally check error messages in the tests so we would normally expect that changing a message would also require a change in the tests.
> 
> so it maybe that there are no tests that hit this or the message is only partially checked. so further investigation welcome.

I'll try locating the tests related to this specific message (if any), and fix (or add) them.

And I'll also try to think of a way for catching similar issues in future tests, but I don't think it'd be easy, as simply saying "fail if no space after period" would catch this one, but would probably have about a thousand false-positives in other messages, because in Python syntax it's perfectly valid to have `some.Thing`. A test for catching issues similar to this one might end up being overkill.
> so it maybe that there are no tests that hit this or the message is only partially checked. so further investigation welcome.

it appears that test_warn_if_chunks_have_mismatched_type[c_low] hits this. so can modify that test to also check message.
`151214 passed, 5674 skipped, 925 xfailed, 4 xpassed, 1 warning in 1126.89s (0:18:46)`
@MicaelJarniac can you merge master and we'll try to get this in
closing as stale. @MicaelJarniac if you'd like to rebase and fix this up pls ping.
CI: Mark network test as xfail
State: closed

- [x] xref #39155
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

https://github.com/pandas-dev/pandas/issues/39155#issuecomment-759783358


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues

Comments:
great ping on green
@jreback green 
thanks @phofl 
@meeseeksdev backport 1.2.x
CLN remove 'import pandas as pd' from pandas/core/generic.py
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

---

will do the rest (excluding `pandas/tests`) and add a check if this is the kind of thing we want

Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
nice
thanks @MarcoGorelli
ENH: Allow Series.apply to accept list-like and dict-like
State: closed

- [x] closes #39140
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

I only added the Series.apply list/dict-like addition to the whatsnew. For both Series and DataFrame .agg, this PR also enables passing a Series as a dict-like, but other dict-likes already work. It didn't seem notable to add this to the whatsnew.

Labels:
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
if my comment works ok, otherwise lgtm. merge either way.
Thanks @jreback - your comment did work, but I found that just removing the keyword `axis` also works.
thanks @rhshadrach
ASV: Add xlrd engine for xls file
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Is one parametrized function more efficient than multiple functions during setup and teardown? If not I would make a separate function for the xls benchmark

cc @jorisvandenbossche 


Labels:
name: IO Excel
description: read_excel, to_excel
name: Benchmark
description: Performance (ASV) benchmarks

Comments:
> * [x]  Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
> 
> Is one parametrized function more efficient than multiple functions during setup and teardown? If not I would make a separate function for the xls benchmark
> 
> cc @jorisvandenbossche

this is fine, the setup time doesn't make much difference.
@phofl thanks for the follow-up!
PERF: cythonize kendall correlation
State: closed

- [x] closes #28329 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

ASV's. 

       before           after         ratio
     [a6bc6ecc]       [1eb86441]
     <perf-cythonize-kendall~2^2>       <perf-cythonize-kendall>
-             81M            57.6M     0.71  stat_ops.Correlation.peakmem_corr_wide('kendall')
-      1.41±0.02s          362±7ms     0.26  stat_ops.Correlation.time_corr_wide_nans('kendall')
-      1.57±0.04s          398±4ms     0.25  stat_ops.Correlation.time_corr_wide('kendall')
-        34.6±4ms      8.75±0.06ms     0.25  stat_ops.Correlation.time_corr('kendall')

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.

Labels:
name: Performance
description: Memory or execution speed performance

Comments:
@jreback this is now ready for review.
cc @jbrockmendel and @dsaxton(if you guys are interested). 
A small slightly unrelated change that I made in this PR was in the docstring of the min_periods arg for corr. It used to state that min_periods was only valid for spearman and pearson correlations, however, I believe that it always worked for kendall correlation. 
couple nitpicks, generally looks good.  i take it we have sufficient testing?
@jbrockmendel I think testing coverage should be fine. There is a test that compares the results between scipy and our implementation. (see https://github.com/pandas-dev/pandas/blob/master/pandas/tests/frame/methods/test_cov_corr.py#L89)
thanks @lithomas1 very nice!
DOC: fixed a minor mistake worksheet -> workbook
State: closed

A small typo related to #38990
A worksheet can't have multiple worksheets, but a workbook can. Minor fix but may confuse the newcomers

- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Docs
name: IO Excel
description: read_excel, to_excel

Comments:
thanks @pilyavetsb