BrandonFors/pandas-github-issues huggingface dataset:
REGR: prefer user-provided mode
State: closed

follow-up to #39253

- [x] closes #39247
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Make sure that the user-provided mode is always preferred (if it contains 't' or 'b') and extend list of text-only classes.

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
@twoertwein ping when ready here (merge master as well)
I assume that the occasional errors in `test_server_and_default_headers` can be fixed with `pytest --dist=loadfile`?
> I assume that the occasional errors in `test_server_and_default_headers` can be fixed with `pytest --dist=loadfile`?

yeah that makes sense. separate PR though
thanks @twoertwein 
@meeseeksdev backport 1.2.x
Something went wrong ... Please have  a look at my logs.
Perhaps I'm misusing the API, but this seems to have broken the ability to write gzipped JSON.  I haven't fully walked the code path, but here is what I'm seeing:

Reproduction:
```python
from io import BytesIO
import pandas as pd

dataframe = pd.DataFrame([1, 2, 3], columns=['a'])
object_stream = BytesIO()
dataframe.to_json(object_stream, compression='gzip', orient='records', lines=True)
```

Result:
```python
        if path_or_buf is not None:
            # apply compression and byte/text conversion
            with get_handle(
                path_or_buf, "wt", compression=compression, storage_options=storage_options
            ) as handles:
>               handles.handle.write(s)
E               TypeError: a bytes-like object is required, not 'str'

../../.env/lib/python3.7/site-packages/pandas/io/json/_json.py:105: TypeError
```

***

From what I can tell, _json's get_handle uses a static 'wt' mode which now, with the latest changes to common's _is_binary_mode, will always return false rather than check the path_or_buf instance type, thereby omitting the b flag on the mode passed to the rest of get_handle.

Is this expected behavior/am I misusing the Pandas API? Not the reproduction code works with Pandas 1.2.1
BUG: RangeIndex concatenating incorrectly a single object of length 1
State: closed

- [x] closes #39401
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
@jreback All green! Good to merge?
thanks @AnnaDaglis very nice. keep em coming!
ENH: Add compression to read_stata and StataReader
State: closed

Add support for reading compressed dta files directly

xref #26599

- [X] closes #26599
- [X] tests added / passed
- [X] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [X] whatsnew entry


Labels:
name: IO Stata
description: read_stata, to_stata

Comments:
cc @twoertwein if any comments
@bashtage can you merge master and ping on green
@jreback green, except for unrelated failure.

thanks @bashtage
BUG: Assert_frame_equal always raising AssertionError when comparing extension dtypes
State: closed

- [x] closes #39410
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Regression
description: Functionality that used to work in a prior pandas version
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
LGTM pending green
thanks @phofl 
@meeseeksdev backport 1.2.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.2.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 201d263aefecabc84ee49add65f37d06b45966e9
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #39423: BUG: Assert_frame_equal always raising AssertionError when comparing extension dtypes'
```

4. Push to a named branch :

```
git push YOURFORK 1.2.x:auto-backport-of-pr-39423-on-1.2.x
```

5. Create a PR against branch 1.2.x, I would have named this PR:

> "Backport PR #39423 on branch 1.2.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
cc @phofl if you wouldn't mind doing the backport here
Done
BUG: Subset slicer on Styler failed on MultiIndex with slice(None) 
State: closed

- [x] closes #33562
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

The problem with above issue was with the function `non_reducing_slice`. It didn't correctly re-format slicing tuples on multiindices when rows or cols also contained a `slice(None)`. An additional check is now done. This method is only used for `Styler` currently.

Tests Added / reformmatted.

Labels:
name: Bug
name: MultiIndex
name: Styler
description: conditional formatting using DataFrame.style

Comments:
ready for review..
can you merge master
cc @phofl @jbrockmendel if you'd have a look
lgtm
thanks @attack68
Read hdf returns unexpected values for categorical
State: closed

- [V] closes #39189
- [V] tests added / passed
- [V] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [V] whatsnew entry

This bug happens when filtering on categorical string columns and choose a value which doesn't exist in the column. Instead of returning an empty dataframe, we get some records. It happens because of the usage in numpy `searchsorted(v, side="left")` that find indices where elements should be inserted to maintain order (and not 0 in case that the value doesn't exist), like was assumed in the code.
I changed it to first the for the value, and use `searchsorted` only if value exists, I also added a test for this specific use case. I think in the long run, maybe we should refactor this area in the code since one function covers multiple use cases which makes it more complex to test.

In addition, I moved the logic to a new method to keep the single-responsibility principle and to make it easier to test.

Labels:
name: IO HDF5
description: read_hdf, HDFStore
name: Categorical
description: Categorical Data Type

Comments:
note that you don't need to open a new PR, you can simply push to the original one. since you already did, then ok.
I think the failures here are not related to my changes.
thanks @nofarm3
DOC: Update mypy version in whatsnew
State: closed

- [x] whatsnew entry

Forgot this in #39407

Labels:
name: Docs
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
Is backporting here necessary? Meaning regression fixes which do not pass with old mypy?
> Is backporting here necessary?

no we keep the style checks etc on the backport branch unchanged.

so ci on 1.2.x uses mypy 0.782 and we need to make changes to autobackports if they fail the ci.
Ah thanks for the info
update link to config.py in options user guide
State: closed

- `config.py` now `config_core.py` ?!
- maybe also [pandas/_config/config.py](https://github.com/pandas-dev/pandas/blob/master/pandas/_config/config.py)

- link in [`user_guide/options` ](https://pandas.pydata.org/pandas-docs/stable/user_guide/options.html) changed. 

Labels:
name: Docs

Comments:
thanks @enryH
BUG: make Index.where behavior mirror Index.putmask behavior
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Index
description: Related to the Index class or subclasses

Comments:
needs a rebase. can you add a whatsnew on what is changing here
one more rebase
rebased + green
can you merge master
rebased + finally green
TYP: Make mypy 0.800 compatible
State: closed

- [x] closes #39399
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
LGTM cc @simonjayhawkins 
@simonjayhawkins green.

Could you point me to an example of something like ``str | bytes``? Searched for something like this but could not find it
> Could you point me to an example of something like `str | bytes`? Searched for something like this but could not find it

from http://mypy-lang.blogspot.com/

Two new Python features improve this situation and are now supported by mypy:

PEP 585 lets you use list[int] instead of List[int] (no need to import List and other generic collections from typing).
PEP 604 lets you write X | Y instead of Union[X, Y], and X | None instead of Optional[X] (no need to import Union or Optional from typing).
Note: Using list[int] requires Python 3.9 and X | Y requires Python 3.10 (alpha) in order to work at runtime. To use them on older versions of Python, use from \_\_future__ import annotations. This allows them to be used in type annotations, but the older variants (or string literal escaping) may be required in non-annotation contexts, such as in type aliases. See the docs for more details.
thanks very much, should have looked there and not in our codebase :)
@phofl also need to update the version in the doc/source/whatsnew/v1.3.0.rst
Thx, did not think about this. #39419
ENH: implement EA.delete
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Mmm @jbrockmendel what was the motivation for this change (can you just always provide that, for future readers?) In general I think we should carefully consider adding new methods, since that increases the burden on extension array maintainers to write efficient implementations.

If we are adding methods to the public API then they should be documented. This would need a docstring and it should be included in the class docstring.
An arbitrary EA-backed Index will need a `delete` implementation.
I assume there are calls to `np.delete` in core/indexes or core/indexing? Can those calls be rewritten to use `take`  instead of `np.delete`?
There is an Index.delete method that would need to be implemented.  So de-facto we have to implement a delete method or something equivalent to it.

(np.delete wont work directly until we get `__array_function__` up and running)
> There is an Index.delete method that would need to be implemented. So de-facto we have to implement a delete method or something equivalent to it.

meta-comment: this type of context is useful for evaluating PRs. Can you include motivation when creating a PR?

My preference would be to rewrite `Index.delete` to use a `take` or masked-based implementation, if it's not slower than the `np.delete` implementation. And `np.delete` seems to be implemented in Python, so I imagine we can make it equally fast.


> if it's not slower than the np.delete implementation

Maybe there's a faster take/getitem-based imlpementation, but

```
In [3]: arr = np.arange(1000)

In [4]: %timeit np.delete(arr, 500)
4.49 µs ± 35.4 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [5]: %timeit mask = np.delete(np.arange(1000), 500); arr.take(mask.nonzero()[0])
13.6 µs ± 284 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

In [6]: %timeit mask = np.delete(np.arange(1000), 500); arr[mask]
7.83 µs ± 279 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)
```
Clean up DataFrame.setitem behavior for duplicate columns
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

cc  @jbrockmendel This cleans the edgy behavior up for duplicate columns.

Previous test was wrong.

Labels:
name: Refactor
description: Internal refactoring of code
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
If you're interested, there might be something worth salvaging from _setitem_array [here](https://github.com/jbrockmendel/pandas/blob/setitem-frame-no-defer/pandas/core/frame.py#L3244).  Avoids going through iloc so that we dont have to worry about inplace vs not-inplace there (xref #38896).

If/when I get #39302 in shape to be a real PR, something like that branch will have to be a part of it, but I'm optimistic you'll get around to it before I do.
That looks good. I will try to incorporate this after this is merged.
thanks @phofl 

was this technically a bug fix? or just an incorrect test? 
Bug fix yes, but is covered by the whatsnew from #39341, follow up from that one
BUG: re-render CSS with `styler.apply` and `applymap` non-cleared ` _todo`
State: closed

- [ x] closes #39395
- [ x] tests added / passed
- [ x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
@jreback, changes added, is green.
thanks @attack68
ENH: Fix support for matplotlib's constrained_layout (#25261)
State: closed

- [x] closes #25261
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Visualization
description: plotting

Comments:
The failing tests seem totally unrelated!?
thanks @mdruiter very nice!
ENH: `Styler.apply` accept ndarray return with `axis=None` for consistency 
State: closed

- [x] closes #39359
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry

Also minor edits to docs to extend information and show examples.

cc @TomAugspurger 

Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
this is ready for review.

It fixes the functionality and edits the docs where necessary, including some better examples.
> not sure we should change this. why can't the user just wrap the return value _inside_ the apply.

Dont know why this restriction was placed in the first place. If you use `axis in [0,1]` then your function must return 1-dim array like (numpy array allowed), but `axis=None` must need a `DataFrame`. This makes it quite painful to implement simple numpy functions like `nanmax` cause the user has to deal with the different cases and match the return type.

The builtin functions always have, for example, logic like this:
```
        if data.ndim == 1:  # Series from .apply
            return [attr if v else "" for v in extrema]
        else:  # DataFrame from .tee
            return pd.DataFrame(
                np.where(extrema, attr, ""), index=data.index, columns=data.columns
            )
```

I  dont really understand the downside to just handling an array function output in all cases for axis.
@jreback changes done. mostly green. fails unrelated it seems
thanks @attack68
CI: split benchmarks from other checks
State: closed

That job was the longest-running, so this should parallelize the various code checks with the benchmarks to get done in less wall clock time.

This pull request builds on https://github.com/pandas-dev/pandas/pull/39745.

cc @fangchenli

- [ ] ~~closes #xxxx~~
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] ~~whatsnew entry~~


Labels:
name: Refactor
description: Internal refactoring of code
name: CI
description: Continuous Integration
name: Benchmark
description: Performance (ASV) benchmarks
name: Stale

Comments:
> not sure this is actually running we had 19 checks before

Ah, whoops, I thought that was a GitHub Actions glitch, but as usual, Problem Exists Between Chair and Keyboard 😛  Hadn't realized that when using [an action in the same repository](https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#example-using-action-in-the-same-repository-as-the-workflow):

- The path needs to start with `./`
- The job needs to run `actions/checkout` before that action can be used (which makes sense, now that I think about it)
- Need to specify the `shell` for every step (`defaults` aren't inherited)

20 checks running now, as intended!
@afeld can you merge master
can you merge master
i merged this one: https://github.com/pandas-dev/pandas/pull/39745
Hmm, went from the longest run being `CI / Checks` at [48 minutes](https://github.com/pandas-dev/pandas/runs/1884091523?check_suite_focus=true) to `CI / Benchmarks` taking 46. Better, but not a _lot_ better 😕

Flaky tests failing, but `CI` ones are 🟢. 
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
@afeld is this ready?
Not sure if this is worth. We've got an overheat of 15 minutes, and we are saving 5 from 50 for an extra build. No strong opinion, but I think our CI should be faster without this with the amount of builds we usually have running (unless we've got many more than the 20 GH actions jobs, that I don't think we did).
> Not sure if this is worth. We've got an overheat of 15 minutes, and we are saving 5 from 50 for an extra build. No strong opinion, but I think our CI should be faster without this with the amount of builds we usually have running (unless we've got many more than the 20 GH actions jobs, that I don't think we did).

@afeld thoughts on this?
@afeld closing as stale. feel free to reopen if you want to continue with this. (@datapythonista has a question)
CI: save site+docs builds as artifacts
State: closed

 Makes the downloadable for previewing locally. Not as good as having a URL to preview it, but better than nothing.

- [x] closes https://github.com/pandas-dev/pandas/issues/17921
- [ ] tests ~~added /~~ passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] ~~whatsnew entry~~

Labels:
name: Docs
name: CI
description: Continuous Integration
name: Web
description: pandas website

Comments:
Personally, I think I'd merge both directories and upload just one artifact. Not a big deal, happy to get this merged as is too. But I think it should make things simpler, and being able to navigate from the web to the docs could be useful.
Thanks @afeld 

I was going to say that indeed if it's not too hard, it would be nice to have a single download. But a potential reason to keep them separate is that the docs are a huge download compared to the website. So if you only want to check the website, having a separate download might be nicer. 

Now, I am wondering if we can't get the size down (the docs are now 180Mb), but it seems the largest part is simply all the individual API pages, so no easy way around this with the current generated API docs.


@afeld pls update when you can
Updated to include both the site and docs in the same download. Sorry for the delay!
thanks @afeld
PERF: Rolling.cov/corr
State: closed

- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Will attempt to improve the performance of `groupby.rolling` next.

```
       before           after         ratio
     [37b5800a]       [ee9a56c1]
     <master>         <ref/rolling_cov_corr>
-        10.4±2ms       8.90±0.2ms     0.86  rolling.Pairwise.time_pairwise(10, 'cov', True)
-        10.2±1ms       8.52±0.1ms     0.83  rolling.Pairwise.time_pairwise(None, 'cov', True)
-      2.48±0.1ms      1.07±0.01ms     0.43  rolling.Pairwise.time_pairwise(1000, 'cov', False)
-      2.55±0.2ms      1.08±0.02ms     0.42  rolling.Pairwise.time_pairwise(10, 'cov', False)
-     2.33±0.04ms          918±6μs     0.39  rolling.Pairwise.time_pairwise(None, 'cov', False)
-      4.02±0.6ms      1.46±0.02ms     0.36  rolling.Pairwise.time_pairwise(10, 'corr', False)
-      4.08±0.4ms       1.42±0.1ms     0.35  rolling.Pairwise.time_pairwise(1000, 'corr', False)
-     3.69±0.07ms      1.17±0.03ms     0.32  rolling.Pairwise.time_pairwise(None, 'corr', False)

SOME BENCHMARKS HAVE CHANGED SIGNIFICANTLY.
PERFORMANCE INCREASED.
```

Added a `ddof` argument to `corr` which was implicitly accepted (and used) by `kwargs` before; making it an explicit kwargs in this PR. Will document it after https://github.com/pandas-dev/pandas/pull/39219 is in.

Labels:
name: Performance
description: Memory or execution speed performance
name: Window
description: rolling, ewma, expanding

Comments:
thanks @mroeschke
ENH: PandasArray.value_counts
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Refactor
description: Internal refactoring of code

Comments:
sure, merge master just to be sure
rebased + green

neat, just about out of tests/extension xfails that i can address ATM