BrandonFors/pandas-github-issues huggingface dataset:
DEPR: squeeze argument in read_csv/read_table/read_excel
State: closed

- [ ] closes #43242
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: IO CSV
description: read_csv, to_csv
name: IO Excel
description: read_excel, to_excel
name: Deprecate
description: Functionality to remove in pandas

Comments:
Wanted some feedback on the new approach to verify stacklevel in the tests.

Also, I'm not sure how to deprecate this for ``ExcelFile.parse``. It will probably warn there as well, but IDK if we need to update the ExcelFile docs.
th.assert_produces_warning should check the stacklevel and raise if is not correct. This should be enough I think?
> Wanted some feedback on the new approach to verify stacklevel in the tests.
> 
> Also, I'm not sure how to deprecate this for `ExcelFile.parse`. It will probably warn there as well, but IDK if we need to update the ExcelFile docs.

ok to tackle in a followup
thanks @lithomas1 
@phofl Are there other comments that I should address in a follow-up? The current approach is using tm.assert_produces_warning in the conftest since thats where the warnings should be raised.

If OK, I will make another PR updating all the other tests, and changing their stacklevels if necessary.
REF: MultiIndex._engine followup to #43370
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
ty
Backport PR #43150 on branch 1.3.x (BUG: GroupBy.quantile fails with pd.NA)
State: closed

Backport PR #43150

Labels:
name: Groupby
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
@jreback @simonjayhawkins 
There is a difference in what's happening with the index when df has nullable dtypes in 1.3.x vs. master (comparison below).

So I think there are two options 
1. backport the PR which changes this behaviour (I was not able to find the PR though).
2. hardcode the test to pass (which I don't think we should do).

```
In [60]: pd.__version__
Out[60]: '1.3.2+16.g6a54ac4a0a'

In [61]: df = DataFrame({"x": [1, 1], "y": [0.2, np.nan]})

In [62]: df.dtypes
Out[62]: 
x      int64
y    float64
dtype: object

In [63]: df.groupby("x")["y"].quantile(0.5).index
Out[63]: Int64Index([1], dtype='int64', name='x')

In [64]: df=df.astype({"x":"Int64"})

In [65]: df.dtypes
Out[65]: 
x      Int64
y    float64
dtype: object

In [66]: df.groupby("x")["y"].quantile(0.5).index
Out[66]: Index([1], dtype='object', name='x')
```

```In [13]: pd.__version__
Out[13]: '1.4.0.dev0+598.g35d52ff2d5'

In [14]: df = DataFrame({"x": [1, 1], "y": [0.2, np.nan]})

In [15]: df.dtypes
Out[15]: 
x      int64
y    float64
dtype: object

In [16]: df.groupby("x")["y"].quantile(0.5).index
Out[16]: Int64Index([1], dtype='int64', name='x')

In [17]: df=df.astype({"x":"Int64"})

In [18]: df.dtypes
Out[18]: 
x      Int64
y    float64
dtype: object

In [19]: df.groupby("x")["y"].quantile(0.5).index
Out[19]: Int64Index([1], dtype='int64', name='x')

thanks @debnathshoham
Backport PR #43150 on branch 1.3.x (BUG: GroupBy.quantile fails with pd.NA)
State: closed

Backport PR #43150: BUG: GroupBy.quantile fails with pd.NA

Labels:
name: Groupby
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
@debnathshoham 


need to change the `any_float_dtype` fixtures as that is changed in 1.4 e.g. https://github.com/pandas-dev/pandas/pull/43408/checks?check_run_id=3515311784

and update the PR to this branch


Will have to change `any_int_ea_dtype` to `any_nullable_int_dtype`, and `any_float_dtype` to `any_float_allowed_nullable_dtype`.
Again, since not my PR will need help from maintainer.

Little confused why `test_groupby_quantile_allNA_column` is failing though
@debnathshoham you can fork this PR and create a new one (or I can do it.. LMK)
hi @simonjayhawkins - not sure how to fork this PR (didn't get anything on google; can try if you can point me to something that explains how to do it).
Or, are you suggesting I fork the `1.3.x branch`? 
There's an "open with" button in the top right next to the issue title. (if using github through the web)
Closing in favor of #43417
BUG: DataFrame.__setitem__ sometimes operating inplace
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

The fixed bug is the currently-xfailed test test_setitem_same_dtype_not_inplace (see #39510, which has its own subsection in the 1.3.0 release notes).  Pretty much all the other altered tests are changed because those tests rely on the incorrect behavior.

The main non-test change is the addition of an `inplace` check in `BlockManager.iset`.  We want inplace to be False when this is reached via `DataFrame.__setitem__` and True when reached via `loc/iloc.__setitem__`

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
cc @jorisvandenbossche 
cc @phofl 
@jorisvandenbossche @phofl gentle ping
@jorisvandenbossche gentle ping
@jreback gentle ping, doesn't sound like the others are weighing in
I think this looks good. Have to figure out the boolean case in the future, but otherwise is good
let me look
id like to get this in before tackling #43996
gentle ping; has a bearing on #40380
cool, ideally can simplify the caching checks at some point (e.g. the maybe_update_cacher is getting more complicated).
`df.assign` uses `DataFrame.__setitem__`:

https://github.com/pandas-dev/pandas/blob/965ceca9fd796940050d6fc817707bba1c4f9bff/pandas/core/frame.py#L4841-L4845

> We want inplace to be False when this is reached via `DataFrame.__setitem__` and True when reached via `loc/iloc.__setitem__`

Did this change intend to modify `df.assign`'s behavior? Or do you think `assign` should be using the in-place mode again through `loc/iloc`?
@SimeonStoykovQC no, `assign` should not modify the existing data so not be in-place. I guess an in-place modification on a *copy* would be fine though
@SimeonStoykovQC I misunderstood what you meant. You meant to use `[...]` instead of `.loc[:, ...]` on the copy, right? That makes sense if `.loc` is faster (which I don't know)
No, the assign change is intended.
It caused a 10x performance regression in some of our code. We will try to come up with a reproducer. 
> No, the assign change is intended.

In the meantime, could you elaborate? What's wrong with in-place allocations if [`df.assign` is creating a deep copy (prior version 2.0)](https://github.com/pandas-dev/pandas/blob/2e218d10984e9919f0296931d92ea851c6a6faf5/pandas/core/frame.py#L4886)?
Yeah I know, but this was fixed for 2.0, we had a nasty performance bottleneck in 1.5 unfortunately. Everything is back to normal with 2.0
TYP: type some un-typed decorators
State: closed

-  xref #33455
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
thanks @twoertwein
REF: `styler.to_latex` siunitx changes: render {} only when needed.
State: closed

adds item on #41649 for better backwards compat, and this is probably slightly better anyway.

When:

 - `siunitx=True` columns headers are now rendered like `{} & {A} & {B} \\`
 - `siunitx=False` (default) then they are now rendered like ` & A & B \\` 

Rndering column headers with or without curly braces has no effect in the LaTeX output, except when using `siunitx` where they are required for it to work properly.


Labels:
name: IO LaTeX
description: to_latex
name: Styler
description: conditional formatting using DataFrame.style

Comments:
sure
BUG: dtype lost in DataFrame.append
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

1) For BlockManager, this improves dtype-retention, see test_append_same_columns_type
2) For ArrayManager, this fixes an unwanted casting to object, see test_append_empty_frame_to_series_with_dateutil_tz
3) I'm increasingly confident that with this, we can get rid of several kludges in internals.concat
4) This does change the order of columns in some cases, see test_append_empty_frame_to_series_with_dateutil_tz.  AFAICT this is due to a bug in union_indexes (but fixing it breaks other things that i need to run down), xref #43375

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
whatsnew added + green
Backport PR #43152 on branch 1.3.x (BUG: Outer/right merge with EA dtypes cast to object)
State: closed

Backport PR #43152: BUG: Outer/right merge with EA dtypes cast to object

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.

Comments:
Is this because, maybe `any_numeric_ea_dtype` is not defined in 1.3.x?
Hello @meeseeksmachine! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-09-04 18:47:28 UTC
Rats. The line got too long when I updated the name in the web editor. I'll fix it if all other CI passes.
@github-actions pre-commit
thanks @debnathshoham  and @lithomas1
ENH: `multicol` naive implementation. part2
State: closed

follows #43369

Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
@ivanovmg you fancy reviewing this one too?
@ivanovmg changes made.

I disagree with your philosophy on logic in tests. Personally I prefer parametrizations to focus as much as possible on as few parameters, and be an indicator to the purpose of the test.
The purpose of the test is not really to check `siunitx` affects the column format and level1 header, it just happens to do this so the test has to be structured correctly. This test is small enough so it doesn't really matter either way but that is at least my reasoning.
Auto-update now occurs monthly instead of weekly
State: closed

Fixes issue #43364

- [x] closes #44364
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
Thanks for the PR. Closing since https://github.com/pandas-dev/pandas/pull/43378 is addressing this.
REF: reindex_indexer use np.void to avoid JoinUnit
State: closed

ATM we do a kludgy thing where we sometimes reindex before calling concat (e.g. in DataFrame.append) and that leads to all-NA columns that we choose to ignore when choosing the dtype for the concatenated column.  This leads to a) funky behavior when we encounter an "real" all-NA column and b) a big performance hit in checking for all-NA columns.

Instead of doing that, this PR implements the keyword use_na_proxy (chosen to match ArrayManager's keyword) that ends up creating ndarrays with np.void dtype instead of all-NA float64 ndarrays.

This will allow us to simplify a bunch of code in internals.concat (e.g. get rid of case where `unit.block is None`) and improve the funky behavior mentioned above.  Both of these are left to follow-ups to keep the diff manageable.

Labels:
name: Refactor
description: Internal refactoring of code
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
greenish, just test_to_csv_compression_encoding_gcs
BUG: rolling.groupby with on and __getitem__ doesn't mutate underlying object
State: closed

- [x] closes #43355
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [x] whatsnew entry


Labels:
name: Groupby
name: Window
description: rolling, ewma, expanding

Comments:
Thanks @mroeschke This does not close the issue? (as indicated in the OP)

There are still the issues of the inconsistent indexes between `grp.count()` and `grp["y"].count()` and the issue of the change in index from 1.2.5?
Sorry, since https://github.com/pandas-dev/pandas/issues/43355 reported 2 bugs, this just addresses one of them.

I just opened up https://github.com/pandas-dev/pandas/issues/43405 separately since this PR does not address the `index` inconsistency.
> I just opened up #43405 separately since this PR does not address the `index` inconsistency.

Thanks @mroeschke 

> There are still the issues of the inconsistent indexes between `grp.count()` and `grp["y"].count()` and the issue of the change in index from 1.2.5?

there was a typo here. the index didn't change from 1.2.5, the result columns did. There was a change in index from 1.1.5. I've commented on the issue. https://github.com/pandas-dev/pandas/issues/43405#issuecomment-913105340
thanks @mroeschke
CLN: Minor cleanup of deprecations in parser code
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them
- [ ] whatsnew entry

Some spaghetti that I noticed and was annoying.

Labels:
name: IO CSV
description: read_csv, to_csv
name: Clean

Comments:
thanks @lithomas1
PERF: MultiIndex.get_indexer
State: closed

Avoids expensive construction of MultiIndex.values, followed by expensive casting of sequence-of-tuples back to tuple-of-Indexes.

```
import pandas as pd

mi = pd.MultiIndex.from_product([range(100), range(100), range(100)])

%timeit mi.get_indexer(mi[:-1])
698 ms ± 18.7 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)  # <- master
157 ms ± 1.21 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- PR
```

Follow-up will move _extract_level_codes out of cython so that Engine._get_indexer and Engine._get_indexer_non_unique are restored to having consistent signatures.

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance
name: MultiIndex

Comments:
lgtm but if you can add some comments, ping on green
ok fair enough all of this is fine for a followon.
also since this is a huge speedup this migth improve some concat / join asvs for multi-multi (if so that is worth a whatsnew note0
ENH: `multirow` naive implementation `Styler.to_latex` part1
State: closed

this is also for compatability with `DataFrame.to_latex`. #41649 

I think its a nice feature though, although opinions on the argument input choice are welcome. Maybe `"none"` is better than `"naive"`.

When implementing this for multicol there will be the option for left or right alignment, so the likes of `"naive-left"` or `"naive-r"` will be used. see #43382 


Labels:
name: IO LaTeX
description: to_latex
name: Styler
description: conditional formatting using DataFrame.style

Comments:
seems ok, ideally let's have a latex user comment here. umm @bashtage IIRC?
cc @ivanovmg if any comments
> Looks like a nice tests clean-up.
> A couple of follow-up comments.

sure. done.
CLN: add arg validation for `caption` 
State: closed

this is needed since ``DataFrame.to_latex`` validates the ``caption`` arg. 

(Personally not a fan of this type of explicit validation)


Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
looks fine. i think ok to validate this (though the fact that we are accepting a tuple here is odd)
> looks fine. i think ok to validate this (though the fact that we are accepting a tuple here is odd)

latex captions have a short version and long version. written up in docs.
MAINT: rename IOError -> OSError
State: closed

Since Python 3.3,  IO and OS exceptions were re-worked as per [PEP 3151](https://www.python.org/dev/peps/pep-3151/). The new main base class is [OSError](https://docs.python.org/3/library/exceptions.html#OSError), with several other exceptions merged, including IOError (i.e. `assert IOError is OSError`).

This PR changes many of the old alias to OSError, with a few exceptions:

- Versioneer EnvironmentError were not modified (this change has been merged upstream)
- In `pandas/_libs/parsers.pyx`, this is not an OSError, as there is no issue reading the file. The exception is reclassified as a [TypeError](https://docs.python.org/3/library/exceptions.html#TypeError).

Are any of these changes worthy of a whatsnew entry?


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
@mwtoews can you add a whatsnew note in 1.4., I/O Bug fix section is fine, just noting that we changed this. (and ref this issue). pls merge master and ping on green.
@jreback added a whatsnew entry to describe the user-visible change, let me know if it suites or needs rephrasing. I don't understand the Azure "The operation was canceled" CI failure.

The scope of this PR has been reduce to a pure maintenance change, which shouldn't need any whatsnew entry, since any IOError class thrown was viewed as OSError to the user, so no changes should be visible.

The more involved bug fix that changes the exception class, and works across different parsers is at #43443

@mwtoews - can you take a look at the test failures?
The third commit in this PR fixes the excel readers, which allow bytes for the first parameter (e.g. [pandas.read_excel](https://pandas.pydata.org/docs/reference/api/pandas.read_excel.html)). Since `get_handle` expects FilePathOrBuffer, these bytes need to be passed as BytesIO buffers.

I'd appreciate a review from anyone familiar with `pandas/io/excel/_base.py`. As these are internal changes to the excel readers, I'm not expecting any further whatsnew notes on this aspect, but please correct me if it should.
It seems that support for reading bytes for the excel readers was added with #30519

Also, feel free to re-title and re-classify this PR, as the scope has shifted since initially submitted.
> It seems that support for reading bytes for the excel readers was added with #30519
> 
> Also, feel free to re-title and re-classify this PR, as the scope has shifted since initially submitted.

no its fine, just the question on where the bytes are wrapped is still likely hit
pre commit issue: https://github.com/pandas-dev/pandas/pull/43366/checks?check_run_id=3555699486
@github-actions pre-commit
thanks @mwtoews
TST: added test for groupby when calling first, last, nth (GH29645)
State: closed

- [x] closes #29645
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Provided test for `groupby` when callling `first`, `last`, `nth` on a `Series` with `None` or `np.nan`.

Tests and linter pass locally.


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Groupby

Comments:
thanks @lorenzophys
CI: Fix Failing Doc Build
State: closed

https://github.com/pandas-dev/pandas/pull/43145 had failing code checks

https://github.com/pandas-dev/pandas/pull/43347/checks?check_run_id=3488498926

```
/home/runner/work/pandas/pandas/doc/source/whatsnew/v1.3.3.rst:36: WARNING: Bullet list ends without a blank line; unexpected unindent.
```

Labels:
name: Docs
name: CI
description: Continuous Integration

Comments:
wow! merge on green.
actually need to backport this
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
$ git checkout 1.3.x
$ git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
$ git cherry-pick -m1 62c12cf09cdd77b35d404667d859fd90190aeb70
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
$ git commit -am 'Backport PR #43349: CI: Fix Failing Doc Build'
```

4. Push to a named branch :

```
git push YOURFORK 1.3.x:auto-backport-of-pr-43349-on-1.3.x
```

5. Create a PR against branch 1.3.x, I would have named this PR:

> "Backport PR #43349 on branch 1.3.x (CI: Fix Failing Doc Build)"

And apply the correct labels and milestones.

Congratulation you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove `Still Needs Manual Backport` label once the PR gets merged.

If these instruction are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@meeseeksdev backport 1.3.x
thanks @alimcmaster1 
grr i should have merged the other first :-<
ENH: kw `names` to `Styler.hide_index`
State: closed

adds an item to #41649 and #41693

Im ambiguous about this feature but for better compatability when refactoring `DataFrame.to_html` and `DataFrame.to_latex` to use `Styler` it is needed.

![Screen Shot 2021-09-01 at 19 18 49](https://user-images.githubusercontent.com/24256554/131715321-34925946-f42c-4c79-ae9e-e07fc9a1381a.png)


Labels:
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: Styler
description: conditional formatting using DataFrame.style

Comments:
> Im ambiguous about this feature but for better compatability when refactoring DataFrame.to_html and DataFrame.to_latex to use Styler it is needed.

yeah i see; its just one more option, but ok i guess.
BUG: styler.format options and validator tests
State: closed

sorry, this is a fix for recent #43256.
the input types were not working using the instance factory approach so this is explicit.


Labels:
name: Styler
description: conditional formatting using DataFrame.style

Comments:
cool, once more rebase pls.
grr frequent network issues calling the conda repos hopefully https://github.com/pandas-dev/pandas/issues/42580 helps with this.

```
Collecting package metadata (repodata.json): ...working... failed

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://repo.anaconda.com/pkgs/main/win-64/repodata.json>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.

If your current network has https://www.anaconda.com blocked, please file
a support request with your network engineering team.

'https://repo.anaconda.com/pkgs/main/win-64'
```
/azp run
<samp>
Azure Pipelines successfully started running 1 pipeline(s).<br>

</samp>