BrandonFors/pandas-github-issues huggingface dataset:
CLN/PERF: avoid double-checks
State: closed

Mostly CLN with a couple of PERF things tacked on at the last minute

1) replace can_hold_element checks with np_can_hold_element checks and only for non-dtlike numpy dtypes.  This avoids double-validating in non-raising cases.

2) Avoid up/downcasting in Block.where no-op cases

```
import numpy as np
import pandas as pd

df = pd.DataFrame(range(10**6), dtype=np.int32)
mask = np.ones(df.shape, dtype=bool)

%timeit res = df.where(mask, -1)
7.69 ms ± 13.7 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <- master
1.31 ms ± 104 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)  # <- PR
```

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
CI/TST: Use pytest.mark.skipif instead of marker for CI clipboard tests
State: closed

- [ ] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

Allows us not to have to always set the clipboard marker and will just skip those tests if not available in CI tests.

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration

Comments:
Looks like when determining the clipboard in the CI it tries to probe the clipboard in a crashable way. This approach doesn't work. Closing.
REF: dtypes.astype
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Tighter scope than dtypes.cast.

Labels:
name: Refactor
description: Internal refactoring of code
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
so just a rename?
> so just a rename?

Yes.
DOC: comprehensive example of `Styler.to_latex` features
State: closed

can be backported as all features needed are now in 1.4.0

rendered:
![Screen Shot 2022-01-06 at 17 45 07](https://user-images.githubusercontent.com/24256554/148418704-8bad8197-7f07-457b-a73a-03c041a64c79.png)
![Screen Shot 2022-01-06 at 17 45 16](https://user-images.githubusercontent.com/24256554/148418715-7229b12f-eac3-4249-b3ad-48bdea026194.png)



Labels:
name: Docs
name: IO LaTeX
description: to_latex

Comments:
wow crazy that you can now do this :-> thanks @attack68
BUG: raise on wrong keyword arguments in Timedelta
State: closed

- [x] closes #45108
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: Timedelta
description: Timedelta data type

Comments:
couple of linting failures in the Code Checks job
thanks @deponovo 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -m1 e681fcdfa6261e99d902b019a6ffabc175d664aa
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #45227: BUG: raise on wrong keyword arguments in Timedelta'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-45227-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #45227 on branch 1.4.x (BUG: raise on wrong keyword arguments in Timedelta)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@meeseeksdev backport 1.4.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -m1 e681fcdfa6261e99d902b019a6ffabc175d664aa
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #45227: BUG: raise on wrong keyword arguments in Timedelta'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-45227-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #45227 on branch 1.4.x (BUG: raise on wrong keyword arguments in Timedelta)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: Fix lost precision when reading from Oracle Number column GH34988
State: closed

Fixes: https://github.com/pandas-dev/pandas/issues/34988

- [x] closes #34988
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but appears it has gone stale. If you'd like to continue, please address the review, move the whatsnew note to v.1.5 and we can reopen
MISC: Check that min versions are aligned in CI and import_optional_dependency
State: closed

Follow up to include `install.rst`


Labels:
name: Code Style
description: Code style, linting, code_checks

Comments:
when you update pre-commit hooks, does one have to update dev environment with some git command, or is the addition of new files and new code enough for the hooks to operate as expected?
> when you update pre-commit hooks, does one have to update dev environment with some git command, or is the addition of new files and new code enough for the hooks to operate as expected?

Nope. pre-commit will run (and install new dependencies if needed) with the changes of `.pre-commit-config.yaml` and addition of scripts. IIRC pre-commit runs in it's _own_ virtual environment.
also pls rebase
@mroeschke can you rebase this to make sure ok (as merged @lithomas1 change)
https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=72445&view=logs&j=e2350215-d973-503e-a41e-d287ff823797&t=751e993b-1954-5330-832a-a54bd1064f7d are these failing on master?

> https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=72445&view=logs&j=e2350215-d973-503e-a41e-d287ff823797&t=751e993b-1954-5330-832a-a54bd1064f7d are these failing on master?

Yes it appears so. 

https://dev.azure.com/pandas-dev/pandas/_build/results?buildId=72425&view=logs&j=e2350215-d973-503e-a41e-d287ff823797&t=751e993b-1954-5330-832a-a54bd1064f7d&l=44
it's not urgent (i.e. block 1.4) but would be good to keep the ci files in sync on 1.4.x and main until they need to diverge for say Python 3.11 so that ci backports have less chance of needing manual intervention.

could either sync ci/deps/actions-38-minimum_versions.yaml or backport this PR.
Might be better to backport this PR since it has some additional changes to `pandas/compat/_optional.py`, but no strong opinion on my side.
@meeseeksdev backport 1.4.x
Adding unixtime property to Timestamp and DatetimeIndex
State: closed

This PR implements #43975 (https://github.com/pandas-dev/pandas/issues/43975), adding a unixtime property to Timestamp and Datetime index.

In the course of developing this PR, I discovered that the Timestamp class already has a method called timestamp() which calculates a unixtime. I deprecated the timestamp method, as I found the name ambiguous (there's nothing about the phrase timestamp that leads me to think of unixtimes). 

This is my first PR against pandas, so apologies for any violations of conventions for code or PRs.

Labels:
name: Enhancement
name: Datetime
description: Datetime data dtype
name: Stale

Comments:
Hello @fergalm! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-01-31 00:44:13 UTC
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
closing as stale. we would still take this if rebased & updated. pls ping to reopen if interested.
REGR: Index(pd.array(foo), dtype=object)
State: closed

- [x] closes #45206
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Could try to get into 1.4 to avoid the regression (in which case no whatsnew needed)

Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: Index
description: Related to the Index class or subclasses
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
This patch fixes the issues we had (in our codebase) that led originally to issue: #45206
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
CLN: suppress warnings
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Clean

Comments:
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
TST/REF: port coercion tests to SetitemCastingEquivalents
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Dtype Conversions
description: Unexpected or buggy dtype conversions

Comments:
@jreback gentle ping; lots more tests to implement
great!
DOC: improve IO & General Functions API reference
State: closed

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them


* Improve formatting of reference pages.
* Improve some doc summary lines so they are clearer in the reference pages.


Labels:
name: Docs

Comments:
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
DOC: fix doctests for depr `closed` arg
State: closed

``closed`` is deprecated for ``inclusive``, and this fixes the docstring warning.

should be backported 1.4.0

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Docs

Comments:
@meeseeksdev backport 1.4.x
thanks @attack68 
Something went wrong ... Please have a look at my logs.
Backport PR #45195 on branch 1.4.x (BLD: updated pydata-sphinx-theme to master in environment.yml)
State: closed

Backport PR #45195: BLD: updated pydata-sphinx-theme to master in environment.yml

Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
hmm don't we need to update github actions here to have 1.4 backports run?

cc @lithomas1 
> hmm don't we need to update github actions here to have 1.4 backports run?

will do after #45201 is merged.
i *think* may need to rebase this to trigger all the CI

oh nvm
it worked
Backport PR #45199 on branch 1.4.x (CI: activate azure pipelines/github actions on 1.4.x)
State: closed

Backport PR #45199: CI: activate azure pipelines/github actions on 1.4.x

Labels:
name: CI
description: Continuous Integration

Comments:
ahh i see @simonjayhawkins merge when ready
> ahh i see @simonjayhawkins merge when ready

I think safe to merge now. did restart since there were some failures which are likely unrelated.
thanks !
DOC: start v1.5.0 release notes
State: closed

~~NOT TO BE MERGED till after we branch at 1.4.0rc0~~



Labels:
name: Docs

Comments:
Is the merge moratorium over?
yup

can merge to master (and backport appropriate things)

ping if things are appropriate for 1.4
I'll self merge this one so that we can start to instruct some contributors to move their release notes.
DOC: Increased minimum version for Python
State: closed

https://github.com/pandas-dev/pandas/issues/41957#issuecomment-1005592239

Labels:
name: Docs

Comments:
@meeseeksdev backport 1.4
thanks @simonjayhawkins 
Something went wrong ... Please have a look at my logs.

 It seems that the branch you are trying to backport to does not exist.
BLD: updated pydata-sphinx-theme to master in environment.yml
State: closed

- [X] closes #45123
- [X] Ensure all linting tests pass

Changed pip `pydata-sphinx-theme` to `master` from `41764f5` in `environment.yml`
Pre-commit hook changed `requirements-dev.txt` accordingly.

Can be changed to master because issue fixed with `pydata-sphinx-theme` master. 
[pydata-issue](https://github.com/pydata/pydata-sphinx-theme/issues/523)
 


Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
thanks @prithvijitguha 
@meeseeksdev backport 1.4
Something went wrong ... Please have a look at my logs.

 It seems that the branch you are trying to backport to does not exist.
API/DEPR: reintroduce errors kwd in where, mask, deprecate inconsistent behaviors
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

xref #45190 which does the same thing for fillna

Lots of new code here, but most of it goes away once deprecations are enforced.

Many new paths here don't have test coverage.

Existing docstring says we accept "ignore", "raise", while this PR assumes "raise" and "coerce".  So this would need to be updated/deprecated.

Labels:
name: API - Consistency
description: Internal Consistency of API/Behavior

Comments:
I like this as along with the `.fillna` and *hopefully* `.astype` changes we get to a consistent place. this would be ok for 1.4 (as this is reversing a current deprecation).
> hopefully .astype changes

AFAIK that's not on the horizon
> this would be ok for 1.4 (as this is reversing a current deprecation).

i think we should reverse the current deprecation in 1.4 and do this (with appropriate tests) for 1.5
cool, pls rebase
Mothballing while I try to sort out some of the kinks (and to clear the queue)
REF/PERF: dont use hashtable in IndexEngine.__contains__
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

This is going to have some tradeoffs:

```
import pandas as pd
import numpy as np

idx = pd.Index(np.arange(1000))
idx2 = idx.repeat(2)

In [5]: %timeit 400.0 in idx
1.27 µs ± 46.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <- master
421 ns ± 3.81 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <- PR

In [6]: %timeit idx.get_loc(400)
1.11 µs ± 49.9 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <- master
625 ns ± 49.8 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)   # <- PR

In [6]: 400 in idx2
1.05 µs ± 24.4 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <- master
1.85 µs ± 43.7 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)  # <- PR

In [7]: %timeit idx2.get_loc(400)
3.43 µs ± 309 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <- master
2.07 µs ± 52.1 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)  # <- PR
```

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Performance
description: Memory or execution speed performance

Comments:
i think this is fine
cc @phofl if comments
can you merge master. cc @phofl if comments.
rebased + green
POC/API/DEPR: errors kwd for fillna
State: closed

xref #45153

Upsides: 1) users get to control the behavior, 2) we can deprecate the inconsistent default behaviors to become consistent
Downsides: 1) Larger API surface area, 2) many more test cases to cover (which ATM this PR does none of), 3) increased code complexity, though most of that goes away once deprecations are enforced.

Could implement the same for Index.fillna to keep it consistent.

If we go down this road, I'd want to do the same for other sometimes-casting methods: putmask, where, replace.  (which just leaves `__setitem__` is inconsistent)

Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: API - Consistency
description: Internal Consistency of API/Behavior
name: Mothballed
description: Temporarily-closed PR the author plans to return to

Comments:
Agreed with the goal to get things consistent. I don't have a strong preference on raising vs. coercing by default, or having a keyword or not.
+1 on doing this (didn't look in detail), of course should deprecate downcast at the same time....
> Agreed with the goal to get things consistent. I don't have a strong preference on raising vs. coercing by default, or having a keyword or not.

i think coercing by default is ok (its what pandas is good at generally) and keeps things natural while allowing user control if it really matters
I think we're all on board, so im going to start writing tests/whatsnew for this.

Are we OK with only "coerce" and "raise" being allowed here?  In other contexts we have "ignore", but I just see that as adding unnecessary API/test surface area.
hmm for categorical in particular there is a question about what "coerce" should mean.  e.g. Categorical._replace has coercion logic that is probably what we want here.
Mothballing while I try to sort out some of the kinks (and to clear the queue)
DEPR: inplace setting of Categorical.categories
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

I thought we had gotten all of these, but apparently not.

Labels:
name: Categorical
description: Categorical Data Type
name: Deprecate
description: Functionality to remove in pandas
name: Mothballed
description: Temporarily-closed PR the author plans to return to

Comments:
hmm the suggested alternative is to use rename_categories, but that will make a copy, whereas what we really want to offer is to make a view.
moving to 1.5
Mothballing while troubleshooting locally.
TST: Create test for #30122 about display.precision
State: closed

- [x] closes #30122 
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
I have merged master and pushed the code again.
thanks @dhruvsamdani
BUG: Index.get_loc always raise InvalidIndexError on listlike
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Shouldn't be user-facing unless user is using Index.get_loc directly since we catch this in the relevant Series/DataFrame indexing methods.  The upshot here is this will allow us to simplify `Series.__setitem__` (and maybe others)

Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
mypy complaints are unrelated, just opened a PR that should address them
BUG: encoding_errors=None with read_csv c-engine
State: closed

The c-engine did not handle `encoding_errors=None`.

The test case for `replace` was simply wrong in case of read_csv.

Un-blocks @MarcoGorelli's #45173.

Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
If `TextReader` had a pyi file and if it wasn't called with `**kwargs`, this should have been caught by mypy.
@jreback If it isn't too late, it would be great to have this for 1.4.
thanks @twoertwein
ENH: `Styler.show` shorthand alternative to `hide`
State: closed

- [x] closes #42533
- [x] tests added / passed
- [x] whatsnew entry

Provides a convenient shorthand for displaying only specified rows as data exploration:

### Example

<img width="772" alt="Screenshot 2022-01-03 at 17 59 15" src="https://user-images.githubusercontent.com/24256554/147958089-66137713-647c-49e2-b1e5-2577a04e4ad1.png">

### Rendered Docstring

<img width="554" alt="Screenshot 2022-01-03 at 18 03 07" src="https://user-images.githubusercontent.com/24256554/147958483-ea378ee8-ff6e-438f-82cd-53e14147f1ab.png">
<img width="538" alt="Screenshot 2022-01-03 at 18 03 16" src="https://user-images.githubusercontent.com/24256554/147958492-4d5cd2e8-8ea2-43fa-97a8-652a7dbd957e.png">



Labels:
name: Enhancement
name: Styler
description: conditional formatting using DataFrame.style
name: Stale

Comments:
why not just expose `.head()`, `.tail()`, `.iloc[...]` ?
> why not just expose `.head()`, `.tail()`, `.iloc[...]` ?

This would have to be a separate Styler implementation of the above. The point is to be able to apply the style functions to the table in entirety but just display certain rows. I.e. highlighting a max in a full-column, yet the section displayed might not contain that max. If you subselect prior to the styling application it loses its purpose in most cases.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Hi @attack68 , it seems that the "pandas-dev.pandas" build has failed. (conflicts in a branch). So this issue is still open.
Closing this, I have a different idea for **docs** instead which does not include new methods
> Closing this, I have a different idea for **docs** instead which does not include new methods

What are your ideas? Where can they be found in the docs? 

It's frustrating that `max_rows` can not be applied to a stylized dataframe as discussed in #42533