BrandonFors/pandas-github-issues huggingface dataset:
BUG: Frame.iat item_cache invalidation bug
State: closed

- [x] closes #45684
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves

Comments:
updated + green
@meeseeksdev backport 1.4.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -m1 5d0f6edac58f31788f54751210ddb3ee64963845
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #45706: BUG: Frame.iat item_cache invalidation bug'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-45706-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #45706 on branch 1.4.x (BUG: Frame.iat item_cache invalidation bug)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.4.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -m1 5d0f6edac58f31788f54751210ddb3ee64963845
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #45706: BUG: Frame.iat item_cache invalidation bug'
```

4. Push to a named branch:

```
git push YOURFORK 1.4.x:auto-backport-of-pr-45706-on-1.4.x
```

5. Create a PR against branch 1.4.x, I would have named this PR:

> "Backport PR #45706 on branch 1.4.x (BUG: Frame.iat item_cache invalidation bug)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
@jbrockmendel if you can push the manual backport here
TST: Disable testing that hangs Azure CI
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them


Precursor to https://github.com/pandas-dev/pandas/pull/45478

This PR doesn't necessarily guarantee that we won't see random Azure timeouts, but based on working on #45478, these were tests that noticeably caused the CI to hang  



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
@mroeschke backport?
> @mroeschke backport?

Sure might be helpful for the 1.4.x branch as well.
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
Use ccache in GHA pipelines
State: closed

Saves around 5–10 minutes in each of the GHA builds.

Also cleans up some duplicated code.

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: CI
description: Continuous Integration

Comments:
Hello @jonashaag! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:



There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-02-28 08:52:23 UTC
@jonashaag can you merge main
ping @mroeschke 
Trying Sccache right now because it has support for Windows/Visual Studio compiler
I can't figure out how to tell `build_ext` to use a different compiler for Windows. It always seems to use `cl.exe`. @mroeschke @jbrockmendel any ideas?
Awesome, this seems to work great.
Awesome, this also works for Windows now. Not yet implemented for Azure pipelines though, so this only applies to the `python-dev` Windows builds. Would like to test this for a few days and then port it to Azure pipelines as well.
Looks pretty good! Have you observed pushing a commit a day later and seeing the ccache being rebuilt?
I have observed that numerous times while developing this but we can also retrigger this specific version of the PR tomorrow and have a look.
There is one important caveat with this PR that I forgot to mention, which is that it currently uses an unmerged version of `ccache-action` (https://github.com/hendrikmuhs/ccache-action/pull/47)
can you rebase again (as merged the mamba change).
I'm working on an extension of this that also uses Micromamba for env setup (where I recently implemented caching of entire Conda envs). It removes another 5 minutes from the builds. https://github.com/pandas-dev/pandas/pull/46346
Example before, almost 10 minutes for Conda setup + Pandas build
<img width="95" alt="Screenshot 2022-03-12 at 22 46 43" src="https://user-images.githubusercontent.com/175722/158036241-f3acf38f-60a2-463c-adc2-cd5c41af8983.png">

Example after, less than 2.5 minutes for Conda setup and Pandas build (both w/ hot cache)
<img width="93" alt="Screenshot 2022-03-12 at 22 46 50" src="https://user-images.githubusercontent.com/175722/158036242-90b30ce9-bbc1-43a8-b4fc-6ac3eaaba5a3.png">

With hot cache most of the time of the Pandas build is spent during cythonization, maybe we can cache that too.
BUG: also use Float Nullable dtypes in read_parquet when use_nullable_dtypes=True
State: closed

fixes #45694 
Add float types in the mapping dictionary




Labels:
name: IO Parquet
description: parquet, feather
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays
name: Stale

Comments:
Thanks for the PR! Please add tests, whatsnew line, and link this PR to the original issue.
**Before Fix** :  float64 is returned instead of Float64 #45700 
```
>>> import pandas as pd
>>> df = pd.DataFrame({'a':[1, 2, 3], 'b':[1.0, 2.0, 3]})
>>> df['c'] = pd.Series([1, 2, 3], dtype='uint8')
>>> df.to_parquet('a')
>>> pd.read_parquet('a').dtypes
a      int64
b    float64
c      uint8
dtype: object
>>> pd.read_parquet('a', use_nullable_dtypes=True).dtypes
a      Int64
b    float64
c      UInt8
dtype: object
```


**After Fix :**  Float64 is returned instead of float64 and bug is fixed
```
>>> import pandas as pd 

>>> df = pd.DataFrame({'a':[1, 2, 3], 'b':[1.0, 2.0, 3]})

>>> df['c'] = pd.Series([1, 2, 3], dtype='uint8')

>>> pd.read_parquet('a').dtypes
a      int64
b    float64
c      uint8
dtype: object

>>> df.to_parquet('a')

>>> pd.read_parquet('a', use_nullable_dtypes=True).dtypes
a      Int64
b    Float64
c      UInt8
dtype: object
```
@Jaafarben2 - the tests need to be added to `pandas.tests`, see [the dev docs](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#test-driven-development-code-writing) for more details.
Thanks for the PR!

To be explicit for the required test, there is an existing test for this functionality at

https://github.com/pandas-dev/pandas/blob/5c757e9c4cf6a74fabfa1983dcd72b4759a88e5f/pandas/tests/io/test_parquet.py#L603-L645

This tests the roundtrip of a dataframe with some columns that have a nullable dtype variant in pandas. I think it should be sufficient to add two extra columns to the test dataframe (float32 and float64) in that existing test.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
DOC: Remove obsolete reference to CONTRIBUTING.md generation
State: closed

It seems that a note was left over after https://github.com/pandas-dev/pandas/pull/45063 from the old CONTRIBUTING.md times ([still there in devdocs](https://pandas.pydata.org/docs/dev/development/contributing_documentation.html#about-the-pandas-documentation)). The note referred to generating the file from `contributing.rst` via pandoc, but that seems to no longer apply. Suspiciously these were also the only remaining hits for `git grep CONTRIBUTING`.

Removal seems straightforward enough, but I ran pre-commit just in case.

Labels:
name: Docs

Comments:
Thanks @adeak!
TST/CLN: organize SparseArray tests
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Sits on top of #45691

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Sparse
description: Sparse Data Type

Comments:
can you rebase
rebased + greenish
TST: Remove unused fixtures
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Looks like recent commits are trial/error for [something].  If thats sorted out ill take a look, tentatively looks good.
> Looks like recent commits are trial/error for [something]. If thats sorted out ill take a look, tentatively looks good.

This was an attempt at fixing https://github.com/pandas-dev/pandas/runs/4992372435?check_suite_focus=true but I cant reproduce locally
Ready for another look
thanks @mroeschke
BUG: Series[sparse].where losing fill_value
State: closed

- [ ] closes #xxxx
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Sparse
description: Sparse Data Type

Comments:
Excellent.  This along with #45673 and #45582 put us within striking instance of sharing some more Block methods
CI: Fix docbuild upload web step on main
State: closed

@datapythonista the main branch docbuild step has been failing on the web upload with

```
Run rsync -az --delete --exclude='pandas-docs' --exclude='docs' web/build/ docs@***:/usr/share/nginx/pandas
  rsync -az --delete --exclude='pandas-docs' --exclude='docs' web/build/ docs@***:/usr/share/nginx/pandas
  shell: /usr/bin/bash -e {0}
  env:
    ENV_FILE: environment.yml
    PANDAS_CI: 1
rsync: write failed on "/usr/share/nginx/pandas/Pandas_Cheat_Sheet.pdf": No space left on device (28)
rsync error: error in file IO (code 11) at receiver.c(393) [receiver=3.1.1]
Error: Process completed with exit code 11.
```

It was suggested in https://serverfault.com/questions/688680/rsync-write-failed-no-space-left-on-device-28 that rsync should use the --inplace flag to help resolve. Not sure if there's anything else that should be considered.

Labels:
name: CI
description: Continuous Integration
name: Web
description: pandas website

Comments:
The problem is that the disk in our web server is full:
```
$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            989M     0  989M   0% /dev
tmpfs           200M   21M  179M  11% /run
/dev/xvda1       39G   39G     0 100% /
tmpfs           998M     0  998M   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           998M     0  998M   0% /sys/fs/cgroup
tmpfs           200M     0  200M   0% /run/user/1007
```

I don't think the inplace will help. I'm having a look to see if there is anything in the server that can be removed, but either we delete something or we get more disk.
I've got 5Gb free in the disk, by removing an old backup of the docs, from when we started using rsync instead of uploading the docs manually. That should solve the problem for a while. There are some discussions at NumFOCUS on what to do with several servers including the pandas one. The server used to be free but not anymore. Hopefully by the time the disk is full again, we moved to a new server with more space.
Okay thanks for the quick resolution. I'll close this PR then since things will be good for a while. 
For context, all the different versions of the documentations that we store until now it's 14Gb, and we've got 5Gb free now. So, when I said for a while, I think it should be at least couple of years or more I guess.
Doc Build and Upload is again failing on main with 

```
Run rsync -az --delete --exclude='pandas-docs' --exclude='docs' web/build/ docs@***:/usr/share/nginx/pandas
rsync: write failed on "/usr/share/nginx/pandas/Pandas_Cheat_Sheet.pdf": No space left on device (2[8](https://github.com/pandas-dev/pandas/runs/5156040978?check_suite_focus=true#step:8:8))
rsync error: error in file IO (code 11) at receiver.c(3[9](https://github.com/pandas-dev/pandas/runs/5156040978?check_suite_focus=true#step:8:9)3) [receiver=3.1.1]
Error: Process completed with exit code 11.
```
Strange, I'll have a look later, thanks for the heads up.
I fixed the problem for now, and created #45944 for a longer term solution.
Thanks @datapythonista
REF: make Block.delete return a new Block
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

I lost track of the original use case that motivated this, but in general i think we're better off keeping block.values fixed.

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
https://github.com/pandas-dev/pandas/runs/4989389923?check_suite_focus=true

are these unrelated?
> are these unrelated?

Yes.  These were fixed in #45707
Nice.  Now #45352 is all that's left between us and block.values being immutable
TST: Use less autouse=True when unecessary
State: closed

- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them



Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
thanks @mroeschke
ENH: Allow safe access to `.book` in ExcelWriter
State: closed

Towards #45572 

- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them

The plan to expose both `sheets` and `book` as public attributes of `ExcelWriter`. We need to make sure that `sheets` is correct if the user modifies the state via accessing `book`.

This accesses a protected attribute for xlwt; I looked at the xlwt code and there is no public access to determine the sheets. The xlwt repository is read-only, we only support the most recent version, and xlwt will be removed in pandas 2.0. So I think this is okay.

Labels:
name: Enhancement
name: IO Excel
description: read_excel, to_excel

Comments:
Failures look related?
Thanks @phofl - wasn't using a context manager for ExcelWriter
Failure is unrelated.

```
FAILED pandas/tests/computation/test_eval.py::TestEval::test_unary_in_array[python-python]
FAILED pandas/tests/computation/test_eval.py::TestEval::test_unary_in_array[python-pandas]
FAILED pandas/tests/computation/test_eval.py::TestOperations::test_column_in[python-python]
FAILED pandas/tests/computation/test_eval.py::TestOperations::test_column_in[python-pandas]
```
You did not add a test for xlsxwriter, was this on purpose?
> You did not add a test for xlsxwriter, was this on purpose?

Yes, but I realize now that was a mistake. I removed the `sheets` attribute from the writer for xlsxwriter because it was never used (hence, no tests). But that creates an inconsistency between the writers, so I've added it back on.
@jreback  - added docstrings. I'd like to hold off updating the docs until the API is ready to be public.
BUG: read csv not breaking lines for warn messages
State: closed

- [x] closes #41710
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry

We should probably unify the warning messages too

Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO CSV
description: read_csv, to_csv

Comments:
great
Adding DuckDB as a SQL Database for the to_sql() function
State: closed

This PR adds DuckDB as a direct possibility to the to_sql() function. 
It basically uses DuckDBs internal pandas integration to load data frames as a duckdb table.

I think I've added duckdb correctly as an optional dependency, I'm happy to fix it if I made any mistake :-)

I'm also really supporting the optional ```if_exists``` attribute, if any other attribute is a 'must-have' I'm also happy to do so.

Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Hey, thanks for the PR, DuckDb looks interesting. Might be worth opening an issue + getting agreement from core team that this is something we want to support before continuing dev here
agree with @mroeschke recommendations here. 
Thanks for the pull request, but as mentioned, in https://github.com/pandas-dev/pandas/issues/45678#issuecomment-1024456727, pandas would need to support a DB engine interface first before supporting DuckDB directly. Closing for now, but happy to have progress on the interface!
BUG: consistent downcast on fillna no-ops
State: closed

- [x] xref #45423
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate

Comments:
thanks, left the issue open if any api discussion left.
REF: implement LossySetitemError
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry

Catch this specifically instead of ValueError, which can have unrelated causes xref https://github.com/numpy/numpy/issues/20923

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
> So this is not a user-exposed error?

That’s the idea, yes

ping(ish)
can you rebase
hmm rebase and i think some related failures
> hmm rebase and i think some related failures

rebased + green
@jbrockmendel looks like the doctests and ASV are failing as this error is being surfaced

https://github.com/pandas-dev/pandas/runs/5014539859?check_suite_focus=true
https://github.com/pandas-dev/pandas/runs/5014540218?check_suite_focus=true

(odd that the build didn't catch this)

Nevermind. I see you're on it :)
BUG: unnecessary FutureWarning in sort_values
State: closed

- [x] closes #45618
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Sparse
description: Sparse Data Type

Comments:
hmm failing tests though
> hmm failing tests though

yah places where we are testing for futurewarning and no longer getting them.  will update.
> yah places where we are testing for futurewarning and no longer getting them. will update.

updated + greenishx
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
CI/TST: Skip external s3 bucket test that hangs on min build
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: CI
description: Continuous Integration
name: IO Network
description: Local or Cloud (AWS, GCS, etc.) IO Issues

Comments:
Looks like one of the processes got stuck on `test_s3_parser_consistency`

```
2022-01-27T23:11:31.0333115Z [gw1] PASSED pandas/tests/io/xml/test_xml.py::test_unsuported_compression[etree] 
2022-01-27T23:11:35.6861543Z pandas/tests/io/xml/test_xml.py::test_s3_parser_consistency 
2022-01-27T23:11:35.6944349Z [gw0] PASSED pandas/tests/io/sas/test_sas7bdat.py::TestSAS7BDAT::test_iterator_loop 
```

In https://github.com/pandas-dev/pandas/pull/45665/commits/1fd4e923accf92bed8bedd4dab274f2e3d36a592, I don't see that test ever complete in the logs.

@jbrockmendel min build is green here. Wdyt?
All green on the min build. Going to merge.
Investigate Conda env caching
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:

Comments:
Can you give this a meaningful title and some description of what you're doing or why?
CLN: suppress distutils warnings, assorted 
State: closed

- [ ] closes #xxxx
- [ ] tests added / passed
- [ ] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [ ] whatsnew entry


Labels:
name: Clean

Comments:
@mroeschke do you have a gameplan for fixing the py38 GHA or should i spend some time troubleshooting it?
I'm looking into it, no lead yet, but some help would be appreciated.

IIRC I saw this yesterday too but the logs are not that helpful

```
2022-01-27T19:21:29.7660840Z Found online and idle hosted runner in the current repository's organization account that matches the required labels: 'ubuntu-latest'
2022-01-27T19:21:29.8118611Z Waiting for a Hosted runner in the 'organization' to pick this job...
2022-01-27T19:21:30.1206185Z Job is waiting for a hosted runner to come online.
2022-01-27T19:21:34.4331926Z Job is about to start running on the hosted runner: GitHub Actions 31 (hosted)
```
On other builds, it appears like another test is hanging since builds are exiting with 143 sigterms
@mroeschke thoughts here?  id like to get the warning suppression in
BUG: Fix joining overlapping IntervalIndex objects (GH-45661)
State: closed

Replacing calls to `get_indexer()` with `get_indexer_for()` as
`IntervalIndex`es can be unique and overlapping.

Similar to #44588

- [x] closes #45661
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Interval
description: Interval data type
name: Index
description: Related to the Index class or subclasses

Comments:
thanks @johannes-mueller very nice
@meeseeksdev backport 1.4.x
Something went wrong ... Please have a look at my logs.
@jbrockmendel this has been backported. The PR that is the cause of the regression was https://github.com/pandas-dev/pandas/pull/43692 which was some changes specifically targeted at IntervalIndex handling. The changes in this PR affect all Index types? (and the function changed is no longer appropriately named?) Can you give this a quick look and thumbs up before we release 1.4.1 (end of this week)
This looks fine to me.