BrandonFors/pandas-github-issues huggingface dataset:
BUG: melt losing ea dtype
State: closed

- [x] closes #41570 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

```
       before           after         ratio
     [1613f26f]       [6b93f528]
-      15.5±0.2ms      9.72±0.07ms     0.63  reshape.Melt.time_melt_dataframe('Float64')
```

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl
CI: Unpin pyarrow<10
State: closed

- [x] closes #50058 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
no opinion, could do but don't have to I think
Thanks @phofl
DOC: Add to the io documentation of `on_bad_lines` to alert users of silently skipped lines.
State: closed

This PR adds to the documentation of `on_bad_lines` for reading CSV files. Specifically, when using user-defined callables for `on_bad_lines` any bad line that isn't a 'line with too many fields' is silently skipped. The user should be aware of this behaviour.

The issue has been left open for further discussion of what is considered a 'bad line' and whether any code change should be made to accomodate the new definition of a bad line.

- [ ] closes #48020
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
friendly ping @mroeschke 
@mroeschke 
Thanks @kostyafarber
ENH: Implement fastpath for masked indexes in join
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

As of right now,  masked indexes go through object even though we only use the join functions when we are monotonic, e.g. no NA present. We can simply use the numpy array to perform the join.

```
       before           after         ratio
     [026a83e0]       [d804bced]
-     5.82±0.02ms      1.38±0.02ms     0.24  index_object.SetOperations.time_operation('monotonic', 'ea_int', 'intersection')
-     4.08±0.03ms         452±10μs     0.11  index_object.SetOperations.time_operation('monotonic', 'ea_int', 'union')
```

This is related to #49420, will make it a bit easier to review since it solves the join changes


Labels:
name: Performance
description: Memory or execution speed performance
name: Index
description: Related to the Index class or subclasses
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl
BUG: misleading message when using to_datetime with format='%V %a'
State: closed

- [ ] closes #50308 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Good call @mroeschke to factor this out! I think it's simpler like this, and it helped find some untested code, and other code where the message was misleading

---

Performance is unaffected:

On this branch:
```python
In [1]: format = '%Y-%d-%m %H:%M:%S%z'

In [2]: dates = pd.date_range('1900', '2000').tz_localize('+01:00').strftime(format).tolist()

In [3]: %%timeit
   ...: pd.to_datetime(dates, format=format)
   ...: 
   ...: 
234 ms ± 470 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

upstream/main (after having recompiled):
```python
In [1]: format = '%Y-%d-%m %H:%M:%S%z'

In [2]: dates = pd.date_range('1900', '2000').tz_localize('+01:00').strftime(format).tolist()

In [3]: %%timeit
   ...: pd.to_datetime(dates, format=format)
   ...: 
   ...: 
234 ms ± 885 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Labels:
name: Datetime
description: Datetime data dtype

Comments:
Thanks @MarcoGorelli
PERF: construction with integer dtype
State: closed

Motivated by this arithmetic example:

```
In [2]: ser = pd.Series(range(10_000_000))

In [3]: %timeit ser + 1
23.3 ms ± 304 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)  # <- main
15.1 ms ± 110 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)  # <- PR
```

Labels:
name: Performance
description: Memory or execution speed performance
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations

Comments:
> mypy complains.

fixed

> So copy wasn't used anyway?

correct
thx @jbrockmendel
CLN: clean IntervalArray._simple_new
State: closed

There is a lot of code in `IntervalArray._simple_new`. I suggest moving all the code there that checks that the input data is correct into its own method (`_ensure_simple_new_inputs`), so we have a better split between building the ÌntervalArray` and checking that its inputs are correct.

This PR changes no functionality.

Labels:
name: Interval
description: Interval data type

Comments:
The arm failure looks unrelated.
I've updated the PR.
thanks @topper-123
BUG: to_datetime with M or Y unit and non-round float
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype

Comments:
If non-round values aren't allowed, would this supersede https://github.com/pandas-dev/pandas/pull/50183?
> If non-round values aren't allowed, would this supersede https://github.com/pandas-dev/pandas/pull/50183?

Its only for unit="M" or unit="Y"
Thanks @jbrockmendel
WARN fixup warning in pandas/tests/apply/test_frame_apply.py
State: closed

- [ ] closes #50292 
- [ ] All code checks passed
@MarcoGorelli 


Labels:
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
you've got irrelevant commits here, can you do
```
git fetch upstream
git rebase -i upstream/main
```
then, in the pop-up, only select the last commit. then
```
git push origin HEAD --force
```
Thanks I was wondering why all the other commits were showing up.
Use ea interface to calculate accumulator functions for datetimelike
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @jbrockmendel Follow up from the other pr

As mentioned on the other pr, this changes the behavior (highlighted by the test). 

Labels:
name: Datetime
description: Datetime data dtype
name: Timedelta
description: Timedelta data type

Comments:
Thanks @phofl
DEV speed up `pip install -r requirements-dev.txt` by nearly 2000%
State: closed

- [ ] closes #50239 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

it looks like it may not be necessary to list boto3 and botocore, and they get pulled in anyway by s3fs and moto

EDIT:

yup, this works! and we can stop listing even more unpinned dependencies!

- botocore and boto3 are pulled in by s3fs
- beautifulsoup4 is pulled in by nbsphinx
- coverage is pulled in by pytest-cov
- fsspec is pulled in by s3fs
- gitdb is pulled in by GitPython
- ipykernel is pulled in by ipywidgets
- ipython is pulled in by ipywidgets
- jinja2 is pulled in by moto
- nbformat is pulled in by nbsphinx
- psutil is pulled in by ipywidgets
- pyyaml is pulled in by dask
- requests is pulled in by gcsfs
- sphinx is pulled in by nbsphinx

By removing them, `pip install -r requirements-dev.txt` goes from taking over 54 minutes, to just under 3 minutes!

EDIT2: only removing boto3 and botocore. Timing is still under 3 minutes 🥳 

Labels:
name: Dependencies
description: Required and optional dependencies

Comments:
I am slightly -1 on this. I think it is clearer that _any_ package that we use _directly_ is listed in the requirements (which eg was not the case for aiobotocore). This is more explicit and also more robust (in case some other package changes its dependencies). 

It is pip's "fault" that it cannot handle this properly. Of course, given that this makes the pip install so unusable, we can be pragmatic and help that on our side. But then I would personally limit it to removing the biggest culprits (eg just removing boto3/botocore already resolves most of it?), and keep being explicit for the others.
>  But then I would personally limit it to removing the biggest culprits (eg just removing boto3/botocore already resolves most of it?)

OK, sure, as long as we can do that because `pip` installation is currently unusable
Agree with @jorisvandenbossche's reasoning. Would be good to keep an issue open regarding adding back boto3 and botocore if pip can more quickly resolve this in the future. Maybe also adding lower pins on all our dependencies would help the solver?
> Agree with @jorisvandenbossche's reasoning. Would be good to keep an issue open regarding adding back boto3 and botocore if pip can more quickly resolve this in the future. Maybe also adding lower pins on all our dependencies would help the solver?

sure, done https://github.com/pandas-dev/pandas/issues/50300
Can we backport? Otherwise the docker change might take forever
DOC: Make use_nullable_dtypes example easier to understand
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: IO CSV
description: read_csv, to_csv

Comments:
Thanks @phofl
REF: Rename mode.nullable_backend to mode.dtype_backend
State: closed

From the dev call yesterday, it was discussed to make the new global configuration for 2.0 `mode.nullable_backend="pyarrow"|"pandas"` more generically named. I have changed it to `mode.dtype_backend` as we may decide to not always associate this option with the `use_nullable_dtype` keyword argument in the IO readers

cc @phofl @jrbourbeau 

Labels:
name: Refactor
description: Internal refactoring of code
name: Arrow
description: pyarrow functionality

Comments:
@phofl any thoughts on `dtypes_backend` vs. `dtypes` vs. some other name? 
The nice thing about the old name: It implies that you have to use nullable dtypes that the option can take effect. If you don't use nullable_dtypes then nullable_backend does not make much sense.

The new name implies that it can work globally, which is not possible right now. In general, I'd be ok with this, but we have to clarify in the docs that this works only in specific situations right now
Agreed with the concern, at least for 2.0 where we will be paring this closely with nullable types, but I like that this change future-proofs this option to be more applicable to the non-nullable types in the future
thx @mroeschke
ENH: Add use_nullable_dtypes for read_fwf
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: IO Fixed Width
description: read_fwf
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl
BUG: sql ignores dtype when chunksize is set and result is empty
State: closed

- [x] closes #50245 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
Thanks @phofl
Incorrect identically-labeled DataFrame objects Exception comes with df.compare method #50083
State: closed

- [ ] closes #50083
- [ ] All code checks passed



Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
First time contributing to the code base and hopefully everything goes smoothly.
I will try to fix the failing tests tonight.
@MarcoGorelli  would the following work as a whatsnew note?
**Whatsnew Note**

- Updated the Exception messages for df.compare method to include that both index and columns that are identically-labeled can be compared.
- Updated the test associated with these Exceptions.

Edit: sorry for all the commits please see the most recent one with the message "fixed formatting"; if you would like I will try to get rid of the two before the last
You can just say that you improved the error message - I think there's some similar entries in the what's new file

No need to mention tests, it's a user facing note
@MarcoGorelli I have updated the whats new file. Please review whenever you have a moment
I have updated the whatsnew comment and added back in the all the other changes. I wanted to make sure all the checks were passed before requesting a review but some of the checks keep failing with the following error message `ERROR  - SystemError: initialization of _internal failed without raising an exception`. Is this due to the change requested? Also sorry this is taking so much or your time @MarcoGorelli  but I appreciate your time as this process as taught me alot.
Hey - CI failures are unrelated, they're showing up in other PRs, I'll see what's going on there
Thanks @tpackard1
ENH: Add use_nullable_dtypes for read_html
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Enhancement
name: IO HTML
description: read_html, to_html, Styler.apply, Styler.applymap
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks @phofl 
Hi Folks, I'm not sure if this is the right venue for comments on patches after the fact, but just updated my codebase from pandas 1.5.3 to the current version (at time of this post it is 2.2), and noticed that at 2.0, there was a change to the nullable string values added to na_values: https://pandas.pydata.org/docs/whatsnew/v2.0.0.html#:~:text=Added%20%22None%22%20to%20default%20na_values%20in%20read_csv()%20(GH%2050286

<b>Changing "None" to NaN ended up introducing a breaking change to my script</b>, where it still ran without runtime errors, but processed the data differently causing errors in the output dataset.  I had a csv file with "None" intentionally present in some columns in order to show the word on a dashboard.  The issue didn't actually present until that null value showed up in an np.where() where the condition checked to see if it was "None".  The observation then followed an undesired logic path.

I addressed this by copying the default na_values list from pandas 1.5.3 and overriding the one in pandas 2.2 (as I'd noticed a number of new values showed up in the default list in addition to "None").  

I'm not sure I can recommend a better way to introduce a change like this, or a way to better communicate this to users, and the change was mentioned pretty far down the release notes..  You probably don't want to put FutureWarnings in read_csv() for everyone who uses it as it'd get pretty annoying.  At any rate, I wanted to make a note of this, as adding/removing values from the default na_values list might introduce a "soft" breaking change when moving to new pandas versions.

Cheers,
REV: revert deprecation of Series.__getitem__ slicing with IntegerIndex
State: closed

- [x] closes #49612 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Deprecate
description: Functionality to remove in pandas

Comments:
thx @jbrockmendel 
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.5.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 1613f26ff0ec75e30828996fd9ec3f9dd5119ca6
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #50283: REV: revert deprecation of Series.__getitem__ slicing with IntegerIndex'
```

4. Push to a named branch:

```
git push YOURFORK 1.5.x:auto-backport-of-pr-50283-on-1.5.x
```

5. Create a PR against branch 1.5.x, I would have named this PR:

> "Backport PR #50283 on branch 1.5.x (REV: revert deprecation of Series.__getitem__ slicing with IntegerIndex)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
                
was this backported? Otherwise we should do this manually before 1.5.3