BrandonFors/pandas-github-issues huggingface dataset:
ENH: add calamine excel reader (close #50395)
State: closed

- [x] closes #50395
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Based on #50581, which was closed as stale and in favor of #53005, which was rejected. Now, calamine support datetime and sheet types, that more than pyxlsb supports, for example.

Now fail a few tests with ods:
- #54994 (5 tests);
- #55045 (1 test) - bad ods file in test_reader_seconds and, I think, a [bug](https://github.com/pandas-dev/pandas/blob/main/pandas/io/excel/_odfreader.py#L226) in the OdsReader (see `office:time-value` in [specification](http://docs.oasis-open.org/office/v1.2/os/OpenDocument-v1.2-os-part1.html#refTable13));
- two bugs in calamine, which can be fixed in the future (one of them - problem with specification - is fixed with [hack](https://github.com/pandas-dev/pandas/blob/main/pandas/io/excel/_odfreader.py#L198) in ODFReader for example).

My English is too bad for writing documentation and I will be happy any help with it.

cc @kostyafarber 

Labels:
name: Enhancement
name: IO Excel
description: read_excel, to_excel

Comments:
@WillAyd recipe was [added](https://anaconda.org/conda-forge/python-calamine) to conda-forge.
> [dimastbk](https://github.com/dimastbk) [force-pushed](https://github.com/pandas-dev/pandas/compare/df8b523037fa9f009912a809aa70ab52424ca909..d6002fe1476ea05cd5c6f7f030ad34c5c4d9d2a0) the issue-50395 branch from [df8b523 ](https://github.com/pandas-dev/pandas/commit/df8b523037fa9f009912a809aa70ab52424ca909)to [d6002fe ](https://github.com/pandas-dev/pandas/commit/d6002fe1476ea05cd5c6f7f030ad34c5c4d9d2a0)
[4 hours ago](https://github.com/pandas-dev/pandas/pull/54998#event-10323838878)

This makes is quite more time consuming to review.
Do you mean about force-push instead of merging "main" and adding new commits? Sorry.
> Do you mean about force-push instead of merging "main" and adding new commits? Sorry.

Correct - not a big problem but wanted you to be aware.
Thanks, I fixed it.
Thanks @dimastbk  - impressive work on this
BUG: is_string_dtype(pd.Index([], dtype='O')) returns False
State: closed

- [x] closes #54661
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit)
- [x] Added an entry in `doc/source/whatsnew/v2.2.0.rst`

Corrected the definition of `is_all_strings`. Now for object arrays, which have no items `is_string_dtype` returns `True` instead of `False`.

Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Strings
description: String extension data type and string data

Comments:
@jbrockmendel, could you please take a look at this PR?
@jbrockmendel, I corrected my PR, could you please take a look at the update?
LGTM.  pls merge main and ping on green
> LGTM. pls merge main and ping on green

thanks, I merged main, ci - green
thanks @natmokval
BUG: boolean/string value in OdsWriter (#54994)
State: closed

- [x] closes #54994
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: IO Excel
description: read_excel, to_excel

Comments:
Can you add a test for this? I don't fully understand the issue from the changelog alone
I added test. Also bug is described in #54994.

Hm, test fails on win. I need some time for fixing it.
@dimastbk - can you merge main here; I'd like to just make sure the tests pass with #54998 now in main.
I merged main and removed 3 xfails from tests.
Thanks @dimastbk
DOC: Grammatically updated the tech docs
State: closed

-Added an entry in doc/source/getting_started/into_tutorials/01_table_oriented.rst
-Changed highlighted part in attached screenshot.

![image](https://github.com/pandas-dev/pandas/assets/139927657/e00708d4-eb7d-480e-998b-e48d788c423c)


Labels:
name: Docs

Comments:
Thanks @mhb143
DOC: expanded pandas.DataFrame.to_sql docstring
State: closed

- [x] closes #54712
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Implements the suggested fix from @mattatark to `pandas.DataFrame.to_sql`


Labels:
name: Docs

Comments:
Thanks @pauluhlenbruck
BUG: pct_change showing unnecessary FutureWarning
State: closed

- [ ] closes #54981 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
@phofl I arrived here because I was about to submit a bug report on this issue. However, looking at the commited change above I notice it won't fix the issue for DataFrames: the 3 changed lines in [pandas/core/generic.py](https://github.com/pandas-dev/pandas/pull/54983/commits/50c453b30e2cd094010380d63816775ae7db7a2a#diff-1a2e3df0db7dd8bddc2ec4bff9de8a7a55e328e6c32e2cecde761dc9549fcd46) only work if `self` is a Series, not when it's a DataFrame, I believe.
good point, fixed
Thanks @phofl
REG: filter not respecting the order of labels
State: closed

- [ ] closes #54980 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Looks good just a merge conflict
ENH: add Series.struct accessor for ArrowDtype[struct]
State: closed

Features:

* Series.struct.dtypes -- see dtypes and field names
* Series.struct.field(name_or_index) -- extract a field as a Series
* Series.struct.to_frame() -- convert all fields into a DataFrame

- [x] closes #54938 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Closes #54938 

Labels:
name: Enhancement
name: Arrow
description: pyarrow functionality
name: Accessors
description: accessor registration mechanism (not .str, .dt, .cat)

Comments:
@mroeschke Thanks for the feedback! I believe I've now addressed all of your comments.
@mroeschke OK to merge now?
Nice thanks @tswast
TYP: fix a few types
State: closed

I fixed a few type issues discovered by using the pandas-stubs type tests with the pandas annotations (there are many errors).

I'm surprised that the two Excel-classes being marked as abstract wasn't detected by the CI, they are clearly meant to be instantiated, so they cannot be abstract.

Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
@twoertwein - looks like isort is making a modification
> @twoertwein - looks like isort is making a modification

All green now :)
Thanks @twoertwein
ENH: Move factorizing for merge to EA interface
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


cc @jbrockmendel thoughts here? This would still need docs obviously

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: ExtensionArray
description: Extending pandas with custom dtypes or arrays.
name: Stale

Comments:
Generally looks good, mostly bikeshedding names
a handful of nitpicks, otherwise im very happy to see this refactor
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Sorry this fell off my radar, updated
@MarcoGorelli is there a way to shut up the build about the missing example? There are a bunch of ea methods without examples...
> is there a way to shut up the build about the missing example? There are a bunch of ea methods without examples...

+1.  Many of the examples are not particularly useful (I know because I wrote them to get the CI to shut up)
yeah I'd expect pasting into 

https://github.com/pandas-dev/pandas/blob/13e034aaf68894079c61ace35e2f9d541fd64a1e/scripts/validate_docstrings.py#L44-L58

to work
Looks like this has gone stale, so I'm bumping off the milestone
Looks close to merge but has gone stale. Going to close to clear the queue but feel free to reopen when you have time to circle back
Include pyarrow_numpy string in efficient merge implementation
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Performance
description: Memory or execution speed performance
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
I wouldn't add a whatsnew, not widely enough used that it would really matter I think
Thanks @phofl
BUG: Fix BaseWindowGroupby.aggregate where as_index is ignored
State: closed

- [x] closes #31007
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Apply
description: Apply, Aggregate, Transform, Map
name: Window
description: rolling, ewma, expanding

Comments:
Also realized that this is failing with a list-like parameter as well so extended testing
TST: add test case of ngroup with NaN value
State: closed

- [x] closes #50100
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests).
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added new tests to `GroupBy.ngroup` method, which ensure consistency returning a 0 value.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @josemayer
BUG: IntervalIndex.get_indexer incorrectly matching ints to datetimes
State: closed

- [x] closes #47772 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Interval
description: Interval data type

Comments:
Thanks @jbrockmendel 
This patch may have induced a performance regression. If it was a necessary behavior change, this may have been expected and everything is okay.

Please check the links below. If any ASVs are parameterized, the combinations of parameters that a regression has been detected for appear as subbullets.

 - [reshape.Cut.time_qcut_timedelta](https://asv-runner.github.io/asv-collection/pandas/#reshape.Cut.time_qcut_timedelta)
   - [bins=4](https://asv-runner.github.io/asv-collection/pandas/#reshape.Cut.time_qcut_timedelta?p-bins=4)
 - [reshape.Cut.time_qcut_float](https://asv-runner.github.io/asv-collection/pandas/#reshape.Cut.time_qcut_float)
   - [bins=10](https://asv-runner.github.io/asv-collection/pandas/#reshape.Cut.time_qcut_float?p-bins=10)
   - [bins=4](https://asv-runner.github.io/asv-collection/pandas/#reshape.Cut.time_qcut_float?p-bins=4)

Subsequent benchmarks may have skipped some commits. The link below lists the commits that are between the two benchmark runs where the regression was identified.

https://github.com/pandas-dev/pandas/compare/a98be065e365c3d673d759414586345b14de50ac...28f274cb107ef0a252eb4cb954c1cfbe43968395
BUG: DataFrame.stack with future_stack=True failing when columns are tuples
State: closed

- [x] closes #54948 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

In the previous implementation, we assumed that if `idx` is a tuple that we were stacking multiple levels and hence did not need to wrap it in a tuple. However you can have columns consist of tuples that are not a MultiIndex, and then when stacking (the only level) you still want to wrap in a tuple.

Labels:
name: Bug
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
thx @rhshadrach
ENH: Use more arrow compute functions for string[pyarrow] dtype
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
Thanks @phofl
REGR: read_csv splitting on comma with delim_whitespace
State: closed

- [ ] closes #54931 (Replace xxxx with the GitHub issue number)
- [ ] closes #54918 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO CSV
description: read_csv, to_csv

Comments:
Lgtm if green - thanks!
REGR: Arrow backed objects not propagating exceptions
State: closed

- [ ] closes #54944 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Arrow
description: pyarrow functionality

Comments:
Hm, this is kinda annoying so I think it should go into the release.

I'm not familiar enough to review this part of the code, though.

@mroeschke can you take another quick look
Thanks @phofl
BUG: Categorical.isin raising for overlapping intervals
State: closed

- [x] closes #34974
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Categorical
description: Categorical Data Type
name: Interval
description: Interval data type

Comments:
thx @lukemanley
BLD: improvements to meson.build files
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The second commit is a bug fix: run `generate_version.py` with a shebang, not 'python'. The way this was before can result in build failures. It assumed that `python` is a working Python 3.x interpreter, and that is not always true. See for example this bug report for the exact same thing in NumPy, where `python` isn't working for Sage: https://github.com/numpy/numpy/issues/24514
    
Meson guarantees that .py scripts with a shebang on the top line will be run with a Python interpreter (if there's none on the PATH, it can use the one Meson itself is run with). Hence this is the most robust way of using `run_command` on a .py script.

The first commit is cleanups to make `meson.build` files more idiomatic:

- Use `pure: false` only in a single place. This is recommended for robustness, this way you can't forget it in a subdirectory and end up with a subtly broken package only on niche Linux distros that split purelib and platlib directories.
- Use `py.install_sources` with a list input rather than in a foreach loop.
- Remove the `werror` comment: it's never a good idea to enable `-Werror` by default in the build config of a library, that can easily break builds. This should be done in one or more CI jobs instead.



Labels:
name: Build
description: Library building on various platforms

Comments:
Lgtm. @lithomas1 
thanks @rgommers.
TST: Use (unused) window parameter of test_freq_window_not_implemented
State: closed

Realized that `window` parameter is unused so this is fixing it. Calling `sum` function as the exception is only raised when a function is called with `VariableOffsetWindowIndexer`. Not sure if I need to add an entry to whatsnew for this

- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
Thanks @ihsansecer
REGR: MultiIndex.append raising for overlapping IntervalIndex levels
State: closed

- [x] closes #54934
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.1.1.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Regression
description: Functionality that used to work in a prior pandas version
name: MultiIndex
name: Interval
description: Interval data type

Comments:
Thx @lukemanley