BrandonFors/pandas-github-issues huggingface dataset:
BUG: read_excel not forward filling index columns when no names are given
State: closed

- [x] closes #47487 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


The prevoious has_index_name routine was too simple. This version is aligned with the TextReader version that is used a few lines below

Labels:
name: IO Excel
description: read_excel, to_excel

Comments:
```
pandas/io/excel/_base.py:793: error: List item 0 has incompatible type "Union[int, Sequence[int]]"; expected "int"  [list-item]
pandas/io/excel/_base.py:797: error: Argument 1 to "len" has incompatible type "Union[int, Sequence[int], None]"; expected "Sized"  [arg-type]
pandas/io/excel/_base.py:798: error: Argument 1 to "len" has incompatible type "Union[int, Sequence[int], None]"; expected "Sized"  [arg-type]
Found 3 errors in 1 file (checked 1391 source files)
```
Thx, fixed
thanks @phofl
BUG: read_csv ignoring non existing header rows for python engine
State: closed

- [x] closes #47400 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Initially, I thought this would be a case for deprecation, but this was simply a bug and was supposed to work before.
Also, I aligned the error message of the python parser with the c parser


Labels:
name: Bug
name: IO CSV
description: read_csv, to_csv

Comments:
Thanks @phofl
ENH: Move PyperclipException and PyperclipWindowsException to error/_…
State: closed

- [x] xref #27656. this GitHub issue is being done in multiple parts
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

I noticed that there weren't a lot of test around the functions that used PyperclipException & PyperclipWindowsException. So, I took a stab at adding some. Let me know if this unwarranted.

Also, the github only mentioned  PyperclipException, so not sure if we also want to move PyperclipWindowsException, but I did so just in case. 

Labels:
name: Enhancement
name: Error Reporting
description: Incorrect or improved errors from pandas

Comments:
Thanks again @dataxerik !
CI: Run all code checks even if one fails
State: closed

xref https://github.com/pandas-dev/pandas/pull/47469#issuecomment-1163790491=

Solution gleaned from: https://stackoverflow.com/a/58859404

Want to `always()` run all these steps _If_ pandas was built correctly

Note: Cannot use `continue-on-error` because if a check fails the build will still pass which we don't want https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepscontinue-on-error=

Labels:
name: CI
description: Continuous Integration

Comments:
If we have this does it make sense to split the steps into multiple jobs? And have one upstream job that builds pandas so that work isn't duplicated. We could use that upstream job for ASV and docs as well.
+1 to have one build pandas job for all the checks that share the `environment.yml` dependencies. I am slightly partial to the current job grouping though. Is there a max number of jobs that can run in parallel across workflows? I couldn't immediately find it in the docs 
I monitored the load on the Pandas CI pipeline for the past 14 days (with the help of another GHA pipeline :)) and this is the quantiles

```
0.0: 0
5.0: 0
10.0: 0
15.0: 0
20.0: 0
25.0: 0
30.0: 0
35.0: 2
40.0: 2
45.0: 2
50.0: 3
55.0: 5
60.0: 5
65.0: 5
70.0: 7
75.0: 8
80.0: 10
85.0: 14
90.0: 20
```

Checked every 13 minutes. 65% of the time we have < 5 workflows running, for a maximum of 5 * 17 = 85 parallel jobs (we have 17 jobs in the Ubuntu workflow). So I'm not sure it's worth reducing the number of parallel jobs right now. We might want to optimize latency-to-first-valuable-feedback instead.
BUG: Fix issues with numeric_only deprecation
State: closed

Part of #46560

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

ref: https://github.com/pandas-dev/pandas/pull/47265#issuecomment-1150415159

Issues were introduced as part of 1.5, no need for a whatsnew note.

Labels:
name: Bug
name: Groupby
name: Nuisance Columns
description: Identifying/Dropping nuisance columns in reductions, groupby.add, DataFrame.apply

Comments:
Thanks @rhshadrach
REGR: maybe_convert_objects ignoring uints
State: closed

- [x] closes #47294 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I think this is closely related to #37258, though it does not impact any code examples there.

cc @TomAugspurger, @jbrockmendel, @jorisvandenbossche 

Labels:
name: Bug
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
thanks @rhshadrach
BUG: to_sql with method=callable not returning int raising TypeError
State: closed

- [x] closes #46891 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v1.4.4.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query

Comments:
@mroeschke 1.4.3 is (i think) ready for release first thing tomorrow. I know nothing about sql so not reviewing. If this goes in, I should probably redo some pre-release checks. Not a problem. but would prefer not to have broken ci, broken release note links, manual backports needed etc. etc.. All these sap time waiting for the ci to complete.
> @mroeschke 1.4.3 is (i think) ready for release first thing tomorrow. I know nothing about sql so not reviewing. If this goes in, I should probably redo some pre-release checks. Not a problem. but would prefer not to have broken ci, broken release note links, manual backports needed etc. etc.. All these sap time waiting for the ci to complete.

No problem. Will push to 1.4.4/1.5

@meeseeksdev backport 1.4.x
Could not push to auto-backport-of-pr-47474-on-1.4.x due to error, aborting.
BUG: GH33912 fixed issue with precision of the the left end
State: closed

- [ ] closes #33912 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: cut
description: cut, qcut

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
TYP: Fix typing errors on main
State: closed


- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.


Labels:
name: Typing
description: type annotations, mypy/pyright type checking

Comments:
slightly off-topic for this PR. but it appears that when the typing validation fails,  "Run docstring validation script tests" is skipped, so have been living dangerously ignoring the ci failures today.
Thanks @mroeschke
ENH/TST: Add BaseInterfaceTests tests for ArrowExtensionArray PT2
State: closed

xref https://github.com/pandas-dev/pandas/pull/47377#issuecomment-1163230392=


* Min pyarrow version with timezones type coerces to integer when setting (xfailing)
* ArrayManger tests with duration types oddly checks for a ndim attribute (xfailing)

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Arrow
description: pyarrow functionality

Comments:
Ready for another look @jreback
DEPS/TST: tzdata is optional, not required
State: closed

- [ ] closes #47332  (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Timezones
description: Timezone data dtype
name: Dependencies
description: Required and optional dependencies

Comments:
OK, this works now. We should probably specify a minimum tzdata version too, though.

@jbrockmendel Do you have anything in mind?
no idea, cc @pganssle ?
seems fine, does this need backport?
> We should probably specify a minimum tzdata version too, though.

My 2c is might as well have the min version be the latest available version since this is a new feature.

> does this need backport?

Don't think so since `ZoneInfo` support will be new for 1.5.
> > does this need backport?
> 
> Don't think so since `ZoneInfo` support will be new for 1.5.

correct. the failures were only on nightlies (main). 1.4.x is passing tests (or was when I last run the build test which is running again now ready for release https://github.com/MacPython/pandas-wheels/pull/173)

Zoneinfo is also kinda wonky, since it doesn't really need the tzdata package, just a copy of the IANA tz database. I'm not sure we have a way to check the version of the IANA tz database, though.
> Zoneinfo is also kinda wonky, since it doesn't really need the tzdata package, just a copy of the IANA tz database. I'm not sure we have a way to check the version of the IANA tz database, though.

I don't think we should necessarily runtime check `tzdata` in pandas, but it would be good to state that the minimum version we test with `tzdata=2022.1`
This is blocking #47442, so we should probably merge this now, and open an issue to discuss tzdata version, since there is not yet enough consensus on that.
IMO setting `tzdata=2022.1` should be okay for now.
OK, so I've updated this PR again. It's too difficult/costly to check the version of the system IANA tz db if present, so we're just going to specify a minimum version.

If tzdata is installed, though, the minimum version will be enforced through a warning, even if they have an up to date IANA tz db, since we would want to warn them of the mismatch.
Hello @lithomas1! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:









There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2022-08-09 13:55:50 UTC
I'm going to self merge this in a couple of days if no other comments, so we can get the Python 3.11 testing in.
thanks @lithomas1 this is great
Fix signature of initObjToJSON
State: closed

This patch is required to make Pandas compile with `-Wl,--fatal-warnings`. It is also needed to prevent an "indirect call signature mismatch" error when using pandas in wasm. This is the only patch Pyodide currently applies to Pandas.

Labels:
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Strange. Does this line not need to change too?

https://github.com/pandas-dev/pandas/blob/45f094d1fdbda6d09954dd49f72bebfeebf0de3b/pandas/_libs/src/ujson/python/ujson.c#L46
Sorry misread that. This lgtm
For the record the error we get without this is:
```
wasm-ld: error: function signature mismatch: initObjToJSON
>>> defined as () -> void in build/temp.emscripten_3_1_13_wasm32-cpython-310/pandas/_libs/src/ujson/python/ujson.o
>>> defined as () -> i32 in build/temp.emscripten_3_1_13_wasm32-cpython-310/pandas/_libs/src/ujson/python/objToJSON.o
```
Thanks @hoodmane
Update ecosystem.rst
State: closed

Fix capitalization for some headings

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the PR, but as mentioned these titles are as intended. If interested in contributing feel free to tackle any issue labeled `good first issue`. Closing.
Backport PR #47393 on branch 1.4.x (CI/TST: Don't require length for construct_1d_arraylike_from_scalar cast to float64)
State: closed

Backport PR #47393: CI/TST: Don't require length for construct_1d_arraylike_from_scalar cast to float64

Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
not yet got down to why this is not failing in main. made the code change anyway for now to see if tests pass.
> not yet got down to why this is not failing in main. made the code change anyway for now to see if tests pass.

@mroeschke Numpy Dev is green. waiting on the others. OK to merge or allow time to investigate further? (and fix for 1.4.4 instead). I don't think this is urgent.


> > not yet got down to why this is not failing in main. made the code change anyway for now to see if tests pass.
> 
> @mroeschke Numpy Dev is green. waiting on the others. OK to merge or allow time to investigate further? (and fix for 1.4.4 instead). I don't think this is urgent.

Agreed this is not urgent, but the CI looks "green" (Windows failures appear to the memory errors and the ASV failure is a conda connection error"). So can go in 1.4.3 or 1.4.4
OK let's do for 1.4.3. Since this is numpy-dev related it doesn't affect wheel builds. but the code changes will get exercised by users and if any issues identified could then be resolved in 1.4.4.
Cache Conda env
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: CI
description: Continuous Integration

Comments:
> 1. Can `cache-downloads: true` be set?

Yes we can do it. I'm not sure it's worth it given that the total amount of data that can be cached per repository is 10 GB (https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy) and some of the envs are 0.5 GB large.
> 2\. Could you give an example before/after improvement in the setup?

Sure, here we have 2 min -> 0.5 min https://github.com/pandas-dev/pandas/runs/7000499676?check_suite_focus=true https://github.com/pandas-dev/pandas/runs/7007202415?check_suite_focus=true and here we have 4.5 min to 1.5 min https://github.com/pandas-dev/pandas/runs/6998916048?check_suite_focus=true https://github.com/pandas-dev/pandas/runs/7001171440?check_suite_focus=true

In my experience env setup can be considerably slower if there are network problems. For example here we have 8 min, although I'm not sure how much of it is related to using Mamba instead of Micromama https://github.com/pandas-dev/pandas/runs/7028218282?check_suite_focus=true.
> > 1. Can `cache-downloads: true` be set?
> 
> Yes we can do it. I'm not sure it's worth it given that the total amount of data that can be cached per repository is 10 GB (https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#usage-limits-and-eviction-policy) and some of the envs are 0.5 GB large.

I see. I would still be partial in trying it as I've noticed we sometimes still hit connection errors from conda although I extended the timeouts in the `condarc.yml`. If environments are evicted from the cache during every build due to the size then yeah I can see it not being worth it
Thanks @jonashaag. These CI improvements are awesome!
Manual Backport PR #47287 on branch 1.4.x (DEPS: Sync environment.yml with CI dep files)"
State: closed

PR #47287

Labels:
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
moving to 1.4.4 as the typing issues on main need sorting too.
@mroeschke the typing failures on 1.4.x I'm also seeing locally with numpy 1.23.1 on main. downgrading to 1.22.4 and mypy checks pass. (So I'm assuming the mypy errors reported on 1.4.x are correct)

With the xarray failure here, I'm not sure what the status of mypy is here. (IIRC you did a PR that run checks even if previous checks failed, we could maybe backport that one)

I'm assuming the environment for the mypy checks on main is incorrect?

I need to understand the issues here and purpose of this more before merging. (As mentioned above, we have tended not to update mypy on the backport branches before and although only a dev dependency, we do list the change in the 1.5 release notes, so updating on 1.4.x could lead to confusion?)


This PR just aimed to align the CI changes between main and 1.4.x, namely ensuring all the jobs that utilize `environment.yml` (all non-testing related checks) include all the dependencies. Looks like in turn this changed typing validations due to bumped versions.

IMO maybe it's not worth back-porting this as it might risk change in behavior in 1.4.x, so closing
Improve error message for DataFrame.from_dict when wrong orient is provided
State: closed

- [X] closes #47450
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [X] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors

Comments:
I don't think any tests are needed for this, since it really is such a small fix. Could be wrong though :/
Damn yall really test everything lol. Ok I will fix the failing test, at least now I have a good idea of which test to fix.
I think these failures are unrelated
Ah. Ok I will write new tests then
thx @KianShah
Backport PR #47327 on branch 1.4.x (REGR: Fix fillna making a copy when dict was given as fill value and inplace is set)
State: closed

Backport PR #47327

Labels:
name: Copy / view semantics

Comments:
@phofl XPASS on `test_fillna_dictlike_value_duplicate_colnames`. will fix-up later or feel free to push to this branch.
This is interesting, do you know if this was an intended change? Otherwise we should open an issue about it
> This is interesting, do you know if this was an intended change? Otherwise we should open an issue about it

not sure. As this is 1.4.x, I'm not sure array manager changes are a concern. @jbrockmendel ?
Sorry that wasn't clear. I meant: If this is passing on 1.4.x and failing on main, this looks like a regression on main compared to the 1.4 release
yeah. can open an issue for main. for this branch, i'm sure we just skip any failing array manager tests to get ci green.
if there is a problem and a fix on main, we could then maybe consider backporting the fix for 1.4.4
> not sure. As this is 1.4.x, I'm not sure array manager changes are a concern. @jbrockmendel ?

i wont lose any sleep over array manager tests failing
Deprecate non-keyword arguments for rsplit
State: closed

- [x] closes #47423
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v1.5.0.rst` file if fixing a bug or adding a new feature.


Labels:
name: Strings
description: String extension data type and string data
name: Deprecate
description: Functionality to remove in pandas

Comments:
@MarcoGorelli @datapythonista Thanks for reviewing! Year, sure, I'll do the same for `split` method today in this PR. Fails look unrelated to PR, I hope it disappears after updating branch with new master :)
https://github.com/pandas-dev/pandas/runs/7007844136?check_suite_focus=true

hmm these are all unrelated i supposed

cc @mroeschke