BrandonFors/pandas-github-issues huggingface dataset:
BUG: Fix `Timestamp('now')` and `Timestamp.now` unit inconsistency
State: closed

This fix make `Timestamp('now')` and `Timestamp('today')` return a Timestamp with unit 'us', the same as `Timestamp.now()` and `Timestamp.today()`.

- [x] closes #55879(Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @yuanx749
DEP: update python-calamine to 0.1.7
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Update python-calamine to 0.1.7:
* Fix one test. There are two failed tests left.
* Handling `file_rows_needed` in rust code.
* Add build for python 3.12 on conda.

Should I add what's new entry? This PR is part of #54998, that has not yet been released. 

Labels:
name: IO Excel
description: read_excel, to_excel
name: Dependencies
description: Required and optional dependencies

Comments:
Thanks @dimastbk (no whatsnew needed)
DOC: Added documentation for codes
State: closed

- [X] closes #55435 
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).



Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
TST/CLN: Remove makeTime methods
State: closed


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
Hi. To all people finding this: In a similar spirit like [verlib2](https://pypi.org/project/verlib2/), we rescued the corresponding testing helper methods `makeTimeDataFrame` and `makeMixedDataFrame`, and made them available on behalf of the [pueblo](https://pypi.org/project/pueblo/) package, for all downstream packages which absolutely need them.

- https://github.com/pyveci/pueblo/pull/59
BUG: support non-nano times in ewm
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Window
description: rolling, ewma, expanding
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
updated+green
Thanks @jbrockmendel
Support e.g. modern MacBooks running Linux containers
State: closed

Modern MacBooks have aarch64 CPUs. When using containers (or virtualized Linux in general), they now have to build Pandas from source if installing from the PyPI index. This costs a lot of time and complexity that is better dealt with in the Pandas product build environment.

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
Yes, many container images including ours are based on Alpine Linux, which is the main 'container distro'. It is based on musl libc.
Generally we generate wheels for platforms we have CI testing for, and currently we don't test on musl aarch. Once testing is added, we'd probably be more comfortable releasing wheels for it since there would be "support"
I'll gladly implement testing. Any pointers?
> I'll gladly implement testing. Any pointers?

I would suggest trying to modify `unit-tests.yml` and copy the job contents of the `Linux-Musl` job except to use this image: https://quay.io/repository/pypa/musllinux_1_2_aarch64
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
I'm going to close this PR, but I merged in another PR a while back adding arm64 musllinux wheels (which you can expect to see in the next release).
Less C89, most const, static funcs
State: closed

Will have to do this in a few passes for other files, but generally we:

1.  Are removing C89 style variable declarations at the top of functions
2. Adding `const` qualifiers where we can
3. Making more functions static that aren't intended to be linked outside of their translation unit

Point 1 helps with code readability, Point 2 can help with code safety and compiler optimization. Point 3 helps with code structure and makes functions "local" to their translaction unit instead of "global"

Labels:
name: Internals
description: Related to non-user accessible pandas implementation

Comments:
Thanks @WillAyd
FIX: Solving Int64 precision loss when read_csv(StringIO(data), dtype…
State: closed

…={"a": "Int64"}, engine=pyarrow)

[] closes [BUG: read_csv losing precision when reading Int64[pyarrow] data with N/A values #56135](https://github.com/pandas-dev/pandas/issues/56135) and [BUG: read_csv loses precision when engine='pyarrow' and dtype Int64 #56136](https://github.com/pandas-dev/pandas/issues/56136)
     [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
     All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
     Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.

Enable the integer_object_nulls option by setting it to True in the following line of code in the "arrow_parser_wrapper.py" file:
table.to_pandas(types_mapper=self.kwds["dtype"].get, integer_object_nulls=True)
This adjustment is crucial when using the "pyarrow" engine as a parameter for the 'read_csv' function. It ensures that columns containing integers (Int64) with null elements within a dataframe maintain their precision. Without this setting, numpy may treat columns with null elements by converting the entire column data to Float, leading to precision loss. Enabling 'integer_object_nulls' prevents this issue and preserves the precision of integer data.


Labels:

Comments:
Thanks for the PR, but appears this is already being worked on in https://github.com/pandas-dev/pandas/pull/56251 so closing in favor of that PR
JSON code removals and cleanups
State: closed

Not sure this code ever did anything - seems to be optimized out. Still more to be done but this works for a first pass

As far as the functions go, adding `static` is a best practice for functions that aren't meant to be exported from a shared library (C defaults functions to extern visibility, which you could argue is unfortunate)

Labels:
name: Internals
description: Related to non-user accessible pandas implementation
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Thanks @WillAyd
FIX: Solving Int64 precision loss when read_csv(StringIO(data), dtype…
State: closed

…={a:Int64}, engine=pyarrow)

- [] closes #56135 and #56136
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.


Enable the integer_object_nulls option by setting it to True in the following line of code in the "arrow_parser_wrapper.py" file:
table.to_pandas(types_mapper=self.kwds["dtype"].get, integer_object_nulls=True)
This adjustment is crucial when using the "pyarrow" engine as a parameter for the 'read_csv' function. It ensures that columns containing integers (Int64) with null elements within a dataframe maintain their precision. Without this setting, numpy may treat columns with null elements by converting the entire column data to Float, leading to precision loss. Enabling 'integer_object_nulls' prevents this issue and preserves the precision of integer data.

Labels:

Comments:
Doing what you are describing here has huge performance implications, I don't want to do this here
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: Read CSV on python engine fails when skiprows and chunk size are specified (#55677, #56323) 
State: closed

-Added support for the python parser to handle using skiprows and chunk_size options at the same time to ensure API contract is met.

-Added a regression test to ensure this #55677 can be quickly caught in the future if it reappears.

-Fixed a flawed test case that now screens for #56323 regressions.

- [✅ ] closes #55677 (Replace xxxx with the GitHub issue number)
- [✅ ] closes #56323 (Replace xxxx with the GitHub issue number)
- [✅] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [✅] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [✅] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [✅] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO CSV
description: read_csv, to_csv

Comments:
pre-commit.ci autofix
Does this support the pyarrow engine as well?
> Does this support the pyarrow engine as well?

pyarrow doesn't support chunksize, so it shouldn't be affected at all. This PR only modifies the python engine.
Hey Will! Left a couple comments I want you to check out and then I'll implement the changes the way you'd prefer from there.
Passes test, looking for feedback / to merge if ready.
Would love to see pyarrow support for this

On Mon, Dec 4, 2023, 2:13 PM William Ayd ***@***.***> wrote:

> ***@***.**** approved this pull request.
>
> lgtm @mroeschke <https://github.com/mroeschke> any thoughts?
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/56250#pullrequestreview-1763430889>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAA5E3JABHICEUHPDQOXDH3YHY4HBAVCNFSM6AAAAABAAOIJRKVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMYTONRTGQZTAOBYHE>
> .
> You are receiving this because you commented.Message ID:
> ***@***.***>
>

Thanks @Flytre
ENH: Disallowed ambiguous units to pd.Timedelta.
State: closed

- [ ] closes #53801 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Added deprecation to the check_exact field in assert_series_equal.
State: closed

- [x] closes #56132 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v2.2.0.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: __eq__ raising for new arrow string dtype for incompatible objects
State: closed

- [ ] closes #56008 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Strings
description: String extension data type and string data

Comments:
cc @mroeschke gentle ping
cc @mroeschke
DEPR: Deprecate dtype inference on pandas objects
State: closed

- [ ] closes #56012 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

This is very weird to begin with and becomes a real PITA when we infer strings as arrow backed strings

Labels:
name: Deprecate
description: Functionality to remove in pandas

Comments:
There was some weird additional inference going on in cat, which was uncovered by the deprecation, is fixed now but doesn't really need a whatsnew since we merged another fix today as well
cc @mroeschke 

good to merge here?
Should this affect pd.array?
> Should this affect pd.array?

Yes, but it already preserves the dtype, so we should be good there
TST: dt64 units
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

aimed at trimming the diff in #55901

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
DEPR: Default of observed=False in DataFrame.pivot_table
State: closed

- [x] closes #56236 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Categorical
description: Categorical Data Type
name: Deprecate
description: Functionality to remove in pandas

Comments:
Thanks @rhshadrach
Table cell styling
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DOC: Fix typo in copy_on_write docs
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @smij720
DOC: Update release instructions
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Labels:
name: Docs

Comments:
Thanks @lithomas1
CoW warning mode: enable chained assignment warning for DataFrame setitem in default mode
State: closed

Follow-up on https://github.com/pandas-dev/pandas/pull/55522#discussion_r1406831714

This ensures the chained assignment warning is shown in the default mode as well. 
This does mean you currently get some noise cases of both this new warning and the existing SettingWithCopyWarning. But either this SettingWithCopy was a false positive and it was setting on a view (eg a far-fetched case like `df[0:3][0:2] = 10`), and then we really need to have the new warning as well because it will change behaviour. Or either it was a correct warning (e.g. `df[mask][other_indexer] = ..`, and in that case this has never been working, and adding an additional warning for a case that never works does no harm (and so a user can only have by accident, or potentially unaware it doesn't do what they intend).

xref https://github.com/pandas-dev/pandas/issues/56019

Labels:
name: Copy / view semantics

Comments:
Thanks for the fixup
BUG: DataFrame.update not operating in-place for datetime64[ns, UTC] dtype
State: closed

- [ ] closes #56227 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Copy / view semantics
name: inplace
description: Relating to inplace parameter or equivalent

Comments:
@phofl any idea what the copy-on-write warnings CI jobs are failing here?
Thanks @MarcoGorelli
Switch arrow type for string array to large string
State: closed

- [ ] closes #56259
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


large string is a more sensible default (take concatenates the chunks in pyarrow which can cause overflows pretty quickly), large string should avoid this

one todo for a follow up:

- ensure interoperability with ``"string[pyarrow]"``

Let's see if CI likes this

Labels:
name: Strings
description: String extension data type and string data
name: Arrow
description: pyarrow functionality

Comments:
cc @mroeschke this is green now (pending merge conflicts)
this is green now, so should be ready to merge