BrandonFors/pandas-github-issues huggingface dataset:
PDEP-16: Consistent missing value handling
State: open

Note: this is only a draft, partial proposal. It already lists the main points of the proposal at a high level, but of course we need to add a lot of the details and specifically about backwards compatibility and how we can move towards the end goal (I think those last aspects will actually require the most discussion eventually, the devil is in the details here ;)). 

But, I thought to already open this PR to start the discussion on those main high-level points of the proposal (we can already discuss those and see if there is agreement about that), and to have a reference for this NA proposal (since it has come up in a lot of other tangentially related discussions as well), and also to push myself to actually write up those other sections as well ;)

A summary of all (potentially breaking) behaviour changes that I am aware of can be found here: https://github.com/pandas-dev/pandas/issues/58243#issuecomment-2053656616

**Abstract**

This proposal aims to unify the missing value handling across all dtypes:

- All data types support missing values and use `pd.NA` exclusively as the user-facing missing value indicator.
- All data types implement consistent missing value "semantics" corresponding to the current nullable dtypes using `pd.NA`

This proposal tries to provide a high level way forward, and is not meant to address all implementation details (i.e. all dtypes should follow the above principles, but we don't need to decide in this PDEP for every single dtype how this is exactly achieved)

cc @pandas-dev/pandas-core @pandas-dev/pandas-triage

Labels:
name: PDEP
description: pandas enhancement proposal
name: PDEP missing values
description: Issues that would be addressed by the Ice Cream Agreement from the Aug 2023 sprint

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@jorisvandenbossche @phofl are you planning to continue work on this? If not, should we just close and pass the torch to somebody else? Another option is to vote on the part already described here, and let other issues or PDEPs discuss specific details if needed?

Keeping this open in its stale status seems to be harming the project more than being of any value.
I would personally like to revive this discussion (although I haven't fully caught up with all other ongoing discussions from the last days to know how that would potentially impact this PDEP). 

I can try to fill out the details (of what is currently summarized) in the coming weeks. But also if someone else is interested in helping to move this forward, that is certainly welcome.
ENH: add "right" argument to "qcut"
State: closed

- [x]  #58933 
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I know that it does not pass tests and cannot be merged, but could this be a good starting point? Could you give me some advice on how to proceed? Thanks in advance!

Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: make JSONTableWriter fail if no index.name and 'index' in columns
State: closed

This commit is intended to fix GH #58925. If index.name is empty it will use set_default_names inside __init__ to make check on overlapping names fail. Otherwise it's done during schema creation and not reflected on the dataframe itself which creates inconsistency between the data and its schema.

Probably new behavior should be mentioned in the `to_json` docs? I added the docs change as a separate commit to ease squashing or reverting. 

- [x] closes #58925
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Error Reporting
description: Incorrect or improved errors from pandas
name: IO JSON
description: read_json, to_json, json_normalize

Comments:
Thanks @taranarmo
BUG: Fix issue with negative labels in `group_cumsum`
State: closed

- [x] closes #58811  (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Reduction Operations
description: sum, mean, min, max, etc.

Comments:
Sorry for careless, and `whatsnew` is added now.
Thanks @luke396
Handling None values as NaN in combine_first
State: closed

- [x] closes #58977
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks @asishm. I'll add a test soon!
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Fix: Add to_pandas_kwargs to read_parquet for PyArrow engine
State: closed

Adds the `to_pandas_kwargs` parameter to `pd.read_parquet` to allow passing arguments to `pyarrow.Table.to_pandas`. This addresses issues that may arise during Parquet-to-DataFrame conversion, such as handling microsecond timestamps.

Fixes #49236

- [ ] closes #49236 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.


Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG: byteorder option in to_stata is not honoured when writing strL data
State: closed

- [x] closes #58969  (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

I have created a local variable to indicate whether the file is being saved to the native byteorder in two places, but am happy to calculate this once in `__init__` and store in `self` for later access if you prefer.

Labels:
name: IO Stata
description: read_stata, to_stata

Comments:
> but am happy to calculate this once in __init__ and store in self for later access if you prefer.

I think this would be preferable. Over time we've wanted to avoid mutating state variables in methods if possible
> > but am happy to calculate this once in **init** and store in self for later access if you prefer.
> 
> I think this would be preferable. Over time we've wanted to avoid mutating state variables in methods if possible

Thanks, that's useful to know. I have now changed this.
Thanks @cmjcharlton
DOC: Update typo in missing_data.rst
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @DaxServer
API/CoW: take view of underlying Block values for shallow (deep=False) copies
State: open

Testing this on CI, will update description

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Copy / view semantics
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Plan to pursue this?
PERF: Use shallow copies/remove unnecessary copies in reshaping
State: closed

In most of these cases, the axes is just modified. There is a case in `_generate_marginal_results` where I think a copy was made in order to avoid an old `SettingWithCopy` warning

Labels:
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode
name: Copy / view semantics

Comments:
Looks like tests are passing here so merging
PERF: Only copy in plotting when needed
State: closed

It appears there was an unconditional copy before a plot was made. In some paths, the data (the axes or attributes) was not even modified. In other cases, I _think_ with CoW an eager copy is no longer needed. 

Labels:
name: Visualization
description: plotting
name: Performance
description: Memory or execution speed performance

Comments:
Going to merge since test are passing here. Happy to follow up if needed
DOC: added see also to docstrings in mean and median
State: closed

Added, See also for mean and median in doctring 

see https://github.com/pandas-dev/pandas/issues/58065

Labels:
name: Docs

Comments:
Thanks for the PR! Changes look good but can you take a look at the failing checks?
Thanks! I believe  ci/code_checks.sh needs to be updated. 
Yes, could you update it to remove the ones you fixed?
Sure, no problem!
Thanks @BDixon808
Fix branch 58865
State: closed

Add edge cases for:                                                                                                        
>> File with only comments
>> File with no pandas references
Add two more tests:
>> test_edge_cases
>> test_mixed_good_file
Fix issue #58865"

Labels:

Comments:
Thanks for this PR, but as mentioned on your other PR, I think the changed files were adequate as is.

Closing, but happy to have your contributions toward other issues labeled `good first issue`
DOC: Add missing parameters to offsets class: Tick
State: closed

- [x] partially complete #52431
Updated documentation for offsets class: Tick (added missing parameters).

Labels:
name: Docs

Comments:
Thank you @Aloqeely! I have committed to the suggested change.
Could you address these failing check? 

```
Error: None:None:EX01:pandas.tseries.offsets.Tick:No examples section found
Error: None:None:GL08:pandas.tseries.offsets.Tick:EXPECTED TO FAIL, BUT NOT FAILING
```
hi @Anh-HoDac - this looks to be on the right track, if you can add an example to the docstring I think it can go in
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
CLN: code consolidation
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
that escalated quickly 
Thanks for the PR, but IMO I don't think these code consolidations make the code any clearer or necessary for these files.

Closing, but happy to have your contributions towards issues labeled `good first issue`
BUG: Inconsistent types for groupby group names when grouping with a length-1 list
State: closed

xref #58858
fixed bug in groupby group names when grouping with a length-1 list

Labels:
name: Bug
name: Groupby
name: Stale

Comments:
in this PR I changed group `keys` type to `1-tuples` when grouping with a `length-1` list for consistency with the type of group `names`.

To fix the inconsistency with the types, I added the parameter `key_dtype_str` in the `Grouping` constructor. However, these changes have caused some tests to fail, such as: `pandas/tests/groupby/test_grouping.py -k test_groups`

I think my changes should not break existing tests. @rhshadrach, could you please take a look at my changes?

> However, these changes have caused some tests to fail, such as: `pandas/tests/groupby/test_grouping.py -k test_groups`

This test is checking the behavior that you're changing, the test needs to be updated to the new behavior. In any case, I think this needs to be deprecated before this change can be made.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
BUG : fix Series.mode throwing exception with dropna=False and no nul…
State: closed

…ls present (#58926)

- [x] closes #58926
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Fixes edge case in #58926 with a series of nullable dtypes but no null values.


Labels:
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
ENH: Align Timedelta fractional Seconds (+ replace) 
State: closed

- [x] closes #57188
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

We created a property in TimedeltaProperties called **adjusted**, that aligns timedelta fractional seconds. As mentioned in the comments of this issue we also decided to create a **replace** method for timedeltas.

Labels:

Comments:
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
Proofreading/editing Getting Started for readability. Attempt 2, this time without extra files.
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

From my original PR:
_I noticed that the Getting Started page had some typos and issues with sentence structure. I went through and modified index.rst to reflect what I hope are some quality changes in readability.
Thanks for checking out my ideas!_

In the original PR I had accidentally committed an extra dozen images of graphs. After some fumbling with GitHub on my part, @mroeschke suggested a new PR would be the easiest way to get going. 

Labels:
name: Docs

Comments:
Thanks @shindelr
Feat/remediate CVE 2020 13091 audit hooks
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Apologies, I'm not able to delete this PR. It was raised in the wrong (remote) repo rather than my fork.
BUG: Handle unit count when converting from `np.datetime64`
State: closed

- [x] closes #39977
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
I think this case is rare enough I’d rather not support it at all than support is inconsistently across the codebase.  Iirc @seberg was thinking of deprecating it in numpy too
Yeah, I suspect very few use that really and unless someone would shout, I would be fine with deprecating it.  I would also suspect NumPy also has bugs around it due to it having at least almost no users in practice.
Any interest in adding a warning or putting a note in the docs? Otherwise, I'll close this PR.
Looks like there not much appetite for this change so closing
ENH: Implemented a find_min_subset_sum function that solves the change-making problemFeature
State: closed

The input is an array of integers, a target integer sum
and the following optional boolean flags:

closest: if a close approximation is accepted instead
of the target sum

verbose: prints debug information

greedy: uses a greedy approach to the problem,
minimizing complexity while accepting sub-optimal results

The output is a dictionary containing the subset array
and an integer with the achieved sum

Added tests for the function


Labels:

Comments:
Thanks for the pull request, but it appears there is no associated issue with this feature request. A new feature like this would need an issue and approval by the core team to proceed with a PR.

Closing as it's to early to have a PR for this feature. Also, happy to have you contribute to other issues labeled `good first issue`
ENH: Add **kwargs to pivot_table to allow the specification of aggfunc keyword arguments #57884
State: closed

Add the option of passing keyword arguments to DataFrame.pivot_table and pivot_table's aggfunc through **kwargs.

- [X] closes #57884 
- [X] Tests added and passed after adding a new feature
- [X] All code checks passed
- [X] Added an entry in the latest `doc/source/whatsnew/v3.0.0.rst` file after adding a new feature.

Labels:
name: Enhancement
name: Reshaping
description: Concat, Merge/Join, Stack/Unstack, Explode

Comments:
Thanks @PF2100!