BrandonFors/pandas-github-issues huggingface dataset:
PDEP0004: implementation
State: closed

- [x] closes #12585 (Replace xxxx with the Github issue number)
- [x] closes #46071
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Finally ready for review - I've separated the types of changes by commit to try to make the review easier. I've also split out a few changes into precursor PRs, which have now been merged.
This changes a lot of files, but I think it's hard to split it out further

PDEP for reference: https://pandas.pydata.org/pdeps/0004-consistent-to-datetime-parsing.html

Labels:
name: Datetime
description: Datetime data dtype
name: PDEP
description: pandas enhancement proposal

Comments:
Thanks for your reviews, much appreciated!

There are still some open issues related to datetime parsing, but I don't think any of them should be blockers to this:
- #50072
- #50071
- #50051
- #50025

I've addressed the previous blocker, so it'd be good if we could move this along @jbrockmendel @mroeschke 
just a couple of comments
Thanks all! @Dr-Irv there's still a "requested changes" from you - any further comments?
> Thanks all! @Dr-Irv there's still a "requested changes" from you - any further comments?

I think there are 2 things that I'd like clarification on (listing them here so I don't have to find them again):

- https://github.com/pandas-dev/pandas/pull/49024/files#r1046027911
- https://github.com/pandas-dev/pandas/pull/49024/files#r1046036247
> I'm good now - all issues resolved.

Thanks! Really appreciate all the reviews here

That's 4 explicit approvals from core members, unless there's objections I'll merge soonish then - there's already 110+ messages here and it's getting hard to navigate

If minor objections come up I'll address them in a follow-up, and if there's any major ones we can always revert
Clarify pipe documentation
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
Thanks @fredrike 
Is this part of #hacktoberfest 😉.
CLN: tseries/offsets base tests
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Moving tests in `pandas/tests/tseries/offsets/common.py::Base` to it's own file `pandas/tests/tseries/offsets/test_common.py`


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Frequency
description: DateOffsets
name: Clean

Comments:
thx @mroeschke
BUG: resolution in datetimelike add/sub timedeltalike scalar
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Datetime
description: Datetime data dtype
name: Numeric Operations
description: Arithmetic, Comparison, and Logical operations
name: Timedelta
description: Timedelta data type
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
API: retain non-nano timedelta64 dtype in DataFrame/Series/Index constructors
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Timedelta
description: Timedelta data type
name: Constructors
description: Series/DataFrame/Index/pd.array Constructors
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution

Comments:
Thanks @jbrockmendel
48855 pylint import error
State: closed

- [X] Partially addresses #48855 (Replace xxxx with the Github issue number)
- [N/A] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [N/A] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [N/A] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

The huge number of errors that were reported by `pre-commit` that resulted in the disabling of the `import-error` message in the first place I think stem from  the way in which `pre-commit` was configured to run`pylint` which was to use the action from the `pylint` repository. This meant a dedicated virtual environment was used and none of the packages in `requirements-dev.txt` were installed in the isolated virtual environment that is used (see `pre-commit` [#157](https://github.com/pre-commit/pre-commit-hooks/issues/157)). The solution as described in the discussion of that thread (see [comment](https://github.com/pre-commit/pre-commit-hooks/issues/157#issuecomment-265516255)) is to create a local hook and have it use the system environments `pylint` which is why `.pre-commit-config.yaml` has been modified.

In addition I have...

* Explicitly ignored `versioneer.py` from being linted because it is not a component of `pandas` and failed linting.
* Explicitly ignored ǜalidate_min_versions_in_sync.py` as it failed to import key libraries. Its a script used in another `pre-commit` local hook that checks minimum version of dependencies are aligned and not a component `pandas`.
* Added `vulture` as a required package to `requirements-dev.py`/ `environment.yaml` as it was the cause of some `pylint` `import-error`.
* Disabled the `pylint` `import-error` around...
  *  `import zoneinfo` (and updated the code that checks for compatability with Python 3.9) in `pandas/tests/indexes/datetimes/test_constructors.py`.
  * `from py.path import local as LocalPath` in `pandas/tests/io/excel/test_readers.py` / `pandas/tests/io/pytables/test_read.py` / `pandas/tests/io/sas/test_sas7bdat.py`


Labels:

Comments:
~~Closing as missing essential `pylint` package.~~

Oops, hadn't realised that adding a commit to my fork would make it through to here.

@MarcoGorelli : Ok not a problem, I don't have sufficient knowledge to comment on that. I'll leave it to others to decide.
no worries, thanks anyway for your PR - closing then, but there's lot's of other issues to work on if you'd like
differences in Series.map with defaultdict with different dtypes
State: closed

- [X] closes #48813
- [X] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [X] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [X] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Missing-data
description: np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
name: Series
description: Series data structure

Comments:
> Does this fix the defaultdict mutation issue? #47527

Just tested this ( & followed the older threads on this issue ).
Yes, these changes does fix the defaultdict mutation issue.

Eg -
Before changes - 
```
s = pd.Series([1, 2, np.nan])
default_map = defaultdict(lambda: "missing", {1: "a", 2: "b", np.nan: "c"})

s.map(default_map) 
# 0              a
# 1              b
# 2    missing
# dtype: object

# defaultdict is mutated
default_map 
# defaultdict(<function <lambda> at 0x7f3e792833a0>, {1: 'a', 2: 'b', nan: 'c', nan: 'missing'})
```

After changes - 
```
s = pd.Series([1, 2, np.nan])
default_map = defaultdict(lambda: "missing", {1: "a", 2: "b", np.nan: "c"})

s.map(default_map) 
# 0    a
# 1    b
# 2    c
# dtype: object

# defaultdict is not mutated
default_map 
# defaultdict(<function <lambda> at 0x7fe002b773a0>, {1: 'a', 2: 'b', nan: 'c'})
```
>  Yes, these changes does fix the defaultdict mutation issue.

Could you add a test to check that the default dict isn't mutated?
cc @phofl since you made a fix here recently
> > Yes, these changes does fix the defaultdict mutation issue.
> 
> Could you add a test to check that the default dict isn't mutated?

Updated the PR with a testcase to check defaultdict mutation.
Thx @dannyi96
BUG/PERF: MultiIndex setops with sort=None
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/v1.6.0.rst` file if fixing a bug or adding a new feature.


Simplify and improve perf in algos.safe_sort which improves perf in MultiIndex setops when sort=None.

```
           before           after         ratio
     [55dc3243]       [f00149ce]
     <main>           <safe-sort-multiindex>
-        73.3±2ms       20.2±0.4ms     0.28  multiindex_object.SetOperations.time_operation('monotonic', 'ea_int', 'intersection', None)
-        71.0±2ms       19.0±0.6ms     0.27  multiindex_object.SetOperations.time_operation('monotonic', 'int', 'intersection', None)
-       104±0.7ms         24.0±3ms     0.23  multiindex_object.SetOperations.time_operation('non_monotonic', 'string', 'intersection', None)
-        98.7±2ms       21.4±0.3ms     0.22  multiindex_object.SetOperations.time_operation('monotonic', 'string', 'intersection', None)
-       114±0.5ms       24.7±0.4ms     0.22  multiindex_object.SetOperations.time_operation('non_monotonic', 'ea_int', 'intersection', None)
-       109±0.7ms         21.3±2ms     0.20  multiindex_object.SetOperations.time_operation('non_monotonic', 'int', 'intersection', None)
-        99.3±1ms         18.5±2ms     0.19  multiindex_object.SetOperations.time_operation('monotonic', 'datetime', 'intersection', None)
-         157±6ms         21.4±2ms     0.14  multiindex_object.SetOperations.time_operation('non_monotonic', 'datetime', 'intersection', None)
```



I updated the expected value in `test_union_nan_got_duplicated` which was added in #38977. 

`MultiIndex.union` sorts by default but the expected value in that test was not sorted.

For reference, here are the docs for the sort parameter:

https://github.com/pandas-dev/pandas/blob/55dc32437ea43a238975439ddb6c9dda81b33020/pandas/core/indexes/base.py#L3297-L3305



Labels:
name: Bug
name: Performance
description: Memory or execution speed performance
name: MultiIndex
name: setops
description: union, intersection, difference, symmetric_difference

Comments:
Thanks @lukemanley
ENH: retain reso in Timestamp(dt64_obj)
State: closed

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Non-Nano
description: datetime64/timedelta64 with non-nanosecond resolution
name: Timestamp
description: pd.Timestamp and associated methods

Comments:
Thanks @jbrockmendel
DOC: Fix typos and standardize spelling of "GitHub"  and "macOS"
State: closed

## Proposed changes

1. Fix some typos. They were found with:

   ```shell
   brew install aspell
   
   find . -type f -exec cat '{}' \; |
     aspell -a |
     grep -v '*' |
     sed '/^$/d' |
     cut -d ' ' -f 2 |
     sort -u
   ```

   The output was visually inspected to find the typos.

3. Standardize spelling of "GitHub". The official spelling of "GitHub" can be seen in the title of page <https://github.com/>.
4. Standardize spelling of "macOS". The Apple operating system was called "Mac OS X" in versions 10.0-10.7, then "OS X" in versions 10.8-10.11, and finally "macOS" since version 10.12, released in 2016 (<https://en.wikipedia.org/wiki/MacOS#Release_history>).

## Checklist

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs

Comments:
thx @gcbeltramini
CLN/TST: Use fixture instead of setup_method
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
thx @mroeschke
Backport PR #48412 on branch 1.5.x (BUG: safe_sort losing MultiIndex dtypes)
State: closed

Backport PR #48412: BUG: safe_sort losing MultiIndex dtypes

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: MultiIndex

Comments:
cc @mroeschke 
This should cover the parts that we need for the other pr
Thanks @phofl
CLN/TST: Remove testing ArrowExtensionArray in favor of `extensions/test_arrow.py`
State: closed

closes https://github.com/pandas-dev/pandas/issues/46008


1. `extension/arrow/test_timestamp.py`: The original issue https://github.com/pandas-dev/pandas/issues/34986 (ExtensionDtypes with kind="M") is more-narrowly tested in https://github.com/pandas-dev/pandas/pull/48640 and the DataFrame construction is tested in `pandas/tests/extension/test_arrow.py::TestConstructors` as `ArrowDtype(pa.timestamp())` has kind="M"

2. `extension/arrow/test_string.py`: Moved test to `pandas/tests/extension/test_string.py`

3. `extension/arrow/test_bool.py`: Moved `test_is_bool_dtype` and `test_basic_equals` to `extensions/test_arrow.py` The rest of the test appeared to be a subset of what is tested in `test_arrow.py`



Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean
name: Arrow
description: pyarrow functionality

Comments:
thx @mroeschke
CLN: test_nanops/take.py
State: closed

- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Clean

Comments:
thx @mroeschke
Backport PR #48987 on branch 1.5.x (REG: Fix regression in assert_frame_equal for categorical and check_like)
State: closed

Backport PR #48987: REG: Fix regression in assert_frame_equal for categorical and check_like

Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Yikes this is tricky, the changes in safe_sort are not on 1.5.x
@mroeschke any opinion on back porting  the safe sort improvement too? Basically we have to get a MultiIndex back from safe_sort. Not sure if there is another way around this.
> @mroeschke any opinion on back porting the safe sort improvement too? Basically we have to get a MultiIndex back from safe_sort. Not sure if there is another way around this.

Could you link the PR(s) that made the change(s) to safe_sort?
#48412 and (unfortunately) parts of #48498


We could revert the change that caused this, but since this was a bug fix, we would loose functionality
IMO I think it's okay to backport both PRs. They seem fairly narrow toward what they fixed. I guess you can transfer that bug fix whatsnew note from 1.6.0 to 1.5.1.
I think I would prefer only back porting the algorithm part from the second one, the other one ties into a few other things we've done, is probably better to keep them together
> I think I would prefer only back porting the algorithm part from the second one, the other one ties into a few other things we've done, is probably better to keep them together

Okay sounds good.
REGR: to_parquet raising with bytes filename
State: closed

- [x] closes #48944 (Replace xxxx with the Github issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Regression
description: Functionality that used to work in a prior pandas version
name: IO Parquet
description: parquet, feather

Comments:
Thanks @phofl
Merge pull request #1 from pandas-dev/main
State: closed

UPDATE

- [ ] closes #xxxx (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:

Comments:
Hi, looks like something went wrong here. Closing for now. Feel free to ping to reopen, when you can provide a bit more context
DOC: replace developer meeting with contributor community page
State: closed

Updating the developer meeting page to include the new contributor meeting and slack
Along the way, I decided to include other communication channels and did my best to describe them, making this more of an internal contributor community page. 

Inspired by [SciPy](https://scipy.org/community/) and [NumPy](https://numpy.org/community/)'s community pages, and Matplotlib's [home page resources](https://matplotlib.org/) and [meeting documentation](https://matplotlib.org/stable/devel/contributing.html#get-connected), and [readme](https://github.com/matplotlib/matplotlib#contact). 

I tried to keep this pandas' community page as much as possible. Please provide feedback on what you would like to see changed, so it's more appropriate.

I would suggest we add redirects for the old page to this one. If you have an example for how this was done before, I could add it to this PR. Otherwise, I will investigate if Sphinx supports this.

Labels:
name: Docs

Comments:
> I would suggest we add redirects for the old page to this one. If you have an example for how this was done before, I could add it to this PR. Otherwise, I will investigate if Sphinx supports this.

We currently have some manual setup for redirects in conf.py, but tailored to generated API pages. Either you could adapt that a bit to work with generic pages, or otherwise we can use some sphinx extension (eg https://documatt.gitlab.io/sphinx-reredirects/index.html)
I think I just added the redirect – but check me 😉
Looks good, labelling 1.5.1 for now so that we can backport if there aren't any objections from others
The doc build failure seems to be related:

```
 /home/runner/work/pandas/pandas/doc/source/development/index.rst:26: WARNING: toctree contains reference to nonexisting document 'development/meeting'
looking for now-outdated files... none found
pickling environment... done
/home/runner/work/pandas/pandas/doc/source/development/community.rst: WARNING: document isn't included in any toctree
```

The `/docs/source/development/index.rst` file still needs to be updated
> Looks good, labelling 1.5.1 for now so that we can backport if there aren't any objections from others

Generally we don't do the effort of backporting doc fixes, but given that it would be useful to have this updated soon and this shouldn't give conflicts, sounds good to me.
CI: Pin pydata sphinx theme
State: closed

- [x] xref #48988 (Replace xxxx with the Github issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Docs
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
Thanks @phofl