BrandonFors/pandas-github-issues huggingface dataset:
BUG: Series[object].astype(td64_unsupported)
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Already has a whatsnew in 2.0.


Labels:
name: Bug
name: Astype

Comments:
Thanks @jbrockmendel
REGR: memory_usage showing unnecessary FutureWarning
State: closed

- [x] closes #50066 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Regression
description: Functionality that used to work in a prior pandas version
name: DataFrame
description: DataFrame data structure

Comments:
Either need conditional logic on 32 vs 64-bit machines or just not check the result; I'm okay with the latter for this.
lets try intp 
It's not the dtype, it's the value. You're counting bytes on 32bit vs 64bit machines.
Thx, should have checked the error message...
Thanks @phofl
BUG: infer-objects raising for bytes Series
State: closed

- [x] closes #49650 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

cc @jbrockmendel 

Labels:
name: Dtype Conversions
description: Unexpected or buggy dtype conversions
name: Regression
description: Functionality that used to work in a prior pandas version

Comments:
Needs rebase
rebased
thanks @phofl
BUG: loc.setitem raising when expanding empty frame with array value
State: closed

- [x] closes #49972 (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Bug
name: Indexing
description: Related to indexing on series/frames, not to indexes themselves
name: Stale

Comments:
@phofl worth talking about in today's meeting?
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DEPR: Enforce DataFrameGroupBy.__iter__ returning tuples of length 1
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas

Comments:
thx @rhshadrach
DEPR: Enforce groupby.transform aligning with input index
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: Groupby
name: Deprecate
description: Functionality to remove in pandas
name: Apply
description: Apply, Aggregate, Transform, Map

Comments:
thx @rhshadrach
TST: test to confirm rmod works for large series
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

test for #29602 

Labels:
name: Stale

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Looks like #29602 was addressed so closing
TST: add test for nested OrderedDict in constructor
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


closes #18166

Labels:
name: Testing
description: pandas testing functions or related to the test suite

Comments:
out of draft mode but CI is failing though looks completely unrelated :/
Thanks @codamuse
DOC: Remove all versionchanged / versionadded directives < 1.0.0
State: closed

**Description**
Remove all `versionchanged` / `versionadded` directives for versions < `1.0.0`

**Checks**
- [x] closes https://github.com/noatamir/pydata-global-sprints/issues/10)
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).

Labels:
name: Docs
name: Sprints
description: Sprint Pull Requests

Comments:
thx @iofall
BUG: read_csv with engine pyarrow parsing multiple date columns
State: closed

- [ ] closes #47961 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Labels:
name: IO CSV
description: read_csv, to_csv
name: Arrow
description: pyarrow functionality

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
thx @lithomas1 

can you open an issue about the remaining case?
CI: Pin pyarrow smaller than 10
State: closed

lets see if this fixes ci, matplotlib is a bit weird

Labels:
name: CI
description: Continuous Integration

Comments:
>  matplotlib is a bit weird

🤔 yeah, why is version 3.3.2 getting installed?
merging to fix CI, thanks @phofl !
3.6.2 was pulled on main, same version as before but plotting tests were failing anyway
are you sure? [this build from main](https://github.com/pandas-dev/pandas/actions/runs/3614121669/jobs/6090344581) had matplotlib 3.3.2, which was causing the plotting tests to fail
good point, checked a build where only the arrow tests were failing
Guess: A dependency of a dependency pinned arrow recently and we pulled a really old build there that did not pin arrow but matplotlib.
@mroeschke I think we should revisit our conversation from a while back to pin everything to be at least our minimum allowed dependency
this needs backporting, 1.5.x CI is failing
@meeseeksmachine backport 1.5.x
@meeseeksdev backport to 1.5.x
Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

```
git checkout 1.5.x
git pull
```

2. Cherry pick the first parent branch of the this PR on top of the older branch:
```
git cherry-pick -x -m1 4c5db2e435d13c5ca91f98d07edde18a498ae6dd
```

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

```
git commit -am 'Backport PR #50055: CI: Pin pyarrow smaller than 10'
```

4. Push to a named branch:

```
git push YOURFORK 1.5.x:auto-backport-of-pr-50055-on-1.5.x
```

5. Create a PR against branch 1.5.x, I would have named this PR:

> "Backport PR #50055 on branch 1.5.x (CI: Pin pyarrow smaller than 10)"

And apply the correct labels and milestones.

Congratulations — you did some good work! Hopefully your backport PR will be tested by the continuous integration and merged soon!

Remember to remove the `Still Needs Manual Backport` label once the PR gets merged.

If these instructions are inaccurate, feel free to [suggest an improvement](https://github.com/MeeseeksBox/MeeseeksDev).
BUG: to_datetime with decimal number doesn't fail for %Y%m%d
State: closed

- [x] closes #50051 (Replace xxxx with the GitHub issue number)
- [x] closes #17410
- [x] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [x] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.

Previously, %Y%m%d was excluded from the ISO8601 fast path, and went down its own special path.

This:
- introduced complexity
- introduced bugs (like the one linked)

%Y%m%d is listed as an example of ISO8601 date on Wiki https://en.wikipedia.org/wiki/ISO_8601#Calendar_dates:

> the standard allows both the "YYYY-MM-DD" and YYYYMMDD formats for complete calendar date representations

So, I'd suggest just getting rid of the special %Y%m%d path, and just going down the ISO8601 path

---

Performance degrades, and is now on-par with other non-ISO8601 formats. I think this is fine - if the code is buggy, it doesn't matter how fast it is.

On upstream/main:
```python
In [1]: dates = pd.date_range('1700', '2200').strftime('%Y%m%d')

In [2]: %%timeit
   ...: to_datetime(dates, format='%Y%m%d')
   ...: 
   ...: 
68.6 ms ± 2.18 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
```

On this branch:
```python
In [1]: dates = pd.date_range('1700', '2200').strftime('%Y%m%d')

In [2]: %%timeit
   ...: to_datetime(dates, format='%Y%m%d')
   ...: 
   ...: 
212 ms ± 6.57 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

Note that this is comparable performance with other non-ISO8601 formats:
```python
In [4]: %%timeit
   ...: to_datetime(dates, format='%Y foo %m bar %d')
   ...: 
   ...: 
235 ms ± 7.99 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

The fastest thing in this example would actually be to not specify a `format`, but then that would fail on 6-digit `%Y%m%d` dates (see https://github.com/pandas-dev/pandas/issues/17410#issuecomment-1337636330)

Labels:
name: Bug
name: Datetime
description: Datetime data dtype
name: Performance
description: Memory or execution speed performance

Comments:
closing, probably superseded by #50242
BUG: fix  AttributeError: CompileError object has no attribute orig
State: closed

- [x] closes #50062 (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [x] Added an entry in the latest `doc/source/whatsnew/v1.5.2.rst` file if fixing a bug or adding a new feature.


Labels:
name: Stale

Comments:
Pandas version: 1.5.2.
AttributeError: CompileError object has no attribute orig


Function pandas.DataFrame.to_sql() writes records stored in a DataFrame to a SQL database. Databases supported by SQLAlchemy are supported. This function will call 'sql.py' in 'pandas/pandas/io/sql.py'.

When applied to a database supported by SQLAlchemy, 'sql.py' will use the class 'SQLAlchemyEngine(BaseEngine)', which has this line causing the reported bug: https://github.com/pandas-dev/pandas/blob/main/pandas/io/sql.py#L1392.

Basically, method 'insert_records' handles 'SQLAlchemyError' exceptions. But not all child class of base class 'SQLAlchemyError' has the attribute 'orig'. Doc: https://docs.sqlalchemy.org/en/14/core/exceptions.html

Therefore, adding a checking attribute step will fix this bug.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
Thanks for the pull request, but it appears to have gone stale. If interested in continuing, please merge in the main branch, address any review comments and/or failing tests, and we can reopen.
DEPR: remove Index._is backward_compat_public_numeric_index
State: closed

Removes the attribute  `Index._is backward_compat_public_numeric_index`and related stuff. This is the last PR before actually removing Int64/UInt64/Float64Index from the code base.

This PR builds on top of #49560 (which is not yet merged), so please exclude the first commit  from review of this PR.

I'd also truly appreciate if someone could review #49560, as it's quite a heavy PR with lots of dtype changes.

Labels:
name: Deprecate
description: Functionality to remove in pandas
name: Index
description: Related to the Index class or subclasses

Comments:
Could someone take a review? E.g. @jbrockmendel or @mroeschke?

I've rebased this after #49560 has been merged.

After #49560 and this will have been merged, the removal of Int/Uint/Float64Index should become quite straight forward. 

Thanks @topper-123
DOC: Improve groupby().ngroup() explanation for missing groups
State: closed

The exact prose and examples are open to improvements if you have ideas.

I figure this didn't require a separate issue, please let me know if I'm missing some step in the process.

Labels:
name: Docs
name: Groupby

Comments:
@datapythonista @mroeschke @rhshadrach this is ready for re-review! Sorry for the tag if this is already in your queue :)

That failing test looks unrelated to this change.
Thanks @NickCrews
ENH: Add nullable keyword to read_sql
State: closed

Sits on top of #50047 

Functionality wise, this should work now.

More broadly, I am not sure that this is the best approach we could take here. Since the ``convert_to_nullable_type`` in ``lib.maybe_convert_objects`` is not used right now except here, we could also make this strict and return the appropriate Array from the Cython code, not only when nulls are present. This would avoid the re-cast in the non-cython code part.

Labels:
name: IO SQL
description: to_sql, read_sql, read_sql_query
name: NA - MaskedArrays
description: Related to pd.NA and nullable extension arrays

Comments:
I think makes sense and like your vision on the interaction with Cython. Is there any performance difference here moving away from the `.from_records` construction of the DataFrame?
Also discussed this with @mroeschke in another pr, he leaned more towards keeping maybe_convert_objects as is.

Maybe a bit faster? The previous implementation was running through the same function that we are calling directly now. Definitively no slowdown
>  Since the convert_to_nullable_type in lib.maybe_convert_objects is not used right now except here, we could also make this strict and return the appropriate Array from the Cython code, not only when nulls are present

I think in a future PR it might make sense to do this; I didn't realized this was the only location where this is used. I guess the only slight downside is run time importing the Arrays in the Cython code?
Thanks @phofl
Docker Hub Sample images
State: closed


Still working on mamba. Opened an open source account inquiry with Docker so will eventually place there. In the interim can place on personal account for testing

Labels:

Comments:
Just to understand: This is for development with different environment sizes?
Yea it could help target different development goals. Alpine would be a super small image (~300 MB total), the minimal variants for pip/mamba just give you everything you need to compile / test most of the repository and come in at ~1 GB. The full variants would provide all packages for a complete environment, at the cost of more disk space (pip ~ 3 GB mamba ~ 6 GB)

I put the minimal images up at https://hub.docker.com/r/willayd/pandas-dev/tags if you'd like to try
Not a goal at the moment but there *might* also be a future where instead of rebuilding a testing image for every CI run we have a pre-built docker image that we just download and run from. Might save a lot of build time
I see that the pip vs mamba minimals are pretty close in size while the mamba ones have a ~1G difference.

Was the thought of providing pip vs mamba due to a size difference? If the size difference isn't significant, I would slightly favor just having the mamba environments, especially this is for scientific python development in which mamba/conda is well more equipped than pip
Is that the size you see after installing locally? Might have some artifacts left over locally that I'm not aware of but the mamba full install for me was almost double the size of the pip image. 

The mamba image is also a little more complicated to deal with if you want to run pre commit. Not sure if that would even work but the pip images should
Oh that's just the compressed sizes I'm seeing in https://hub.docker.com/r/willayd/pandas-dev/tags


Ah gotcha. Yea would be curious if anyone does `docker pull willayd/pandas-dev:mamba-all` and `docker pull willayd/pandas-dev:pip-all` if you see a difference. Here's what mine yield

```sh
$ docker image list
REPOSITORY                      TAG                      IMAGE ID       CREATED         SIZE
willayd/pandas-dev              mamba-all                6336333277b7   2 hours ago     5.68GB
willayd/pandas-dev              pip-all                  b209d2a0b20b   2 days ago      2.63GB
```

There could be something simple missing from the mamba Dockerfile that would minimize that, but I think it will always use more disk space than pip
We now an official pandas org set up on docker hub - updated the documentation to point to that
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@WillAyd still active? also a conflict needs resolving.
Closing to clear the queue, but feel free to reopen when you have time to circle back
BUG: Change FutureWarning to DeprecationWarning for inplace setitem with DataFrame.(i)loc
State: closed

- [X] closes #48673 
- [X] [Tests added and passed](
- [ ] Added [type annotations]
- [ ] All [code checks passed].
- [X] Added an entry in the latest `doc/source/whatsnew/v1.5.3.rst` file.


Labels:
name: Warnings
description: Warnings that appear or should be added to pandas

Comments:
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@aneesh98 during today's community dev meeting we decided to change this warning to `DeprecationWarning` instead of fully removing it. Would you be interested in continuing and making that change?
Hi, @mroeschke I am very much interested in completing the change.
Hi @mroeschke, just to confirm should be any change to text content of the warnings?
> just to confirm should be any change to text content of the warnings?

No change to the text. Also, could you reopen this PR targeting the 1.5.x branch directly?
Sorry for the noise, but changing the base branch was a bit trickier than expected. I also addressed the pending comments here.

If CI finishes green, I think this is ready to be merged. If this and #50682 are ready in the next hours, happy to start the release of 1.5.3 tomorrow.

CC: @pandas-dev/pandas-core
Push a commit to fix some tests
Thanks @aneesh98 for the work on this. As you've seen we finished the last couple of issues, as we need this for the 1.5.3 release which we are going to release today.

And thanks @mroeschke for finishing it.
test: Add test for loc assignment changes datetime dtype
State: closed

- [x] closes #49837
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).


Labels:
name: Needs Tests
description: Unit test(s) needed to prevent regressions

Comments:
@phofl ping 
Thanks for the review!
thx @alonme