PDEP-15: Reject PDEP-10
State: closed

- [ ] closes #xxxx (Replace xxxx with the GitHub issue number)
- [ ] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [ ] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [ ] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions.
- [ ] Added an entry in the latest `doc/source/whatsnew/vX.X.X.rst` file if fixing a bug or adding a new feature.


Just something quick and dirty I threw up that we should talk about tomorrow at the dev call.

Labels:
name: Closing Candidate
description: May be closeable, needs more eyeballs
name: PDEP
description: pandas enhancement proposal

Comments:
/preview
Website preview of this PR available at: https://pandas.pydata.org/preview/pandas-dev/pandas/58623/
Ugh, looks like the bullets aren't rendering correctly.
Did I miss an official vote on rejecting this? I am not sure yet that I would want to reject, and am still leaning towards keeping in spite of some negative feedback
Nope, just opening since I said I would in the discussion issue. 

We'll still need a formal vote - I'm just kicking off the discussion here.
cc @pandas-dev/pandas-core @pandas-dev/pandas-triage 

xref https://github.com/pandas-dev/pandas/issues/57073#issuecomment-2080683080 for context


As discussed in dev meeting on 5/8/24, suggestion is to do a new PDEP that reverts PDEP-10, and keeps any parts we want to keep.

> I am not sure yet that I would want to reject, and am still leaning towards keeping in spite of some negative feedback

I'm now leaning towards approving the rejection. My approval of the original PDEP was based solely on improvements to default inference for other dtypes. Despite some recent comments about this, no discussion/clarification has followed on this topic. I'd need to see some positive evidence that the original PDEP-10 authors still intend to support delivering the promised enhancements in this area. Now that the implications of using pd.NA as a default has been discussed in more depth, I suspect that any improved inference would need a couple of dtype variants.
> As discussed in dev meeting on 5/8/24, suggestion is to do a new PDEP that reverts PDEP-10, and keeps any parts we want to keep.

Yep, I'm planning on updating this current PR to do that, so if anyone has any objections or whatever, we can still discuss here.
Minor note - do we need to rename this PR? Right now PDEP-10 shows twice on the website

![image](https://github.com/pandas-dev/pandas/assets/609873/694d839f-3c8e-474b-87e2-b72968049403)
Yeah, I'll probably change the name to PDEP-15 once I get around to moving this to a separate PDEP (probably tomorrow).

I was travelling the past week, so didn't really have time then.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@lithomas1 I think you should update the PDEP based on comments from @WillAyd so we can move this forward and have a vote.
My vote on this will heavily depend if we can come to an agreement on PDEP 14
Thanks for the reminder, this slipped off my radar.

> My vote on this will heavily depend if we can come to an agreement on PDEP 14

Yeah, ideally, we'd vote on these at the same time. 
I haven't been involved in maintenance for a while, so no idea how far along that one is.
Marking as draft as although I've pushed up another commit, I haven't fully finished and refined my thoughts yet.

I also need to fix the formatting issues in this PDEP.
@lithomas1 now that PDEP-14 is accepted, we don't need to require `pyarrow` for pandas 3.0, so we should resolve this PDEP and get a vote going.
I was out for the summer, but I think this PDEP should be updated such that all comments were addressed.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
If there are no objections I'll open a vote on this shortly. While in general we've been assuming that we don't want to make pyarrow a required dependency as previously agreed, no formal vote has happened (other than in other PDEPs that assumed this one was already approved). While the discussion here could continue for longer, I think it had enough iterations to be ready for a vote.

CC @pandas-dev/pandas-core 
> > I am not sure yet that I would want to reject, and am still leaning towards keeping in spite of some negative feedback
> 
> I'm now leaning towards approving the rejection. My approval of the original PDEP was based solely on improvements to default inference for other dtypes. Despite some recent comments about this, no discussion/clarification has followed on this topic. I'd need to see some positive evidence that the original PDEP-10 authors still intend to support delivering the promised enhancements in this area. Now that the implications of using pd.NA as a default has been discussed in more depth, I suspect that any improved inference would need a couple of dtype variants.

Thinking some more I would now probably vote against rejecting PDEP-10. PDEP-10 states "NumPy object dtype will be avoided as much as possible." and gives a list of new dtypes that would be implemented in the future. Without these being included in another **approved** PDEP, rejecting PDEP-10 would be effectively removing this agreed upon direction from our roadmap which I don't think provides clarity to the community. If the issue is just the timing of the pyArrow requirement, then an update to the timelines in PDEP-10 may suffice?
This PDEP only rejects PDEP-10, and doesn't discuss how to move forward, which will have to be decided later (for example via [PDEP-16](https://github.com/pandas-dev/pandas/pull/58988) if this one is approved).

So I think the sooner we vote on this, the sooner we can decide what exactly to do for pandas 3.0, and the sooner we can start working on it.

I started the voting at #61596. The deadline for voting is 15 days, until June 22nd (I guess if everybody who can votes cast their vote before, we can move accept or reject the PDEP earlier).
in https://github.com/pandas-dev/pandas/issues/61596#issuecomment-2952390414 @rhshadrach wrote

> > A call for vote is in violation of PDEP-1.
> 
> > * After 30 days, with a note that there is at most 30 days remaining for discussion, and that a vote will be called for if no discussion occurs in the next 15 days.
> > * After 45 days, with a note that there is at most 15 days remaining for discussion, and that a vote will be called for in 15 days.
> >   ...
> >   After 30 discussion days, in case 15 days passed without any new unaddressed comments, the authors may close the discussion period preemptively, by sending an early reminder of 15 days remaining until the voting period starts.
> 
> The timeline requires a 15 day announcement to commence with voting. This has not occurred.

I may be misunderstanding or our PDEP guidelines may need to be updated but these cases are for when we  move to an early vote. The PDEP process states "A PDEP discussion will remain open for up to 60 days.". we have far exceeded this requirement and so we are not calling an early vote?

also may need clarification but PDEP-1 also states

> Invalid PDEP[ #](https://pandas.pydata.org/pdeps/0001-purpose-and-guidelines.html#invalid-pdep)
For submitted PDEPs that do not contain proper documentation, are out of scope, or are not useful to the community for any other reason...

going stale is IMO a good reason to declare this PDEP invalid? We need better guidelines in PDEP-1 about when a PDEP can be closed automatically to eliminate the situation we find ourselves in here and other PDEPs such as PDEP-16. Stale PDEP do not provide clarity to the community on the direction of pandas and the intent of the development team. 
I officially call the vote for this PDEP.

No fully agree this is needed, as this PDEP exceeded the discussion period by almost a year, but that gives people 15 days to finish the discussion. Note that the PDEP author is not an active pandas maintainer, and I'm not aware of anyone who is planning to lead the discussion or update the PDEP with the feedback. So, while it feels just like a waste of time, I'll reopen the voting issue in 15 days. I'll also be sending an email as stated in PDEP-1, but note that may vote on the PDEP-1 is -1, so my intention here is not to get the PDEP accepted, but to make a decision regarding PDEP-10 (requiring PyArrow in pandas 3.0), which so far continues to be approved.
@simonjayhawkins 

> I may be misunderstanding or our PDEP guidelines may need to be updated but these cases are for when we move to an early vote. The PDEP process states "A PDEP discussion will remain open for up to 60 days.". we have far exceeded this requirement and so we are not calling an early vote?

In all cases there needs to be a notice that there will be a vote, even in the case of early voting.

> by sending an early reminder of 15 days remaining until the voting period starts.

I do not think this needs updating. We need to give proper notice that a vote will be taking place. Anyone with final thoughts on a PDEP needs time to voice them prior to the vote starting.

> going stale is IMO a good reason to declare this PDEP invalid? We need better guidelines in PDEP-1 about when a PDEP can be closed automatically to eliminate the situation we find ourselves in here and other PDEPs such as PDEP-16.

Agreed that deciding not to put a PDEP up for a vote does not require a notice. That is not what happened here. I don't think we need better guidelines - the guidelines are clear on what needs to happen in order to have a vote. If that doesn't happen, then there is no vote. The author of the PDEP can close it, or we can close one due to inactivity as in any other PR. It can always be reopened, and if another core member wants to take up a PDEP someone else has backed away from they are free to do so.
> I do not think this needs updating. We need to give proper notice that a vote will be taking place. Anyone with final thoughts on a PDEP needs time to voice them prior to the vote starting.

I find it a paradox that we acted like PDEP-10 was already rejected immediately just because of some informal discussions, and now we need to wait one month to have confirmation if people really wanted it. Anyway, Marco brought some very good points, maybe we do benefit from some extra discussion time, we'll see.
This is a bit of a mess at this point.  

PDEP-10 states:
> Starting in pandas 2.2, pandas raises a FutureWarning when PyArrow is not installed in the users environment when pandas is imported. This will ensure that only one warning is raised and users can easily silence it if necessary. This warning will point to the feedback issue.

The feedback issue is here:  https://github.com/pandas-dev/pandas/issues/54466 

We then received a lot of complaints, and we held a vote on Feb. 20, 2024, at https://github.com/pandas-dev/pandas/issues/57424 to remove the `pyarrow` warning, with the idea that PDEP-15 would be created.    We then removed the warning in version 2.2.1 .

So if we are to require `pyarrow` in `pandas` 3.0, we really haven't given the community the proper warning from versions 2.2.1 through 2.3.

One option we have is to release a 2.3.1 with the warning put back in so that people get the appropriate warning.  We could also create a new feedback issue to see if the community still feels the requirement is onerous.  After a month or two of collecting feedback, we can then have the vote on PDEP-15.

Personally, I'm really uncomfortable requiring `pyarrow` for 3.0 because we haven't provided the warning for over a year, so if we do make the requirement without having the warning present for a reasonable amount of time, we may get some pretty negative feedback once a 3.0 is released that requires `pyarrow`.




That vote took 6 days to be decided based on the issue log. It was won by just one vote, with no previous time for discussion, and with an ad-hoc voting not in the governance, and not discusssed in advance. Everybody seems ok with it (including myself).

Can't understand the double standards compared to adding 15 extra days delay here for something that people has been discussing and thinking about for more than two years. And that it just needs a final decision so things can be unblock so devs and users can act accordingly.

Good point about the warning. Personally not too bothered about it. Most users won't even realize if pandas installed some extra depemdencies or not. And users know in major versions things change (particularly from 2 to 3, if we think of Python or Gnome). In any case, PDEP-15 is about rejecting the previous PDEP-10. Whether it's accepted or not, another discussion will be required to see how to move forward.
Going through the feedback issue, the concerns voiced by users that still remain are:

 - Package size
 - Users on systems that need to compile their own wheels

I too am concerned about making PyArrow a hard dependency. Some of that is based on not understanding how many users we will leave without an upgrade path, and some of it is based on what maintenance PyArrow itself will receive (I've heard the situation there is not great, but don't have a good understanding).

> One option we have is to release a 2.3.1 with the warning put back in so that people get the appropriate warning.

Assuming we go forward with this, I'm on board with adding back the warning if we also add a config to disable it. Many complained about the inability to silence the warning globally. Being able to silence it from the system environment is also a need-to-have in my opinion.
Thanks @rhshadrach. In my mind this is not a binary issue. There are three options:

1. reject PDEP-10 and effectively remove the future intentions of improved Arrow interoperability and improvements to default inference of some new datatypes from the pandas roadmap. That's achieved by accepting this PDEP.
2. do not reject PDEP-10 and make pyArrow a required dependency in pandas 3.0. That's achieved by rejecting this PDEP.
3. postpone making pyArrow a required dependency until the next major release and continue development assuming that pyArrow will be a required dependency in the future. That's achieved by rejecting this PDEP and approving a revision to PDEP-10 instead.

Surely welcome suggestions from others for alternative options.

Now my rejection of this PDEP in the "invalid" vote is based on not wanting to remove some previously agreed roadmap items.

in https://github.com/pandas-dev/pandas/issues/61596#issuecomment-2952281881 I said I may vote differently if the motivation for rejecting PDEP-10 was to keep PyArrow an optional dependency indefinitely. I am not adverse to that but would want to clarify the pandas roadmap for default inference of some new dtypes before agreeing to that. I would probably want to see some object fallback versions of these dtypes for default inference when PyArrow is not installed.