WEB: add note to PDEP-10 about delayed timeline for requiring pyarrow
State: open

This is the PR that I _should_ have done a year ago after PDEP-14 got voted on (https://github.com/pandas-dev/pandas/pull/58551, voting issue at https://github.com/pandas-dev/pandas/issues/59160). With that PDEP, and at the time with the intent of releasing pandas 3.0 already somewhere last year, the plan was to add this new default string dtype that would use pyarrow, but only if installed and otherwise still fallback on the object-dtype based implementation of the string dtype. 
Essentially that PDEP detailed the plan for a string dtype mentioned in PDEP-10, while also delaying the rest of PDEP-10 until after pandas 3.0 (i.e. not yet making pyarrow a required dependency for pandas 3.0).

I think that if I would have done this PR directly after the vote on PDEP-14, it would have been clear in that context and that such an amendment to a PDEP would not necessarily require a vote or a separate PDEP (since the amendment is reflecting that a subsequent PDEP changed/superseded a part of a previous PDEP).

However, given the current context with the revived discussions about defaulting to pyarrow dtypes and requiring pyarrow and the vote on the PDEP about rejecting PDEP-10 (and with this PR being done long after the PDEP-14 acceptance), I entirely understand that this PR is now a lot more controversial (all the more reason that I should have only opened this PR with proper context .. again apologies about that).

To conclude, with this PR I mostly want to illustrate what I personally would do on the short term for pandas 3.0, as I also mentioned in the PDEP-15 (reject PDEP-10) discussion two days ago at https://github.com/pandas-dev/pandas/pull/58623#issuecomment-2996630716. 
(and apologies for not having done this PR much sooner .. I should have handled the public communication after PDEP-14's acceptance better)

Labels:
name: Web
description: pandas website

Comments:
@datapythonista sorry for opening this PR without the proper context, I should just have waited with opening it until I had written that up.

[moved the context I added here to the top post]

EDIT: and with the above, I hope that I clarified that it is not my intent to apply double standards

Now that we finally could vote on PDEP-15 and know what the team wants, and since it seems like most people is happy to consider PDEP-10 officially rejected, I'm ok with this.

I still think it's very poor team working to fully ignore the governance procedure when it's convenient, and use it as if we were lawyers in other cases (not that you personally did the latter).
Perhaps this should wait until PDEP-14 is actually implemented. The new string dtype is behind a "future/experimental/options" whatever we want to call it flag until we are ready to make it the default. None of the code samples in https://github.com/pandas-dev/pandas/pull/61705 raise any warnings to end users that the behavior is changing. Since PDEP-14, the team has approved PDEP-17, Backwards compatibility and deprecation policy, and we need at least 2 releases to allow "Breaking changes should go through a deprecation cycle before being implemented if possible."

Now, this would be a further delay to 3.0 that probably nobody wants.

We perhaps should have a milestone/gated/readiness, whatever you want to call it, review of the status of the PDEP-14 implementation. If not "flipping the switch" on the new dtype we could probably get 3.0 out the door tomorrow?
> None of the code samples in https://github.com/pandas-dev/pandas/pull/61705 raise any warnings to end users that the behavior is changing.

@simonjayhawkins That is on purpose, because we are planning to do this as a breaking change (as described in PDEP-14, see its section on "Backward compatibility"). If you want to reconsider this, please open a separate discussion issue to bring this up. 

(yes, we have PDEP-17 to describe our _standard_ deprecation policy, and it mentions that breaking changes should go through a deprecation cycle _if possible_. But so there are cases where this is not possible (because it would be very hard to implement or too noisy for users), and PDEPs are especially the place to agree on such "not possible" cases). 
The first part of the comment was...

> Perhaps this should wait until PDEP-14 is actually implemented.

My reasoning being that updating PDEP-10 with regard to PDEP-14 may be premature as PDEP-14 is not  yet fully implemented. However, I am also all for clarifying the current situation to the community.

So wording along the lines that none of the reasons for requiring PyArrow for 3.0 are yet applicable and therefore the requirement is deferred. 

> If you want to reconsider this, please open a separate discussion issue to bring this up.

I won't open an issue about the warnings specifically, but maybe a discussion on the readiness of the dtype with a frank discussion on realistic timescales for 3.0rc release. I have made some comments in #61590 towards the goal of establishing a realistic release date.
> The first part of the comment was...

I know, and I haven't yet responded to that. I just wanted to quickly point out a (in my mind) misunderstanding about the plan for the string dtype (that we would do a deprecation cycle for this). I don't really agree with your reasoning for delaying adding this note, but that requires some more time to write a thoughtful response.
> I don't really agree with your reasoning for delaying adding this note

maybe the note could be as simple as following feedback from the community the requirement has been deferred and that the pandas team still intend to deliver the benefits of PyArrow in a future release?
> that the pandas team still intend to deliver the benefits of PyArrow in a future release?

assuming PDEP-15 is rejected of course.
> > that the pandas team still intend to deliver the benefits of PyArrow in a future release?
> 
> assuming PDEP-15 is rejected of course.

so @jorisvandenbossche I guess we are both guilty of getting ahead of ourselves and assuming that PDEP-15 is already rejected:)

> I don't really agree with your reasoning for delaying adding this note

let's then wait for the outcome of the vote? What if PDEP-15 is accepted? Would this PR still be required?

I think the vote closes on 8th July. By then we should have more clarity.
> as PDEP-14 is not yet fully implemented

PDEP-14 is implemented, I would say, except that we indeed did not yet switch the default on main. We had discussed before to do that after releasing 2.3.0, but so that now has happened.

> maybe the note could be as simple as following feedback from the community the requirement has been deferred and that the pandas team still intend to deliver the benefits of PyArrow in a future release?

I am fine with rewording to something more like that. But essentially that community feedback let to the creation of PDEP-14 (which mentions this), so that is very similar in my mind (but I probably also too much with my head into it to judge what is a good wording)

> I guess we are both guilty of getting ahead of ourselves and assuming that PDEP-15 is already rejected:)

FWIW, I don't consider myself getting ahead of things. Of course the situation is now a bit different with the ongoing vote, but as I mentioned in the top post above, IMO I _should_ have done this PR directly after PDEP-14's acceptance, so I am actually much behind instead of ahead ;)

Joking aside, sure let's wait until there is clarity around what the vote on PDEP-15 actually means ..
> FWIW, I don't consider myself getting ahead of things. Of course the situation is now a bit different with the ongoing vote, but as I mentioned in the top post above, IMO I _should_ have done this PR directly after PDEP-14's acceptance, so I am actually much behind instead of ahead ;)
> 
> Joking aside, sure let's wait until there is clarity around what the vote on PDEP-15 actually means ..

That's a fair point since I assume that PDEP-15 would have been withdrawn long ago if this had been done sooner.

With our current governance only people who participated in the discussion can give a negative vote on PDEP-15, and I think most have done that. So the outcome is perhaps known even though we need to wait until we can tally the figures.
Thanks @jorisvandenbossche for this PR, and for getting pandas ready for 3.0.

Since PDEP-15 will be rejected and PDEP-10 won't be enforced but theoretically still approved, I think it'd be good to highlight this note more. What I'd do is to add the admonition mardown extension in `web/pandas/config.yml`, and then you can use the next syntax to render as a proper note or warning (it'll probably need adding styles in our css too):

```
!!! warning
    Your message in the note.
```

Also, if this note is the main place where we communicate that we changed our plans (we had the warning, social media posts..., so I'd bet many people is still expecting PyArrow in pandas 3.0), I'd provide more information. There is some in PDEP-15, but it'll be rejected, and it's outdated, so probably this note is the main reference.

In particular, I think it'll be good for users to understand WHY we changed the plans, not just letting them know that we did. I assume the main reason is the extra 288Mb in disk pandas would necessarily have if installed always with PyArrow (not sure if people had other reasons as they weren't shared in PDEP-15 vote). Also, since my guess is that just a minority of users would really care about the extra disk space, I think it'd also be good to explain why we don't simple add PyArrow to the `pandas` package, and release a second package `pandas-lite` for these users who care.

I'd also prefer waiting to merge this until the vote in PDEP-15 is over. I don't expect any surprise, and I'm not a big fan of following our governance blindly when it doesn't make sense. But if this doesn't delay the release, which I don't think it does, seems a bit better to not assume the result of votes while they are still happening.
@jorisvandenbossche we've had some PDEPs that have their status updated to "Implemented". Would we also want to do that here for PDEP-14? or separate PR?