PDEP-1 first revision (scope)
State: closed

This proposal revises the PDEP definition and guidelines. **Feedback is welcome**.

Context and summary of the changes:

- Clarify the scope and purpose of PDEPs to make it clearer when a PDEP would be needed and when not (this is a discussion we basically didn't really have in the initial PR (https://github.com/pandas-dev/pandas/pull/47444), and this wasn't very clear for several people involved): this is done by expanding the "PDEP definition, purpose and scope" section, and by adding a list of examples at the bottom of the page.
- Updated the "PDEP authors" section to expand the explanation for non-core-team members and include they need a sponsor. IIRC the reason for this is two-fold: to have _some_ form of quality assurance (to not be "spammed" by many drive-by PDEPs that might not need to be a PDEP) and to help the non-core-team submitter (the sponsor would be a core-team member who takes on the responsibility of guiding the PDEP author through the process).

Co-authored-by: @MarcoGorelli, @rhshadrach, @Dr-Irv, @jorisvandenbossche, @mroeschke 

Labels:
name: PDEP
description: pandas enhancement proposal

Comments:
Thanks @Dr-Irv! I'm fixing it up. Should pandas-stubs also be `pandas-stubs`? 
> Thanks @Dr-Irv! I'm fixing it up. Should pandas-stubs also be `pandas-stubs`?

Yes.  And I may have missed a few other places where backquotes are needed.
> > Thanks @Dr-Irv! I'm fixing it up. Should pandas-stubs also be `pandas-stubs`?
> 
> Yes. And I may have missed a few other places where backquotes are needed.

Yeah, I'm adding a bunch more following your examples =)
> Yeah, I'm adding a bunch more following your examples =)

Let me know when you're done with your edits, and I'll take another look. Thanks!
I haven't been to a governance meeting in a while(had a time conflict), but is there a procedure for revising a PDEP/deferring parts of a PDEP?

I am asking since I am going to open PDEP-7 soon, and there are some parts where I would like to survey users, but not have that block acceptance of a PDEP.

It would also be nice to have since this PR itself revises a PDEP :).
> I haven't been to a governance meeting in a while(had a time conflict), but is there a procedure for revising a PDEP/deferring parts of a PDEP?
> 
> I am asking since I am going to open PDEP-7 soon, and there are some parts where I would like to survey users, but not have that block acceptance of a PDEP.
> 
> It would also be nice to have since this PR itself revises a PDEP :).

Looking through the existing text, it looks like there's some process for this, but it would be nice to clarify whether changes would need to be accepted again and who can propose revisions.
Nice, thanks for writing this up! 🙌 
> Looking through the existing text, it looks like there's some process for this, but it would be nice to clarify whether changes would need to be accepted again and who can propose revisions.

I believe that as long as you mean changes during the discussion phase, you don't need a process for amending your PDEP. Only once it has been accepted you'd need to follow a revision process. 

We are working on a definition for the decision making process (e.g. voting and acceptance) at the moment. 

> > Looking through the existing text, it looks like there's some process for this, but it would be nice to clarify whether changes would need to be accepted again and who can propose revisions.
> 
> I believe that as long as you mean changes during the discussion phase, you don't need a process for amending your PDEP. Only once it has been accepted you'd need to follow a revision process.

There is indeed some text around revisions currently. But it's maybe something we should discuss a bit more to what extent small revisions (after acceptance) would need to follow the full normal decision process. 
For example, Marco has made some small revisions to PDEP4 while implementing it and from initial feedback with the implementation (https://pandas.pydata.org/pdeps/0004-consistent-to-datetime-parsing.html, see at the bottom there is a "revision history"). It might again become subjective call on whether something is a "significant" change or not. 

Preview docs at: https://pandas.pydata.org/preview/51417/
> This proposal revises the PDEP definition and guidelines.

Why?

PDEPs are predominantly to engage the community in significant changes to pandas? So this should also apply to PDEP revisions too? It's not clear to me what the motivation is for revising the PDEP.
It's just to clarify a few things

> PDEPs are predominantly to engage the community in significant changes to pandas?

That, and to have a process with which to make significant changes
> It's just to clarify a few things

Thanks @MarcoGorelli 

I think the bar for revising a PDEP should be as high as the initial PDEP itself and therefore the motivation for the changes should be transparent (to the community).

maybe the revision status should indicate what needed clarifying. And also the change to requiring core-team sponsors explicitly mentioned? 



Sure, good point

For context, the revision comes out of the biweekly governance meeting, quite some discussion went into it

And yes a brief summary of the revision would be good (e.g. in [pdep4](https://pandas.pydata.org/pdeps/0004-consistent-to-datetime-parsing.html) there's a sentence to say why it was revised, even though that particular revision was really minor)
> For context, the revision comes out of the biweekly governance meeting, quite some discussion went into it

Yes. I do read the minutes. But we are now transferring that outcome to the wider community.

We introduced the PDEP as a formal process for pandas changes. I don't want to see that process compromised by allowing or encouraging revisions that would not have the same scrutiny or oversight.

(to be clear. I'm not trying to block the changes in this PR per se.)


  
The top post could indeed have provided a brief summary and rationale of the proposed changes, to give some context to the diff. Now, I _think_ apart from clarification of existing text (what Marco mentioned), the "sponsor" seems to be the only substantial change. Potential summary:

- Clarify the scope and purpose of PDEPs to make it clearer when a PDEP would be needed and when not (this is a discussion we basically didn't really have in the initial PR (https://github.com/pandas-dev/pandas/pull/47444), and this wasn't very clear for several people involved): this is done by expanding the "PDEP definition, purpose and scope" section, and by adding a list of examples at the bottom of the page.
- Updated the "PDEP authors" section to expand the explanation for non-core-team members and include they need a sponsor. IIRC the reason for this is two-fold: to have _some_ form of quality assurance (to not be "spammed" by many drive-by PDEPs that might not need to be a PDEP) and to help the non-core-team submitter (the sponsor would be a core-team member who takes on the responsibility of guiding the PDEP author through the process).

The "sponsor" term is inspired on the PEP process (https://peps.python.org/pep-0001/#submitting-a-pep), but we should probably explain that in a bit more detail in the text here.

@noatamir if that sounds about right, I (or you) can update the top post to include this summary.




> We introduced the PDEP as a formal process for pandas changes. I don't want to see that process compromised by allowing or encouraging revisions that would not have the same scrutiny or oversight.

The first PDEP-1 was done as a PR, this revision is using the exact same process: here is a PR with proposed changes that is open for discussion and needs to be approved by the core team.
Thanks @jorisvandenbossche 

That summary reducing the changes to 2 key points would indeed be helpful to guide the discussion and gaining acceptance.

On the first point, my initial reaction was that the (additional) information could instead be included in the pandas documentation and therefore be updated regularly outside of the PDEP process.

On the second point I have no strong preference at this time. I see @datapythonista POV that if it's not broken.... And also since the voting process is also being discussed that maybe this change may be premature. (for instance... why stop at only 1 sponsor? and what if it's decided that co-authors don't have voting rights)
> On the first point, my initial reaction was that the (additional) information could instead be included in the pandas documentation and therefore be updated regularly outside of the PDEP process.

I think our general idea is that this PDEP is the document that fully describes the process in enough detail, and doesn't need separate documentation to explain in more detail what the PDEP-1 summarizes. 

Of course it's also an option to keep PDEP-1 to the essentials and have more extended guidelines elsewhere (I don't think this is a point that really has been discussed already?). Personally I am not sure if this would make it easier/clearer. Because also when doing such documentation updates, we will still need to evaluate whether that's just a minor tweak/clarification, or a larger change that is worth being mentioned in the PDEP itself. And it gives you two sources of truth that need to be kept up to date with each other.

> [on the sponsors] And also since the voting process is also being discussed that maybe this change may be premature.

At the moment, I don't think we have intentions to let it influence the voting process (being a sponsor or co-author doesn't impact whether you are eligible to vote or not). And also if it does, we can still update that when we discuss the exact details of the voting process. So I think we can discuss it now on its own merits, regardless of potential voting.

Thank you all for your feedback! 🌷 
@pandas-dev/pandas-core I think this PR should be more or less ready now. Please give it another look, and also if you are fine with the latest changes, please give it an explicit approval.
+1
> PDEPs are appropriate for user facing changes

In #52140 @dr-irv wrote "Given that you are proposing a change in behavior (which would require a deprecation cycle anyway), I believe that a PDEP is warranted."

The idea that any behavior change warrants a PDEP is absurd.  This is not viable.
I understand your concerns Brock, and if the suggestion was that any behaviour change warranted a PDEP, then that would indeed be absurd. But if a change is fundamental, then it could benefit from the PDEP process.

In https://github.com/pandas-dev/pandas/issues/52140 specifically, the issue starts with "These changes are somewhat fundamental", and so I'm inclined to agree that it should be a PDEP.

Regarding the changes being proposed in this PR - are you on board with them? If not, that's fine, but could you please point out which specific parts you'd like to see changed?

Looking forward, we're going to have to make big decisions around nullable dtype and pyarrow, which I know you care about. By improving governance and decision-making, the expectation is that we will all together be able to make pandas even better
@jbrockmendel can you respond to Marco's answer? Does that answer your concern, or do you still have concrete feedback for _this_ PR?
The problematic sentence is not made worse by this PR, so no objection.  I only commented here because irv pointed to the PR specifically as relevant.
Is there anything stopping this being merged now? (we don't have the voting procedure until the next revision of this document is accepted)
No, after the edit for your last comment yesterday, I was just waiting for CI to pass ;) So going to merge now.


Thanks @noatamir and others.
Thanks @noatamir for leading the effort, and thanks to everyone who contributed feedback both in the meetings as here on the PR!