Add a Roadmap
State: closed

This PR adds a roadmap document. This is useful when pursing funding; we can point to a list of known items that we'd like to do if we had the person time (and funding) to tackle them.

Let's have two discussions

1. Do we want this? Roadmaps tend to go stale. How can we keep this up to date?
2. If so, what items should go on it? I've mostly picked from https://docs.google.com/document/d/151ct8jcZWwh7XStptjbLsda6h2b3C0IuiH_hfZnUA58/edit#heading=h.qm48l6dargmd plus a couple pet peeves of mine.

cc @pandas-dev/pandas-core 

Labels:
name: Admin
description: Administrative tasks related to the pandas project

Comments:
> Do we want this? Roadmaps tend to go stale. How can we keep this up to date?

Conditional on having a good way to keep it up to date, I have no strong opinion.  Otherwise I'd rather not have it.

> If so, what items should go on it?

Depends largely on: who is the target audience?
> 1. Do we want this?

I think it adds a lot of value to users, developers of other tools of the ecosystem, and also companies or institutions who can be incentivaized to provide funds to see the points in the roadmap implemented. Also to ourselves, at least to me it's good to have an overview or what are the points we all think we should be moving towards.

I think having this page in the top level of the docs will make it visible enough to not be obviously outdated. But in any case, I think all the points will take months (or years) to be implemented, I don't see the roadmap being outdated easily. We can add a check to the release process to double check that items completed are deleted from the roadmap. We may forget to add new points, or remove points that stop being a priority, but I think we can keep it updated enough, and even if it's not, to me it's better to have a not very accurate roadmap, than not having it.
> But in any case, I think all the points will take months (or years) to be implemented, I don't see the roadmap being outdated easily.

That's my hope. I don't expect many of these to be checked off within the year.

> Since we mention the benchmarks, may be we could also add a section about the CI?

Happy to have something here like that. Would you mind pushing it here, or as a followup? I was struggling with summarizing what remains to be done with those linked issues.
Touching on the Target Audience thing again, two ideas for niches this might help with
- clarifying what Big Picture things are being actively worked on (possibly by whom)
- we have Good First Issue tags but there is a gap between newcomer and veteran where contributors might appreciate ideas on where help is needed
Maybe too technical, or just not important enough for a "big picture" talk, but simplifying the indexing code is certainly on the roadmap (thinking of https://github.com/pandas-dev/pandas/wiki/(Tentative)-rules-for-restructuring-indexing-code )
I will look soon
Added a short numba section if you want to take a look. I emphasized the rolling / window `.apply` aspect.

---

Going forward, it'd be nice if we had a GitHub issue dedicated to each of things to scope out the problem and serve as a discussion site for specifics (like https://github.com/pandas-dev/pandas/pull/27478#discussion_r305548712). During those discussions we may determine that the roadmap item is not appropriate and remove it from the roadmap. We shouldn't take these items as gospel handed down from on high.
Haven't reviewed deeply yet but just to bring attention to it we also have something like this on our donations page:

https://pandas.pydata.org/donate.html

So probably want to align these
Updated the arrow section.

I'll followup with a PR updating the website to point to this document.

Are people generally OK with this? I'd rather have something in and then iterate on individual items. This is intended to be a living document.
Thanks for starting this Tom!

> Do we want this? Roadmaps tend to go stale. How can we keep this up to date?

As @datapythonista said, I think having a roadmap has a lot of value (for communication to the community). 
I also think that an important part of the value for *us* are the processes around it: having and recording the discussion to get those items on which there is broad agreement, making sure we have processes to regularly update the roadmap, ..

Do we also want a section that tries to describe the goals and/or scope of pandas? (not necessarily in this PR, but I think it fits naturally in a roadmap (discussion)) 
Eg numpy did this when adding their roadmap (https://numpy.org/neps/scope.html), and also xarray has some wording on the focus of the package (http://xarray.pydata.org/en/stable/roadmap.html)

@TomAugspurger I think it would be good to mention this on the pandas-dev mailing list as well


> I also think that an important part of the value for us are the processes around it: having and recording the discussion to get those items on which there is broad agreement, making sure we have processes to regularly update the roadmap, ..

https://github.com/apple/swift-evolution/blob/master/process.md may be worth reading through on this.
Added da01cb42cc658d9ebdc5a4419f3fc053494759b3 with a discussion about how to change the roadmap.

As a followup, I'd like to go through the process (probably with String dtype).
I like the idea of a roadmap, and would like to suggest an addition, which is that things that are available on columns should be available when using a `MultiIndex` as the index for rows and vice versa.  For example:
1) I can use `query` and include the name of an index level and the name of a column, and don't have to think about which is which.
2) But if I use `assign`, I can only refer to column names, not index level names
3) I can use merge keys that are a mix of column names and index names, but if I don't use the full index, other index levels get dropped
4) I can rename selected column names using `DataFrame.rename()` with a `dict` as an argument, but I can't do that for index names.  (issue #20421)
5) With `groupby`, I can specify a mix of column names and index level names, but not with `pd.Grouper` (issue #19542)

There are other examples as well if I did a careful inspection of the APIs.  IMHO, if you have an operation on a `DataFrame` and you refer to column names, it shouldn't matter whether the name is in the index or in the columns.  `query` and `groupby` are nice that way, but other operators are not.
@Dr-Irv I might ask you to write that up as an issue / design document and submit it as a proposed addition to the roadmap once this is in.
> Do we also want a section that tries to describe the goals and/or scope of pandas? (not necessarily in this PR, but I think it fits naturally in a roadmap (discussion))
Eg numpy did this when adding their roadmap (https://numpy.org/neps/scope.html), 

My $2c: this is very valuable to have, I'm referring to the NumPy one quite regularly. You can do better than the current NumPy one though. That scope doc starts with "this is not an aspirational scope/vision ....". Being more forward-looking helps, for attracting volunteers/funding/....
Thanks for the updates!

Regarding the "Roadmap evolution" you added, one concern I have is about using github issues for this, as it is very easy to miss an issue in the large flood of github notifications.
I agree that a "scope" document would be helpful. But in thinking it over for about 2 minutes I'll request that we defer as I expect the conversation to be even more contentious than this one. https://github.com/pandas-dev/pandas/issues/27649

> Regarding the "Roadmap evolution" you added, one concern I have is about using github issues for this, as it is very easy to miss an issue in the large flood of github notifications.

Indeed. I'm not sure what's best here. I don't want to implement too much overhead, especially as we're still learning what process works best for pandas. As a compromise, perhaps we require that the mailing list also be notified?
> @Dr-Irv I might ask you to write that up as an issue / design document and submit it as a proposed addition to the roadmap once this is in.

@TomAugspurger See what I put together in #27652
Thanks @Dr-Irv, will take a look today.

---

Any other comments here?
Added a note to the top about which items go in the roadmap, and that something not being on the roadmap doesn't mean we won't consider it (to avoid making non-roadmap work feel like it's "second class" to roadmap work).
De-Coupling the Code Base
----------------------------
We want to make it easier to reason about, understand, and test parts of the code in isolation.  There is an ongoing effort to reduce the circular dependencies among the code-base:

- pandas._config is "upstream" of all other pandas modules.
- pandas._libs.tslibs is _almost_ independent of the rest of pandas.
  - The exception it relies on pandas.tseries.offsets and parts of pandas.tseries.frequencies.
  - tslibs can be made entirely self-contained once tseries.offsets classes are made into cython classes and the relevant parts of tseries.frequencies are moved into tslibs. See ????.
- pandas._libs, pandas.compat, pandas.errors, and pandas.util._* are _almost_ independent of the rest of pandas.
- pandas.core.dtypes is _mostly_ independent of the rest of pandas.core, with some lingering dependencies on pandas.core.arrays.
- Several modules in pandas.core are _mostly_ independent of the ExtensionArray/Index/Series/DataFrame machinery: accessor, algorithms, base, common, consturction, indexers, missing, nanops, sorting.
- pandas.core.arrays has dependencies on pandas.core.dtypes and the modules in the previous bulletpoint, but _mostly_ do not depend on Index/Series/DataFrame machinery.

Refactors that finish the decoupling of these modules will improve the maintainability of pandas.

Some more troublesome areas:
- core.internals and DataFrame._data should be exposed in very few places.  io.pytables, io.msgpack, core.groupby, core.reshape, core.indexing all access internals directly.  We should try to find alternative implementations for these use cases.
- pandas.core.reshape.concat is imported at runtime in many places.
@jbrockmendel would you mind writing that up as a separate issue, so we can try out the roadmap evolution process once this is merged? 
Any objections to merging this as is?
@TomAugspurger you said you'd take a look at my ideas in #27652 and see if they are worth including.  Thanks.
Right, I'm still mulling that over. Once that discussion gets to the point
where we think it's ready we can go through the evolution process to add it
to this document.

On Thu, Aug 1, 2019 at 8:40 AM Irv Lustig <notifications@github.com> wrote:

> @TomAugspurger <https://github.com/TomAugspurger> you said you'd take a
> look at my ideas in #27652
> <https://github.com/pandas-dev/pandas/issues/27652> and see if they are
> worth including. Thanks.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/27478?email_source=notifications&email_token=AAKAOIUNUYIBM3XLKE2N4MLQCLRWBA5CNFSM4IFG47G2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3KUGOA#issuecomment-517292856>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AAKAOIVONKMH333P7AQHPZDQCLRWBANCNFSM4IFG47GQ>
> .
>

@Dr-Irv and thanks for opening that issues! (it's also on my to do list to look at it) But it is not a blocker of merging this PR, this PR is only meant to kickstart a roadmap document
ok by me. I don't think we need a more formal process than a PR, though IIUC you are suggesting a *higher* bar for major changes here?
@TomAugspurger @jorisvandenbossche Thanks for replying. No reason to stand in front of the merge

> I don't think we need a more formal process than a PR, though IIUC you are suggesting a higher bar for major changes here?

Do you mean for adding to the roadmap, or for inclusion to pandas?

For adding to pandas, no. That'll be the same process (just a PR).

For adding to the roadmap, I suppose it's a higher bar in the sense that there was no bar before :)
> > I don't think we need a more formal process than a PR, though IIUC you are suggesting a higher bar for major changes here?
> 
> Do you mean for adding to the roadmap, or for inclusion to pandas?
> 
> For adding to pandas, no. That'll be the same process (just a PR).
> 
> For adding to the roadmap, I suppose it's a higher bar in the sense that there was no bar before :)

right, once we merge, then we should have a 'policy' of what the bar is to include in the roadmap. ok for now just as consensus.