DOC: Design drafts to assist with next-gen pandas internals discussion
State: closed

Sprinted on my accumulated design ideas. There's more to do, but I'm going to take a break for a bit. 

I published this as a sphinx site temporarily to https://wesm.github.io/pandas2-design/ so it's easier to read. This needs to be a team effort, so depending on where the discussions go perhaps we can build a set of documents that we can make PRs to and discuss specific matters either in GitHub issues or on the mailing list. Let me know what everybody thinks!


Labels:
name: Docs
name: API Design
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
## [Current coverage](https://codecov.io/gh/pydata/pandas/pull/13944?src=pr) is 85.30% (diff: 100%)

> Merging [#13944](https://codecov.io/gh/pydata/pandas/pull/13944?src=pr) into [master](https://codecov.io/gh/pydata/pandas/branch/master?src=pr) will not change coverage

``` diff
@@             master     #13944   diff @@
==========================================
  Files           139        139          
  Lines         50157      50157          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
  Hits          42785      42785          
  Misses         7372       7372          
  Partials          0          0          
```

> Powered by [Codecov](https://codecov.io?src=pr). Last update [b7abef4...c7819cf](https://codecov.io/gh/pydata/pandas/compare/b7abef4949fb1ba7fd1004feba4f47ace7004282...c7819cf806914c5fb9d375d01b19ab850ab405d8?src=pr)

On the topic of cleaning up API's, IMO there should be a section about the `read_csv` API.  This API is a giant monolith and supporting two implementations doesn't make much sense anymore given that their behaviours and support are inconsistent.  Defining a new, sleeker interface (most likely based in the C engine) would be nice to have.

One general point of observation so far: this coupling with `numpy` probably will need some more fleshing out, especially when it comes to defining data types in `pandas`.  Finding that good balance between just enough separation from `numpy` to be able to operate largely independently of it while at the same time providing nice inter-op capabilities between `pandas` objects and `numpy` objects is going to be tricky.

@gfyoung I would appreciate some help making very specific analysis (with code examples) of potential concerns around insulating users from NumPy-specific implementation details. For example: I do not believe it is pandas's responsibility to maximize its "substitutability" with ndarrays. It's a "nice to have", but the original "Series is an ndarray" design was a mistake in retrospect.

I agree that coming up with a plan to improve `read_csv` is a good idea — this is basically orthogonal to the data structure internals. The implementation of `read_csv` will obviously have to change (in a good way, e.g. "unavoidable consolidation" can be avoided in the new design).

Also, I would appreciate instead of using the term "data type" rather to use either "physical type" or "logical type".  In NumPy there is no separation of these concepts (except perhaps with `datetime[unit]`) and this is big reason why we are in this mess. See https://wesm.github.io/pandas2-design/internal-architecture.html#some-definitions

Thanks for putting this together!

A topic that is somewhat orthogonal to the internals, but one thing I've been thinking about in the context of pandas 2.0, is interaction with JIT compilers - mostly for UDFs.  I'm using `numba` as a frame of reference, but thinking generally.

I came to pandas from SAS - there is _plenty_ to dislike about it - but one thing I actually do miss is a feeling of safety, that if you can write it, it will basically be fast.  Of course with pandas/numpy you have to be much more defensive, and sometimes try really hard to vectorize - just as an example I remember, this [SO answer](http://stackoverflow.com/a/32524908/3657742) from @unutbu is really clever, but I would have never come up with it.

I think this need to be vectorized adds to the pandas api  size, and I also think adds significantly to the learning curve.  I'm sure like many others, I've taken a hard look at Julia - it wasn't there yet (and maybe it won't ever be?) but there is a big allure to that model.  As a user, I'd love to be able to just write this, and just have it be fast.

``` python
df.groupby('key').transform(lambda x: (x - x.mean()) / x.std())
```

This may be more of a stretch, but as a contributor who's not a C++ wizard, it would also be nice if some portion of pandas was implementable at a JIT-able level, ala @shoyer's [numbagg](https://github.com/shoyer/numbagg).  I get this may not possible - you probably couldn't even make `numba` a required dep.

So I was just curious if you had any thoughts / roadmap ideas for this topic?  A C/C++ API would open the door a lot of the way, but I think there may be value in tighter integration?

@chris-b1 several things on the "faster groupby" front:
- if the inner loops of group-wise operations are happening in compiled code with limited contact wtih the Python interpreter, then the microperformance of each operation will be much better
- hypothetically (with much effort) one could create a useful enough C API such that the groupby operators could accept a ctypes function pointer -- this is basically (IIUC) how numba is implementing custom ufuncs. obviously the low-level bits of numerical arrays are simpler than pandas objects but the same general approach would be valid. 

so we have tons of issues related to this, see [here](https://github.com/pydata/pandas/issues?q=is%3Aopen+is%3Aissue+label%3AInternals).

I will post a mini-list of issue that I am closing.

As for C vs C++, one factor that I find convenient about C++ is that you have the STL for standard data structures builtin. NumPy actually uses Python's C API when it needs a dict or set internally, for example, which is kind of insane.

Note that the STL containers (like `std::unordered_map` for hash tables) have pretty acceptable performance:

http://incise.org/hash-table-benchmarks.html

We should obviously do our own investigations, but we always have the option to pull in 3rd party libraries (we might decide to continue using klib or decommission depending on benchmarks).

I'm going to let discussions collect here for a bit, then incorporate feedback into the documents. 

It would be useful to start a separate document that is changes/improvements/refactorings that we would like to do in pandas 2.0 that do not involve changes to the internals (like, read_csv was brought up), how do you want to manage that? 

One thing that surfaced recently in #13395 that I would like consider for pandas 2.0 is to expose the contiguous one dimensional arrays that store the values of DataFrame colums as part of the public API. These would be similar to the existing `Categorical` type, but done in more systematic manner and returned by pandas objects when you call `Series.values` and `Series.unique()`.

This would be complementary to existing scalar types like `Period`/`Datetime`/`Timestamp`. If the existing scalar type has the name `pandas.Scalar`, I would call the array version `pandas.ScalarArray`, inheriting from `pandas.BaseArray`.

Having such an API would be useful for third party libraries that integrate with pandas in a deep way (such as seaborn, sklearn, statsmodels and xarray), because it doesn't always make sense to work with labeled values.

@shoyer I'm totally with you here. This isn't explicitly called out in these docs but that was kind of what I intended here:

https://wesm.github.io/pandas2-design/internal-architecture.html#pandas-array-types

So basically every logical type (whether Int32Array, StringArray, CategoricalArray, etc.) would have a corresponding public Python API. These would also have `__array__` for NumPy interoperability as able. 

I realigned the milestones with some 'approx' dates.

@wesm Thanks for the extensive explanations!

Two small things I wondered (have to read it further in more detail):
- There has been discussion about the the datetime resolution (and resulting span) which is limited. I suppose with the better logical dtype system, it should be possible to tackle this? But maybe it is worth mentioning it somewhere?
- Also regarding the dtype system, to what extent would it be possible to make defining custom dtypes externally work? (other libraries that provide a dtype that can be used in pandas dataframes)

> It would be useful to start a separate document that is changes/improvements/refactorings that we would like to do in pandas 2.0 that do not involve changes to the internals (like, read_csv was brought up), how do you want to manage that? 

Agreed. 
Would it be worth to create a separate repo where those documents can live, together with the ones of this PR? (a pandas-design, pandas-discussion, pandas-dev, ...) This way, we can also use the issues/PRs over there to discuss the several topics without adding to the already huge number of open issues in the pandas repo, keeping it a bit separated. 
But this has the disadvantage that when we start with actual code PRs, it will again be in the pandas repo.

Otherwise, I would just add documents describing those possible changes to this same `pandas-2.0` directory.

@jorisvandenbossche I'd be happy to create a pandas-design repo — this has the added benefit of not bloating the main pandas git history if we check in images or other assets. Otherwise we can keep working in this doc/pandas-2.0 directory and indicate which docs are "internals-related" and which are "stuff that depends on the internals" (like the CSV reader, indexing, etc.)

I'm going to start incorporating the feedback into this document. I'll write a follow up PR for the Copy-On-Write discussion. I'll indicate in the document pages that are Internals related and those that are separate from Internals and maybe we can make more PRs digging into some of the other refactoring / cleanup we want to do. 

Let's start discussing on the mailing list how we intend to proceed with two branches of pandas? I would suggest we should keep the pandas 2.0 branch "rebaseable" for as long as possible, but at some point there will be some necessary divergence. 

What do others think of putting those documents and PRs in a separate repo?

I'm +0 on a separate repo (pydata/pandas-design or something), if only to make the discussion issues more accessible to outsiders (vs. having to seek them out in the main pandas issue tracker)

I'll go ahead and do that. We can always move the docs back here if the process is not working well. 

I also would like more people to actively watch the design discussion. Having a dedicated repo will make the GitHub emails separate from the usual pandas firehose. 

I've addressed many of the comments in here in pydata/pandas-design#1 -- let's move the discussion there and see how it goes. I still need to fix up the github permissions so that all who have push rights here can push there