PDEP-10: Add pyarrow as a required dependency
State: closed

- [x] closes #52509 (Replace xxxx with the GitHub issue number)

I've tried to summarize the motivations and drawbacks from https://github.com/pandas-dev/pandas/issues/52509 in this PDEP. Please let me know if there are any reasons on either side that I am missing.

Feel free to continue the discussion in https://github.com/pandas-dev/pandas/issues/52509, but as a reminder the voting will take place here. 

cc @pandas-dev/pandas-core 

Labels:
name: Dependencies
description: Required and optional dependencies
name: Arrow
description: pyarrow functionality
name: PDEP
description: pandas enhancement proposal

Comments:
Will there also be a version cap?  NumPy is extremely conservative with breaking changes. I can't recall a case where a cap would have helped avoid issues with NumPy changes, especially deprecations and removals.  Is pyarrow similarly stable?  Does pyarrow have an implicit or explicit deprecation policy? If not, would there need to be a cap on each release too?

Recently in a number of projects I've been downstream of SciPy which has been doing a lot of long-needed but painful clean up.  This has resulted in cases where not too old releases of downstream projects are breaking against the most recently released SciPy. 
In one of the linked issues there is the comment about pandas being backend agnostic. Is it possible that this PDEP can broach that and consider how making pyarrow a dependency defines that objective as being unacheivable or how it would fit in to such a concept. This is not clear to me, or whether that is indeed a goal.

Primarily I have used numpy for linear algebra and pandas as an extension for better indexing, data wrangling and output. the added bloat when incorporating into web apps with limited build space is concerning.
This isn't a really useful discussion to have right now (and IMO shouldn't be in scope here). Even if we get to a point where NumPy is optional (this is years away), NumPy is still a required dependency of PyArrow
Is it a good comparison to say that trying to develop arrow-backed arrays without pyarrow as a dependency is like trying to develop NumPy-backed arrays/blocks without NumPy as a dependency?
> Is it a good comparison to say that trying to develop arrow-backed arrays without pyarrow as a dependency is like trying to develop NumPy-backed arrays/blocks without NumPy as a dependency?

For development cases where pyarrow functionality is still optional, yes that's a good comparison (ignoring that the internals are still numpy based), as development is made a bit more inconvenient always checking if pyarrow is available.
According to BigQuery about 1.7% of our current PyPi downloads are source distributions (at least over past 30 days)

```sql
SELECT file.type, COUNT(*) AS num_downloads
FROM `bigquery-public-data.pypi.file_downloads`
WHERE file.project = 'pandas'
  AND DATE(timestamp)
    BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
    AND CURRENT_DATE()
GROUP BY 1
```

```json
[{
  "type": "sdist",
  "num_downloads": "1937866"
}, {
  "type": "bdist_wininst",
  "num_downloads": "4292"
}, {
  "type": "bdist_wheel",
  "num_downloads": "110852691"
}]
```
I'm not super-keen on this idea, but...

I want a not-too-distant future where:

```
pd.Series(["a", "b", "c"])
pd.Series([b"1", b"2", b"3"])
pd.Series([date(2016, 1, 2), date(2016, 1, 3)])
pd.Series([time(2, 3), time(3, 4)])
pd.Series([Decimal(1), Decimal(2)])

# [...] copy/paste the same sequences as object-dtype ndarrays

some_series.astype(str)
some_series.astype(bytes)
```

all come back with pyarrow dtypes.   The string one being by far the most common/important case.  What if don't "require" pyarrow, but do raise in these cases with a message telling users to install pyarrow?

> Looks good - just got one request: could we add a timeline section please? As in, do we want to make it required in 2.x, or in 3.0? Do we want to make it required right away, or wait for pyarrow to shrink the size of its package?

The proposal is to make it a dependency immediately, the first bullet point in the abstract says `PyArrow becomes a runtime dependency starting with pandas 2.1`. Am I missing something, or shouldn't this be enough?
sorry, you're right, I'd glossed over the abstract and couldn't find it in the main body. this is fine, minimising then
From https://github.com/pandas-dev/pandas/pull/52711#issuecomment-1518717765 (see code snippets in that comment)

> all come back with pyarrow dtypes. The string one being by far the most common/important case. What if don't "require" pyarrow, but do raise in these cases with a message telling users to install pyarrow?

I think that would be entirely reasonable for new data types. We can certainly start inferring those scalars (we actually already do for all of decimal, bytes, date, time in `infer_dtype`), and (optionally) use that information to construct with a different dtype. For example for supporting to infer bytes in `pd.array(..)`, with this small patch:

<details>

```diff
diff --git a/pandas/core/construction.py b/pandas/core/construction.py
index d626afa0c6..c28c884be5 100644
--- a/pandas/core/construction.py
+++ b/pandas/core/construction.py
@@ -364,6 +364,14 @@ def array(
         elif inferred_dtype == "boolean":
             return BooleanArray._from_sequence(data, copy=copy)
 
+        elif inferred_dtype == "bytes":
+            if not lib.is_pyarrow_installed():
+                # raise error, fallback to object dtype, ... (whatever we decide)
+                pass
+            else:
+                from pandas.core.arrays.arrow import ArrowExtensionArray
+                return ArrowExtensionArray._from_sequence(data, copy=copy)
+
     # Pandas overrides NumPy for
```

</details>

I get

```
In [1]: pd.array([b"a", b"b"])
Out[1]: 
<ArrowExtensionArray>
[b'a', b'b']
Length: 2, dtype: binary[pyarrow]
```

(for `Series(..)` it's a bit more complicated because inference in Series relies on `maybe_convert_objects` instead of `infer_dtype`, so that would need some more code updates. But ideally we would also converge those two paths)

But I think we will initially want to do this optionally, which doesn't require a hard dependency on pyarrow to implement this (the simplified patch above can easily be done without pyarrow being always available, we just need to decide what behaviour we want in such a case). 
> But ideally we would also converge those two [infer_dtype/maybe_convert_objects] paths

+1!

> raise error, fallback to object dtype, ... (whatever we decide)

A big part of the motivation for me is having _fewer_ code paths to maintain.  I really really don't want to give users different behavior based on the presence or lack of pyarrow.  Raising in some of these no-pyarrow cases is a compromise on that front.
> [@Dr-Irv] I'd really like to understand the burden on development if we left things as they are in terms of it being an optional dependency. How much simpler does the code base become if PyArrow is required?

I would also like some more argumentation about this from the proponents of this PDEP. Personally, I think the simplification to the development is quite limited. 
Some examples that were given before were:

* That it is awkward to check for arrow inputs. I think it is quite easy to add a helper for this, as shown in https://github.com/pandas-dev/pandas/pull/52830
* That it is difficult to properly identify arrow scalars in all kinds of ops (e.g. indexing ops). I did an attempt to fix one of the mentioned bugs in https://github.com/pandas-dev/pandas/pull/52833, and I don't think that having pyarrow as an optional dependency makes this hard to fix (the patch is somewhat complex, but that doesn't come from the conditional pyarrow import, but just because we need to add code to recognize/handle pyarrow scalars is various places that are often already complex right now, but that's something we need to do anyway if we want to properly handle pyarrow scalars)

Thanks everyone for weighing in.

To be honest, I am feeling a bit uneasy about this. The point about making installation from source trickier is valid. I'm also thinking about what this would signal, in terms of the pyarrow backend being "ready", which as far as I understand it isn't quite

To improve support for this, I'll join the call to expand on the benefits of `pyarrow` being a runtime depenency. The proposal says

> Currently, there are 17 import_optional_dependency("pyarrow") checks throughout the pandas code base

but doesn't say why that's a bad thing. And regarding

> Avoiding NumPy object data types more by default for analogous types that have native PyArrow support such as string, decimal, binary, and nested types

,  the suggestion to default to pyarrow but error if it's not installed sounds fine to me

---

I'm not opposed to this, just a bit on-the-fence
> , the suggestion to default to pyarrow but error if it's not installed sounds fine to me

Can I get a very-preliminary show of hands on this?  For the time being this refers only to the dtypes mentioned in https://github.com/pandas-dev/pandas/pull/52711#issuecomment-1518717765.  If there's interest I'll move forward with a proof of concept.
> Can I get a very-preliminary show of hands on this? For the time being this refers only to the dtypes mentioned in [#52711 (comment)](https://github.com/pandas-dev/pandas/pull/52711#issuecomment-1518717765). If there's interest I'll move forward with a proof of concept.

Are you asking if pyarrow should be required to use strings (and other less important types)?  I think making it required for strings will de facto make it a hard requirement given how common and essential strings are.  If this is the consensus, then I think it makes more sense and not to special case these datatypes and just make it a hard req.
> > , the suggestion to default to pyarrow but error if it's not installed sounds fine to me
> 
> Can I get a very-preliminary show of hands on this? For the time being this refers only to the dtypes mentioned in [#52711 (comment)](https://github.com/pandas-dev/pandas/pull/52711#issuecomment-1518717765). If there's interest I'll move forward with a proof of concept.

I think I need more clarity here if we go in this direction, which I would understand would be to make `pyarrow` optional except if you use certain dtypes.

And if that's the case, if you read in a CSV file, would it automatically raise if the file had strings or dates?

Maybe you would need to have an option to indicate whether to raise or not.  i.e., there has to be a way to use existing code without it causing an error.  Or if `pyarrow` is not installed, you just issue a warning (not an error!), but maintain current behavior.




The "compromise" was that we would _raise_ (after a deprecation cycle) in the relevant cases.  I am very very against having different non-raising behavior based on the presence or absence of pyarrow.  That is the worst option vis-a-vis maintenance burden.
> The "compromise" was that we would _raise_ (after a deprecation cycle) in the relevant cases. I am very very against having different non-raising behavior based on the presence or absence of pyarrow. That is the worst option vis-a-vis maintenance burden.

Strong +1

I’ll add more context ti the examples later
> To be honest, I am feeling a bit uneasy about this.

Maybe we should split the conversation in two different parts.

Correct me if I'm wrong, but I think there is agreement on moving torwards Arrow, in particular for what NumPy is not good at (strings, missing values...). For example NumPy/PyObject supported strings is something that exists because there wasn't a better alternative when pandas was first implemented. And our custom integer with missing values types or categorical type were only implemented because we didn't have Arrow implementing those for us when pandas implemented them. In that sense, we'd eventually (probably far in the future for backward compatibility) have pandas use Arrow structures by default, and will have to depend on an Arrow implementation.

If we all agree on the above, and that PyArrow packaging is far from optimal, we've got 3 main options:
1. Keep the status quo indefinitely, having PyArrow as an optional dependency
2. Make PyArrow a required dependency
3. Migrate from PyArrow to another implementation of Arrow (Rust seems to be the most appropriate)

I think there are two different groups of users that will be affected by the decision we make, besides how it affects pandas development:
a. The ones that can easily install PyArrow and use Arrow structures. No big difference for them, the main one is that now they can save some megabytes of disk space if they don't want to use Arrow. I don't think this is a main concern to everyone if I read the above comments correctly
b. Users who that don't have PyArrow conda packages or wheels available for their system, but they do have NumPy packages. They have the option to spend a significant amount of time and energy installing PyArrow manually. But if they don't, the difference is:
  - Option 1: They will have a limited version of pandas, pandas without Arrow
  - Option 2: They won't be able to use pandas

From the comments about, the users in group b are currently the WASM users. Alpine doesn't have NumPy wheels for what Thomas said, and pandas is broken in PyPy anyway for what Joris said. pandas support for WASM is less than a year old, and not really useful IMHO. We couldn't manage to make it working decently in our own website. And WASM limitations on CORS prevent it from loading data from urls, which means that only data that can be used is the one added to the WASM binary, or potentially one that the users upload via JavaScript, which I haven't seen anyone doing it yet.

So, unless there are actual users being affected by this (and nobody came to this thread to opposed, as it happend for example with #35407), I'm surely +1 on requiring PyArrow. If there are few exceptional cases, I'm still +1 on this, as it makes our life easier to develop pandas. If there are too many users affected by this, I think we should consider using a better Arrow implementation that users can install.
> we've got 3 main options:
> 
> 1. Keep the status quo indefinitely, having PyArrow as an optional dependency
> 2. Make PyArrow a required dependency
> 3. Migrate from PyArrow to another implementation of Arrow (Rust seems to be the most appropriate)

The PDEP is proposing (2).  "- PyArrow becomes a runtime dependency starting with pandas 2.1"  

In light of this comment from @jbrockmendel 
> "The "compromise" was that we would raise (after a deprecation cycle) in the relevant cases.

I think the PDEP should indicate the transition path (deprecation cycle).  Something along the lines of:
1. We issue warnings if pyarrow is not installed for certain dtypes (and maybe we need the list in the PDEP??).  
2. If pyarrow is installed, then for those dtypes, we use don't use pyarrow unless it is asked for (e.g. `dtype_backend`)
3. Then when pyarrow is required (probably not 2.1 - maybe it is 3.0 -  as proposed in the PDEP), we always use pyarrow for those dtypes

I may have the steps wrong above, but I think the transition path needs to be outlined in the PDEP.


>  I think making it required for strings will de facto make it a hard requirement given how common and essential strings are

I think it would be OK to:
- by default, something like `pd.Series(['Italy', 'Croatia'])` would use pyarrow string dtype (and raise if pyarrow isn't installed)
- if someone can't install pyarrow, then they can specify `dype=object` and keep using pandas without pyarrow as before.

Having said that:

>  If there are few exceptional cases, I'm still +1 on this

Agreed - if something benefited 99%+ of users, and held back 1%, then I think it'd be unfair to the 99% to not do it.
[Will's comment](https://github.com/pandas-dev/pandas/pull/52711#issuecomment-1518709540) suggests it may be more than just a few exceptional cases, but still, not a huge number

I'll post this on the mailing list if that's OK
> b. Users who that don't have PyArrow conda packages or wheels available for their system, but they do have NumPy packages. They have the option to spend a significant amount of time and energy installing PyArrow manually. But if they don't, the difference is:
> 
>     * Option 1: They will have a limited version of pandas, pandas without Arrow
> 
>     * Option 2: They won't be able to use pandas

Option 1 has been our status quo since the inception of pandas. I'm less concerned about that than option 2

> From the comments about, the users in group b are currently the WASM users. Alpine doesn't have NumPy wheels for what Thomas said, and pandas is broken in PyPy anyway for what Joris said. pandas support for WASM is less than a year old, and not really useful IMHO. We couldn't manage to make it working decently in our own website. And WASM limitations on CORS prevent it from loading data from urls, which means that only data that can be used is the one added to the WASM binary, or potentially one that the users upload via JavaScript, which I haven't seen anyone doing it yet.

Where has the WASM effort been happening? I would be interested to know more about what tripped that up. Again one of the motivating factors for moving our build system away from setuptools was support for cross-compilation, so I think there is conflict in the goals of PDEP-10 and PDEP-0002. Also curious why you think it isn't useful?


> pandas is broken in PyPy

This is only when trying to use Timedelta in PyPy right? I don't think PyPy support is critical part of our developer roadmap, but at the same time don't want to overstate how broken it is
> Where has the WASM effort been happening? I would be interested to know more about what tripped that up. Again one of the motivating factors for moving our build system away from setuptools was support for cross-compilation, so I think there is conflict in the goals of PDEP-10 and PDEP-0002. Also curious why you think it isn't useful?

I don't know much about the effort to make pandas run in WASM. The effort to have jupyterlite in our website with a working pandas environment happened few months ago, but the binary was big and very slow to download, then it was slow to set up the environment, and in some cases it didn't even finish loading.

The reason why I think pandas in WASM is not useful is for what I mentioned of the CORS limitations. You want to use pandas with data, and your data won't be in the WASM filesystem, so you probably want to load it from a url, but that's also not an option (unless the data is in the same domain, which wasn't working either from our I/O, since pandas uses a library for http which is not the one you need to use in WASM).

I may be missing something, and I guess it can eventually be useful to have pandas in WASI for something relevant. But pandas in WASM seems to be a toy and mostly irrelevant for what I know. Happy to hear more and change my mind if someone knows of significant use cases, but I don't know of any other than a terminal to try pandas, and we couldn't make that work well enough.
Just for reference, I use pandas in a pyscript project, so it does work, but the QoL isn't great (since pyodide takes forever to load).

https://lithomas1.github.io/platypi/packageinfo.html is the link in case you're interested.
(It is a PyPi stats tracking page, that shows wheel vs sdist downloads, platforms wheels were downloaded on, etc.. The UI is still kinda bad since I haven't invested a lot of time into it since February). 

There are a lot more interesting pyscript examples here https://pyscript.net/examples/. 
> https://lithomas1.github.io/platypi/packageinfo.html is the link in case you're interested.

Thanks for sharing @lithomas1, this is very interesting. Feels like the PyPI webservice allows CORS requests, right? I see another demo using data from a Mapbox API that seems to do the same. Do you have more information about this, and whether it's a common practice?

For me personally it still feels too niche and unusual use case to be worth stopping this PDEP. But it'd be good to understand better what's the potential for pandas in the browser.
I've added more context to the motivation section.

This discussion mostly focuses on a very small subset of our users. They would have to put more effort into installing pandas. The vast majority of our users does not have the "from source installation issues" but would benefit immensely from PyArrow dtypes in certain situations. Especially strings are a huge improvement compared to NumPy object.

A couple of observations from interacting with users:

- PyArrow strings are hugely popular, users are generally enthusiastic when talking about Arrow support.
- proper Decimal support came up multiple times during our AMA session on reddit, this would be pretty helpful in finance
- Nested dtype support is something that I've heard people complain about regularly, although not as strong as the other 2 points

Fastparquet engine is another pain point, the implementation is buggy wrt dtypes and does not support our dtype backends. I'd very much like to get rid of it to reduce maintenance burden.


> > https://lithomas1.github.io/platypi/packageinfo.html is the link in case you're interested.
> 
> Thanks for sharing @lithomas1, this is very interesting. Feels like the PyPI webservice allows CORS requests, right? I see another demo using data from a Mapbox API that seems to do the same. Do you have more information about this, and whether it's a common practice?

Yeah, this calls to something I have running that allows CORS, but I'm pretty sure that others are probably either using services that allow CORS or use a proxy to get around it.

Pyodide/WASM stuff shouldn't be a blocker, but I definitely can see it being used in things like dashboards.

xposting from https://github.com/pandas-dev/pandas/issues/52509#issuecomment-1528793286


> We recently enabled jupyter lite on our examples on the scikit-learn side, and by default we're including pandas. Looking at the size of the wheels for pyarrow, they're somewhere between 20-35MB based on the platform. That's a significant increase for users who would be loading the example in their browser.
> 
> We also have been having trouble with the latest pandas when pip installed in a mamba environment, since conda-lock fails in this case, and that's because pandas=2.0 requires tzdata, and the env ends up with two tzdata. It's not a pandas bug per say, but it complicates workflows (we're sticking with an older pandas for now for that env). That is to say even a small dependency like tzdata which should be present in most environments anyway, complicates things.
> 
> Pandas is a library on which, other libraries depend, and any dependency added here adds to the dependency tree to those environments, and pyarrow is by far not a trivial dependency.
> 
> All of that is to say, from my perspective, it would be a pity if pyarrow becomes a mandatory dependency.

Sorry for xposting, I was fooled by this ;) 

> Feel free to continue the discussion in https://github.com/pandas-dev/pandas/issues/52509, but as a reminder the voting will take place here.
> a. The ones that can easily install PyArrow and use Arrow structures. No big difference for them, the main one is that now they can save some megabytes of disk space if they don't want to use Arrow. I don't think this is a main concern to everyone if I read the above comments correctly

If you run a microservice or a quantitatively backed web-app you might typically install to your environment, numpy, pandas, scipy and matplotlib (or maybe use a javascript based front end grapher), maybe you add sqlalchemy, flask/django etc. The environment size limit on Heroku for a server running such a service is 500mb so an additional 35mb(?) is not negligible. It becomes a constraint. 
I already avoid installing scipy now to save space for the few algorithms I need from it. I end up coding them internally if its just a few. An option to only install parts of packages that you need is helpful in these circumstances.

I would happily extend this to say try to reduce some of the other pandas dependencies to shrink its size if they are of reasonable size, although this is out of scope.