BLD: add optional dependencies as extras_require in setup.py
State: closed

EDIT: This was originally a limited scope PR to bring recommended dependencies into pandas extras_require, but discussion  evolved into 'solve packaging for all optional dependencies' rather than just solve for the `recommended` optional dependencies.  The logic behind this is the same as originally argued for `recommended` dependencies but I updated the text of the post to keep it aligned with the PR contents

---------

Closes #47335. Closes #39164 . Closes #48361 .

Hoping to make this part of the 1.5 milestone.

Optional dependencies should have package mgmt facilitated through pandas.  This will make mgmt of pandas in production docker environments a significantly simpler and allow pandas explicit control over the versions of optional packages it should operate with.

## Actions
Use setup.cfg options as [documented here](https://setuptools.pypa.io/en/latest/userguide/declarative_config.html?highlight=options.extras_require#configuring-setup-using-setup-cfg-files) to make  optional extras for a pandas install.

Once this is done, a user will be able to ensure version alignment by
```
# in requirements.txt or setup.py
pandas[recommended, s3] == 1.22.4  # install pd 1.22.4 and compatible numexpr, bottleneck, s3fs versions
```

The proposed solution is simple to do, simple to maintain, and already done by major libraries in similar situations ([moto](https://github.com/spulec/moto/blob/master/setup.py), [aiobotocore](https://github.com/aio-libs/aiobotocore/blob/master/setup.py#L19)).

Not having this solution wastes user time by forcing them to manually maintain versions of pandas dependencies that are not clearly linked to pandas, save in the brains of users and by manually referring to an install docs page.

## checklist
- [x] closes #47335 , closes #39164 
- [NA] [Tests added and passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#writing-tests) if fixing a bug or adding a new feature
- [x] All [code checks passed](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit).
- [NA] Added [type annotations](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#type-hints) to new arguments/methods/functions. (no new functions)
- [x] Added an entry in the latest `doc/source/whatsnew/v1.4.3.rst` file if fixing a bug or adding a new feature. (in `Other` section)


Labels:
name: Build
description: Library building on various platforms
name: Dependencies
description: Required and optional dependencies

Comments:
Test failure `Windows-MacOS / windows-latest actions-39.yaml (pull_request)` seemingly unrelated to modified code.  

This same test fails in PRs that are approved to be merged so unclear to me if action is required.

```
INTERNALERROR> AssertionError: ('pandas/tests/plotting/frame/test_frame_color.py::TestDataFrameColor::test_mpl2_color_cycle_str', <WorkerController gw0>)
INTERNALERROR> assert not 'pandas/tests/plotting/frame/test_frame_color.py::TestDataFrameColor::test_mpl2_color_cycle_str'
```
> This same test fails in PRs that are approved to be merged so unclear to me if action is required.

Unrelated, don't worry about it.
Updates made as per discussion in #39164
@mroeschke Ready for review.
> Does this include all the optional dependencies in `actions-38-minimum_versions.yaml`

EDIT: YES, following updated commits.
- Currently missing `compression` tools as a single option, but they are in `all`.  I can bundle them into an option (although I doubt users will want them simultaneously, I don't want to name three individual optional extras for each compression lib).
- Missing `odfpy`, because that is undocumented elsewhere in installs or other sources I looked at in #39164. Will add to the excel options and the install doc (alternative of making a new category for the non-excel csv-read lib seems inefficient) 
 - `pyreadstat` for spss.  I will add.
Test failure seems unrelated to the PR:

```
FAILED pandas/tests/scalar/timedelta/test_timedelta.py::TestTimedeltas::test_round_sanity[round]
FAILED pandas/tests/scalar/timedelta/test_timedelta.py::TestTimedeltas::test_round_sanity[floor]
FAILED pandas/tests/scalar/timedelta/test_timedelta.py::TestTimedeltas::test_round_sanity[ceil]
FAILED pandas/tests/scalar/timestamp/test_unary_ops.py::TestTimestampUnaryOps::test_round_sanity[round]
FAILED pandas/tests/scalar/timestamp/test_unary_ops.py::TestTimestampUnaryOps::test_round_sanity[ceil]
```
Merging in upstream changes from dev will hopefully fix?
> Merging in upstream changes from dev will hopefully fix?

No, the test itself has corner cases that sometimes get hit.
> [JMBurley](https://github.com/JMBurley) requested a review from [simonjayhawkins](https://github.com/simonjayhawkins) [28 minutes ago](https://github.com/pandas-dev/pandas/pull/47336#event-7029612344)

PR looks OK although would maybe would need to repeat the usage in doc/source/whatsnew/v1.5.0.rst in doc/source/getting_started/install.rst for future readers. Also keeping the minimum versions in sync is probably going to be a nightmare.

I've not been involved in the issue discussion so will leave to others to give approval to merge.
Thanks @simonjayhawkins!

Quick thoughts:
>  Also keeping the minimum versions in sync is probably going to be a nightmare.

It will take work, but an explicit bit of code that asserts what the minimum versions are will be better maintained than documentation that merely suggests versions in a manner disconnected from code (the latter we will always struggle to keep up-to-date.  For example, in writing this PR I found packages that were missing from install.rst).

I expect we will have users submitting PRs that keep the minimum versions in `extras_require` correct.   And globally across all pandas users it will save a LOT of time to be able to get versions programmatically rather than reading docs and manually writing versions of pandas sub-packages into requirements/pyproject.toml/setup file

@mroeschke passing tests and read for review.

Thanks for the iterations thus far.
> minimum versions in sync is probably going to be a nightmare

We have a precommit hook to ensure the versions are consistent between in `pandas/compat/_optional.py` and `actions-38-minimum_versions.yaml` are in sync. The script is in `scripts/validate_min_versions_in_sync.py`. It would be good to include the dependencies in `setup.cfg` as well 
> The script is in `scripts/validate_min_versions_in_sync.py`. It would be good to include the dependencies in `setup.cfg` as well

Good idea, although I'm considering it out of scope for this PR.

There will need to be a larger refactor to read the versions from setup.cfg (which would be potentially wasted effort if they are then moved to pyproject.toml).  And if you can do that version reading, the versions verification should be refactored to be managed directly from that reading of the ground truth file and not the hard-pasted VERSIONS dict in `_optional.py`
@mroeschke ready for rereview: comments addressed, tests passing (except the edge case timedelta in linux that jbrockmendel said was fine) 
@mroeschke is this good to merge ahead of the 1.5 milestone at the end of this week?
@mroeschke ready for re-review.
@mroeschke Ready for re-review.
@mroeschke All comments addressed, ready for re-review.
Sorry for the delay here @JMBurley. I will probably have more time to review this after the 1.5 release unless other core team members have time to weigh in. 
Thanks for the update @mroeschke .

My 2 cents are that this would be a big boon to get in the 1.5 release, as it is really helpful for any team working with multiple packages if pandas can define its own optional extras. If there are any problems with how I've structured this they will be best found by the myriad of open source users and can be patched via minor versions.

Obviously I know that there are a variety of competing efforts for inclusion in 1.5 (and only so much team time to go around), but this PR should be an easy win for inclusion.   We know this code works; decisions from here are a matter of taste.  The important taste feedback is that from general users, so we should get it sooner rather than later.
This pull request is stale because it has been open for thirty days with no activity. Please [update](https://pandas.pydata.org/pandas-docs/stable/development/contributing.html#updating-your-pull-request) and respond to this comment if you're still interested in working on this.
@mroeschke with pandas 1.5 out, is there now capacity to work on this?

Obviously I'll need to redo some the documentation now that the whatsnew isn't 1.5, and merge conflicts have been created by not moving this in.  

I will resolve both of those if there is confirmation that the work won't go stale again.
Hi @JMBurley so sorry for the delay here. Yes, I should have time later this week to review this PR. I think this is definitely a worthwhile change to include in 2.0
Main updates done.  Wait to see if tests pass.

Then moving comments into latest `doc/source/whatsnew/` .
@mroeschke Thanks for the comments.  ~~Will address today.~~ [updates pushed]
> Looks pretty good. Just noting a few followup tasks (separate PR) that would be good, if you're interested:
> 
> 1. A CI check that `pip install pandas[foo]` works and install the expected dependencies
> 2. A pre-commit check that ensure that the min version in setup.cfg are aligned with other areas we specify min version (would mean augmenting `scripts/validate_min_versions_in_sync.py`)

Added issues #48942 #48949 to track.  Agree they should be distinct PRs.  Not sure on my availability over next few weeks.
Pending tests passing, I think we are done.  First enhancement for pandas 1.6.0 in the bag!

Will mark for re-review when tests pass.
@mroeschke  Ready for review.  Tests pass with a cancellation/timeout on `Python Dev / actions-311-dev (macOS-latest) (pull_request)`.  
Thanks for comments.  All resolved
> Great thanks for your work and patience here, LGTM. Will leave open just in case others want to chime in

Awesome, glad to have this in.  Appreciate the reviews to keep this PR on track.  Optional dependencies are **really** helpful for container mgmt in production DS.

Note: there is now a conflict with main branch where `whatsnew/v1.6.0.rst` has been renamed to `whatsnew/v2.0.0.rst`.  Is pandas 2.0.0 now the next major release instead of pandas 1.6.0?

I'll let someone provide guidance before I alter documentation for the merge.
>  Is pandas 2.0.0 now the next major release instead of pandas 1.6.0?

Correct, 2.0 will be the next major release so a rebase is needed