BUILD: Simplifying contributor dependencies
State: closed

Unifying dev and optional dependencies, moving them to their standard locations, and create script to generate the pip file, and make sure in the CI they are always sync

- [X] refs #23115
- [ ] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry


Labels:
name: Build
description: Library building on various platforms
name: CI
description: Continuous Integration
name: Dependencies
description: Required and optional dependencies

Comments:
Hello @datapythonista! Thanks for submitting the PR.

 - There are no PEP8 issues in the file [`scripts/conda_to_pip.py`](https://github.com/datapythonista/pandas/blob/1de24029c87470dbf7954af11348b5f79ab68df7/scripts/conda_to_pip.py) !


@jreback having just a `environment.yml` was my first option too. But @TomAugspurger said he preferred to have also dependencies to install via `pip`. That's why I have the two files, `environment.yml` for conda users, and `requirements.txt` for pip users (and the script `scripts/conda_to_pip.py` to generate the pip file from the conda file, and validate in the CI that they have the same dependencies).

I'm happy to remove `requirements.txt` and the script if @TomAugspurger is ok with it.
My reasoning for the names of the files, is that I don't think anyone will install pandas from source unless it's for development. And when installing from pip or conda, the dependencies are obtained from `setup.cfg`.

Personally, I think it makes sense to have `environment_dev.yml` and `requirements_dev.yml` if we have `environment.yml` (or `environment_prod.yml`) for production, and same for requirements. But as I doubt those files won't be used, I think the current names are the simplest option.

I'd really like to keep `environment.yml` as this is the de-facto standard by conda, and I think the future is that in any project you will simply do `conda env create` to get the environment (and `conda env update` to update it). And I think it's very convenient when you start to be used to it, and users will sooner than later know about this standard. Knowing that this will install from `environment.yml` is to me like knowing that you have your vim options in `~/.vimrc`, something that you're expected to know once you start using the tool more.

But let me know if you're not convinced, and you still prefer to make the changes.
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23522?src=pr&el=h1) Report
> Merging [#23522](https://codecov.io/gh/pandas-dev/pandas/pull/23522?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/383d0525831d7301ae4ab26bb8c2d475ee4ab72a?src=pr&el=desc) will **decrease** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/23522/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/23522?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #23522      +/-   ##
==========================================
- Coverage   92.24%   92.23%   -0.01%     
==========================================
  Files         161      161              
  Lines       51278    51288      +10     
==========================================
+ Hits        47299    47304       +5     
- Misses       3979     3984       +5
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.62% <ø> (-0.01%)` | :arrow_down: |
| #single | `42.26% <ø> (-0.02%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/23522?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/tseries/offsets.py](https://codecov.io/gh/pandas-dev/pandas/pull/23522/diff?src=pr&el=tree#diff-cGFuZGFzL3RzZXJpZXMvb2Zmc2V0cy5weQ==) | `97.07% <0%> (-0.15%)` | :arrow_down: |
| [pandas/core/dtypes/concat.py](https://codecov.io/gh/pandas-dev/pandas/pull/23522/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZHR5cGVzL2NvbmNhdC5weQ==) | `96.26% <0%> (-0.09%)` | :arrow_down: |
| [pandas/io/pytables.py](https://codecov.io/gh/pandas-dev/pandas/pull/23522/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3B5dGFibGVzLnB5) | `92.34% <0%> (-0.09%)` | :arrow_down: |
| [pandas/io/parsers.py](https://codecov.io/gh/pandas-dev/pandas/pull/23522/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnNlcnMucHk=) | `95.55% <0%> (-0.07%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/23522/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.03% <0%> (-0.01%)` | :arrow_down: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/23522/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `96.81% <0%> (ø)` | :arrow_up: |
| [pandas/core/window.py](https://codecov.io/gh/pandas-dev/pandas/pull/23522/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvd2luZG93LnB5) | `96.4% <0%> (ø)` | :arrow_up: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/23522/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `93.7% <0%> (ø)` | :arrow_up: |
| [pandas/core/indexes/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/23522/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9iYXNlLnB5) | `96.45% <0%> (ø)` | :arrow_up: |
| [pandas/core/arrays/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/23522/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvYXJyYXlzL2RhdGV0aW1lcy5weQ==) | `98.44% <0%> (+0.02%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23522?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23522?src=pr&el=footer). Last update [383d052...6f30a04](https://codecov.io/gh/pandas-dev/pandas/pull/23522?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

I'd like to keep the requirements.txt around, but am not going to block things if others decide to remove it :)
Made explicit the requirements file for conda in the docs.

I think pip is not the primary anywhere (the script creates the pip file from the conda file already), and in the docs is already clear that conda should be installed, and pip instructions are provided as an extra.

So, I think this should be all right now.
I think we should name ``requirements-dev.txt`` I believe ``requirements.txt`` is what you need to install your project; ours includes lots of other things, so shouldn't confuse it.
lgtm. ping on green.
The script is called in `code_checks.sh` to make sure that the pip dependencies are synchronized with the conda ones, and it fails with a message asking to run the script if it's not the case.

I think the script is named `requirements_dev.txt` and not `requirements-dev.txt` now, but made the change, doesn't make a big difference.

I think at some point would be nice to have all the scripts explained in the documentation. But I'd leave that for another PR, as I think some clean up is needed, and personally I think we should merge the content of `scripts` into `ci`, as for what I know, it's mostly arbitrary which scripts go to one, and which go to the other (I guess there was a reason originally, but not sure if that applies anymore).
thanks!