Repository: pyjanitor-devs/pyjanitor

Issue:
Latest requirements.txt seems to be over-specifying dependencies
This is the latest version requirements.txt file:

```
black>=19.3b0
darglint
hypothesis>=4.4.0
interrogate
ipykernel
isort>=4.3.18
jupyter_client
lxml
natsort
nbsphinx>=0.4.2
pandas-flavor
pandas-vet
pre-commit
pyspark
pytest-azurepipelines
pytest-cov
pytest>=3.4.2
scikit-learn
seaborn
setuptools>=38.5.2
sphinxcontrib-fulltoc==1.2.0
unyt
xarray
```

Those are inject in the setup.py as mandatory dependencies but a lot in there looks like docs, dev, or optional. Is pyjanitor really require all that at run time?


Discussion:
@ocefpaf thanks for chiming in! Yes, I needed them in `requirements.txt` to ensure that Netlify deploy docs could work properly.

It seems like this breaks the conda-forge builds, is that right? Do you have ideas on how we might be able to get around this conflict?
Doesn't really break b/c we can add them there too but it is odd to ship all that to use pyjanitor.

One solution would be stop using the requirements.txt file and specify the deps in the pyproject.toml, going full pep517/518. (I can send a PR if you want to go that route.)

PS: I don't know how netfly works but the current setup defeats the purpose of the optional deps in setup.py and the requeriments-dev.txt. The latter is a duplicate of requeriments.txt at the moment.
> PS: I don't know how netfly works but the current setup defeats the purpose of the optional deps in setup.py and the requeriments-dev.txt. The latter is a duplicate of requeriments.txt at the moment.

Totally agree. Netlify is a bit too opinionated, but the ability to preview docs changes was too alluring for me.

> One solution would be stop using the requirements.txt file and specify the deps in the pyproject.toml, going full pep517/518. (I can send a PR if you want to go that route.)

Looks like `pyproject.toml` is the right way to go here! I would love to get your help setting this up, as I've not tried this before, so seeing the pattern at work IRL will be super educational for me.
I can also suggestion using [pip-tools](https://github.com/jazzband/pip-tools). With this set up, you would have a `requirements.in` file that specified you "base" requirements, which then is used to generate a `requirements.txt` file that you can use for CI or Netlify. It's not as flashy as pyproject.toml / poetry, but it's worked well for me for several years now.

Recent versions of pip-tools would allow you to use your `setup.py` file in place of a separate `requirements.in` file as well.
> I can also suggestion using [pip-tools](https://github.com/jazzband/pip-tools). With this set up, you would have a `requirements.in` file that specified you "base" requirements, which then is used to generate a `requirements.txt` file that you can use for CI or Netlify. It's not as flashy as pyproject.toml / poetry, but it's worked well for me for several years now.

I don't think that pyproject.toml is flashy, in fact I'm not really a fan, but it is the community standard.

> Recent versions of pip-tools would allow you to use your `setup.py` file in place of a separate `requirements.in` file as well.

I hear great things about pip-tools but I never used it. I don't see how that setup can solve the problem here though. Setup.py would still be 'bloated'' with all the dev and docs specifications to create the requirements.txt for netfly, no? Unless we add them all in separate categories and pip-tools aggregate them in a single file. (Which can also be done with pyproject.toml.)

I'll wait for your expertise on pip-tools here before sending a PR so we can choose a more stable route that would avoid another change in the short term.

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -1,15 +1,82 @@
 """Setup script."""
-
+import codecs
+import os
 import re
 from pathlib import Path
+from pprint import pprint
 
 from setuptools import find_packages, setup
 
+HERE = os.path.abspath(os.path.dirname(__file__))
+
+
+def read(*parts):
+    # intentionally *not* adding an encoding option to open
+    return codecs.open(os.path.join(HERE, *parts), "r").read()
+
 
-def requirements():
-    """Reader for requirements.txt"""
-    with open("requirements.txt", "r+") as f:
-        return f.read()
+def read_requirements(*parts):
+    """
+    Return requirements from parts.
+
+    Given a requirements.txt (or similar style file),
+    returns a list of requirements.
+    Assumes anything after a single '#' on a line is a comment, and ignores
+    empty lines.
+
+    :param parts: list of filenames which contain the installation "parts",
+        i.e. submodule-specific installation requirements
+    :returns: A compiled list of requirements.
+    """
+    requirements = []
+    for line in read(*parts).splitlines():
+        new_line = re.sub(  # noqa: PD005
+            r"(\s*)?#.*$",  # the space immediately before the
+            # hash mark, the hash mark, and
+            # anything that follows it
+            "",  # replace with a blank string
+            line,
+        )
+        new_line = re.sub(  # noqa: PD005
+            r"-r.*$",  # link to another requirement file
+            "",  # replace with a blank string
+            new_line,
+        )
+        new_line = re.sub(  # noqa: PD005
+            r"-e \..*$",  # link to editable install
+            "",  # replace with a blank string
+            new_line,
+        )
+        # print(line, "-->", new_line)
+        if new_line:  # i.e. we have a non-zero-length string
+            requirements.append(new_line)
+    return requirements
+
+
+# pull from requirements.IN, requirements.TXT is generated from this
+INSTALL_REQUIRES = read_requirements(".requirements/base.in")
+
+EXTRA_REQUIRES = {
+    "dev": read_requirements(".requirements/dev.in"),
+    "docs": read_requirements(".requirements/docs.in"),
+    "test": read_requirements(".requirements/testing.in"),
+    "biology": read_requirements(".requirements/biology.in"),
+    "chemistry": read_requirements(".requirements/chemistry.in"),
+    "engineering": read_requirements(".requirements/engineering.in"),
+    "spark": read_requirements(".requirements/spark.in"),
+}
+
+# add 'all' key to EXTRA_REQUIRES
+all_requires = []
+for k, v in EXTRA_REQUIRES.items():
+    all_requires.extend(v)
+EXTRA_REQUIRES["all"] = set(all_requires)
+
+for k1 in ["biology", "chemistry", "engineering", "spark"]:
+    for v2 in EXTRA_REQUIRES[k1]:
+        EXTRA_REQUIRES["docs"].append(v2)
+
+pprint(EXTRA_REQUIRES)
 
 
 def generate_long_description() -> str:
@@ -42,12 +109,6 @@ def generate_long_description() -> str:
     return long_description
 
 
-extra_spark = ["pyspark"]
-extra_biology = ["biopython"]
-extra_chemistry = ["rdkit"]
-extra_engineering = ["unyt"]
-extra_all = extra_biology + extra_engineering + extra_spark
-
 setup(
     name="pyjanitor",
     version="0.20.10",
@@ -58,15 +119,8 @@ def generate_long_description() -> str:
     license="MIT",
     # packages=["janitor", "janitor.xarray", "janitor.spark"],
     packages=find_packages(),
-    install_requires=requirements(),
-    extras_require={
-        "all": extra_all,
-        "biology": extra_biology,
-        # "chemistry": extra_chemistry, should be inserted once rdkit
-        # fixes https://github.com/rdkit/rdkit/issues/1812
-        "engineering": extra_engineering,
-        "spark": extra_spark,
-    },
+    install_requires=INSTALL_REQUIRES,
+    extras_require=EXTRA_REQUIRES,
     python_requires=">=3.6",
     long_description=generate_long_description(),
     long_description_content_type="text/x-rst",


Test patch:
diff --git a/.requirements/testing.in b/.requirements/testing.in
new file mode 100644
--- /dev/null
+++ b/.requirements/testing.in
@@ -0,0 +1,6 @@
+pytest-azurepipelines
+pytest-cov
+pytest>=3.4.2
+hypothesis>=4.4.0
+interrogate
+pandas-vet
diff --git a/.requirements/testing.txt b/.requirements/testing.txt
new file mode 100644
--- /dev/null
+++ b/.requirements/testing.txt
@@ -0,0 +1,29 @@
+#
+# This file is autogenerated by pip-compile
+# To update, run:
+#
+#    pip-compile ./.requirements/testing.in
+#
+attrs==20.2.0             # via hypothesis, interrogate, pandas-vet, pytest
+click==7.1.2              # via interrogate
+colorama==0.4.4           # via interrogate
+coverage==5.3             # via pytest-cov
+flake8==3.8.4             # via pandas-vet
+hypothesis==5.37.3        # via -r ./.requirements/testing.in
+iniconfig==1.1.1          # via pytest
+interrogate==1.3.1        # via -r ./.requirements/testing.in
+mccabe==0.6.1             # via flake8
+packaging==20.4           # via pytest
+pandas-vet==0.2.2         # via -r ./.requirements/testing.in
+pluggy==0.13.1            # via pytest
+py==1.9.0                 # via interrogate, pytest
+pycodestyle==2.6.0        # via flake8
+pyflakes==2.2.0           # via flake8
+pyparsing==2.4.7          # via packaging
+pytest-azurepipelines==0.8.0  # via -r ./.requirements/testing.in
+pytest-cov==2.10.1        # via -r ./.requirements/testing.in
+pytest==6.1.1             # via -r ./.requirements/testing.in, pytest-azurepipelines, pytest-cov
+six==1.15.0               # via packaging
+sortedcontainers==2.2.2   # via hypothesis
+tabulate==0.8.7           # via interrogate
+toml==0.10.1              # via interrogate, pytest