Repository: Lightning-AI/pytorch-lightning

Issue:
Make use of package extras in setup.py
## 🚀 Feature
<!-- A clear and concise description of the feature proposal -->

Right now, to install testing dependencies, you must follow the steps in [Contributing.md](https://github.com/PyTorchLightning/pytorch-lightning/blob/master/.github/CONTRIBUTING.md):

```
python -m pip install -r requirements/devel.txt
python -m pip install -r requirements/examples.txt
python -m pip install pre-commit
```

 If we make use of the `extras_require` kwarg of `setup()` in setup.py, we can install these extras like this:

```
pip install -e ".[testing]"
# or
pip install -e ".[dev]"
``` 

### Motivation

<!-- Please outline the motivation for the proposal. Is your feature request related to a problem? e.g., I'm always frustrated when [...]. If this is related to another GitHub issue, please link here too -->

I would rather run one command to setup my dev/testing environment than having to run multiple.

### Pitch

<!-- A clear and concise description of what you want to happen. -->

All we have to do is construct a dictionary in `setup.py` that collects arguments for different 'extra' options.

```python
extras = dict()
extras['dev'] = ['numpy>=1.16.4', 'torch>=1.3', 'tensorboard>=1.14', ...]
extras['test'] = extras['dev'] + ['codecov>=2.1', 'pytest>=3.0.5', ...]
...

setup(
    ...
    extras_require=extras,
)
```

### Alternatives

<!-- A clear and concise description of any alternative solutions or features you've considered, if any. -->

- We could get rid of `requirements/` folder entirely and let `setup.py` explicitly define the extra requirements
- We could probably populate the `extras` dict from the `requirements/` folder's files if we wanted to, but I think this makes things a touch more confusing (and I'm not sure if you'd have to include the requirements folder when you build the artifact to avoid setup.py from breaking on missing files).

### Additional context

<!-- Add any other context or screenshots about the feature request here. -->

I can submit PR for this if we're interested in doing it. 😄 


Discussion:
we shall use the requirements listed in folder `requirements` as we do with the base requirements anyway
the question is if we want to have extras as default option? @PyTorchLightning/core-contributors 

@nateraw have you done this before? :]

for the record: https://setuptools.readthedocs.io/en/latest/setuptools.html
@Borda I have done something like this before and I really think we should not use it as a default, but instead document what is installed when and really make sure to keep those up to date.
no defaults... this is not a common thing people know/do (i have never used it actually). 

We are not re-inventing how installs are done today with PL. We need to keep it simple as it is right now. 

If users want a package they can install it separately 
> Borda I have done something like this before and I really think we should not use it as a default, but instead document what is installed when and really make sure to keep those up to date.

just check the docs and it is very simple, just add this one dict, which would be loaded from our existing requirements

> If users want a package they can install it separately

this hold, we just thinking about the option to install extras also with pip package which is not possible in a simple way now...
is it for development only or for all users?
This would be available for all users then

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -12,21 +12,23 @@
 
 # https://packaging.python.org/guides/single-sourcing-package-version/
 # http://blog.ionelmc.ro/2014/05/25/python-packaging/
-
 PATH_ROOT = os.path.dirname(__file__)
 builtins.__LIGHTNING_SETUP__ = True
 
 import pytorch_lightning  # noqa: E402
 
 
-def load_requirements(path_dir=PATH_ROOT, comment_char='#'):
-    with open(os.path.join(path_dir, 'requirements', 'base.txt'), 'r') as file:
+def load_requirements(path_dir=PATH_ROOT, file_name='base.txt', comment_char='#'):
+    with open(os.path.join(path_dir, 'requirements', file_name), 'r') as file:
         lines = [ln.strip() for ln in file.readlines()]
     reqs = []
     for ln in lines:
         # filer all comments
         if comment_char in ln:
-            ln = ln[:ln.index(comment_char)]
+            ln = ln[:ln.index(comment_char)].strip()
+        # Make slight syntax alteration to git dependency for PL's sphinx theme
+        if ln.startswith('git') and file_name == 'docs.txt':
+            ln = f'pt_lightning_sphinx_theme @ {ln}#egg=pt-lightning-sphinx-theme'
         if ln:  # if requirement is not empty
             reqs.append(ln)
     return reqs
@@ -43,6 +45,19 @@ def load_long_description():
     return text
 
 
+# https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras
+# Define package extras. These are only installed if you specify them.
+# From remote, use like `pip install pytorch-lightning[dev, docs]`
+# From local copy of repo, use like `pip install ".[dev, docs]"`
+extras = {
+    'docs': load_requirements(file_name='docs.txt'),
+    'examples': load_requirements(file_name='examples.txt'),
+    'extra': load_requirements(file_name='extra.txt'),
+    'test': load_requirements(file_name='test.txt')
+}
+extras['dev'] = extras['extra'] + extras['test']
+extras['all'] = extras['dev'] + extras['examples'] + extras['docs']
+
 # https://packaging.python.org/discussions/install-requires-vs-requirements /
 # keep the meta-data here for simplicity in reading this file... it's not obvious
 # what happens and to non-engineers they won't know to look in init ...
@@ -67,7 +82,8 @@ def load_long_description():
     keywords=['deep learning', 'pytorch', 'AI'],
     python_requires='>=3.6',
     setup_requires=[],
-    install_requires=load_requirements(PATH_ROOT),
+    install_requires=load_requirements(),
+    extras_require=extras,
 
     project_urls={
         "Bug Tracker": "https://github.com/PyTorchLightning/pytorch-lightning/issues",


Test patch: