Repository: iterative/dvc
Pull request: Coalesce `requirements.txt` into `setup.py`
* [x] Have you followed the guidelines in our
      [Contributing document](https://dvc.org/doc/user-guide/contributing)?

* [x] Does your PR affect documented changes or does it add new functionality
      that should be documented? If yes, have you created a PR for
      [dvc.org](https://github.com/iterative/dvc.org) documenting it or at
      least opened an issue for it? If so, please add a link to it.

Issue Link: https://github.com/iterative/dvc.org/issues/387

Fixes #1979 

-----

These changes unify `requirements` and `tests/requirements.txt` and add their contents in a list in `setup.py`. This removes duplication in `setup.py`.

Problem statement:
dvc: unify requirements files and setup.py
We have duplication between `requirements.txt` and `setup.py` `install_requires`. We also use three `pip install` lines in docs to set things up. Ideally we would just say:
```bash
pip install -e .[tests]
# or
pip install -e .[all,tests]  # for all remotes
```
So this contains several parts:
- [ ] include `requirements.txt` in test requirements with `-r` syntax,
- [ ] parse requirements files in `setup.py` or drop requirement files 
- [ ] update contributing docs

Hints from issue discussion:

Commit message: Append extra dependenices in test_requirements in setup.py

Patch to review:
diff --git a/requirements.txt b/requirements.txt
deleted file mode 100644
index 6bf6bb725b..0000000000
--- a/requirements.txt
+++ /dev/null
@@ -1,33 +0,0 @@
-azure-storage-blob==1.3.0
-boto3==1.9.115
-ply>=3.9 # See https://github.com/pyinstaller/pyinstaller/issues/1945
-configparser>=3.5.0
-zc.lockfile>=1.2.1
-future>=0.16.0
-google-cloud-storage==1.13.0
-PyInstaller==3.4
-colorama>=0.3.9
-configobj>=5.0.6
-networkx>=2.1
-gitpython>=2.1.8
-setuptools>=34.0.0
-nanotime>=0.5.2
-pyasn1>=0.4.1
-schema>=0.6.7
-paramiko>=2.4.1
-jsonpath-ng>=1.4.3
-requests>=2.22.0
-wheel>=0.31.1
-futures>=3.2.0; python_version == "2.7"
-grandalf==0.6
-pydot>=1.2.4
-asciimatics>=1.10.0
-distro>=1.3.0
-appdirs>=1.4.3
-treelib>=1.5.5
-inflect>=2.1.0
-humanize>=0.5.1
-dulwich>=0.19.11
-ruamel.yaml>=0.15.91
-pathlib2==2.3.3; python_version == "2.7"
-oss2==2.6.1
diff --git a/setup.py b/setup.py
index 74871f7118..1afeac6258 100644
--- a/setup.py
+++ b/setup.py
@@ -69,6 +69,34 @@ def run(self):
 ssh = ["paramiko>=2.4.1"]
 all_remotes = gs + s3 + azure + ssh + oss
 
+# Extra dependecies to run tests
+tests_requirements = [
+    "PyInstaller==3.4",
+    "wheel>=0.31.1",
+    'futures>=3.2.0; python_version == "2.7"',
+    "pydot>=1.2.4",
+    'pathlib2==2.3.3; python_version == "2.7"',
+    # Test requirements:
+    "pytest>=4.4.0",
+    "pytest-timeout>=1.3.3",
+    "pytest-cov>=2.6.1",
+    "pytest-xdist>=1.26.1",
+    "pytest-mock>=1.10.4",
+    "flaky>=3.5.3",
+    "mock>=3.0.0",
+    "xmltodict>=0.11.0",
+    "awscli>=1.16.125",
+    "google-compute-engine",
+    "pywin32; sys_platform == 'win32'",
+    "Pygments"  # required by collective.checkdocs,
+    "collective.checkdocs",
+    "black==19.3b0 ; python_version >= '3.6'",
+    "flake8",
+    "flake8-docstrings",
+    "jaraco.windows==3.9.2",
+    "mock-ssh-server >= 0.5.0",
+]
+
 setup(
     name="dvc",
     version=version,
@@ -88,6 +116,7 @@ def run(self):
         "ssh": ssh,
         # NOTE: https://github.com/inveniosoftware/troubleshooting/issues/1
         ':python_version=="2.7"': ["futures", "pathlib2"],
+        "tests": tests_requirements,
     },
     keywords="data science, data version control, machine learning",
     python_requires=">=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*",
diff --git a/tests/requirements.txt b/tests/requirements.txt
deleted file mode 100644
index c7e1627afe..0000000000
--- a/tests/requirements.txt
+++ /dev/null
@@ -1,18 +0,0 @@
-pytest>=4.4.0
-pytest-timeout>=1.3.3
-pytest-cov>=2.6.1
-pytest-xdist>=1.26.1
-pytest-mock>=1.10.4
-flaky>=3.5.3
-mock>=3.0.0
-xmltodict>=0.11.0
-awscli>=1.16.125
-google-compute-engine
-pywin32; sys_platform == 'win32'
-Pygments  # required by collective.checkdocs
-collective.checkdocs
-black==19.3b0 ; python_version >= '3.6'
-flake8
-flake8-docstrings
-jaraco.windows==3.9.2
-mock-ssh-server >= 0.5.0


Code review comments:
diff_hunk: @@ -69,6 +69,34 @@ def run(self):
 ssh = ["paramiko>=2.4.1"]
 all_remotes = gs + s3 + azure + ssh + oss
 
+# Extra dependecies to run tests
+tests_requirements = [
+    "PyInstaller==3.4",
+    "wheel>=0.31.1",
+    'futures>=3.2.0; python_version == "2.7"',
+    "pydot>=1.2.4",
+    'pathlib2==2.3.3; python_version == "2.7"',
+    # Test requirements:
+    "pytest>=4.4.0",
+    "pytest-timeout>=1.3.3",
+    "pytest-cov>=2.6.1",
+    "pytest-xdist>=1.26.1",
+    "pytest-mock>=1.10.4",
+    "flaky>=3.5.3",
+    "mock>=3.0.0",
+    "xmltodict>=0.11.0",
+    "awscli>=1.16.125",
+    "google-compute-engine",
+    "pywin32; sys_platform == 'win32'",
+    "Pygments"  # required by collective.checkdocs,
+    "collective.checkdocs",
+    "black==19.3b0 ; python_version >= '3.6'",
+    "flake8",
+    "flake8-docstrings",
+    "jaraco.windows==3.9.2",
+    "mock-ssh-server >= 0.5.0",
path: setup.py
text: @user1:
```suggestion
    "mock-ssh-server>=0.5.0",
```

diff_hunk: @@ -69,6 +69,34 @@ def run(self):
 ssh = ["paramiko>=2.4.1"]
 all_remotes = gs + s3 + azure + ssh + oss
 
+# Extra dependecies to run tests
+tests_requirements = [
+    "PyInstaller==3.4",
+    "wheel>=0.31.1",
+    'futures>=3.2.0; python_version == "2.7"',
+    "pydot>=1.2.4",
+    'pathlib2==2.3.3; python_version == "2.7"',
+    # Test requirements:
+    "pytest>=4.4.0",
+    "pytest-timeout>=1.3.3",
+    "pytest-cov>=2.6.1",
+    "pytest-xdist>=1.26.1",
+    "pytest-mock>=1.10.4",
+    "flaky>=3.5.3",
+    "mock>=3.0.0",
+    "xmltodict>=0.11.0",
+    "awscli>=1.16.125",
+    "google-compute-engine",
+    "pywin32; sys_platform == 'win32'",
+    "Pygments"  # required by collective.checkdocs,
path: setup.py
text: @user1:
```suggestion
    "Pygments",  # required by collective.checkdocs
```