Repository: iterative/dvc
Pull request: Optional dependencies for cache strategies
An attempt to fix #348.

Problem statement:
allow installing and running only without all cloud dependencies
DVC has dependencies on multiple cloud services (aws, google), though a typical instalation will only use one of them

is it possible to make the package not use or require unneeded packages?

in runtime, it's usually possible to do 'lazy' imports, that will be called only when the specific cloud service is configured

Hints from issue discussion:
Hi @ophiry 

It is a good idea. But it might confuse some users - after changing a cloud in config DVC can start crashing. From `pip` point of view, we might (in theory) separate dvc package into a few packages: `dvc`, `dvc-aws`, `dvc-gcp`. This approach has advantages as well as disadvantages. We should think about this.

PS:
In the new version, we are going to provide binary packages for all OSs. All the dependencies will be inside the binary packages - no issues with dependencies.
IIRC there's no need to separate packages on pip, you can use conditional dependencies and let people install `dvc`, `dev[aws]` or `dvc[all]` for example.
@villasv thank you! I will take a look at it.
I just had to install `dvc` inside a `pipenv` and boy, the GCP packages are a ton of dependencies. It feels like I now have a `node_modules` inside my project. Seriously, GCP dependencies are about 30% of the `Pipfile.lock`, while `dvc` totals 90%.

I believe this issue can be extended to other optional dependencies, like HDFS and so on. [It doesn't seem too difficult to implement optional dependencies.](https://setuptools.readthedocs.io/en/latest/setuptools.html#declaring-extras-optional-features-with-their-own-dependencies) I might be able to send a PR this week.

Help wanted?
PRs are always appreciated :) Thank you for looking into it!
Alright. Time to get it done. What are all cache integrations that we could apply this to? I know we have `s3`, `gs` and `hdfs`, though I'm a bit unsure on which dependencies each of those require that are not required by dvc "core".
@villasv All remote drivers are located in dvc/remote. There are four right now: s3(boto3), gs(google-cloud* stuff), hdfs(no pip dependencies required, since it uses hadoop CLI utilitiy) and ssh(paramiko). Everything that uses these dependencies is isolated in respective dvc/remote/*.py files, so it should be pretty easy. Thank you for looking into it! Please feel free to ping me if you need anything.
I noticed that `ply` is intalled with `>3.11` by default, but there's a comment mentioning that `3.8` is required by `google-cloud`. Should I leave `3.8` as a core package?
Yeah, I would use the same versions that are currently listed in requirements.txt(so ply==3.8) unless there is a problem that requires adjustments.
Got it. How should I run the tests?

Commit message: Add dependency checking as config validation

Patch to review:
diff --git a/dvc/remote/__init__.py b/dvc/remote/__init__.py
index 40a44b5d3d..1648e1ec8d 100644
--- a/dvc/remote/__init__.py
+++ b/dvc/remote/__init__.py
@@ -21,6 +21,6 @@ def supported_url(url):
 
 def Remote(project, config):
     for r in REMOTES:
-        if r.supported(config):
+        if r.supported(config, check_dependencies=True):
             return r(project, config)
     raise DvcException('Remote \'{}\' is not supported.'.format(config))
diff --git a/dvc/remote/base.py b/dvc/remote/base.py
index 6fb0a791f2..f526ae1c35 100644
--- a/dvc/remote/base.py
+++ b/dvc/remote/base.py
@@ -33,14 +33,16 @@ def __init__(self, msg):
 
 class RemoteBase(object):
     REGEX = None
+    REQUIRES = []
 
     def __init__(self, project, config):
         pass
 
     @classmethod
-    def supported(cls, config):
+    def supported(cls, config, check_dependencies=False):
         url = config[Config.SECTION_REMOTE_URL]
-        return cls.match(url) != None
+        deps_ok = not check_dependencies or all(cls.REQUIRES)
+        return cls.match(url) != None and deps_ok
 
     @classmethod
     def match(cls, url):
diff --git a/dvc/remote/gs.py b/dvc/remote/gs.py
index 4683fb6488..0ae7ee3b74 100644
--- a/dvc/remote/gs.py
+++ b/dvc/remote/gs.py
@@ -1,6 +1,10 @@
 import os
 import posixpath
-from google.cloud import storage
+
+try:
+    from google.cloud import storage
+except ImportError:
+    storage = None
 
 try:
     from urlparse import urlparse
@@ -16,6 +20,7 @@
 class RemoteGS(RemoteBase):
     scheme = 'gs'
     REGEX = r'^gs://(?P<path>.*)$'
+    REQUIRES = RemoteBase.REQUIRES + [storage]
     PARAM_ETAG = 'etag'
 
     def __init__(self, project, config):
diff --git a/dvc/remote/s3.py b/dvc/remote/s3.py
index ad2c1a832a..2d14162959 100644
--- a/dvc/remote/s3.py
+++ b/dvc/remote/s3.py
@@ -1,9 +1,13 @@
 import os
 import math
-import boto3
 import threading
 import posixpath
 
+try:
+    import boto3
+except ImportError:
+    boto3 = None
+
 try:
     from urlparse import urlparse
 except ImportError:
@@ -31,6 +35,7 @@ def __call__(self, byts):
 class RemoteS3(RemoteBase):
     scheme = 's3'
     REGEX = r'^s3://(?P<path>.*)$'
+    REQUIRES = RemoteBase.REQUIRES + [boto3]
     PARAM_ETAG = 'etag'
 
     def __init__(self, project, config):
diff --git a/dvc/remote/ssh.py b/dvc/remote/ssh.py
index 6becc191fd..211aa54d1d 100644
--- a/dvc/remote/ssh.py
+++ b/dvc/remote/ssh.py
@@ -3,9 +3,13 @@
 import errno
 import getpass
 import filecmp
-import paramiko
 import posixpath
 
+try:
+    import paramiko
+except ImportError:
+    paramiko = None
+
 from dvc.logger import Logger
 from dvc.utils import copyfile, file_md5
 from dvc.progress import progress
@@ -41,6 +45,7 @@ class RemoteSSH(RemoteBase):
     #NOTE: temporarily only absolute paths are allowed
     REGEX = r'^ssh://((?P<user>.*)@)?(?P<host>[^/]*):(?P<path>/.*)$'
 
+    REQUIRES = RemoteBase.REQUIRES + [paramiko]
     PARAM_MD5 = 'md5'
 
     def __init__(self, project, config):
diff --git a/setup.py b/setup.py
index cdbf374751..5bb4fb875a 100644
--- a/setup.py
+++ b/setup.py
@@ -5,12 +5,10 @@
 
 
 install_requires = [
-    "boto3==1.7.4",
     "ply==3.8", # required by google-gax
     "configparser>=3.5.0",
     "zc.lockfile>=1.2.1",
     "future>=0.16.0",
-    "google-cloud==0.32.0",
     "colorama>=0.3.9",
     "configobj>=5.0.6",
     "networkx>=1.11",
@@ -22,13 +20,24 @@
     "nanotime>=0.5.2",
     "pyasn1>=0.4.1",
     "schema>=0.6.7",
-    "paramiko>=2.4.1",
     "binaryornot==0.4.4",
     "jsonpath-rw==1.4.0",
     "reflink==0.2.0",
     "requests>=2.18.4",
 ]
 
+# Extra dependencies for remote integrations
+gs = [
+    "google-cloud==0.32.0",
+]
+s3 = [
+    "boto3==1.7.4",
+]
+ssh = [
+    "paramiko>=2.4.1",
+]
+all_remotes = gs + s3 + ssh
+
 setup(
     name='dvc',
     version=VERSION,
@@ -39,6 +48,12 @@
     download_url='https://github.com/iterative/dvc',
     license='Apache License 2.0',
     install_requires=install_requires,
+    extras_require={
+        'all': all_remotes,
+        'gs': gs,
+        's3': s3,
+        'ssh': ssh,
+    },
     keywords='data science, data version control, machine learning',
     classifiers=[
         'Development Status :: 4 - Beta',


Code review comments:
diff_hunk: @@ -33,14 +33,16 @@ def __init__(self, msg):
 
 class RemoteBase(object):
     REGEX = None
+    REQUIRES = []
 
     def __init__(self, project, config):
         pass
 
     @classmethod
-    def supported(cls, config):
+    def supported(cls, config, check_dependencies=False):
path: dvc/remote/base.py
text: @user1:
I don't think `check_dependencies` flag is required, we can safely always assume that it is True.

diff_hunk: @@ -33,14 +33,16 @@ def __init__(self, msg):
 
 class RemoteBase(object):
     REGEX = None
+    REQUIRES = []
 
     def __init__(self, project, config):
         pass
 
     @classmethod
-    def supported(cls, config):
+    def supported(cls, config, check_dependencies=False):
         url = config[Config.SECTION_REMOTE_URL]
-        return cls.match(url) != None
+        deps_ok = not check_dependencies or all(cls.REQUIRES)
path: dvc/remote/base.py
text: @user1:
It would be great if we could print a warning here that the url is supported but the optional dependency is missing.

@author:
Makes sense. It will raise an assertion error during schema validation that ultimately says that `supported_url(...) should return True`, but if a logging message right before says "URL is valid but required dependencies to use it are missing", it should be clear enough.