Repository: iterative/dvc
Pull request: doctor: show versions for fs requirements
Resolves #6248
Previous output:
```
DVC version: 2.4.3+ce45ec 
---------------------------------
Platform: Python 3.8.5+ on Linux-5.4.0-77-generic-x86_64-with-glibc2.29
Supports: s3, gs, http
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: https
Workspace directory: ext4 on /dev/sda7
Repo: dvc, git
```

Current output:
```
DVC version: 2.4.3+fc7409 
---------------------------------
Platform: Python 3.8.5+ on Linux-5.4.0-77-generic-x86_64-with-glibc2.29
Plugins:
        gs (gcsfs = 2021.6.1),
        http (requests = 2.25.1),
        s3 (s3fs = 2021.6.1, boto3 = 1.17.49),
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: https
Workspace directory: ext4 on /dev/sda7
Repo: dvc, git
```

Current output of all remotes:
```
DVC version: 2.4.3+fc7409 
---------------------------------
Platform: Python 3.8.5+ on Linux-5.4.0-77-generic-x86_64-with-glibc2.29
Plugins:
        azure (adlfs = 0.7.7, knack = 0.7.2, azure-identity = 1.5.0),
        gdrive (pydrive2 = 1.8.1),
        gs (gcsfs = 2021.6.1),
        hdfs (pyarrow = 2.0.0),
        webhdfs (hdfs = 2.5.8),
        http (requests = 2.25.1),
        https (requests = 2.25.1),
        s3 (s3fs = 2021.6.1, boto3 = 1.17.49),
        ssh (paramiko = 2.7.2),
        oss (oss2 = 2.6.1),
        webdav (webdav4 = 0.8.1),
        webdavs (webdav4 = 0.8.1)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: https
Workspace directory: ext4 on /dev/sda7
Repo: dvc, git
```

Problem statement:
doctor: should dvc doctor show remote-required dependency versions
Something that we've been experiencing lately is asking for an extra question regarding the version of their used remote dependency. We could try to annotate the dvc doctor's output with the stuff that we can get (optional). Something like this
```
 $ dvc doctor
DVC version: 2.4.2+f1bf10.mod 
---------------------------------
Platform: Python 3.8.5+ on Linux-5.4.0-77-generic-x86_64-with-glibc2.29
...
Remotes: gs (gcsfs == 2021.06.01, ... == 1.0.2)
...
```

Hints from issue discussion:
CC: @iterative/dvc
This would be amazing! :fire:  Should we bump the priority?
One thing that is a bit problematic is when the all remotes are installed. Normally instead of listing everything, DVC only prints All remotes;
```
 $ dvc doctor
DVC version: 2.4.3+ce45ec.mod 
---------------------------------
Platform: Python 3.8.5+ on Linux-5.4.0-77-generic-x86_64-with-glibc2.29
Supports: All remotes
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: https
Workspace directory: ext4 on /dev/sda7
Repo: dvc, git
```
Though this is probably going to be problematic for us in the future, when we introduce the plugin system the definition of 'All remotes' will probably change. There are 3 changes that we can do;

- Keep it as is, and only annotate DVC installations with limited remotes (for `dvc[all]`, nothing is going to change)
- Make it add version information only when you run `dvc doctor -v` or `dvc doctor -vv` (??, seems like a bad choice)
- Show it by default (too complicated?)
```
DVC version: 2.4.3+ce45ec.mod 
---------------------------------
Platform: Python 3.8.5+ on Linux-5.4.0-77-generic-x86_64-with-glibc2.29
Supports: azure(adlfs = 0.7.7, knack = 0.7.2), gdrive(pydrive2 = 1.8.1), gs(gcsfs = 2021.6.1+0.g17e7a09.dirty), hdfs(pyarrow = 2.0.0), webhdfs(hdfs = 2.5.8), http, https, s3(s3fs = 2021.6.1+5.gfc4489c, boto3 = 1.17.49), ssh(paramiko = 2.7.2), oss(oss2 = 2.6.1), webdav(webdav4 = 0.8.1), webdavs(webdav4 = 0.8.1)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: https
Workspace directory: ext4 on /dev/sda7
Repo: dvc, git
```
or perhaps something like this (much better IMHO);
```
 $ dvc doctor
DVC version: 2.4.3+ce45ec.mod 
---------------------------------
Platform: Python 3.8.5+ on Linux-5.4.0-77-generic-x86_64-with-glibc2.29
Supports:
        azure(adlfs = 0.7.7, knack = 0.7.2),
        gdrive(pydrive2 = 1.8.1),
        gs(gcsfs = 2021.6.1+0.g17e7a09.dirty),
        hdfs(pyarrow = 2.0.0),
        webhdfs(hdfs = 2.5.8),
        http,
        https,
        s3(s3fs = 2021.6.1+5.gfc4489c, boto3 = 1.17.49),
        ssh(paramiko = 2.7.2),
        oss(oss2 = 2.6.1),
        webdav(webdav4 = 0.8.1),
        webdavs(webdav4 = 0.8.1)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: https
Workspace directory: ext4 on /dev/sda7
Repo: dvc, git
```
@isidentical I agree that it is better to just show all remotes that are supported with corresponding versions. Indeed, current `all remotes` won't work with plugins in the future anyways.

@skshetry wdyt about removing `all remotes`?
I'm fine with removing `all remotes`. Though I find `dvc version` to be already complicated and slow, and we are in a transition period for the remote updates, maybe we won't need it in the future?

Anyway, to simplify the output a bit, maybe we can even get rid of remote names and just print libraries, like how pytest does.
```console
plugins: cov-2.12.1, flaky-3.7.0, tap-3.2, docker-0.10.3, xdist-2.3.0, mock-3.6.1, anyio-3.1.0, timeout-1.4.2, forked-1.3.0, lazy-fixture-0.6.3
```

```console
plugins: adlfs-0.7.7, pydrive2-1.8.1, gcsfs-2021.6.1, pyarrow-2.0.0, hdfs-2.5.8, s3fs-2021.6.1, sshfs-2021.06.0, ossfs-2021.6.1, webdav4-0.8.1
```

I think it's okay to not worry about `All`, as rarely users will have all of those installed.
Similarly, `http` remotes are always bundled, so we don't really need them.
Also naming the section as `plugins`, as that is how we are going to consider them in the future.

> I'm fine with removing all remotes. Though I find dvc version to be already complicated and slow, and we are in a transition period for the remote updates, maybe we won't need it in the future?

I don't think that it is going to affect performance that much, an `importlib.metadata.version()` call is `+/- 1 msec`, so at total it will be at most 50-60 msec change (including the import of `importlib.metadata`) to retrieve all the versions we need which is waaay lover compared to importing each module so I don't think it is even noticable.  

> we are in a transition period for the remote updates, maybe we won't need it in the future?

I am not sure about what is the remote update stuff, but I doubt they will completely eliminate the need for checking remote dependency versions. At least not until we stabilize everything on the fsspec side. 

> Anyway, to simplify the output a bit, maybe we can even get rid of remote names and just print libraries, like how pytest does.

Though, it will probably overflow and it is not easy to locate a single dependency bound to a single remote (e.g pyarrow for example). 

>  Similarly, http remotes are always bundled, so we don't really need them.

We could potentially create a list of always installed remotes, or just skip the ones that have no `REQUIRES` (e.g http/https in this case)

>  Also naming the section as plugins, as that is how we are going to consider them in the future.

Makes sense, or `supported remote types:`/`supported file systems:` (a bit too internal terminology?) but yea I think they are all better than regular bare `Supports:`.

Commit message: doctor: show versions for fs requirements

Patch to review:
diff --git a/dvc/fs/http.py b/dvc/fs/http.py
index a05c4112f0..c265db2064 100644
--- a/dvc/fs/http.py
+++ b/dvc/fs/http.py
@@ -31,6 +31,7 @@ class HTTPFileSystem(BaseFileSystem):  # pylint:disable=abstract-method
     PATH_CLS = HTTPURLInfo
     PARAM_CHECKSUM = "etag"
     CAN_TRAVERSE = False
+    REQUIRES = {"requests": "requests"}
 
     SESSION_RETRIES = 5
     SESSION_BACKOFF_FACTOR = 0.1
diff --git a/dvc/info.py b/dvc/info.py
index fdd276c294..bb1158303b 100644
--- a/dvc/info.py
+++ b/dvc/info.py
@@ -2,6 +2,7 @@
 import os
 import pathlib
 import platform
+import sys
 import uuid
 
 import psutil
@@ -20,6 +21,11 @@
 else:
     package = f"({PKG})"
 
+if sys.version_info < (3, 8):
+    import importlib_metadata
+else:
+    import importlib.metadata as importlib_metadata
+
 
 def get_dvc_info():
     info = [
@@ -27,7 +33,7 @@ def get_dvc_info():
         "---------------------------------",
         f"Platform: Python {platform.python_version()} on "
         f"{platform.platform()}",
-        f"Supports: {_get_supported_remotes()}",
+        f"Plugins:{_get_supported_remotes()}",
     ]
 
     try:
@@ -113,19 +119,23 @@ def _get_linktype_support_info(repo):
 
 
 def _get_supported_remotes():
-
     supported_remotes = []
     for scheme, fs_cls in FS_MAP.items():
         if not fs_cls.get_missing_deps():
-            supported_remotes.append(scheme)
-
-    if len(supported_remotes) == len(FS_MAP):
-        return "All remotes"
+            dependencies = []
+            for requirement in fs_cls.REQUIRES:
+                dependencies.append(
+                    f"{requirement} = "
+                    f"{importlib_metadata.version(requirement)}"
+                )
 
-    if len(supported_remotes) == 1:
-        return supported_remotes
+            remote_info = scheme
+            if dependencies:
+                remote_info += " (" + ", ".join(dependencies) + ")"
+            supported_remotes.append(remote_info)
 
-    return ", ".join(supported_remotes)
+    assert len(supported_remotes) >= 1
+    return "\n\t" + ",\n\t".join(supported_remotes)
 
 
 def get_fs_type(path):
diff --git a/setup.py b/setup.py
index 744eaf8168..d09d52da41 100644
--- a/setup.py
+++ b/setup.py
@@ -78,6 +78,7 @@ def run(self):
     "pydot>=1.2.4",
     "speedcopy>=2.0.1; python_version < '3.8' and sys_platform == 'win32'",
     "dataclasses==0.7; python_version < '3.7'",
+    "importlib-metadata==1.4; python_version < '3.8'",
     "flatten_dict>=0.3.0,<1",
     "tabulate>=0.8.7",
     "pygtrie>=2.3.2",
diff --git a/tests/func/test_version.py b/tests/func/test_version.py
index e08f05ce80..08937d6494 100644
--- a/tests/func/test_version.py
+++ b/tests/func/test_version.py
@@ -1,7 +1,7 @@
 import re
 
 from dvc.main import main
-from tests.unit.test_info import PYTHON_VERSION_REGEX
+from tests.unit.test_info import PYTHON_VERSION_REGEX, find_supported_remotes
 
 
 def test_(tmp_dir, dvc, scm, capsys):
@@ -10,7 +10,7 @@ def test_(tmp_dir, dvc, scm, capsys):
     out, _ = capsys.readouterr()
     assert re.search(r"DVC version: \d+\.\d+\.\d+.*", out)
     assert re.search(f"Platform: {PYTHON_VERSION_REGEX} on .*", out)
-    assert re.search(r"Supports: .*", out)
+    assert find_supported_remotes(out)
     assert re.search(r"Cache types: .*", out)
     assert re.search(r"Caches: local", out)
     assert re.search(r"Remotes: None", out)
diff --git a/tests/unit/test_info.py b/tests/unit/test_info.py
index 9989e69f12..25a4a40a6c 100644
--- a/tests/unit/test_info.py
+++ b/tests/unit/test_info.py
@@ -13,6 +13,35 @@
 PYTHON_VERSION_REGEX = r"Python \d\.\d+\.\d+\S*"
 
 
+def find_supported_remotes(string):
+    lines = string.splitlines()
+    index = 0
+
+    for index, line in enumerate(lines):
+        if line == "Plugins:":
+            index += 1
+            break
+    else:
+        return []
+
+    remotes = {}
+    for line in lines[index:]:
+        if not line.startswith("\t"):
+            break
+
+        remote_name, _, raw_dependencies = (
+            line.strip().strip(",").partition(" ")
+        )
+        remotes[remote_name] = {
+            dependency: version
+            for dependency, _, version in [
+                dependency.partition(" = ")
+                for dependency in raw_dependencies[1:-1].split(", ")
+            ]
+        }
+    return remotes
+
+
 @pytest.mark.parametrize("scm_init", [True, False])
 def test_info_in_repo(scm_init, tmp_dir):
     tmp_dir.init(scm=scm_init, dvc=True)
@@ -23,7 +52,7 @@ def test_info_in_repo(scm_init, tmp_dir):
 
     assert re.search(r"DVC version: \d+\.\d+\.\d+.*", dvc_info)
     assert re.search(f"Platform: {PYTHON_VERSION_REGEX} on .*", dvc_info)
-    assert re.search(r"Supports: .*", dvc_info)
+    assert find_supported_remotes(dvc_info)
     assert re.search(r"Cache types: .*", dvc_info)
 
     if scm_init:
@@ -98,7 +127,7 @@ def test_info_outside_of_repo(tmp_dir, caplog):
 
     assert re.search(r"DVC version: \d+\.\d+\.\d+.*", dvc_info)
     assert re.search(f"Platform: {PYTHON_VERSION_REGEX} on .*", dvc_info)
-    assert re.search(r"Supports: .*", dvc_info)
+    assert find_supported_remotes(dvc_info)
     assert not re.search(r"Cache types: .*", dvc_info)
     assert "Repo:" not in dvc_info
 
@@ -107,4 +136,14 @@ def test_fs_info_outside_of_repo(tmp_dir, caplog):
     dvc_info = get_dvc_info()
     assert re.search(r"DVC version: \d+\.\d+\.\d+.*", dvc_info)
     assert re.search(f"Platform: {PYTHON_VERSION_REGEX} on .*", dvc_info)
-    assert re.search(r"Supports: .*", dvc_info)
+    assert find_supported_remotes(dvc_info)
+
+
+def test_plugin_versions(tmp_dir, dvc):
+    from dvc.fs import FS_MAP
+
+    dvc_info = get_dvc_info()
+    remotes = find_supported_remotes(dvc_info)
+
+    for remote, dependencies in remotes.items():
+        assert dependencies.keys() == FS_MAP[remote].REQUIRES.keys()


Code review comments:
diff_hunk: @@ -20,14 +21,19 @@
 else:
     package = f"({PKG})"
 
+if sys.version_info < (3, 8):
+    import importlib_metadata
+else:
+    import importlib.metadata as importlib_metadata
+
 
 def get_dvc_info():
     info = [
         f"DVC version: {__version__} {package}",
         "---------------------------------",
         f"Platform: Python {platform.python_version()} on "
         f"{platform.platform()}",
-        f"Supports: {_get_supported_remotes()}",
+        f"Plugins:{_get_supported_remotes()}",
path: dvc/info.py
text: @user1:
Let's keep `Supports` for now, since we don't have plugins just yet and if we introduce this new term we'll have to change docs too.
```suggestion
        f"Supports: {_get_supported_remotes()}",
```