Repository: iterative/dvc
Pull request: get: implement --show-url to display only url/path to remote
Closes #2994. 

From the product requirements, it deviates on the following issue:
1. For now, it does not support showing URLs for multiple paths.
   So, we can only do `dvc get <url> <path> --show-url` for a single path, so as 
   to make it compatible with `dvc get`.
2. It also does not support specifying remote at the moment, so as to not introduce new flags. (#3183)
3. It does not work with the file inside of a directory (i.e. `dir/file`). Limitation of `dvc.api.get_url()`. Requires adding directory granularity support. (#3180)
	```sh
	➜ dvc get . dir/file --show-url
	ERROR: failed to show url - unable to find DVC-file with output 'dir/file'
	```
4. How to show path/URL to the directory? Currently, it only shows `.dir` file. (#3182)
	```sh
	➜ dvc get . dir --show-url
    ../tmp.fKEyzBMWge/12/11325135bf45fe5f67efa975110a57.dir
	```
   Same is the case with `dvc get` (cc @jorgeorpinel for docs regarding `dvc.api.get_url()`).
   
5. It does not work with non-dvc repos (i.e. git only repos). So, the following won't work (#3181):
   ```sh
   ➜ dvc get https://github.com/schacon/cowsay install.sh --show-url 
   ERROR: failed to show url - URL 'https://github.com/schacon/cowsay' is not a dvc repository.
   ```
6. Custom revision does not work for the local repo.
   Not fixing at the moment. #3179 tracks this.
	```sh
	➜ dvc get . foo --rev=HEAD@{5.days.ago} # works
	➜ dvc get . foo --rev=HEAD@{5.days.ago} --show-url
	ERROR: unexpected error - Custom revision is not supported for local repo
   ```


### Example Usage
```sh
➜ dvc get . foo --show-url
../tmp.uJVvn5ZaLY/d3/b07384d113edec49eaa6238ad5ff00
➜ dvc get https://github.com/iterative/dataset-registry.git get-started/data.xml --rev=HEAD@{5.days.ago} --show-url
https://remote.dvc.org/dataset-registry/a3/04afb96060aad90176268345e10355
```
EDIT: Made changes to format from `<path> <url>` previously to just `<url>`. 

* [x] ❗ Have you followed the guidelines in the [Contributing to DVC](https://dvc.org/doc/user-guide/contributing/core) list?

* [x] 📖 Check this box if this PR **does not** require [documentation](https://dvc.org/doc) updates, or if it does **and** you have created a separate PR in [dvc.org](https://github.com/iterative/dvc.org) with such updates (or at least opened an issue about it in that repo). Please link below to your PR (or issue) in the [dvc.org](https://github.com/iterative/dvc.org) repo.
iterative/dvc.org#930

* [x] ❌ Have you checked DeepSource, CodeClimate, and other sanity checks below? We consider their findings recommendatory and don't expect everything to be addressed. Please review them carefully and fix those that actually improve code or fix bugs.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

---

### TODO
#### Edgecases
- [x] https://github.com/iterative/dvc/pull/3156#discussion_r366942056
     


Problem statement:
get: show url to cache
In some cases, users need a direct link to a data file in the cloud. It might be just a matter of convenience or when DVC is used from automation tools like CD4ML scenarios. We need to provide a way to get a direct link.

I suggest creating a new command `dvc resolve`. Other options can be also be considered. For example, it might be a part of `dvc status` (when it start supporting data files as argument).

```
$ dvc resolve file.txt df.csv
file.txt s3://dvc-temp/dataset1/2d10f00a05f1fe70eaa3c42aa9f44b95
df.csv s3://dvc-temp/dataset1/99370a5386cac00c93c9fdc836076a7d
```

It should support revisions (checksum, tag, branch or HEAD^^):
```
$ dvc resolve file.txt df.csv --rev ada9d973
file.txt s3://dvc-temp/dataset1/d30abac6950583e6a5f1dcd31cb1043b
df.csv gs://project42/cache/2290babda371e52eeca2a2065a358783
```

Note, the commands with `revision` options should resolve old remotes as well (see `s3://` and `gs://` above).

Hints from issue discussion:
@dmpetrov Sounds like some automation helper, that can be worked around by simply reading the dvc file, extracting the checksum and then accessing the remote. Sounds like that wouldn't be a problem for anyone needing this for some automation script, not sure it is useful enough for many users though. 
> Sounds like some automation helper, that can be worked around by simply reading the dvc file

This feature is required in automation scenarios where people prefer to write bash code. Also, in the case of using revisions, it won't be as simple to implement. A single command - would be really great to have.

In terms of convenience, if you ask a user to do that in a manual way - it requires quite deep DVC knowledge and it is easy to make a mistake. The user should do 4 operations: find the corresponding dvc-file, find a right checksum, find default remote in config, combine them. 2x more complex for old revisions - file copies, need to know `get` command.

This command will hide a lot of internals and will simplify the usage. It is about lowering the bar.
This issue has a remote config complication -  do we use a remote from config from specified rev or current one? Both might be useful:
- use current if there was no default remote back then or if we changed default remote and moved all the data
- use the one from rev if we intentionally use different default remotes in e.g. different branches
I think this already exists in `dvc.api.get_url` so may be a matter of wrapping it in a command 🙂 (and figuring the command arguments and their logic, etc.)
Change of plans. Talked with @dmpetrov @shcheklein , and they suggested that `--show-url`(or something similar) for `dvc get`  would be better than introducing a separate command. Let's consider that instead.

Also, what to show for directories?
@efiop, maybe I am missing something, but as per docs, `get` command downloads a file/directory from a DVC and Git repository. And, adding another responsibility to `get`, i.e. getting path/URL to the files in the cache, which in my view, is completely different from previous responsibility.
@skshetry good point! Let me clarify the way I see it. The reasons behind this decision to piggy-back on `dvc get` for this feature is to avoid introducing a new global command (`dvc resolve`) + it feels like a "dry-run" `dvc get` (show an URL it _would_ download, but do not actually download it), especially considering that they have a common interface - URL + path.

Btw, do you have a better place in mind to implement this?
@shcheklein, my argument is for [Unix philosophy](https://en.wikipedia.org/wiki/Unix_philosophy), and have a better interface and not become something similar to [`git checkout`](https://github.blog/2019-08-16-highlights-from-git-2-23/).

> git-checkout - Switch branches or restore working tree files

But, my opinion is based on the discussions here on the issue. I'll check the implementation to see what `get_url()` really does and will make up my mind.
> my argument is for Unix philosophy, and have a better interface and not become something similar to `git checkout`

💯, but we should be reasonable about adding new top level commands. Otherwise we will become like git from a different angle - https://git-man-page-generator.lokaltog.net/

> I'll check the implementation to see what get_url() really does and will make up my mind.

yes, please take a look. Open to other options.
Will `dvc import` also have the same flag? I also think these commands already do too many things tbh. And won't it be confusing that `dvc list` will show paths inside the repo `url` but when you actually run `dvc get/import --show-url` on them, the URL is to the remote instead?

What about adding the option in the new `dvc list` command instead? See #2509
This would give people the alternative to download their tools of choice e.g. `wget` or `aws s3 cp`

Commit message: unit: test: check output of get --show-url command

Patch to review:
diff --git a/dvc/api.py b/dvc/api.py
index 9d6a321d9e..85c5c07775 100644
--- a/dvc/api.py
+++ b/dvc/api.py
@@ -10,7 +10,7 @@
 from voluptuous import Schema, Required, Invalid
 
 from dvc.repo import Repo
-from dvc.exceptions import DvcException
+from dvc.exceptions import DvcException, NotDvcRepoError
 from dvc.external_repo import external_repo
 
 
@@ -43,13 +43,27 @@ class SummonError(DvcException):
     pass
 
 
+class UrlNotDvcRepoError(DvcException):
+    """Thrown if given url is not a DVC repository.
+
+    Args:
+        url (str): url to the repository.
+    """
+
+    def __init__(self, url):
+        super().__init__("URL '{}' is not a dvc repository.".format(url))
+
+
 def get_url(path, repo=None, rev=None, remote=None):
     """Returns an url of a resource specified by path in repo"""
-    with _make_repo(repo, rev=rev) as _repo:
-        abspath = os.path.join(_repo.root_dir, path)
-        out, = _repo.find_outs_by_path(abspath)
-        remote_obj = _repo.cloud.get_remote(remote)
-        return str(remote_obj.checksum_to_path_info(out.checksum))
+    try:
+        with _make_repo(repo, rev=rev) as _repo:
+            abspath = os.path.join(_repo.root_dir, path)
+            out, = _repo.find_outs_by_path(abspath)
+            remote_obj = _repo.cloud.get_remote(remote)
+            return str(remote_obj.checksum_to_path_info(out.checksum))
+    except NotDvcRepoError:
+        raise UrlNotDvcRepoError(repo)
 
 
 def open(path, repo=None, rev=None, remote=None, mode="r", encoding=None):
diff --git a/dvc/command/get.py b/dvc/command/get.py
index 22856f1c07..1386d8ca40 100644
--- a/dvc/command/get.py
+++ b/dvc/command/get.py
@@ -10,7 +10,27 @@
 
 
 class CmdGet(CmdBaseNoRepo):
+    def _show_url(self):
+        from dvc.api import get_url
+
+        try:
+            url = get_url(
+                self.args.path, repo=self.args.url, rev=self.args.rev
+            )
+            logger.info(url)
+        except DvcException:
+            logger.exception("failed to show url")
+            return 1
+
+        return 0
+
     def run(self):
+        if self.args.show_url:
+            return self._show_url()
+
+        return self._get_file_from_repo()
+
+    def _get_file_from_repo(self):
         from dvc.repo import Repo
 
         try:
@@ -55,4 +75,9 @@ def add_parser(subparsers, parent_parser):
     get_parser.add_argument(
         "--rev", nargs="?", help="Git revision (e.g. branch, tag, SHA)"
     )
+    get_parser.add_argument(
+        "--show-url",
+        action="store_true",
+        help="Returns path/url to the location in remote for specified path",
+    )
     get_parser.set_defaults(func=CmdGet)
diff --git a/tests/func/test_api.py b/tests/func/test_api.py
index 6ba3aef47d..b3f0c0aba2 100644
--- a/tests/func/test_api.py
+++ b/tests/func/test_api.py
@@ -6,7 +6,7 @@
 import pytest
 
 from dvc import api
-from dvc.api import SummonError
+from dvc.api import SummonError, UrlNotDvcRepoError
 from dvc.compat import fspath
 from dvc.exceptions import FileMissingError
 from dvc.main import main
@@ -51,6 +51,16 @@ def test_get_url_external(erepo_dir, remote_url):
     assert api.get_url("foo", repo=repo_url) == expected_url
 
 
+def test_get_url_git_only_repo_throws_exception(tmp_dir, scm):
+    tmp_dir.scm_gen({"foo": "foo"}, commit="initial")
+
+    with pytest.raises(UrlNotDvcRepoError) as exc_info:
+        api.get_url("foo", fspath(tmp_dir))
+
+    # On windows, `exc_info` has path escaped, eg: `C:\\\\Users\\\\travis`
+    assert fspath(tmp_dir) in str(exc_info).replace("\\\\", "\\")
+
+
 @pytest.mark.parametrize("remote_url", all_remote_params, indirect=True)
 def test_open(remote_url, tmp_dir, dvc):
     run_dvc("remote", "add", "-d", "upstream", remote_url)
diff --git a/tests/func/test_get.py b/tests/func/test_get.py
index c6bd1adaa7..87d8ceea3f 100644
--- a/tests/func/test_get.py
+++ b/tests/func/test_get.py
@@ -5,6 +5,7 @@
 
 from dvc.cache import Cache
 from dvc.config import Config
+from dvc.main import main
 from dvc.repo.get import GetDVCFileError, PathMissingError
 from dvc.repo import Repo
 from dvc.system import System
@@ -184,3 +185,30 @@ def test_get_from_non_dvc_master(tmp_dir, erepo_dir, caplog):
 
     assert caplog.text == ""
     assert (tmp_dir / dst).read_text() == "some_contents"
+
+
+def test_get_url_positive(tmp_dir, erepo_dir, caplog):
+    with erepo_dir.chdir():
+        erepo_dir.dvc_gen("foo", "foo")
+
+    caplog.clear()
+    with caplog.at_level(logging.ERROR, logger="dvc"):
+        assert main(["get", fspath(erepo_dir), "foo", "--show-url"]) == 0
+        assert caplog.text == ""
+
+
+def test_get_url_not_existing(tmp_dir, erepo_dir, caplog):
+    with caplog.at_level(logging.ERROR, logger="dvc"):
+        assert (
+            main(["get", fspath(erepo_dir), "not-existing-file", "--show-url"])
+            == 1
+        )
+        assert "failed to show url" in caplog.text
+
+
+def test_get_url_git_only_repo(tmp_dir, scm, caplog):
+    tmp_dir.scm_gen({"foo": "foo"}, commit="initial")
+
+    with caplog.at_level(logging.ERROR):
+        assert main(["get", fspath(tmp_dir), "foo", "--show-url"]) == 1
+        assert "failed to show url" in caplog.text
diff --git a/tests/unit/command/test_get.py b/tests/unit/command/test_get.py
index 2c464ecc94..3913324902 100644
--- a/tests/unit/command/test_get.py
+++ b/tests/unit/command/test_get.py
@@ -14,3 +14,19 @@ def test_get(mocker):
     assert cmd.run() == 0
 
     m.assert_called_once_with("repo_url", path="src", out="out", rev="version")
+
+
+def test_get_url(mocker, caplog):
+    cli_args = parse_args(
+        ["get", "repo_url", "src", "--rev", "version", "--show-url"]
+    )
+    assert cli_args.func == CmdGet
+
+    cmd = cli_args.func(cli_args)
+    m = mocker.patch("dvc.api.get_url")
+    m.return_value = "url"
+
+    assert cmd.run() == 0
+    assert "url" in caplog.text
+
+    m.assert_called_once_with("src", repo="repo_url", rev="version")


Code review comments:
diff_hunk: @@ -14,3 +14,19 @@ def test_get(mocker):
     assert cmd.run() == 0
 
     m.assert_called_once_with("repo_url", path="src", out="out", rev="version")
+
+
+def test_get_url(mocker, caplog):
+    cli_args = parse_args(
+        ["get", "repo_url", "src", "--rev", "version", "--show-url"]
+    )
+    assert cli_args.func == CmdGet
+
+    cmd = cli_args.func(cli_args)
+    m = mocker.patch("dvc.api.get_url")
+    m.return_value = "url"
path: tests/unit/command/test_get.py
text: @user1:
```suggestion
```

diff_hunk: @@ -14,3 +14,19 @@ def test_get(mocker):
     assert cmd.run() == 0
 
     m.assert_called_once_with("repo_url", path="src", out="out", rev="version")
+
+
+def test_get_url(mocker, caplog):
+    cli_args = parse_args(
+        ["get", "repo_url", "src", "--rev", "version", "--show-url"]
+    )
+    assert cli_args.func == CmdGet
+
+    cmd = cli_args.func(cli_args)
+    m = mocker.patch("dvc.api.get_url")
path: tests/unit/command/test_get.py
text: @user1:
```suggestion
    m = mocker.patch("dvc.api.get_url", return_value="resource_url")
```

diff_hunk: @@ -14,3 +14,19 @@ def test_get(mocker):
     assert cmd.run() == 0
 
     m.assert_called_once_with("repo_url", path="src", out="out", rev="version")
+
+
+def test_get_url(mocker, caplog):
+    cli_args = parse_args(
+        ["get", "repo_url", "src", "--rev", "version", "--show-url"]
+    )
+    assert cli_args.func == CmdGet
+
+    cmd = cli_args.func(cli_args)
+    m = mocker.patch("dvc.api.get_url")
+    m.return_value = "url"
+
+    assert cmd.run() == 0
+    assert "url" in caplog.text
path: tests/unit/command/test_get.py
text: @user1:
 ```suggestion
     assert "resource_url" in caplog.text
```