Repository: iterative/dvc

Issue:
remote: getting a directory checksum requires uploading a file to an external cache
**Version**: 0.62.1

**Description**: The current approach to calculate a **directory checksum** is to collect its contents' checksum and then "upload" it and get the checksum provided by the remote:

https://github.com/iterative/dvc/blob/a091b67341b00fea8ef43906f59c88cb38f384ff/dvc/remote/base.py#L246-L256

For example, if it is an S3 remote, DVC will **create a file** containing an "index" of the directory **with its checksums** on the S3 cache **and then will try to get the ETag of such file**.

This **is an easy way to check if the directory has changed**. Since the "directory index" is already there, **you are a `head` operation away** from knowing if the directory changed or not. Otherwise, you'll need to recompute the file on the spot.

Right now **we don't require setting up a cache for using external dependencies, but as mentioned above, this is needed for directories.**

Let's discuss if there's another way around uploading files or just settle with that implementation and modify the code and docs accordingly.

**Related**: #1654 


Discussion:
@mroutis I don't understand the issue described, tbh. Could you provide please an example where it's clear what cache is being used, what remote is being used, where directory is located, etc.
Sure, @shcheklein . Take a look at the following script, it setups a mock S3 server with a directory which then is used as a dependency on a `dvc run` command:

```bash
#!/usr/bin/env bash

set -x

export AWS_ACCESS_KEY_ID='testing'
export AWS_SECRET_ACCESS_KEY='testing'
export AWS_SECURITY_TOKEN='testing'
export AWS_SESSION_TOKEN='testing'

moto_server s3 &> /dev/null &

# File structure:
#       dvc-temp
#       ├── data
#       │  ├── alice
#       │  ├── alpha
#       │  └── subdir
#       │     ├── 1
#       │     ├── 2
#       │     └── 3
#       ├── empty_dir
#       ├── empty_file
#       └── foo

python -c '
import boto3

session = boto3.session.Session()
s3 = session.client("s3", endpoint_url="http://localhost:5000")
s3.create_bucket(Bucket="dvc-temp")
s3.put_object(Bucket="dvc-temp", Key="cache/")
s3.put_object(Bucket="dvc-temp", Key="empty_dir/")
s3.put_object(Bucket="dvc-temp", Key="empty_file", Body=b"")
s3.put_object(Bucket="dvc-temp", Key="foo", Body=b"foo")
s3.put_object(Bucket="dvc-temp", Key="data/alice", Body=b"alice")
s3.put_object(Bucket="dvc-temp", Key="data/alpha", Body=b"alpha")
s3.put_object(Bucket="dvc-temp", Key="data/subdir/1", Body=b"1")
s3.put_object(Bucket="dvc-temp", Key="data/subdir/2", Body=b"2")
s3.put_object(Bucket="dvc-temp", Key="data/subdir/3", Body=b"3")
'

repository=$(mktemp -d)
cd $repository
dvc init --no-scm
dvc remote add s3 s3://dvc-temp
dvc remote modify s3 endpointurl http://localhost:5000

dvc remote add cache remote://s3/cache
dvc config cache.s3 cache

dvc run -d remote://s3/data 'echo something'
```

To use `remote://s3/data` as a dependency, DVC needs to create a file on the remote cache with the following content:

```
[{'etag': '6384e2b2184bcbf58eccf10ca7a6563c', 'relpath': 'alice'},
 {'etag': '2c1743a391305fbf367df8e4f069f9f9', 'relpath': 'alpha'},
 {'etag': 'c4ca4238a0b923820dcc509a6f75849b', 'relpath': 'subdir/1'},
 {'etag': 'c81e728d9d4c2f636f067f89cc14862c', 'relpath': 'subdir/2'},
 {'etag': 'eccbc87e4b5ce2fe28308fd9f2a7baf3', 'relpath': 'subdir/3'}]
```

_Note: added newlines for readability but the file shouldn't contain newlines._

Such file would be located at `remote://cache/af/134c33d65939675318e71363241eee.dir` and this is what I called an _index file_ before (probably wrongly named -- but it is a list of the files inside the directory and its checksums).

As I already stated, it is useful for checking the integrity of the directory.

The problem is that setting up a cache for working with external dependencies is not required by DVC.

As an example, you can do `dvc import-url remote://s3/data/alice` and it won't need a cache at all (because is a single file and doesn't run the `_get_dir_info_checksum` method).

Let's say that the following commands are equivalent:

```console
$ dvc import-url remote://s3/data
$ dvc run -d remote://s3/data -o data/ 'aws s3 copy --recursive s3://dvc-temp/data data'
```

Because `data/` is a directory, it would require a configured cache.

For external outputs (it doesn't matter if it is a directory or not), we throw an error when a cache is not set:

```console
$ dvc run -o remote://s3/data/alice 'echo something'

ERROR: failed to run command - no cache location setup for 's3' outputs.
```

The discussion resolves around the behavior of creating such _index file_ and requiring the user to set up a cache for `dvc import-url`ing a directory.

Hope it is more understandable now, @shcheklein :)
Thanks, @mroutis ! It's definitely helpful. Some questions that are still not exactly clear to me:

> This is an easy way to perform an integrity check.

Is it an _integrity_ check or a check to see that a remote dir has changed to rerun a stage if needed? If it's an integrity check, when do we run it?

> `head` operation away from knowing if the directory changed or not

Even if create an index file on the remote cache, how does it help to see if the directory changed or not? We need to calculate that file every time, right?

Am I right that the question is not about creating an index file itself (we will need it anyway, right?) and more about if we need to cache it or just calculate when it's needed?

If so, then can you clarify again what are the benefits of caching it? I don't understand if caching this file saves us from recalculating the dir checksum every time we want to check if something has changed on not.
@mroutis ^
Users are getting confused by this https://discordapp.com/channels/485586884165107732/563406153334128681/653666017368735782 🙁  Need to at least handle it gracefully, until we decide what to do with this.
@efiop @mroutis  count me in - I got confused once :) (even though I was aware about this ticket). My 2cs - I would say that the biggest problem is the UI here - that we left a user facing NPE .
@shcheklein:

> Is it an integrity check or a check to see that a remote dir has changed to rerun a stage if needed? If it's an integrity check, when do we run it?

I used _integrity check_ wrongly, I was talking about the remote directory changing as you pointed out.

> If so, then can you clarify again what are the benefits of caching it? I don't understand if caching this file saves us from recalculating the dir checksum every time we want to check if something has changed on not.

@shcheklein , I'll come back to this one later, will need to think it better
Another user ran into this https://discordapp.com/channels/485586884165107732/485596304961962003/667036162472411144
@efiop , https://github.com/iterative/dvc/issues/2647#event-2951497114 the issue is still `p1`. Not sure if it is intended.
@mroutis Yep, I don't think it is more (or as) important than the other p0s, this is a nasty ui issue though 🙁 . Added to the next sprint though, maybe we'll have time to look into it this sprint though.
First step: at least handle this error gracefully, can think about in-memory hash computation after that.
Simplest script to reproduce:
```
#!/bin/bash

rm -rf repo
mkdir repo
pushd repo

set -ex
dvc init --no-scm -q

dvc run -d s3://{bucket}/data -o result `aws2 s3 ls {bucket}/data/>>result`
```
assuming there is `{bucket}` with `data` folder on it

Resolving patch:
diff --git a/dvc/exceptions.py b/dvc/exceptions.py
--- a/dvc/exceptions.py
+++ b/dvc/exceptions.py
@@ -1,6 +1,6 @@
 """Exceptions raised by the dvc."""
 
-from dvc.utils import relpath
+from dvc.utils import relpath, format_link
 
 
 class DvcException(Exception):
@@ -310,3 +310,18 @@ def __init__(self, path, repo):
             " neither as an output nor a git-handled file."
         )
         super().__init__(msg.format(path, repo))
+
+
+class RemoteCacheRequiredError(DvcException):
+    def __init__(self, path_info):
+        super().__init__(
+            (
+                "Current operation was unsuccessful because '{}' requires "
+                "existing cache on '{}' remote. See {} for information on how "
+                "to set up remote cache."
+            ).format(
+                path_info,
+                path_info.scheme,
+                format_link("https://man.dvc.org/config#cache"),
+            )
+        )
diff --git a/dvc/output/base.py b/dvc/output/base.py
--- a/dvc/output/base.py
+++ b/dvc/output/base.py
@@ -6,7 +6,7 @@
 
 import dvc.prompt as prompt
 from dvc.cache import NamedCache
-from dvc.exceptions import CollectCacheError
+from dvc.exceptions import CollectCacheError, RemoteCacheRequiredError
 from dvc.exceptions import DvcException
 from dvc.remote.base import RemoteBASE
 
@@ -98,14 +98,9 @@ def __init__(
         self.persist = persist
         self.tags = None if self.IS_DEPENDENCY else (tags or {})
 
-        if self.use_cache and self.cache is None:
-            raise DvcException(
-                "no cache location setup for '{}' outputs.".format(
-                    self.REMOTE.scheme
-                )
-            )
-
         self.path_info = self._parse_path(remote, path)
+        if self.use_cache and self.cache is None:
+            raise RemoteCacheRequiredError(self.path_info)
 
     def _parse_path(self, remote, path):
         if remote:
diff --git a/dvc/remote/base.py b/dvc/remote/base.py
--- a/dvc/remote/base.py
+++ b/dvc/remote/base.py
@@ -18,6 +18,7 @@
     DvcException,
     ConfirmRemoveError,
     DvcIgnoreInCollectedDirError,
+    RemoteCacheRequiredError,
 )
 from dvc.ignore import DvcIgnore
 from dvc.path_info import PathInfo, URLInfo
@@ -233,6 +234,9 @@ def _collect_dir(self, path_info):
         return sorted(result, key=itemgetter(self.PARAM_RELPATH))
 
     def get_dir_checksum(self, path_info):
+        if not self.cache:
+            raise RemoteCacheRequiredError(path_info)
+
         dir_info = self._collect_dir(path_info)
         checksum, tmp_info = self._get_dir_info_checksum(dir_info)
         new_info = self.cache.checksum_to_path_info(checksum)


Test patch:
diff --git a/tests/func/test_remote.py b/tests/func/test_remote.py
--- a/tests/func/test_remote.py
+++ b/tests/func/test_remote.py
@@ -11,7 +11,7 @@
 from dvc.main import main
 from dvc.path_info import PathInfo
 from dvc.remote import RemoteLOCAL, RemoteConfig
-from dvc.remote.base import RemoteBASE
+from dvc.remote.base import RemoteBASE, RemoteCacheRequiredError
 from dvc.compat import fspath
 from tests.basic_env import TestDvc
 from tests.remotes import Local
@@ -257,3 +257,14 @@ def test_modify_missing_remote(dvc):
 
     with pytest.raises(ConfigError, match=r"Unable to find remote section"):
         remote_config.modify("myremote", "gdrive_client_id", "xxx")
+
+
+def test_external_dir_resource_on_no_cache(tmp_dir, dvc, tmp_path_factory):
+    # https://github.com/iterative/dvc/issues/2647, is some situations
+    # (external dir dependency) cache is required to calculate dir md5
+    external_dir = tmp_path_factory.mktemp("external_dir")
+    (external_dir / "file").write_text("content")
+
+    dvc.cache.local = None
+    with pytest.raises(RemoteCacheRequiredError):
+        dvc.run(deps=[fspath(external_dir)])