Repository: iterative/dvc

Issue:
Failure when using S3 remote cache
Trying to run the command

`dvc run -d s3://mybucket/data.txt -o s3://mybucket/data_copy.txt aws s3 cp s3://mybucket/data.txt s3://mybucket/data_copy.txt`

Using AWS S3 remote cache.

It fails with the error message `ERROR: unexpected error - Parameter validation failed: Invalid length for parameter Key, value: 0, valid range: 1-inf`

Attached is the log with `-v` turned on.

[log.txt](https://github.com/iterative/dvc.org/files/4370652/log.txt)

and out of `pip freeze`

[pip_freeze.txt](https://github.com/iterative/dvc/files/4370722/pip_freeze.txt)



Discussion:
@helger Could you also show us your `$ dvc version` output, please?
> @helger Could you also show us your `$ dvc version` output, please?

```
 dvc version
DVC version: 0.90.2
Python version: 3.7.4
Platform: Linux-4.4.0-1104-aws-x86_64-with-debian-stretch-sid
Binary: False
Package: pip
Supported remotes: azure, gs, hdfs, http, https, s3, ssh, oss
Filesystem type (workspace): ('ext4', '/dev/xvda1')
```
> > @helger Could you also show us your `$ dvc version` output, please?
> 
> ```
> dvc version
> DVC version: 0.90.2
> Python version: 3.7.4
> Platform: Linux-4.4.0-1104-aws-x86_64-with-debian-stretch-sid
> Binary: False
> Package: pip
> Supported remotes: azure, gs, hdfs, http, https, s3, ssh, oss
> Filesystem type (workspace): ('ext4', '/dev/xvda1')
> ```


Output of `pip freeze`

[pip_freeze.txt](https://github.com/iterative/dvc/files/4370733/pip_freeze.txt)
Discord context https://discordapp.com/channels/485586884165107732/485596304961962003/691688363291312139
This happens because we try to create parent when trying to link from cache to workspace (and, here, parent of `s3://<bucket>/<name>` is `""`, i.e. empty).
https://github.com/iterative/dvc/blob/ee26afe65e8f9ebf11d7236fd3ee16a47f9b4fc6/dvc/remote/base.py#L380-L385

If the file is in `s3://<bucket>/<namespaced>/<name>`, this won't fail.

I don't think, we need to create parent directories for object-based storages except when checking out directory.

Google Cloud Storage should equally be affected here as it's the same logic.
@skshetry indeed adding another level fixes the issue
@skshetry Great catch! :pray: In general we do need to create those, because s3 and s3 tools sometimes create dirs like that, it was discussed in https://github.com/iterative/dvc/pull/2683 . We don't need-need to do that, but it is a compromise to handle both approaches. But we definitely need to handle this corner case better.
@efiop, it depends on how you see it. We can handle this corner case, but, there's no need for us to create a directory in most of the cases (except of #2678 cases?).

i.e. `dvc add  s3://dvc-bucket/data/file` should not even try to create `s3://dvc-bucket/data/` as a directory.

Patch:
diff --git a/dvc/remote/gs.py b/dvc/remote/gs.py
--- a/dvc/remote/gs.py
+++ b/dvc/remote/gs.py
@@ -143,6 +143,9 @@ def walk_files(self, path_info):
             yield path_info.replace(fname)
 
     def makedirs(self, path_info):
+        if not path_info.path:
+            return
+
         self.gs.bucket(path_info.bucket).blob(
             (path_info / "").path
         ).upload_from_string("")
diff --git a/dvc/remote/s3.py b/dvc/remote/s3.py
--- a/dvc/remote/s3.py
+++ b/dvc/remote/s3.py
@@ -238,6 +238,9 @@ def makedirs(self, path_info):
         #
         # We are not creating directory objects for every parent prefix,
         # as it is not required.
+        if not path_info.path:
+            return
+
         dir_path = path_info / ""
         self.s3.put_object(Bucket=path_info.bucket, Key=dir_path.path, Body="")
 


Test patch:
diff --git a/tests/func/test_s3.py b/tests/func/test_s3.py
--- a/tests/func/test_s3.py
+++ b/tests/func/test_s3.py
@@ -2,6 +2,7 @@
 
 import boto3
 import moto.s3.models as s3model
+import pytest
 from moto import mock_s3
 
 from dvc.remote.s3 import RemoteS3
@@ -45,6 +46,30 @@ def test_copy_singlepart_preserve_etag():
     RemoteS3._copy(s3, from_info, to_info, {})
 
 
+@mock_s3
+@pytest.mark.parametrize(
+    "base_info",
+    [RemoteS3.path_cls("s3://bucket/"), RemoteS3.path_cls("s3://bucket/ns/")],
+)
+def test_link_created_on_non_nested_path(base_info, tmp_dir, dvc, scm):
+    remote = RemoteS3(dvc, {"url": str(base_info.parent)})
+    remote.s3.create_bucket(Bucket=base_info.bucket)
+    remote.s3.put_object(
+        Bucket=base_info.bucket, Key=(base_info / "from").path, Body="data"
+    )
+    remote.link(base_info / "from", base_info / "to")
+
+    assert remote.exists(base_info / "from")
+    assert remote.exists(base_info / "to")
+
+
+@mock_s3
+def test_makedirs_doesnot_try_on_top_level_paths(tmp_dir, dvc, scm):
+    base_info = RemoteS3.path_cls("s3://bucket/")
+    remote = RemoteS3(dvc, {"url": str(base_info)})
+    remote.makedirs(base_info)
+
+
 def _upload_multipart(s3, Bucket, Key):
     mpu = s3.create_multipart_upload(Bucket=Bucket, Key=Key)
     mpu_id = mpu["UploadId"]
diff --git a/tests/unit/remote/test_remote.py b/tests/unit/remote/test_remote.py
--- a/tests/unit/remote/test_remote.py
+++ b/tests/unit/remote/test_remote.py
@@ -1,4 +1,6 @@
-from dvc.remote import Remote
+import pytest
+
+from dvc.remote import Remote, RemoteS3, RemoteGS
 
 
 def test_remote_with_checksum_jobs(dvc):
@@ -25,3 +27,15 @@ def test_remote_without_checksum_jobs_default(dvc):
 
     remote = Remote(dvc, name="without_checksum_jobs")
     assert remote.checksum_jobs == remote.CHECKSUM_JOBS
+
+
+@pytest.mark.parametrize("remote_cls", [RemoteGS, RemoteS3])
+def test_makedirs_not_create_for_top_level_path(remote_cls, mocker):
+    url = "{.scheme}://bucket/".format(remote_cls)
+    remote = remote_cls(None, {"url": url})
+    mocked_client = mocker.PropertyMock()
+    # we use remote clients with same name as scheme to interact with remote
+    mocker.patch.object(remote_cls, remote.scheme, mocked_client)
+
+    remote.makedirs(remote.path_info)
+    assert not mocked_client.called