Repository: iterative/dvc

Issue:
dvc.api.get_url: it doesn't find "md5" tag
# Bug Report

<!--
## Issue name

Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug.

Example: `repro: doesn't detect input changes`
-->

## Description

<!--
A clear and concise description of what the bug is.
-->

I'm trying to use `dvc.api.get_url `to read a DataFrame in Google Cloud Storage, but the dvc file didn't create an md5 keyword, but an "etag" keyword in the `.csv.dvc` file. The python command `api.get_url()` looks for an "md5" keyword and generates an error.

### Reproduce

<!--
Step list of how to reproduce the bug
-->

<!--
Example:

1. dvc init
2. Copy dataset.zip to the directory
3. dvc add dataset.zip
4. dvc run -d dataset.zip -o model ./train.sh
5. modify dataset.zip
6. dvc repro
-->

these are the steps I followed in macos with zsh:

1. `python -m venv deploy_env`
2. `source deploy_env/bin/activate`
3. `python -m pip install --upgrade pip`
4. `Create these requirements in path requirements/dev.txt`


```txt
# prod
pandas~=1.4.3
scikit-learn~=1.1.1
matplotlib~=3.5.2
seaborn~=0.11.2
dvc~=2.12.0
dvc[gs]
pyyaml

# dev
jupyter
flake8~=4.0.1
black~=22.6.0
```

5. `pip install -r requirements/dev.txt`
6. `dvc init`
7. `dvc remote add dataset-track gs://model-data-tracker-775/dataset`
8. `dvc remote add model-track gs://model-data-tracker-775/model`
9. `dvc add dataset/finantials.csv --to-remote -r dataset-track`
10. `dvc add model/model.pkl --to-remote -r model-track`
11. It generates a .dvc file like this one:
```dvc
outs:
- etag: 4a265d63db921dcca137b881e47eab6d
  size: 593277
  path: finantials.csv
```
12. Create this script in the `src/prepare.py` path:

```python
from dvc import api
import pandas as pd


finantials_data_path = api.get_url("dataset/finantials.csv", remote="dataset-track")
finantials_df = pd.read_csv(finantials_data_path)
```
14. `rm dataset/finantials.csv`
15. `python src/prepare.py`

### Expected

<!--
A clear and concise description of what you expect to happen.
-->

An error looking for an unexistent "md5" keyword:
```bash
 $ python src/prepare.py
Traceback (most recent call last):
  File "../src/prepare.py", line 5, in <module>
    finantials_data_path = api.get_url("dataset/finantials.csv", remote="dataset-tracker")
  File "../deploy_env/lib/python3.9/site-packages/dvc/api/data.py", line 31, in get_url
    md5 = dvc_info["md5"]
KeyError: 'md5'
```

### Environment information

<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc doctor`:**

```bash
$ dvc doctor
DVC version: 2.12.0 (pip)
---------------------------------
Platform: Python 3.9.12 on macOS-12.4-arm64-arm-64bit
Supports:
        gs (gcsfs = 2022.5.0),
        webhdfs (fsspec = 2022.5.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.5.1),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.5.1)
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: gs, gs
Workspace directory: apfs on /dev/disk3s1s1
Repo: dvc, git
```

**Additional Information (if any):**
When I perform the `pull` command, it doesn't give me any error:

```bash
$ dvc pull dataset/finantials.csv.dvc -r dataset-track            
A       dataset/finantials.csv                                         
1 file added and 1 file fetched  
```
<!--
Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.

If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.
-->



Discussion:
Hi Nelson, can you resolve this bug? I have the same problem
I could resove the problem. This work for me:
fsspec==2022.7.1
dvc==2.12.0
I don't even think this is really a `dvc.api.get_url` problem, but a problem with `dvc add --to-remote` using a cloud remote.

Here's [the example from the docs](https://dvc.org/doc/command-reference/add#example-transfer-to-remote-storage) but with a cloud remote:

```bash
$ dvc remote add -d -f cloud s3://dave-sandbox-versioning/test/
$ dvc add https://data.dvc.org/get-started/data.xml --to-remote
$ dvc pull -v
2023-02-28 07:46:27,683 DEBUG: v2.45.2.dev46+g35b648bbc, CPython 3.10.2 on macOS-13.1-arm64-arm-64bit
2023-02-28 07:46:27,683 DEBUG: command: /Users/dave/miniforge3/envs/dvc/bin/dvc pull -v
2023-02-28 07:46:28,162 WARNING: Output 'data.xml'(stage: 'data.xml.dvc') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
2023-02-28 07:46:28,167 WARNING: No file hash info found for '/Users/dave/repo/data.xml'. It won't be created.
1 file failed
2023-02-28 07:46:28,167 ERROR: failed to pull data from the cloud - Checkout failed for following targets:
/Users/dave/repo/data.xml
Is your cache up to date?
<https://error.dvc.org/missing-files>
Traceback (most recent call last):
  File "/Users/dave/Code/dvc/dvc/commands/data_sync.py", line 31, in run
    stats = self.repo.pull(
  File "/Users/dave/Code/dvc/dvc/repo/__init__.py", line 58, in wrapper
    return f(repo, *args, **kwargs)
  File "/Users/dave/Code/dvc/dvc/repo/pull.py", line 47, in pull
    stats = self.checkout(
  File "/Users/dave/Code/dvc/dvc/repo/__init__.py", line 58, in wrapper
    return f(repo, *args, **kwargs)
  File "/Users/dave/Code/dvc/dvc/repo/checkout.py", line 109, in checkout
    raise CheckoutError(stats["failed"], stats)
dvc.exceptions.CheckoutError: Checkout failed for following targets:
/Users/dave/repo/data.xml
```

Also, there's a question of what `-r` should do in the original post above. It doesn't save that remote info to the `.dvc` file, but it probably should.
For the record: looks like `md5` is not recorded during `--to-remote`, which is a regression. We might've lost it somewhere during cloud versioning work. Will take a closer look.
Yeah, just missing a hash_name option in `Remote`. The fix is simple, but need to migrate tests to `dvc/testing` while we are at it so we test that stuff all the time. Will send a PR soon.

Resolving patch:
diff --git a/dvc/data_cloud.py b/dvc/data_cloud.py
--- a/dvc/data_cloud.py
+++ b/dvc/data_cloud.py
@@ -35,7 +35,7 @@ def odb(self) -> "HashFileDB":
         if self.worktree:
             path = self.fs.path.join(path, ".dvc", "cache")
 
-        return get_odb(self.fs, path, **self.config)
+        return get_odb(self.fs, path, hash_name="md5", **self.config)
 
 
 class DataCloud:


Test patch:
diff --git a/dvc/testing/workspace_tests.py b/dvc/testing/workspace_tests.py
--- a/dvc/testing/workspace_tests.py
+++ b/dvc/testing/workspace_tests.py
@@ -282,3 +282,85 @@ def test_get_url_to_dir(self, cloud, tmp_dir, dname):
     def test_get_url_nonexistent(self, cloud):
         with pytest.raises(URLMissingError):
             Repo.get_url(str(cloud / "nonexistent"), config=cloud.config)
+
+
+class TestToRemote:
+    def test_add_to_remote(self, tmp_dir, dvc, remote, workspace):
+        workspace.gen("foo", "foo")
+
+        url = "remote://workspace/foo"
+        [stage] = dvc.add(url, to_remote=True)
+
+        assert not (tmp_dir / "foo").exists()
+        assert (tmp_dir / "foo.dvc").exists()
+
+        assert len(stage.deps) == 0
+        assert len(stage.outs) == 1
+
+        hash_info = stage.outs[0].hash_info
+        meta = stage.outs[0].meta
+        assert hash_info.name == "md5"
+        assert hash_info.value == "acbd18db4cc2f85cedef654fccc4a4d8"
+        assert (remote / "ac" / "bd18db4cc2f85cedef654fccc4a4d8").read_text() == "foo"
+        assert meta.size == len("foo")
+
+    def test_import_url_to_remote_file(self, tmp_dir, dvc, workspace, remote):
+        workspace.gen("foo", "foo")
+
+        url = "remote://workspace/foo"
+        stage = dvc.imp_url(url, to_remote=True)
+
+        assert stage.deps[0].hash_info.value is not None
+        assert not (tmp_dir / "foo").exists()
+        assert (tmp_dir / "foo.dvc").exists()
+
+        assert len(stage.deps) == 1
+        assert stage.deps[0].def_path == url
+        assert len(stage.outs) == 1
+
+        hash_info = stage.outs[0].hash_info
+        assert hash_info.name == "md5"
+        assert hash_info.value == "acbd18db4cc2f85cedef654fccc4a4d8"
+        assert (remote / "ac" / "bd18db4cc2f85cedef654fccc4a4d8").read_text() == "foo"
+        assert stage.outs[0].meta.size == len("foo")
+
+    def test_import_url_to_remote_dir(self, tmp_dir, dvc, workspace, remote):
+        import json
+
+        workspace.gen(
+            {
+                "data": {
+                    "foo": "foo",
+                    "bar": "bar",
+                    "sub_dir": {"baz": "sub_dir/baz"},
+                }
+            }
+        )
+
+        url = "remote://workspace/data"
+        stage = dvc.imp_url(url, to_remote=True)
+
+        assert not (tmp_dir / "data").exists()
+        assert (tmp_dir / "data.dvc").exists()
+
+        assert len(stage.deps) == 1
+        assert stage.deps[0].def_path == url
+        assert len(stage.outs) == 1
+
+        hash_info = stage.outs[0].hash_info
+        assert hash_info.name == "md5"
+        assert hash_info.value == "55d05978954d1b2cd7b06aedda9b9e43.dir"
+        file_parts = json.loads(
+            (remote / "55" / "d05978954d1b2cd7b06aedda9b9e43.dir").read_text()
+        )
+
+        assert len(file_parts) == 3
+        assert {file_part["relpath"] for file_part in file_parts} == {
+            "foo",
+            "bar",
+            "sub_dir/baz",
+        }
+
+        for file_part in file_parts:
+            md5 = file_part["md5"]
+            assert (remote / md5[:2] / md5[2:]).read_text() == file_part["relpath"]
diff --git a/tests/func/test_add.py b/tests/func/test_add.py
--- a/tests/func/test_add.py
+++ b/tests/func/test_add.py
@@ -898,26 +898,6 @@ def test_add_long_fname(tmp_dir, dvc):
     assert (tmp_dir / "data").read_text() == {name: "foo"}
 
 
-def test_add_to_remote(tmp_dir, dvc, remote, workspace):
-    workspace.gen("foo", "foo")
-
-    url = "remote://workspace/foo"
-    [stage] = dvc.add(url, to_remote=True)
-
-    assert not (tmp_dir / "foo").exists()
-    assert (tmp_dir / "foo.dvc").exists()
-
-    assert len(stage.deps) == 0
-    assert len(stage.outs) == 1
-
-    hash_info = stage.outs[0].hash_info
-    meta = stage.outs[0].meta
-    with open(remote.oid_to_path(hash_info.value), encoding="utf-8") as stream:
-        assert stream.read() == "foo"
-
-    assert meta.size == len("foo")
-
-
 def test_add_to_remote_absolute(tmp_dir, make_tmp_dir, dvc, remote):
     tmp_abs_dir = make_tmp_dir("abs")
     tmp_foo = tmp_abs_dir / "foo"
diff --git a/tests/func/test_import_url.py b/tests/func/test_import_url.py
--- a/tests/func/test_import_url.py
+++ b/tests/func/test_import_url.py
@@ -1,4 +1,3 @@
-import json
 import os
 import textwrap
 from uuid import uuid4
@@ -171,63 +170,6 @@ def test_import_url_preserve_fields(tmp_dir, dvc):
     )
 
 
-def test_import_url_to_remote_single_file(tmp_dir, dvc, workspace, local_remote):
-    workspace.gen("foo", "foo")
-
-    url = "remote://workspace/foo"
-    stage = dvc.imp_url(url, to_remote=True)
-
-    assert stage.deps[0].hash_info.value is not None
-    assert not (tmp_dir / "foo").exists()
-    assert (tmp_dir / "foo.dvc").exists()
-
-    assert len(stage.deps) == 1
-    assert stage.deps[0].def_path == url
-    assert len(stage.outs) == 1
-
-    hash_info = stage.outs[0].hash_info
-    with open(local_remote.oid_to_path(hash_info.value), encoding="utf-8") as fobj:
-        assert fobj.read() == "foo"
-    assert stage.outs[0].meta.size == len("foo")
-
-
-def test_import_url_to_remote_directory(tmp_dir, dvc, workspace, local_remote):
-    workspace.gen(
-        {
-            "data": {
-                "foo": "foo",
-                "bar": "bar",
-                "sub_dir": {"baz": "sub_dir/baz"},
-            }
-        }
-    )
-
-    url = "remote://workspace/data"
-    stage = dvc.imp_url(url, to_remote=True)
-
-    assert not (tmp_dir / "data").exists()
-    assert (tmp_dir / "data.dvc").exists()
-
-    assert len(stage.deps) == 1
-    assert stage.deps[0].def_path == url
-    assert len(stage.outs) == 1
-
-    hash_info = stage.outs[0].hash_info
-    with open(local_remote.oid_to_path(hash_info.value), encoding="utf-8") as stream:
-        file_parts = json.load(stream)
-
-    assert len(file_parts) == 3
-    assert {file_part["relpath"] for file_part in file_parts} == {
-        "foo",
-        "bar",
-        "sub_dir/baz",
-    }
-
-    for file_part in file_parts:
-        with open(local_remote.oid_to_path(file_part["md5"]), encoding="utf-8") as fobj:
-            assert fobj.read() == file_part["relpath"]
-
-
 def test_import_url_to_remote_absolute(tmp_dir, make_tmp_dir, dvc, local_remote):
     tmp_abs_dir = make_tmp_dir("abs")
     tmp_foo = tmp_abs_dir / "foo"