Repository: iterative/dvc

Issue:
dvc push: Unexpected error when pushing to Google Cloud storage or S3
# Bug Report

dvc push: "Unexpected error" when pushing to Google Cloud storage or S3

### Reproduce
```
dvc init
dvc remote add -d s3 s3://bucket  # or gcs gs://bucket
dvc import-url https://data.dvc.org/get-started/data.xml
dvc push -v
```
output (s3):

```
2023-12-27 19:56:42,605 DEBUG: v3.36.1 (pip), CPython 3.9.18 on Linux-5.15.139-93.147.amzn2.x86_64-x86_64-with-glibc2.26
2023-12-27 19:56:42,605 DEBUG: command: /path/bin/dvc push -v
Collecting                                                                                                                           |0.00 [00:00,    ?entry/s]
Pushing                                                                                                                              |0.00 [00:00,     ?file/s]
Collecting my.bucket/key on s3                                                                                                |3.00 [00:00, 4.84entry/s]
2023-12-27 19:56:43,676 ERROR: unexpected error
Traceback (most recent call last):
  File "/path/lib/python3.9/site-packages/dvc/cli/__init__.py", line 211, in main
    ret = cmd.do_run()
  File "/path/lib/python3.9/site-packages/dvc/cli/command.py", line 27, in do_run
    return self.run()
  File "/path/lib/python3.9/site-packages/dvc/commands/data_sync.py", line 64, in run
    processed_files_count = self.repo.push(
  File "/path/lib/python3.9/site-packages/dvc/repo/__init__.py", line 65, in wrapper
    return f(repo, *args, **kwargs)
  File "/path/lib/python3.9/site-packages/dvc/repo/push.py", line 144, in push
    push_transferred, push_failed = ipush(
  File "/path/lib/python3.9/site-packages/dvc_data/index/push.py", line 101, in push
    old = build(data.path, data.fs)
  File "/path/lib/python3.9/site-packages/dvc_data/index/build.py", line 90, in build
    for entry in build_entries(path, fs, ignore=ignore):
  File "/path/lib/python3.9/site-packages/dvc_data/index/build.py", line 55, in build_entries
    walk_iter = fs.walk(path, detail=detail)
  File "/path/lib/python3.9/site-packages/dvc_http/__init__.py", line 162, in walk
    raise NotImplementedError
NotImplementedError

2023-12-27 19:56:43,752 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-12-27 19:56:43,755 DEBUG: Removing '/path/.MHVNkr3eAijD7Q5aau3NRK.tmp'
2023-12-27 19:56:43,755 DEBUG: Removing '/path/.MHVNkr3eAijD7Q5aau3NRK.tmp'
2023-12-27 19:56:43,757 DEBUG: Removing '/path/.MHVNkr3eAijD7Q5aau3NRK.tmp'
2023-12-27 19:56:43,757 DEBUG: Removing '/path/bkw-9036/.dvc/cache/files/md5/.mnnSioPUuXvRUCqUV2ug87.tmp'
2023-12-27 19:56:43,777 DEBUG: Version info for developers:
DVC version: 3.36.1 (pip)
-------------------------
Platform: Python 3.9.18 on Linux-5.15.139-93.147.amzn2.x86_64-x86_64-with-glibc2.26
Subprojects:
	dvc_data = 3.3.0
	dvc_objects = 3.0.0
	dvc_render = 1.0.0
	dvc_task = 0.3.0
	scmrepo = 2.0.2
Supports:
	gs (gcsfs = 2023.12.2.post1),
	http (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2023.12.2, boto3 = 1.33.13)
Config:
	Global: /home/jdt/.config/dvc
	System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme1n1p1
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/nvme1n1p1
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/9d9135fb99d9d827364c4dc5a42cdc60

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-12-27 19:56:43,781 DEBUG: Analytics is enabled.
2023-12-27 19:56:43,860 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmpccxiwrmd', '-v']
2023-12-27 19:56:43,871 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmpccxiwrmd', '-v'] with pid 22406
```

output (gcs):

```
2023-12-27 19:47:22,768 DEBUG: v3.36.1 (pip), CPython 3.9.18 on Linux-5.15.139-93.147.amzn2.x86_64-x86_64-with-glibc2.26
2023-12-27 19:47:22,769 DEBUG: command: /path/bin/dvc push -v
Collecting                                                                                                                           |0.00 [00:00,    ?entry/s]
Pushing                                                                                                                              |0.00 [00:00,     ?file/s]
Collecting bucket/path on gs                                                                                          |3.00 [00:01, 2.84entry/s]
2023-12-27 19:47:24,328 ERROR: unexpected error
Traceback (most recent call last):
  File "/path/lib/python3.9/site-packages/dvc/cli/__init__.py", line 211, in main
    ret = cmd.do_run()
  File "/path/lib/python3.9/site-packages/dvc/cli/command.py", line 27, in do_run
    return self.run()
  File "/path/lib/python3.9/site-packages/dvc/commands/data_sync.py", line 64, in run
    processed_files_count = self.repo.push(
  File "/path/lib/python3.9/site-packages/dvc/repo/__init__.py", line 65, in wrapper
    return f(repo, *args, **kwargs)
  File "/path/lib/python3.9/site-packages/dvc/repo/push.py", line 144, in push
    push_transferred, push_failed = ipush(
  File "/path/lib/python3.9/site-packages/dvc_data/index/push.py", line 101, in push
    old = build(data.path, data.fs)
  File "/path/lib/python3.9/site-packages/dvc_data/index/build.py", line 90, in build
    for entry in build_entries(path, fs, ignore=ignore):
  File "/path/lib/python3.9/site-packages/dvc_data/index/build.py", line 55, in build_entries
    walk_iter = fs.walk(path, detail=detail)
  File "/path/lib/python3.9/site-packages/dvc_http/__init__.py", line 162, in walk
    raise NotImplementedError
NotImplementedError

2023-12-27 19:47:24,370 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-12-27 19:47:24,371 DEBUG: Removing '/path/.fJ4uXqQznknWmbrzzUTXLQ.tmp'
2023-12-27 19:47:24,371 DEBUG: Removing '/path/.fJ4uXqQznknWmbrzzUTXLQ.tmp'
2023-12-27 19:47:24,371 DEBUG: Removing '/path/.fJ4uXqQznknWmbrzzUTXLQ.tmp'
2023-12-27 19:47:24,371 DEBUG: Removing '/path/bkw-9036/.dvc/cache/files/md5/.M6iwnJkjQgKzg54kN6chVi.tmp'
2023-12-27 19:47:24,377 DEBUG: Version info for developers:
DVC version: 3.36.1 (pip)
-------------------------
Platform: Python 3.9.18 on Linux-5.15.139-93.147.amzn2.x86_64-x86_64-with-glibc2.26
Subprojects:
	dvc_data = 3.3.0
	dvc_objects = 3.0.0
	dvc_render = 1.0.0
	dvc_task = 0.3.0
	scmrepo = 2.0.2
Supports:
	gs (gcsfs = 2023.12.2.post1),
	http (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.9.1, aiohttp-retry = 2.8.3)
Config:
	Global: /home/jdt/.config/dvc
	System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme1n1p1
Caches: local
Remotes: gs
Workspace directory: ext4 on /dev/nvme1n1p1
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/9d9135fb99d9d827364c4dc5a42cdc60

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-12-27 19:47:24,379 DEBUG: Analytics is enabled.
2023-12-27 19:47:24,445 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmpk_30nnlt', '-v']
2023-12-27 19:47:24,455 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmpk_30nnlt', '-v'] with pid 15755
```

### Expected

Successful push

### Environment information

<!--
This is required to ensure that we can reproduce the bug.
-->
```
DVC version: 3.36.1 (pip)
-------------------------
Platform: Python 3.9.18 on Linux-5.15.139-93.147.amzn2.x86_64-x86_64-with-glibc2.26
Subprojects:
	dvc_data = 3.3.0
	dvc_objects = 3.0.0
	dvc_render = 1.0.0
	dvc_task = 0.3.0
	scmrepo = 2.0.2
Supports:
	gs (gcsfs = 2023.12.2.post1),
	http (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2023.12.2, boto3 = 1.33.13)
Config:
	Global: /home/jdt/.config/dvc
	System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme1n1p1
Caches: local
Remotes: s3
Workspace directory: ext4 on /dev/nvme1n1p1
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/c9c73dbc105eb09a15137f49a60e6a5b
```

**Additional Information (if any):**



Discussion:
Looks like we are accidentally trying to push to that import-url's fs. Taking a look...
Ok, with index migration we need `storage.read_only` that we can use during index building to say that this remote is read-only. This is similar to odb-only `read_only` that we used to have in the past. Already preparing a fix.
Is it about read-only? Or, is it about traversable/non-traversable remotes? Our implementation of `http` is read+write.
@skshetry We are trying to push to http that we've import-url-ed from and which is not even set up as a remote, so the former.

Resolving patch:
diff --git a/dvc/repo/index.py b/dvc/repo/index.py
--- a/dvc/repo/index.py
+++ b/dvc/repo/index.py
@@ -221,7 +221,7 @@ def _load_storage_from_out(storage_map, key, out):
                     ),
                 )
             )
-        storage_map.add_remote(FileStorage(key, dep.fs, dep.fs_path))
+        storage_map.add_remote(FileStorage(key, dep.fs, dep.fs_path, read_only=True))
 
 
 class Index:


Test patch:
diff --git a/tests/func/test_repo_index.py b/tests/func/test_repo_index.py
--- a/tests/func/test_repo_index.py
+++ b/tests/func/test_repo_index.py
@@ -1,3 +1,4 @@
+import os
 from itertools import chain
 
 import pytest
@@ -338,3 +339,53 @@ def test_param_keys_top_level_params(tmp_dir, dvc):
     tmp_dir.gen("dvc.yaml", top_level_params)
     index = Index.from_repo(dvc)
     assert index.param_keys == {"repo": {("classifier", "custom_params_file.yaml")}}
+
+
+def test_data_index(tmp_dir, dvc, local_cloud, erepo_dir):
+    tmp_dir.dvc_gen(
+        {
+            "foo": b"foo",
+            "dir": {"bar": b"bar", "subdir": {"baz": b"baz"}},
+        }
+    )
+
+    with erepo_dir.chdir():
+        erepo_dir.dvc_gen("efoo", b"efoo", commit="create efoo")
+        erepo_dir.dvc_gen(
+            "edir",
+            {"ebar": b"ebar", "esubdir": {"ebaz": b"ebaz"}},
+            commit="create edir",
+        )
+
+    dvc.imp(os.fspath(erepo_dir), "efoo")
+    dvc.imp(os.fspath(erepo_dir), "edir")
+
+    local_cloud.gen("ifoo", b"ifoo")
+    local_cloud.gen("idir", {"ibar": b"ibar", "isubdir": {"ibaz": b"ibaz"}})
+
+    dvc.imp_url(str(local_cloud / "ifoo"))
+    dvc.imp_url(str(local_cloud / "idir"))
+
+    index = Index.from_repo(dvc)
+    assert index.data_keys == {
+        "local": set(),
+        "repo": {("dir",), ("edir",), ("efoo",), ("foo",), ("idir",), ("ifoo",)},
+    }
+
+    data = index.data["repo"]
+    assert set(data.keys()) == {
+        ("dir",),
+        ("edir",),
+        ("efoo",),
+        ("foo",),
+        ("idir",),
+        ("ifoo",),
+    }
+
+    assert not data.storage_map[("foo",)].remote
+    assert not data.storage_map[("dir",)].remote
+
+    assert data.storage_map[("efoo",)].remote.read_only
+    assert data.storage_map[("edir",)].remote.read_only
+    assert data.storage_map[("ifoo",)].remote.read_only
+    assert data.storage_map[("idir",)].remote.read_only