Repository: iterative/dvc

Issue:
pull: implement a filesytstem.download_file protocol for skipping isdir() check
When pulling caches from the remote storage, we only pull files, not directories. But each `download()` call made by pull (for each cache file we want to retrive) costs an extra `isdir()` call which is not needed at all (considering that pull will always fetch files, not directories). 
https://github.com/iterative/dvc/blob/a3a55339abf133e8073a5c4220e7d61749b5f16f/dvc/remote/base.py#L311-L314
https://github.com/iterative/dvc/blob/a3a55339abf133e8073a5c4220e7d61749b5f16f/dvc/fs/base.py#L276-L280

We can get a pretty straightforward optimization by just implementing a `BaseFileSystem.download_file` method which would skip this check and go directly to the `download_file`. 

This API is fsspec-compatible, as it offers a [`get()`](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.get) API for getting a directory or a file recursively (just like our `download()`) and it offers a single [`get_file()`](https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.get_file) API which fetches a single file from the remote. 

```sh
mkdir /tmp/test_dvc
pushd /tmp/test_dvc
git init > /dev/null
dvc init > /dev/null
dvc remote add -d default s3://dvc-temp/s3fs-migration/h5-s3fs/
python ~/tooling/create_dataset.py dataset1 10000
dvc add dataset1 > /dev/null
time dvc push dataset1 > /dev/null
time dvc status -c > /dev/null
rm -rf dataset1 .dvc/cache
time dvc status -c > /dev/null
time dvc pull dataset1 > /dev/null
popd
```

The performance improvement for the script above (creates 10k files and pulls them) was about %30 (8.30min with download_file, 11.30min on the master). The only concern of mine is whether this operation is going to change in the future ODB systems, but even then I think this API might be useful.




Discussion:
CC: @efiop

Resolving patch:
diff --git a/dvc/fs/base.py b/dvc/fs/base.py
--- a/dvc/fs/base.py
+++ b/dvc/fs/base.py
@@ -1,6 +1,6 @@
 import logging
 from concurrent.futures import ThreadPoolExecutor, as_completed
-from functools import partial
+from functools import partial, partialmethod
 from multiprocessing import cpu_count
 from typing import Any, ClassVar, Dict, FrozenSet, Optional
 
@@ -258,6 +258,7 @@ def download(
         name=None,
         no_progress_bar=False,
         jobs=None,
+        _only_file=False,
         **kwargs,
     ):
         if not hasattr(self, "_download"):
@@ -273,12 +274,14 @@ def download(
         if to_info.scheme != "local":
             raise NotImplementedError
 
-        if self.isdir(from_info):
+        if not _only_file and self.isdir(from_info):
             return self._download_dir(
                 from_info, to_info, name, no_progress_bar, jobs, **kwargs,
             )
         return self._download_file(from_info, to_info, name, no_progress_bar,)
 
+    download_file = partialmethod(download, _only_file=True)
+
     def _download_dir(
         self, from_info, to_info, name, no_progress_bar, jobs, **kwargs,
     ):
diff --git a/dvc/remote/base.py b/dvc/remote/base.py
--- a/dvc/remote/base.py
+++ b/dvc/remote/base.py
@@ -309,7 +309,7 @@ def _process(
         )
 
         if download:
-            func = _log_exceptions(self.fs.download, "download")
+            func = _log_exceptions(self.fs.download_file, "download")
             status = STATUS_DELETED
             desc = "Downloading"
         else:


Test patch:
diff --git a/tests/func/test_external_repo.py b/tests/func/test_external_repo.py
--- a/tests/func/test_external_repo.py
+++ b/tests/func/test_external_repo.py
@@ -52,7 +52,7 @@ def test_cache_reused(erepo_dir, mocker, local_cloud):
         erepo_dir.dvc_gen("file", "text", commit="add file")
     erepo_dir.dvc.push()
 
-    download_spy = mocker.spy(LocalFileSystem, "download")
+    download_spy = mocker.spy(LocalFileSystem, "download_file")
 
     # Use URL to prevent any fishy optimizations
     url = f"file://{erepo_dir}"