Repository: iterative/dvc

Issue:
s3: performance implications of s3fs
This issue is for tracking the progress related s3fs/aiobotocore issues for the performance degradations that we might experience.

- [x] [exists() on non-existent files](https://github.com/dask/s3fs/issues/470)
- [x] [HeadObject calls are slower than the regular boto](https://github.com/aio-libs/aiobotocore/issues/866)


Discussion:
import time itself is not something that effects greatly, but still there is `~0.3` seconds (on HDD, might be different for a SSD user) startup overhead;

## `import boto3`
![image](https://user-images.githubusercontent.com/47358913/117134713-96273a00-adae-11eb-8816-9f85a7e7ea46.png)

## `import aiobotocore, boto3`
![image](https://user-images.githubusercontent.com/47358913/117134781-b35c0880-adae-11eb-8c99-f3be96b1bb1b.png)

One important thing after I noticed working with the dataset-private-registry is that the listing take some time to happen since fsspec doesn't have iterators (not for ls/find) and I feel like this is bad for UX. For small sets, it is not that noticeable but if you have one million files then it is obvious that the dvc is blocked and there is no reactivity of any sort.  

Regarding that, there is not much we can do on the dvc side though if we consider that the `find()` is mostly used for multi-level directories (like our remote cache) then we can do a first level `ls()` and then call `find()` with the each sub level. This definitely increases the responsiveness, and at the same time if we parallelize these calls I think it might be faster than the original `walk_files()` since now it is making N listings concurrently. CC: @efiop @shcheklein 

Just to mention some numbers with a similar implementation (these are obtained through a very WIP implementation, so take these with a pinch of salt):
- `walk_files` on the dataset-private-registry/ => 700-720 files/second with s3fs master
- `walk_files` on the dataset-private-registry/ => 870 files/second with the original implementation 
- `walk_files` on the dataset-private-registry/ => 4446 files/second when `ls()` on the first level directories + `find()` on the each sublevel + pararellized with threading

```py
    def walk_files(self, path_info, **kwargs):
        path = self._with_bucket(path_info)
        top_level_dirs = self.fs.ls(path)

        with ThreadPoolExecutor(max_workers=self.jobs) as executor:
            for results in executor.map(self.fs.find, top_level_dirs):
                for result in self._strip_buckets(results):
                    yield path_info.replace(path=result)
```

Of course, this optimization might slow you down when your cache have like 100-200 files in total separate across different prefixes, so we also have to consider that :hmm: 
@isidentical My guess is that iterators should be there, I suppose they were simply not implemented yet. We should be able to implement them for us and possibly send the patch back to upstream, let's create an issue and ask about it.
> Of course, this optimization might slow you down when your cache have like 100-200 files in total separate across different prefixes, so we also have to consider that :hmm:

We have a special optimization for that, where we try to list the whole remote if we estimate that it is smaller than a certain size. Though you've received pretty interesting results with `ls`, but that is not really `walk_files` anymore, more of a listing by prefix that we use for estimation.
> but that is not really walk_files anymore, more of a listing by prefix that we use for estimation.

What do you mean by that? 
@isidentical Ah, sorry, I'm wrong. Didn't notice the `find` in the executor. Still, it is rather tailored to handle our specific remote dir structure, so it is no longer a general `walk_files` (that is expected to return an iterator) and we could use such a workaround directly where needed. The results are very interesting though.
I started writing a proposal as an issue to the fsspec repo with describing the potential APIs and how would they work
we have narrowed down the issue of when the region is not specified, the underlying boto implementation doing extra work to locate the bucket region which resulted with the aiohttp connection being halted down. so we end up paying the creation of the connection for each request, which is making stuff terrible. 

https://github.com/aio-libs/aiobotocore/issues/866#issuecomment-848185110

we might implement some sort of bucket-region caching, though it is just basically a theory (didn't had time to actually test it out).  
See: https://github.com/dask/s3fs/pull/495

Resolving patch:
diff --git a/dvc/config_schema.py b/dvc/config_schema.py
--- a/dvc/config_schema.py
+++ b/dvc/config_schema.py
@@ -154,6 +154,7 @@ class RelPath(str):
                     "grant_read_acp": str,
                     "grant_write_acp": str,
                     "grant_full_control": str,
+                    "cache_regions": bool,
                     **REMOTE_COMMON,
                 },
                 "gs": {
diff --git a/dvc/fs/s3.py b/dvc/fs/s3.py
--- a/dvc/fs/s3.py
+++ b/dvc/fs/s3.py
@@ -68,12 +68,12 @@ def _load_aws_config_file(self, profile):
         self._transfer_config = None
         config_path = os.environ.get("AWS_CONFIG_FILE", _AWS_CONFIG_PATH)
         if not os.path.exists(config_path):
-            return None
+            return {}
 
         config = load_config(config_path)
         profile_config = config["profiles"].get(profile or "default")
         if not profile_config:
-            return None
+            return {}
 
         s3_config = profile_config.get("s3", {})
         return self._split_s3_config(s3_config)
@@ -123,6 +123,14 @@ def _prepare_credentials(self, **config):
         if shared_creds:
             os.environ.setdefault("AWS_SHARED_CREDENTIALS_FILE", shared_creds)
 
+        if (
+            client["region_name"] is None
+            and session_config["s3"].get("region_name") is None
+            and os.getenv("AWS_REGION") is None
+        ):
+            # Enable bucket region caching
+            login_info["cache_regions"] = config.get("cache_regions", True)
+
         config_path = config.get("configpath")
         if config_path:
             os.environ.setdefault("AWS_CONFIG_FILE", config_path)


Test patch: