Repository: iterative/dvc

Issue:
s3: config load takes an excessive amount of time when initializing an s3 fs
Noticed this when benchmarking the recent ODB changes. For initializing a single s3 fs instance we spend almost an entire second (on my machine) just flattening and unflattening config dictionaries, which seems like a lot of overhead for just merging a couple of small dict configs

https://github.com/iterative/dvc/blob/771b89ddf2aaac4bf05afa907eb34f7ada013b1a/dvc/fs/s3.py#L130-L136

(should note that this is in 3rd party `flatten_dict` library)

![Screen Shot 2021-07-22 at 2 52 20 PM](https://user-images.githubusercontent.com/651988/126595386-c74c763a-a471-4396-94fc-d8b6eef3f784.png)


For perspective, both before and after the odb transfer PR, this is from a `dvc status -c` that takes a total of ~3 seconds to complete. So this config handling is 1/3 of the total execution time, and actual s3/network operations are most of the remaining 2/3. (I would have expected the network I/O to be almost the entire runtime)


Discussion:
@pmrowla I've tested with a basic config and these are the results I get (a handwritten version vs `flatten_dict`;
```py
    def _prepare_credentials(self, **config):
        ...
        def drop_redundant_keys(data):
            mirror = {}
            for key, value in data.items(): 
                if value is None:
                    continue
                elif isinstance(value, dict):
                    value = drop_redundant_keys(value)
                    if len(value) == 0:
                        continue

                mirror[key] = value
            return mirror

        import time, sys
        
        t0 = time.perf_counter()
        a = (drop_redundant_keys(login_info))
        t1 = time.perf_counter()
        
        sys.modules.pop('dvc.utils.flatten')
        t0_1 = time.perf_counter()
        from dvc.utils.flatten import flatten, unflatten
        a_1 = (unflatten(
            {
                key: value
                for key, value in flatten(login_info).items()
                if value is not None
            }
        ))
        t1_1 = time.perf_counter()
        
        print(a)
        print(a_1)
        print("fast: ", t1 - t0)
        print("slow: ", t1_1 - t0_1)
        exit()
```

```py
 $ time dvc status -c
{'key': '323', 'secret': '5234', 'use_ssl': True, 'client_kwargs': {'region_name': 'something'}}
{'key': '323', 'secret': '5234', 'use_ssl': True, 'client_kwargs': {'region_name': 'something'}}
fast:  4.650999926525401e-06
slow:  0.0022961140000461455
dvc status -c  0,96s user 0,09s system 99% cpu 1,058 total
```
I think the problem might be related to `flatten_dict` importing `pkg_resources`, which is notoriously known for being very bad for startup performance (since it traverses all your packages). @pmrowla can you try to run your benchmarks in an isolated virtual env, with lower number of packages? I was able to cut out 0.5 seconds via inlining `pkg_resources` imports. 

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -79,7 +79,7 @@ def run(self):
     "speedcopy>=2.0.1; python_version < '3.8' and sys_platform == 'win32'",
     "dataclasses==0.7; python_version < '3.7'",
     "importlib-metadata>=1.4; python_version < '3.8'",
-    "flatten_dict>=0.3.0,<1",
+    "flatten_dict>=0.4.1,<1",
     "tabulate>=0.8.7",
     "pygtrie>=2.3.2",
     "dpath>=2.0.1,<3",


Test patch: