Repository: iterative/dvc

Issue:
progress: inconsistent nested format when no main bar
## Overview
When storage is local (copy_file) on `dvc pull` first pbar is displayed as normal one ([self.pos = 0](https://github.com/iterative/dvc/blob/f05e6e0529ae147e82e3c392951f03c01b6b0c64/dvc/progress.py#L96)) and all others are nested

## Description
When we do remote.download for all scheme [except local](https://github.com/iterative/dvc/blob/f05e6e0529ae147e82e3c392951f03c01b6b0c64/dvc/remote/base.py#L585) there are [main bar](https://github.com/iterative/dvc/blob/f05e6e0529ae147e82e3c392951f03c01b6b0c64/dvc/remote/base.py#L619)
So only first pbar is normal, and all others are nested
```
$ dvc push grand_files
 51% grand_files/1/file-1|█████████████████████████████████████████████                                           |164M/320M [00:14<00:07,    21.1MB/s]
 38%|███▊      |grand_files/1/another-file-1                                                                       122M/320M [00:14<00:18,    11.3MB/s]
 66%|██████▋   |grand_files/1/second-file-1                                                                        212M/320M [00:14<00:08,    13.0MB/s]
 35%|███▌      |grand_files/1/third-file
```

but expects that all pbars are as 1-pbar

Relates other commands as well:
```
$ dvc import ../repo grand_files
Importing 'grand_files (../repo)' -> 'grand_files'                      
 56% grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/second-file-1|██████████████████████████████                        |178M/320M [00:03<00:06,    23.8MB/s]
 54%|█████▍    |grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/another-file-1                                            174M/320M [00:03<00:06,    23.1MB/s]
 57%|█████▋    |grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/file-1                                                    181M/320M [00:03<00:06,    23.0MB/s]
 54%|█████▍    |grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/third-file-1     
```
```
$ dvc pull -j3 grand_files
 78% ../../../../../../../tmp/tmp3zld67wsdvc-erepo/grand_files/1/another-file-1|██████████████████████████▍       |249M/320M [00:04<00:03,    21.7MB/s]
 80%|████████  |../../../../../../../tmp/tmp3zld67wsdvc-erepo/grand_files/1/file-1                                 256M/320M [00:05<00:04,    15.2MB/s]
 93%|█████████▎|../../../../../../../tmp/tmp3zld67wsdvc-erepo/grand_files/1/second-file-1                          297M/320M [00:04<00:01,    21.5MB/s]
```
Relates: 
 - https://github.com/iterative/dvc/issues/2905

**Please provide information about your setup**
```
$ dvc --version
0.87.0+f05e6e
```


progress: inconsistent nested format when no main bar
## Overview
When storage is local (copy_file) on `dvc pull` first pbar is displayed as normal one ([self.pos = 0](https://github.com/iterative/dvc/blob/f05e6e0529ae147e82e3c392951f03c01b6b0c64/dvc/progress.py#L96)) and all others are nested

## Description
When we do remote.download for all scheme [except local](https://github.com/iterative/dvc/blob/f05e6e0529ae147e82e3c392951f03c01b6b0c64/dvc/remote/base.py#L585) there are [main bar](https://github.com/iterative/dvc/blob/f05e6e0529ae147e82e3c392951f03c01b6b0c64/dvc/remote/base.py#L619)
So only first pbar is normal, and all others are nested
```
$ dvc push grand_files
 51% grand_files/1/file-1|█████████████████████████████████████████████                                           |164M/320M [00:14<00:07,    21.1MB/s]
 38%|███▊      |grand_files/1/another-file-1                                                                       122M/320M [00:14<00:18,    11.3MB/s]
 66%|██████▋   |grand_files/1/second-file-1                                                                        212M/320M [00:14<00:08,    13.0MB/s]
 35%|███▌      |grand_files/1/third-file
```

but expects that all pbars are as 1-pbar

Relates other commands as well:
```
$ dvc import ../repo grand_files
Importing 'grand_files (../repo)' -> 'grand_files'                      
 56% grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/second-file-1|██████████████████████████████                        |178M/320M [00:03<00:06,    23.8MB/s]
 54%|█████▍    |grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/another-file-1                                            174M/320M [00:03<00:06,    23.1MB/s]
 57%|█████▋    |grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/file-1                                                    181M/320M [00:03<00:06,    23.0MB/s]
 54%|█████▍    |grand_files.9vFFJ8yBMA9QLmwq5PL7xj.tmp/1/third-file-1     
```
```
$ dvc pull -j3 grand_files
 78% ../../../../../../../tmp/tmp3zld67wsdvc-erepo/grand_files/1/another-file-1|██████████████████████████▍       |249M/320M [00:04<00:03,    21.7MB/s]
 80%|████████  |../../../../../../../tmp/tmp3zld67wsdvc-erepo/grand_files/1/file-1                                 256M/320M [00:05<00:04,    15.2MB/s]
 93%|█████████▎|../../../../../../../tmp/tmp3zld67wsdvc-erepo/grand_files/1/second-file-1                          297M/320M [00:04<00:01,    21.5MB/s]
```
Relates: 
 - https://github.com/iterative/dvc/issues/2905

**Please provide information about your setup**
```
$ dvc --version
0.87.0+f05e6e
```




Discussion:
CC @casperdcl 
@JIoJIaJIu not sure what the problem is - are you experiencing any unexpected behaviour?
Ah I see it's that the format of the first bar should be the same as the others (nested)...
> Ah I see it's that the format of the first bar should be the same as the others (nested)...

I think we sometimes need to have nested bars and 1 main pbar, for example when [compute cache](https://github.com/iterative/dvc/blob/aef3dfba136f09ac3669c460d6e724cbb2e67a09/dvc/remote/base.py#L178) we would like to see 1 main pbar - `Computing file/dir..` and then large files are displayed as nested

I see the problem here is how we determine [main bar](https://github.com/iterative/dvc/blob/498ee509da49e23a8588893b15328318e73f15ae/dvc/progress.py#L96)
Also I've noticed that it's not guaranteed that main bar gets `0 position`

> I see the problem here is how we determine [main bar](https://github.com/iterative/dvc/blob/498ee509da49e23a8588893b15328318e73f15ae/dvc/progress.py#L96)

It's best to add a main bar in cases where there currently is no main bar (that would fix this issue)

> Also I've noticed that it's not guaranteed that main bar gets `0 position`

Really? When?
> Really? When?

with [the script](https://gist.github.com/JIoJIaJIu/05f3b72982373771da88f313bb5cad9f) you may reproduce the behavior

```
$ python pbar.py 10 --slow
  0% File#724.80|                         |0/100 [00:00<?,       ?it/s]
  3%|▎         |File#120.64            3/100 [00:02<01:16,    1.27it/s]
  0%|          |File#755.66                0/100 [00:00<?,       ?it/s]
  0%|          |File#756.29                0/100 [00:00<?,       ?it/s]
  0%|          |File#292.52                0/100 [00:00<?,       ?it/s]
  3%|▎         |File#117.65            3/100 [00:02<01:14,    1.31it/s]
  2%|▏         |File#164.46            2/100 [00:01<01:20,    1.21it/s]
  0%|          |File#340.28                0/100 [00:00<?,       ?it/s]
  0%|          |Downloading directory       0/10 [00:00<?,    ?Files/s]
  0%|          |File#377.57                0/100 [00:00<?,       ?it/s]
  0%|          |File#463.49                0/100 [00:00<?,       ?it/s]
```
we expect that `Downloading directory` appears on the top
That's a different issue... Opened #3521. Will fix separately.
CC @casperdcl 
@JIoJIaJIu not sure what the problem is - are you experiencing any unexpected behaviour?
Ah I see it's that the format of the first bar should be the same as the others (nested)...
> Ah I see it's that the format of the first bar should be the same as the others (nested)...

I think we sometimes need to have nested bars and 1 main pbar, for example when [compute cache](https://github.com/iterative/dvc/blob/aef3dfba136f09ac3669c460d6e724cbb2e67a09/dvc/remote/base.py#L178) we would like to see 1 main pbar - `Computing file/dir..` and then large files are displayed as nested

I see the problem here is how we determine [main bar](https://github.com/iterative/dvc/blob/498ee509da49e23a8588893b15328318e73f15ae/dvc/progress.py#L96)
Also I've noticed that it's not guaranteed that main bar gets `0 position`

> I see the problem here is how we determine [main bar](https://github.com/iterative/dvc/blob/498ee509da49e23a8588893b15328318e73f15ae/dvc/progress.py#L96)

It's best to add a main bar in cases where there currently is no main bar (that would fix this issue)

> Also I've noticed that it's not guaranteed that main bar gets `0 position`

Really? When?
> Really? When?

with [the script](https://gist.github.com/JIoJIaJIu/05f3b72982373771da88f313bb5cad9f) you may reproduce the behavior

```
$ python pbar.py 10 --slow
  0% File#724.80|                         |0/100 [00:00<?,       ?it/s]
  3%|▎         |File#120.64            3/100 [00:02<01:16,    1.27it/s]
  0%|          |File#755.66                0/100 [00:00<?,       ?it/s]
  0%|          |File#756.29                0/100 [00:00<?,       ?it/s]
  0%|          |File#292.52                0/100 [00:00<?,       ?it/s]
  3%|▎         |File#117.65            3/100 [00:02<01:14,    1.31it/s]
  2%|▏         |File#164.46            2/100 [00:01<01:20,    1.21it/s]
  0%|          |File#340.28                0/100 [00:00<?,       ?it/s]
  0%|          |Downloading directory       0/10 [00:00<?,    ?Files/s]
  0%|          |File#377.57                0/100 [00:00<?,       ?it/s]
  0%|          |File#463.49                0/100 [00:00<?,       ?it/s]
```
we expect that `Downloading directory` appears on the top
That's a different issue... Opened #3521. Will fix separately.

Resolving patch:
diff --git a/dvc/remote/local.py b/dvc/remote/local.py
--- a/dvc/remote/local.py
+++ b/dvc/remote/local.py
@@ -448,9 +448,11 @@ def _process(
                 file_mode=self._file_mode,
             )
             status = STATUS_DELETED
+            desc = "Downloading"
         else:
             func = remote.upload
             status = STATUS_NEW
+            desc = "Uploading"
 
         if jobs is None:
             jobs = remote.JOBS
@@ -466,42 +468,45 @@ def _process(
         dir_plans = self._get_plans(download, remote, dir_status, status)
         file_plans = self._get_plans(download, remote, file_status, status)
 
-        if len(dir_plans[0]) + len(file_plans[0]) == 0:
+        total = len(dir_plans[0]) + len(file_plans[0])
+        if total == 0:
             return 0
 
-        with ThreadPoolExecutor(max_workers=jobs) as executor:
-            if download:
-                fails = sum(executor.map(func, *dir_plans))
-                fails += sum(executor.map(func, *file_plans))
-            else:
-                # for uploads, push files first, and any .dir files last
-
-                file_futures = {}
-                for from_info, to_info, name in zip(*file_plans):
-                    file_futures[to_info] = executor.submit(
-                        func, from_info, to_info, name
-                    )
-                dir_futures = {}
-                for from_info, to_info, name in zip(*dir_plans):
-                    wait_futures = {
-                        future
-                        for file_path, future in file_futures.items()
-                        if file_path in dir_paths[to_info]
-                    }
-                    dir_futures[to_info] = executor.submit(
-                        self._dir_upload,
-                        func,
-                        wait_futures,
-                        from_info,
-                        to_info,
-                        name,
-                    )
-                fails = sum(
-                    future.result()
-                    for future in concat(
-                        file_futures.values(), dir_futures.values()
+        with Tqdm(total=total, unit="file", desc=desc) as pbar:
+            func = pbar.wrap_fn(func)
+            with ThreadPoolExecutor(max_workers=jobs) as executor:
+                if download:
+                    fails = sum(executor.map(func, *dir_plans))
+                    fails += sum(executor.map(func, *file_plans))
+                else:
+                    # for uploads, push files first, and any .dir files last
+
+                    file_futures = {}
+                    for from_info, to_info, name in zip(*file_plans):
+                        file_futures[to_info] = executor.submit(
+                            func, from_info, to_info, name
+                        )
+                    dir_futures = {}
+                    for from_info, to_info, name in zip(*dir_plans):
+                        wait_futures = {
+                            future
+                            for file_path, future in file_futures.items()
+                            if file_path in dir_paths[to_info]
+                        }
+                        dir_futures[to_info] = executor.submit(
+                            self._dir_upload,
+                            func,
+                            wait_futures,
+                            from_info,
+                            to_info,
+                            name,
+                        )
+                    fails = sum(
+                        future.result()
+                        for future in concat(
+                            file_futures.values(), dir_futures.values()
+                        )
                     )
-                )
 
         if fails:
             if download:


Test patch: