Repository: iterative/dvc

Issue:
push: dir file not pushed because of progress bar error
# Bug Report

<!--
## Issue name

Issue names must follow the pattern `command: description` where the command is the dvc command that you are trying to run. The description should describe the consequence of the bug. 

Example: `repro: doesn't detect input changes`
-->

## Description

<!--
A clear and concise description of what the bug is.
-->

While pushing data to our azure remote, errors occur in updating the progress bar for some files ( I've so far seen 1-3 file uploads fail out of just 7 being pushed). This causes dvc to think that the file upload failed and so it decides not to upload the .dir file. Actually the file upload did succeed, so on a retry of the dvc push command only the missing .dir file is uploaded.

### Reproduce

<!--
Step list of how to reproduce the bug
-->

1. dvc repro dvc.yaml (produces a directory dir with nested directories and e.g. 8 files)
2. dvc add dir
3. dvc commit -f dir.dvc
4. dvc push dir.dvc

<!--
Example:

1. dvc init
5. Copy dataset.zip to the directory
6. dvc add dataset.zip
7. dvc run -d dataset.zip -o model ./train.sh
8. modify dataset.zip
9. dvc repro
-->

### Expected

<!--
A clear and concise description of what you expect to happen.
-->

All files in the directory structure are pushed to the remote cache, as well as the .dir file representing them.

### Environment information

<!--
This is required to ensure that we can reproduce the bug.
-->

**Output of `dvc doctor`:**

```console
$ dvc doctor
DVC version: 2.10.2 (pip)
---------------------------------
Platform: Python 3.9.12 on Linux-5.15.0-27-generic-x86_64-with-glibc2.35
Supports:
	azure (adlfs = 2022.4.0, knack = 0.9.0, azure-identity = 1.10.0),
	webhdfs (fsspec = 2022.3.0),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/nvme0n1p3
Caches: local
Remotes: azure
Workspace directory: ext4 on /dev/nvme0n1p3
Repo: dvc, git

```

**Additional Information (if any):**

<!--
Please check https://github.com/iterative/dvc/wiki/Debugging-DVC on ways to gather more information regarding the issue.

If applicable, please also provide a `--verbose` output of the command, eg: `dvc add --verbose`.
If the issue is regarding the performance, please attach the profiling information and the benchmark comparisons.
-->

First try with failed uploads:

```
2022-05-05 13:10:00,228 DEBUG: Preparing to transfer data from '/home/qcify/qcify/myagent/_work/1/s/models/.dvc/cache' to 'dvc/models_v3/some_new_folder'
2022-05-05 13:10:00,228 DEBUG: Preparing to collect status from 'dvc/models_v3/some_new_folder'
2022-05-05 13:10:00,228 DEBUG: Collecting status from 'dvc/models_v3/some_new_folder'
2022-05-05 13:10:00,229 DEBUG: Querying 3 hashes via object_exists
2022-05-05 13:10:01,080 DEBUG: Querying 1 hashes via object_exists
2022-05-05 13:10:03,613 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 13:10:03,614 DEBUG: Estimated remote size: 4096 files
2022-05-05 13:10:03,614 DEBUG: Querying '9' hashes via traverse
2022-05-05 13:10:06,704 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 13:10:06,707 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 13:10:06,714 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 13:10:06,724 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 13:10:06,726 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 13:10:06,727 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 13:10:06,734 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 13:10:06,736 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 13:10:06,755 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 13:10:06,755 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 13:10:06,757 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 13:10:06,773 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 13:10:49,294 DEBUG: Preparing to collect status from '/home/qcify/qcify/myagent/_work/1/s/models/.dvc/cache'
2022-05-05 13:10:49,294 DEBUG: Collecting status from '/home/qcify/qcify/myagent/_work/1/s/models/.dvc/cache'
2022-05-05 13:10:49,648 DEBUG: Uploading '/home/qcify/qcify/myagent/_work/1/s/models/.dvc/cache/43/0e4a44fc39b482c1344b2d2ca286f1' to 'dvc/models_v3/some_new_folder/43/0e4a44fc39b482c1344b2d2ca286f1'
2022-05-05 13:10:50,236 DEBUG: Uploading '/home/qcify/qcify/myagent/_work/1/s/models/.dvc/cache/f9/50bdf84b33643f71b7eefd93864d5d' to 'dvc/models_v3/some_new_folder/f9/50bdf84b33643f71b7eefd93864d5d'
2022-05-05 13:10:50,271 DEBUG: Uploading '/home/qcify/qcify/myagent/_work/1/s/models/.dvc/cache/bb/f019f9da1f79aa4bcd611925756a60' to 'dvc/models_v3/some_new_folder/bb/f019f9da1f79aa4bcd611925756a60'
2022-05-05 13:10:50,279 DEBUG: Uploading '/home/qcify/qcify/myagent/_work/1/s/models/.dvc/cache/0d/69161c5a8d2f5883675e21454c9f5b' to 'dvc/models_v3/some_new_folder/0d/69161c5a8d2f5883675e21454c9f5b'
2022-05-05 13:10:50,300 DEBUG: Uploading '/home/qcify/qcify/myagent/_work/1/s/models/.dvc/cache/5e/c95974eacea034d009e940062eba14' to 'dvc/models_v3/some_new_folder/5e/c95974eacea034d009e940062eba14'
2022-05-05 13:10:50,307 DEBUG: Uploading '/home/qcify/qcify/myagent/_work/1/s/models/.dvc/cache/72/58fd7add28137624d7214a146efac5' to 'dvc/models_v3/some_new_folder/72/58fd7add28137624d7214a146efac5'
2022-05-05 13:11:43,447 ERROR: failed to transfer 'md5: 430e4a44fc39b482c1344b2d2ca286f1' - unsupported operand type(s) for -: 'NoneType' and 'int'
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/data/transfer.py", line 25, in wrapper
    func(fs_path, *args, **kwargs)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/data/transfer.py", line 162, in func
    return dest.add(
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/objects/db.py", line 126, in add
    self._add_file(
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/objects/db.py", line 98, in _add_file
    return fs.utils.transfer(
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/fs/utils.py", line 96, in transfer
    _try_links(links, from_fs, from_path, to_fs, to_path)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/fs/utils.py", line 66, in _try_links
    return _copy(from_fs, from_path, to_fs, to_path)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/fs/utils.py", line 44, in _copy
    return to_fs.upload(from_path, to_path)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/fs/base.py", line 386, in upload
    return self.put_file(from_info, to_info, callback=cb, size=size)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/fs/base.py", line 329, in put_file
    self.fs.put_file(
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/fsspec/asyn.py", line 85, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/fsspec/asyn.py", line 65, in sync
    raise return_result
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/adlfs/spec.py", line 1574, in _put_file
    await bc.upload_blob(
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/tracing/decorator_async.py", line 74, in wrapper_use_tracer
    return await func(*args, **kwargs)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/storage/blob/aio/_blob_client_async.py", line 388, in upload_blob
    return await upload_block_blob(**options)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/storage/blob/aio/_upload_helpers.py", line 129, in upload_block_blob
    return await client.commit_block_list(
File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/storage/blob/_generated/aio/operations/_block_blob_operations.py", line 1006, in commit_block_list
    pipeline_response = await self._client._pipeline.run(request, stream=False, **kwargs)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 215, in run
    return await first_node.send(pipeline_request)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  [Previous line repeated 4 more times]
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/pipeline/policies/_redirect_async.py", line 64, in send
    response = await self.next.send(request)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/storage/blob/_shared/policies_async.py", line 111, in send
    response = await self.next.send(request)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/storage/blob/_shared/policies_async.py", line 90, in send
    response_callback(response)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/adlfs/spec.py", line 298, in wrapper
    callback.absolute_update(current)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/fs/_callback.py", line 122, in absolute_update
    self.progress_bar.update_to(value)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/progress.py", line 121, in update_to
    self.update(current - self.n)
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'
------------------------------------------------------------
2022-05-05 13:11:45,918 ERROR: failed to transfer 'md5: 7258fd7add28137624d7214a146efac5' - unsupported operand type(s) for -: 'NoneType' and 'int'
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/data/transfer.py", line 25, in wrapper
    func(fs_path, *args, **kwargs)
File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/data/transfer.py", line 162, in func
    return dest.add(
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/objects/db.py", line 126, in add
    self._add_file(
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/objects/db.py", line 98, in _add_file
    return fs.utils.transfer(
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/fs/utils.py", line 96, in transfer
    _try_links(links, from_fs, from_path, to_fs, to_path)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/fs/utils.py", line 66, in _try_links
    return _copy(from_fs, from_path, to_fs, to_path)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/fs/utils.py", line 44, in _copy
    return to_fs.upload(from_path, to_path)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/fs/base.py", line 386, in upload
    return self.put_file(from_info, to_info, callback=cb, size=size)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/fs/base.py", line 329, in put_file
    self.fs.put_file(
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/fsspec/asyn.py", line 85, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/fsspec/asyn.py", line 65, in sync
    raise return_result
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/adlfs/spec.py", line 1574, in _put_file
    await bc.upload_blob(
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/tracing/decorator_async.py", line 74, in wrapper_use_tracer
    return await func(*args, **kwargs)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/storage/blob/aio/_blob_client_async.py", line 388, in upload_blob
    return await upload_block_blob(**options)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/storage/blob/aio/_upload_helpers.py", line 129, in upload_block_blob
    return await client.commit_block_list(
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/storage/blob/_generated/aio/operations/_block_blob_operations.py", line 1006, in commit_block_list
    pipeline_response = await self._client._pipeline.run(request, stream=False, **kwargs)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 215, in run
    return await first_node.send(pipeline_request)
File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  [Previous line repeated 4 more times]
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/pipeline/policies/_redirect_async.py", line 64, in send
    response = await self.next.send(request)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/storage/blob/_shared/policies_async.py", line 111, in send
    response = await self.next.send(request)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/core/pipeline/_base_async.py", line 83, in send
    response = await self.next.send(request)  # type: ignore
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/azure/storage/blob/_shared/policies_async.py", line 90, in send
    response_callback(response)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/adlfs/spec.py", line 298, in wrapper
    callback.absolute_update(current)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/fs/_callback.py", line 122, in absolute_update
    self.progress_bar.update_to(value)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/progress.py", line 121, in update_to
    self.update(current - self.n)
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'
------------------------------------------------------------
2022-05-05 13:11:45,920 DEBUG: failed to upload full contents of 'None', aborting .dir file upload
2022-05-05 13:11:45,921 ERROR: failed to upload '/home/qcify/qcify/myagent/_work/1/s/models/.dvc/cache/76/df0b9081b1d33145e688a4b23b5cf3.dir' to 'dvc/models_v3/some_new_folder/76/df0b9081b1d33145e688a4b23b5cf3.dir'
2022-05-05 13:11:45,922 ERROR: failed to push data to the cloud - 3 files failed to upload
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/repo/push.py", line 68, in push
    pushed += self.cloud.push(
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/data_cloud.py", line 85, in push
    return transfer(
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/data/transfer.py", line 176, in transfer
    _do_transfer(
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/data/transfer.py", line 116, in _do_transfer
    raise FileTransferError(total_fails)
dvc.exceptions.FileTransferError: 3 files failed to transfer

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/commands/data_sync.py", line 58, in run
    processed_files_count = self.repo.push(
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/repo/__init__.py", line 48, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/qcify/qcify/myagent/_work/1/s/venv/lib/python3.9/site-packages/dvc/repo/push.py", line 72, in push
    raise UploadError(exc.amount)
dvc.exceptions.UploadError: 3 files failed to upload
------------------------------------------------------------
2022-05-05 13:11:45,932 DEBUG: Analytics is enabled.
2022-05-05 13:11:46,047 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpu8n72eqg']'
2022-05-05 13:11:46,050 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpu8n72eqg']'
```


Second try which just uploads the .dir file:

```
2022-05-05 18:04:51,465 DEBUG: Preparing to transfer data from '/home/qcify/qcify/myagent/_work/1/s/models/.dvc/cache' to 'dvc/models_v3/some_new_folder'
2022-05-05 18:04:51,465 DEBUG: Preparing to collect status from 'dvc/models_v3/some_new_folder'
2022-05-05 18:04:51,466 DEBUG: Collecting status from 'dvc/models_v3/some_new_folder'
2022-05-05 18:04:51,467 DEBUG: Querying 3 hashes via object_exists              
2022-05-05 18:04:52,254 DEBUG: Querying 1 hashes via object_exists              
2022-05-05 18:04:54,340 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 18:04:54,341 DEBUG: Estimated remote size: 4096 files                
2022-05-05 18:04:54,342 DEBUG: Querying '9' hashes via traverse                 
2022-05-05 18:04:57,243 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 18:04:57,257 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 18:04:57,263 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 18:04:57,272 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 18:04:57,281 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 18:04:57,291 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 18:04:57,343 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 18:04:57,360 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 18:04:57,405 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 18:04:57,407 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 18:04:57,418 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 18:04:57,422 DEBUG: 'dvc/models_v3/some_new_folder/00' doesn't look like a cache file, skipping
2022-05-05 18:05:17,034 DEBUG: Preparing to collect status from '/home/qcify/qcify/myagent/_work/1/s/models/.dvc/cache'
2022-05-05 18:05:17,034 DEBUG: Collecting status from '/home/qcify/qcify/myagent/_work/1/s/models/.dvc/cache'
2022-05-05 18:05:17,389 DEBUG: Uploading '/home/qcify/qcify/myagent/_work/1/s/models/.dvc/cache/76/df0b9081b1d33145e688a4b23b5cf3.dir' to 'dvc/models_v3/some_new_folder/76/df0b9081b1d33145e688a4b23b5cf3.dir'
2022-05-05 18:05:17,578 DEBUG: Indexing pushed dir 'md5: 76df0b9081b1d33145e688a4b23b5cf3.dir' with '8' nested files                                            
1 file pushed                                                                   
2022-05-05 18:05:17,599 DEBUG: Analytics is enabled.
2022-05-05 18:05:17,714 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp_8_jinwu']'
2022-05-05 18:05:17,716 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp_8_jinwu']'
```

I already went digging a bit myself and found a temporary fix. The problem seems to be that a response from azure can have None for the current progress, which causes an error when updating the progress bar. I fixed this for now by changing the update_to function where the error happens as follows:

```
def update_to(self, current, total=None):
        if total:
            self.total = total
        if current is not None:
            self.update(current - self.n)
```

I don't know for sure if this is intended behaviour on azure's side, but it does seem like it. By following the traceback from the bottom up I found this https://github.com/Azure/azure-sdk-for-python/issues/11419#issuecomment-628143480 link in their code. The linked comment also checks if current is None, so perhaps the solution is as simple as that?


Comments:
Thanks @rubenpraets for the great research, this is an issue in `adlfs` where it should be fixed.

See https://github.com/fsspec/adlfs/blob/master/adlfs/spec.py#L285-L300 (it even refers to the issue you mentioned 🤷🏼).



Patch:
diff --git a/dvc/data/checkout.py b/dvc/data/checkout.py
--- a/dvc/data/checkout.py
+++ b/dvc/data/checkout.py
@@ -139,7 +139,6 @@ def __call__(self, cache, from_path, to_fs, to_path, callback=None):
                 callback,
                 desc=cache.fs.path.name(from_path),
                 bytes=True,
-                total=-1,
             ) as cb:
                 transfer(
                     cache.fs,
diff --git a/dvc/data/stage.py b/dvc/data/stage.py
--- a/dvc/data/stage.py
+++ b/dvc/data/stage.py
@@ -39,7 +39,7 @@ def _upload_file(from_fs_path, fs, odb, upload_odb, callback=None):
             callback,
             desc=fs_path.name(from_fs_path),
             bytes=True,
-            total=size,
+            size=size,
         ) as cb:
             upload_odb.fs.put_file(stream, tmp_info, size=size, callback=cb)
 
diff --git a/dvc/fs/_callback.py b/dvc/fs/_callback.py
--- a/dvc/fs/_callback.py
+++ b/dvc/fs/_callback.py
@@ -59,6 +59,14 @@ def __exit__(self, *exc_args):
     def close(self):
         """Handle here on exit."""
 
+    def relative_update(self, inc: int = 1) -> None:
+        inc = inc if inc is not None else 0
+        return super().relative_update(inc)
+
+    def absolute_update(self, value: int) -> None:
+        value = value if value is not None else self.value
+        return super().absolute_update(value)
+
     @classmethod
     def as_callback(
         cls, maybe_callback: Optional["FsspecCallback"] = None
@@ -97,21 +105,29 @@ class NoOpCallback(FsspecCallback, fsspec.callbacks.NoOpCallback):
 
 
 class TqdmCallback(FsspecCallback):
-    def __init__(self, progress_bar: "Tqdm" = None, **tqdm_kwargs):
+    def __init__(
+        self,
+        size: Optional[int] = None,
+        value: int = 0,
+        progress_bar: "Tqdm" = None,
+        **tqdm_kwargs,
+    ):
+        tqdm_kwargs["total"] = size or -1
         self._tqdm_kwargs = tqdm_kwargs
         self._progress_bar = progress_bar
         self._stack = ExitStack()
-        super().__init__()
+        super().__init__(size=size, value=value)
 
     @cached_property
     def progress_bar(self):
         from dvc.progress import Tqdm
 
-        return self._stack.enter_context(
+        progress_bar = (
             self._progress_bar
             if self._progress_bar is not None
             else Tqdm(**self._tqdm_kwargs)
         )
+        return self._stack.enter_context(progress_bar)
 
     def __enter__(self):
         return self
@@ -120,18 +136,13 @@ def close(self):
         self._stack.close()
 
     def set_size(self, size):
-        if size is not None:
-            self.progress_bar.total = size
-            self.progress_bar.refresh()
-            super().set_size(size)
-
-    def relative_update(self, inc=1):
-        self.progress_bar.update(inc)
-        super().relative_update(inc)
+        # Tqdm tries to be smart when to refresh,
+        # so we try to force it to re-render.
+        super().set_size(size)
+        self.progress_bar.refresh()
 
-    def absolute_update(self, value):
-        self.progress_bar.update_to(value)
-        super().absolute_update(value)
+    def call(self, hook_name=None, **kwargs):
+        self.progress_bar.update_to(self.value, total=self.size)
 
     def branch(
         self,
@@ -140,72 +151,73 @@ def branch(
         kwargs,
         child: Optional[FsspecCallback] = None,
     ):
-        child = child or TqdmCallback(bytes=True, total=-1, desc=path_1)
+        child = child or TqdmCallback(bytes=True, desc=path_1)
         return super().branch(path_1, path_2, kwargs, child=child)
 
 
 class RichCallback(FsspecCallback):
     def __init__(
         self,
+        size: Optional[int] = None,
+        value: int = 0,
         progress: "RichTransferProgress" = None,
         desc: str = None,
-        total: int = None,
         bytes: bool = False,  # pylint: disable=redefined-builtin
         unit: str = None,
         disable: bool = False,
     ) -> None:
+        self._progress = progress
+        self.disable = disable
+        self._task_kwargs = {
+            "description": desc or "",
+            "bytes": bytes,
+            "unit": unit,
+            "total": size or 0,
+            "visible": False,
+            "progress_type": None if bytes else "summary",
+        }
+        self._stack = ExitStack()
+        super().__init__(size=size, value=value)
+
+    @cached_property
+    def progress(self):
         from dvc.ui import ui
         from dvc.ui._rich_progress import RichTransferProgress
 
-        self.progress = progress or RichTransferProgress(
+        if self._progress is not None:
+            return self._progress
+
+        progress = RichTransferProgress(
             transient=True,
-            disable=disable,
+            disable=self.disable,
             console=ui.error_console,
         )
-        self.visible = not disable
-        self._newly_created = progress is None
-        total = 0 if total is None or total < 0 else total
-        self.task = self.progress.add_task(
-            description=desc or "",
-            total=total,
-            bytes=bytes,
-            visible=False,
-            unit=unit,
-            progress_type=None if bytes else "summary",
-        )
-        super().__init__()
+        return self._stack.enter_context(progress)
+
+    @cached_property
+    def task(self):
+        return self.progress.add_task(**self._task_kwargs)
 
     def __enter__(self):
-        if self._newly_created:
-            self.progress.__enter__()
         return self
 
     def close(self):
-        if self._newly_created:
-            self.progress.stop()
-        try:
-            self.progress.remove_task(self.task)
-        except KeyError:
-            pass
-
-    def set_size(self, size: int = None) -> None:
-        if size is not None:
-            self.progress.update(self.task, total=size, visible=self.visible)
-            super().set_size(size)
-
-    def relative_update(self, inc: int = 1) -> None:
-        self.progress.update(self.task, advance=inc)
-        super().relative_update(inc)
+        self.progress.clear_task(self.task)
+        self._stack.close()
 
-    def absolute_update(self, value: int) -> None:
-        self.progress.update(self.task, completed=value)
-        super().absolute_update(value)
+    def call(self, hook_name=None, **kwargs):
+        self.progress.update(
+            self.task,
+            completed=self.value,
+            total=self.size,
+            visible=not self.disable,
+        )
 
     def branch(
         self, path_1, path_2, kwargs, child: Optional[FsspecCallback] = None
     ):
         child = child or RichCallback(
-            self.progress, desc=path_1, bytes=True, total=-1
+            progress=self.progress, desc=path_1, bytes=True
         )
         return super().branch(path_1, path_2, kwargs, child=child)
 
diff --git a/dvc/objects/db.py b/dvc/objects/db.py
--- a/dvc/objects/db.py
+++ b/dvc/objects/db.py
@@ -134,7 +134,6 @@ def add(
             callback,
             desc=fs.path.name(fs_path),
             bytes=True,
-            total=-1,
         ) as cb:
             self._add_file(
                 fs,
diff --git a/dvc/output.py b/dvc/output.py
--- a/dvc/output.py
+++ b/dvc/output.py
@@ -701,7 +701,6 @@ def download(self, to, jobs=None):
         from dvc.fs._callback import FsspecCallback
 
         with FsspecCallback.as_tqdm_callback(
-            total=-1,
             desc=f"Downloading {self.fs.path.name(self.fs_path)}",
             unit="files",
         ) as cb:
diff --git a/dvc/repo/get.py b/dvc/repo/get.py
--- a/dvc/repo/get.py
+++ b/dvc/repo/get.py
@@ -61,7 +61,6 @@ def get(url, path, out=None, rev=None, jobs=None):
                 fs_path = fs.from_os_path(path)
 
             with FsspecCallback.as_tqdm_callback(
-                total=-1,
                 desc=f"Downloading {fs.path.name(path)}",
                 unit="files",
             ) as cb:
diff --git a/dvc/stage/cache.py b/dvc/stage/cache.py
--- a/dvc/stage/cache.py
+++ b/dvc/stage/cache.py
@@ -239,7 +239,6 @@ def transfer(self, from_odb, to_odb):
             with FsspecCallback.as_tqdm_callback(
                 desc=src_name,
                 bytes=True,
-                total=-1,
             ) as cb:
                 func(from_fs, src, to_fs, dst, callback=cb)
             ret.append((parent_name, src_name))
diff --git a/dvc/ui/_rich_progress.py b/dvc/ui/_rich_progress.py
--- a/dvc/ui/_rich_progress.py
+++ b/dvc/ui/_rich_progress.py
@@ -20,7 +20,15 @@ def render(self, task):
         return ret.append(f" {unit}") if unit else ret
 
 
-class RichTransferProgress(Progress):
+class RichProgress(Progress):
+    def clear_task(self, task):
+        try:
+            self.remove_task(task)
+        except KeyError:
+            pass
+
+
+class RichTransferProgress(RichProgress):
     SUMMARY_COLS = (
         TextColumn("[magenta]{task.description}[bold green]"),
         MofNCompleteColumnWithUnit(),
diff --git a/dvc/utils/fs.py b/dvc/utils/fs.py
--- a/dvc/utils/fs.py
+++ b/dvc/utils/fs.py
@@ -205,7 +205,7 @@ def copyfile(src, dest, callback=None, no_progress_bar=False, name=None):
         with open(src, "rb") as fsrc, open(dest, "wb+") as fdest:
             with FsspecCallback.as_tqdm_callback(
                 callback,
-                total=total,
+                size=total,
                 bytes=True,
                 disable=no_progress_bar,
                 desc=name,


Test patch:
diff --git a/tests/func/test_fs.py b/tests/func/test_fs.py
--- a/tests/func/test_fs.py
+++ b/tests/func/test_fs.py
@@ -3,8 +3,10 @@
 from operator import itemgetter
 from os.path import join
 
+import pytest
+
 from dvc.fs import get_cloud_fs
-from dvc.fs._callback import FsspecCallback
+from dvc.fs._callback import DEFAULT_CALLBACK, FsspecCallback
 from dvc.fs.local import LocalFileSystem
 from dvc.repo import Repo
 
@@ -323,3 +325,35 @@ def test_callback_on_repo_fs(tmp_dir, dvc, scm, mocker):
     assert branch.call_count == 1
     assert branch.spy_return.size == size
     assert branch.spy_return.value == size
+
+
+@pytest.mark.parametrize(
+    "api", ["set_size", "relative_update", "absolute_update"]
+)
+@pytest.mark.parametrize(
+    "callback_factory, kwargs",
+    [
+        (FsspecCallback.as_callback, {}),
+        (FsspecCallback.as_tqdm_callback, {"desc": "test"}),
+        (FsspecCallback.as_rich_callback, {"desc": "test"}),
+    ],
+)
+def test_callback_with_none(request, api, callback_factory, kwargs, mocker):
+    """
+    Test that callback don't fail if they receive None.
+
+    The callbacks should not receive None, but there may be some
+    filesystems that are not compliant, we may want to maintain
+    maximum compatibility, and not break UI in these edge-cases.
+    See https://github.com/iterative/dvc/issues/7704.
+    """
+    callback = callback_factory(**kwargs)
+    request.addfinalizer(callback.close)
+
+    call_mock = mocker.spy(callback, "call")
+    method = getattr(callback, api)
+    method(None)
+    call_mock.assert_called_once_with()
+    if callback is not DEFAULT_CALLBACK:
+        assert callback.size is None
+        assert callback.value == 0