Repository: rclone/rclone (language: go)

Issue:
bisync: retries and delay on locked files
In relation to #7471 i will split off this issue as an extra feature request.

I tried using --retries=1000 --retries-sleep=30 for cases where MS Office files are occupied or locked!
The bisync task will end up with a fatal error and needs to be resynced.

@nielash [posted a preview.](https://github.com/nielash/rclone/releases/tag/v1.65.0-beta.7566.68eb6f5b2.bisync-feature-preview-2024-01-01) **Many THANKS for that!**
I share my test results with this preview beta here:

```
rclone bisync "path1" "C:\TBD\Test" --config="rclone.conf" --workdir="cache" --exclude="{~$*,*.tmp,*.bak}" --check-first --fast-list --remove-empty-dirs --retries=1000 --retries-sleep=30 --conflict-resolve=newer --conflict-loser=delete --resilient --recover --max-lock 5m -v
2024/01/09 06:31:11 NOTICE: bisync is IN BETA. Don't use in production!
2024/01/09 06:31:11 INFO  : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
2024/01/09 06:31:11 INFO  : Bisyncing with Comparison Settings:
{
        "Modtime": true,
        "Size": true,
        "Checksum": false,
        "HashType1": 0,
        "HashType2": 0,
        "NoSlowHash": false,
        "SlowHashSyncOnly": false,
        "SlowHashDetected": true,
        "DownloadHash": false
}
2024/01/09 06:31:11 INFO  : lock file renewed for 5m0s. New expiration: 2024-01-09 06:36:11.3875714 +0100 CET m=+300.422610101
2024/01/09 06:31:11 INFO  : Synching Path1 "path1" with Path2 "C:\TBD\Test\"
2024/01/09 06:31:11 INFO  : Building Path1 and Path2 listings
2024/01/09 06:31:11 INFO  : Path1 checking for diffs
2024/01/09 06:31:11 INFO  : - Path1    File changed: size (smaller), time (newer) - Ship-to form.docx
2024/01/09 06:31:11 INFO  : Path1:    1 changes:    0 new,    1 modified,    0 deleted
2024/01/09 06:31:11 INFO  : (Modified:    1 newer,    0 older,    0 larger,    1 smaller)
2024/01/09 06:31:11 INFO  : Path2 checking for diffs
2024/01/09 06:31:11 INFO  : Applying changes
2024/01/09 06:31:11 INFO  : - Path1    Queue copy to Path2       - C:\TBD\Test\Ship-to form.docx
2024/01/09 06:31:11 INFO  : - Path1    Do queued copies to                - Path2
2024/01/09 06:31:11 INFO  : Local file system at //?/C:/TBD/Test: Running all checks before starting transfers
2024/01/09 06:31:11 INFO  : Local file system at //?/C:/TBD/Test: Checks finished, now starting transfers
2024/01/09 06:31:11 ERROR : Ship-to form.docx.cagiyuw2.partial: partial file rename failed: rename \\?\C:\TBD\Test\Ship-to form.docx.cagiyuw2.partial \\?\C:\TBD\Test\Ship-to form.docx: Access is denied.
2024/01/09 06:31:11 INFO  : Ship-to form.docx.cagiyuw2.partial: Removing failed copy
2024/01/09 06:31:11 ERROR : Local file system at //?/C:/TBD/Test: not deleting files as there were IO errors
2024/01/09 06:31:11 ERROR : Local file system at //?/C:/TBD/Test: not deleting directories as there were IO errors
2024/01/09 06:31:11 NOTICE: copy1to2: Received error: rename \\?\C:\TBD\Test\Ship-to form.docx.cagiyuw2.partial \\?\C:\TBD\Test\Ship-to form.docx: Access is denied. - retrying as --resilient is set. Retry 1/1000
2024/01/09 06:31:11 INFO  : Local file system at //?/C:/TBD/Test: Running all checks before starting transfers
2024/01/09 06:31:11 INFO  : Local file system at //?/C:/TBD/Test: Checks finished, now starting transfers
2024/01/09 06:31:11 ERROR : Ship-to form.docx.zeluram5.partial: partial file rename failed: rename \\?\C:\TBD\Test\Ship-to form.docx.zeluram5.partial \\?\C:\TBD\Test\Ship-to form.docx: Access is denied.
2024/01/09 06:31:11 INFO  : Ship-to form.docx.zeluram5.partial: Removing failed copy
2024/01/09 06:31:11 ERROR : Local file system at //?/C:/TBD/Test: not deleting files as there were IO errors
2024/01/09 06:31:11 ERROR : Local file system at //?/C:/TBD/Test: not deleting directories as there were IO errors
2024/01/09 06:31:11 NOTICE: copy1to2: Received error: rename \\?\C:\TBD\Test\Ship-to form.docx.zeluram5.partial \\?\C:\TBD\Test\Ship-to form.docx: Access is denied. - retrying as --resilient is set. Retry 2/1000
```

This loop runs very quick without any sleep time. So the `--retries-sleep=30` was ignored!

At this moment i unlocked the file.

```
2024/01/09 06:31:26 INFO  : Local file system at //?/C:/TBD/Test: Running all checks before starting transfers
2024/01/09 06:31:26 INFO  : Local file system at //?/C:/TBD/Test: Checks finished, now starting transfers
2024/01/09 06:31:26 INFO  : Ship-to form.docx: Copied (replaced existing)
2024/01/09 06:31:26 ERROR : Local file system at //?/C:/TBD/Test: not deleting files as there were IO errors
2024/01/09 06:31:26 ERROR : Local file system at //?/C:/TBD/Test: not deleting directories as there were IO errors
2024/01/09 06:31:26 NOTICE: copy1to2: Received error: not deleting files as there were IO errors - retrying as --resilient is set. Retry 174/1000
2024/01/09 06:31:26 INFO  : Local file system at //?/C:/TBD/Test: Running all checks before starting transfers
2024/01/09 06:31:26 INFO  : Local file system at //?/C:/TBD/Test: Checks finished, now starting transfers
2024/01/09 06:31:26 ERROR : Local file system at //?/C:/TBD/Test: not deleting files as there were IO errors
2024/01/09 06:31:26 ERROR : Local file system at //?/C:/TBD/Test: not deleting directories as there were IO errors
2024/01/09 06:31:26 NOTICE: copy1to2: Received error: not deleting files as there were IO errors - retrying as --resilient is set. Retry 175/1000
```

The loop keeps running...
So i ended up with sending an exit signal.

```
2024/01/09 06:31:31 INFO  : Local file system at //?/C:/TBD/Test: Running all checks before starting transfers
2024/01/09 06:31:31 INFO  : Local file system at //?/C:/TBD/Test: Checks finished, now starting transfers
2024/01/09 06:31:31 ERROR : Local file system at //?/C:/TBD/Test: not deleting files as there were IO errors
2024/01/09 06:31:31 ERROR : Local file system at //?/C:/TBD/Test: not deleting directories as there were IO errors
2024/01/09 06:31:31 NOTICE: copy1to2: Received error: not deleting files as there were IO errors - retrying as --resilient is set. Retry 281/1000
2024/01/09 06:31:31 INFO  : Local file system at //?/C:/TBD/Test: Running all checks before starting transfers
2024/01/09 06:31:31 INFO  : Signal received: interrupt
2024/01/09 06:31:31 NOTICE: Attempting to gracefully shutdown. (Send exit signal again for immediate un-graceful shutdown.)
2024/01/09 06:31:31 INFO  : Telling Sync to wrap up early.
2024/01/09 06:31:31 INFO  : Canceling Sync if not done in: 30
2024/01/09 06:31:31 INFO  : Local file system at //?/C:/TBD/Test: Checks finished, now starting transfers
2024/01/09 06:31:31 ERROR : Local file system at //?/C:/TBD/Test: not deleting files as there were IO errors
2024/01/09 06:31:31 ERROR : Local file system at //?/C:/TBD/Test: not deleting directories as there were IO errors
2024/01/09 06:31:31 NOTICE: copy1to2: Received error: not deleting files as there were IO errors - retrying as --resilient is set. Retry 282/1000
2024/01/09 06:31:31 NOTICE: copy1to2: Received error: <nil> - retrying as --resilient is set. Retry 283/1000
2024/01/09 06:31:31 NOTICE: copy1to2: Received error: <nil> - retrying as --resilient is set. Retry 284/1000
...
...
2024/01/09 06:31:31 NOTICE: copy1to2: Received error: <nil> - retrying as --resilient is set. Retry 999/1000
2024/01/09 06:31:31 NOTICE: copy1to2: Received error: <nil> - retrying as --resilient is set. Retry 1000/1000
2024/01/09 06:31:31 INFO  : Updating listings
2024/01/09 06:31:31 INFO  : Validating listings for Path1 "path1" vs Path2 "C:\TBD\Test\"
2024/01/09 06:31:31 INFO  : Removing empty directories
2024/01/09 06:31:31 NOTICE: Graceful shutdown completed successfully.
2024/01/09 06:31:31 INFO  : Bisync successful
2024/01/09 06:31:31 INFO  :
Transferred:       14.388 MiB / 14.388 MiB, 100%, 711.230 KiB/s, ETA 0s
Errors:               173 (retrying may help)
Checks:               396 / 396, 100%
Transferred:            1 / 1, 100%
Elapsed time:        20.5s

2024/01/09 06:31:31 Failed to bisync with 173 errors: last error was: rename \\?\C:\TBD\Test\Ship-to form.docx.lebeqic0.partial \\?\C:\TBD\Test\Ship-to form.docx: Access is denied.
2024/01/09 06:31:32 INFO  : Exiting...
```

I was wondering why 14.388 MiB are transferred. Every loop the partial file was copied.
So the `--retries-sleep=30` would be reduce the amount.

My summary of the bisync-feature-preview:
anything causes tons of verbose output.
stability seems good. next run recovers automatically.


Discussion / hints:
Thank you, @martind69! This is great feedback. I think there are two issues here, both with easy fixes:

- Bisync should reset the error count in between each retry (it currently does not -- this is why it kept retrying after you unlocked the file.)
- Bisync should implement support for `--retries-sleep`

I should be able to address these pretty quickly (and post a new version to try, if you'd like!)

> anything causes tons of verbose output.

It is verbose because the `-v` flag was included -- you can omit this flag to make it less verbose (or add `-q` for even less). That said, if there are certain messages you think should be downgraded from `NOTICE` to `INFO` or `INFO` to `DEBUG`, let me know and I'm happy to consider changing them 🙂 

> stability seems good. next run recovers automatically.

Glad to hear it! 🙂 
I'm glad to hear from you!

> I should be able to address these pretty quickly (and post a new version to try, if you'd like!)

That were great! I'm your debugger 🙂

> It is verbose because the `-v` flag was included

I know that 🙂, what i mean is this:

```
2024/01/09 06:31:31 NOTICE: copy1to2: Received error: not deleting files as there were IO errors - retrying as --resilient is set. Retry 282/1000
2024/01/09 06:31:31 NOTICE: copy1to2: Received error: <nil> - retrying as --resilient is set. Retry 283/1000
2024/01/09 06:31:31 NOTICE: copy1to2: Received error: <nil> - retrying as --resilient is set. Retry 284/1000
...
...
2024/01/09 06:31:31 NOTICE: copy1to2: Received error: <nil> - retrying as --resilient is set. Retry 999/1000
2024/01/09 06:31:31 NOTICE: copy1to2: Received error: <nil> - retrying as --resilient is set. Retry 1000/1000
```
why is that? the copy was successfull before

`2024/01/09 06:31:26 INFO  : Ship-to form.docx: Copied (replaced existing)`

And btw, the unit isn't print out.

`2024/01/09 06:31:31 INFO  : Canceling Sync if not done in: 30`
> That were great! I'm your debugger 🙂

I appreciate that! [Here's a new version](https://github.com/nielash/rclone/releases/tag/v1.65.0-beta.7569.5bd90ea42.fix-7555-bisync-retries) which I think should fix these issues.

> why is that? the copy was successfull before

Silly oversight on my part -- I neglected to exit the loop early if `err` came back `nil`. The verbosity was more of a symptom -- it should not have gotten past retry 282 in the first place! 🙂 It should be fixed in this version.

> And btw, the unit isn't print out.

This is actually a 30-second countdown that logs every second during the graceful shutdown sequence -- but yours finished in less than a second so you never got to 29 😄 But I agree it would be clearer with a unit -- I've added it in this new version. Thanks for spotting that 🙂 

Let me know how the new version goes for you if you have a chance to try it!
Good morning! in relation to "goog" i have test results ;)

```
...
2024/01/10 08:59:14 INFO  : Backup listing is valid.
2024/01/10 08:59:14 INFO  : Building Path1 and Path2 listings
2024/01/10 08:59:14 INFO  : Path1 checking for diffs
2024/01/10 08:59:14 INFO  : - Path1    File changed: size (larger), time (newer) - Ship-to form.docx
2024/01/10 08:59:14 INFO  : Path1:    1 changes:    0 new,    1 modified,    0 deleted
2024/01/10 08:59:14 INFO  : (Modified:    1 newer,    0 older,    1 larger,    0 smaller)
2024/01/10 08:59:14 INFO  : Path2 checking for diffs
2024/01/10 08:59:14 INFO  : Applying changes
2024/01/10 08:59:14 INFO  : - Path1    Queue copy to Path2       - C:\TBD\Test\Ship-to form.docx
2024/01/10 08:59:14 INFO  : - Path1    Do queued copies to                - Path2
2024/01/10 08:59:14 INFO  : Local file system at //?/C:/TBD/Test: Running all checks before starting transfers
2024/01/10 08:59:14 INFO  : Ship-to form.docx: Deleted
2024/01/10 08:59:14 INFO  : Ship-to form.docx: Copied (replaced existing)
2024/01/10 08:59:14 NOTICE: \\?\C:\TBD\Test\Ship-to form.docx: Remove detected sharing violation - retry 1/10 sleeping 1ms
2024/01/10 08:59:14 NOTICE: \\?\C:\TBD\Test\Ship-to form.docx: Remove detected sharing violation - retry 2/10 sleeping 2ms
2024/01/10 08:59:14 NOTICE: \\?\C:\TBD\Test\Ship-to form.docx: Remove detected sharing violation - retry 3/10 sleeping 4ms
2024/01/10 08:59:14 NOTICE: \\?\C:\TBD\Test\Ship-to form.docx: Remove detected sharing violation - retry 4/10 sleeping 8ms
2024/01/10 08:59:14 NOTICE: \\?\C:\TBD\Test\Ship-to form.docx: Remove detected sharing violation - retry 5/10 sleeping 16ms
2024/01/10 08:59:14 NOTICE: \\?\C:\TBD\Test\Ship-to form.docx: Remove detected sharing violation - retry 6/10 sleeping 32ms
2024/01/10 08:59:14 NOTICE: \\?\C:\TBD\Test\Ship-to form.docx: Remove detected sharing violation - retry 7/10 sleeping 64ms
2024/01/10 08:59:15 NOTICE: \\?\C:\TBD\Test\Ship-to form.docx: Remove detected sharing violation - retry 8/10 sleeping 128ms
2024/01/10 08:59:15 NOTICE: \\?\C:\TBD\Test\Ship-to form.docx: Remove detected sharing violation - retry 9/10 sleeping 256ms
2024/01/10 08:59:15 NOTICE: \\?\C:\TBD\Test\Ship-to form.docx: Remove detected sharing violation - retry 10/10 sleeping 512ms
2024/01/10 08:59:15 ERROR : Ship-to form.docx: Couldn't delete: remove \\?\C:\TBD\Test\Ship-to form.docx: The process cannot access the file because it is being used by another process.
2024/01/10 08:59:15 INFO  : Local file system at //?/C:/TBD/Test: Checks finished, now starting transfers
2024/01/10 08:59:15 ERROR : Local file system at //?/C:/TBD/Test: not deleting files as there were IO errors
2024/01/10 08:59:15 ERROR : Local file system at //?/C:/TBD/Test: not deleting directories as there were IO errors
2024/01/10 08:59:15 NOTICE: copy1to2: Received error: remove \\?\C:\TBD\Test\Ship-to form.docx: The process cannot access the file because it is being used by another process. - retrying as --resilient is set. Retry 1/1000
2024/01/10 08:59:15 NOTICE: will retry in 10s at 2024-01-10 08:59:25 CET
```

Question: what is the purpose of the 2^0 ... 2^9 [ms] stepping?
`2024/01/10 08:59:14 NOTICE: \\?\C:\TBD\Test\Ship-to form.docx: Remove detected sharing violation - retry 1/10 sleeping 1ms`

Bug: without verbosity we end up in an critical error
```
2024/01/10 08:58:48 NOTICE: \\?\C:\TBD\Test\Ship-to form.docx: Remove detected sharing violation - retry 10/10 sleeping 512ms
2024/01/10 08:58:49 ERROR : Ship-to form.docx: Couldn't delete: remove \\?\C:\TBD\Test\Ship-to form.docx: The process cannot access the file because it is being used by another process.
2024/01/10 08:58:49 ERROR : Ship-to form.docx: Couldn't move into backup dir: remove \\?\C:\TBD\Test\Ship-to form.docx: The process cannot access the file because it is being used by another process.
2024/01/10 08:58:49 ERROR : Bisync critical error: C:\TBD\Test\ delete failed for C:\TBD\Test\Ship-to form.docx: remove \\?\C:\TBD\Test\Ship-to form.docx: The process cannot access the file because it is being used by another process.
2024/01/10 08:58:49 ERROR : Bisync aborted. Must run --resync to recover.
```

Ugly: if i choose error log to `--log-file` it would log tons of ERROR (in case without -v but that's currently not possible). That aren't real errors in my case, it needs a reclassification.

Feature idea: i thinked about using `--suffix` in addition to `--backup-dirN`
But there is no dynamic option yet. I think about `--suffix datetime` like microtime or `--suffix date` only date.
And a really nice option would be a `--backup-cleanup` with max-age definition.
What do you think about that?

> Question: what is the purpose of the 2^0 ... 2^9 [ms] stepping?

These are ["low-level-retries"](https://rclone.org/docs/#low-level-retries-number), which are different from the "high-level retries" controlled by the `--retries` flag. They are built in to rclone and not specific to bisync. Basically the idea is to retry certain low-level errors in case they are just temporary, and wait for longer and longer amounts between each retry.

In your case, the `Remove detected sharing violation` error seems to be related to a Windows-specific quirk -- there is some context here: https://github.com/rclone/rclone/commit/42f0963bf907daacb6cb718ef14a64a591fab3a0 

> Bug: without verbosity we end up in an critical error

I'm not sure if I understand this comment. Do you mean that the behavior is different with and without the `-v` flag? Or is the issue just that rclone is erroring for something you don't consider an error (regardless of the `-v` flag)?

> Ugly: if i choose error log to `--log-file` it would log tons of ERROR 

Not sure I understand... what do you wish would happen instead when there is an ERROR?

> That aren't real errors in my case, it needs a reclassification.

Can you give me more info about the use case and why it isn't a real error? It looks like you are trying to overwrite a `docx` file while it is open for editing on the destination... what do you wish would happen instead in that scenario? Is the idea that rclone should silently skip this file instead of complaining about it?

A few things that would be helpful to test:

- is the behavior different in `sync` vs. `bisync`?
- does using `--ignore-errors` help?
- does using `--low-level-retries 1` help?
- Can you give me instructions to reproduce this scenario? (I do have a Windows machine I can try)

> Feature idea: i thinked about using `--suffix` in addition to `--backup-dirN`
But there is no dynamic option yet. I think about `--suffix datetime` like microtime or `--suffix date` only date.

I agree this would be nice! I actually just developed a nearly identical feature for bisync's `--conflict-suffix` flag -- you can read about the dynamic date variables [here](https://github.com/nielash/rclone/blob/bisync-feature-preview-2024-01-01/docs/content/bisync.md#--conflict-suffix-stringstring-conflict-suffix). Use is like this:

```
--conflict-suffix {DateOnly}-conflict
// result: myfile.txt.2006-01-02-conflict1
```


The main [`--suffix`](https://rclone.org/docs/#suffix-suffix) flag is not specific to bisync, but I think it would be quite easy to move that same code into `operations` and use it for both, if people like that idea 🙂 

> And a really nice option would be a `--backup-cleanup` with max-age definition.
What do you think about that?

Explain a little more about what this would do? Would it be like running `rclone delete` on the backup-dir with a `--min-age` filter? Would it apply to all files or only the ones related to the current sync?
> > Bug: without verbosity we end up in an critical error
> 
> I'm not sure if I understand this comment. Do you mean that the behavior is different with and without the `-v` flag? Or is the issue just that rclone is erroring for something you don't consider an error (regardless of the `-v` flag)?

I mean that the behavior is different with and without the `-v` flag
In other words, it can be used only in verbose mode!

> > Ugly: if i choose error log to `--log-file` it would log tons of ERROR
> 
> Not sure I understand... what do you wish would happen instead when there is an ERROR?

When i set up `--log-level ERROR` then i expect nothing in relation to my tests. But i see many error logs in verbose mode. Maybe it's necessary to do a reclassification because there are no real (top-level) errors. I think that are sub-level errors. Some examples:
`2024/01/10 08:59:15 ERROR : Ship-to form.docx: Couldn't delete: remove \\?\C:\TBD\Test\Ship-to form.docx: The process cannot access the file because it is being used by another process.`
Is not an error but maybe a warning?
`2024/01/10 08:59:15 ERROR : Local file system at //?/C:/TBD/Test: not deleting files as there were IO errors`
Is not an error, it's a result of the above one.

> Is the idea that rclone should silently skip this file instead of complaining about it?

Yes! that's a great idea. Then you can report (log) a notice or info classification.

> A few things that would be helpful to test:
> 
> * is the behavior different in `sync` vs. `bisync`?
> * does using `--ignore-errors` help?
> * does using `--low-level-retries 1` help?
> * Can you give me instructions to reproduce this scenario? (I do have a Windows machine I can try)

I will test it soon, sorry no time today.

> I actually just developed a nearly identical feature for bisync's `--conflict-suffix` flag

Perfect, good idea!

> The main [`--suffix`](https://rclone.org/docs/#suffix-suffix) flag is not specific to bisync

I think you can implement this feature to bisync. My idea using `--suffix datetime` like microtime or `--suffix date` only date. For scenarios that no data loss can be accepted, this flag renames the files unique before move to `--backup-dirN`

> Explain a little more about what this would do? Would it be like running `rclone delete` on the backup-dir with a `--min-age` filter? Would it apply to all files or only the ones related to the current sync?

Yes that's correct! The idea is seen global - not to a specific sync. purge all where the `--min-age` filter is applicable.
Then users can plan with a storage budget and doesn't purge manually.
> I mean that the behavior is different with and without the `-v` flag
> In other words, it can be used only in verbose mode!

Can you send a complete log with `-v` and also without `-v`, so that I can see what you mean? (Your last `-v` log cuts off at `NOTICE: will retry in 10s at 2024-01-10 08:59:25 CET`, so I can't see what happened after that... did the retries succeed or fail?) 

So far, I have not seen anything unexpected in my tests, or in the excerpts you posted from your logs. Your OS failed to copy the file, so rclone logged this as an error and exited with an error status. That is what I'd expect to happen.

> When i set up `--log-level ERROR` then i expect nothing in relation to my tests. But i see many error logs in verbose mode.

`--log-level ERROR` still reports errors -- it just suppresses everything else. It sounds to me like the log-level is more of a side-effect, and the real issue here is that rclone is considering something to be an error that you do not consider to be an error. I think what you are saying is that you are not concerned about this `Couldn't delete` error, because bisync can try again on a future run, and sooner or later it will succeed, once the file is closed. Is that right? That sounds reasonable to me. What is tricky though is: how should rclone decide which errors are concerning to the user, and which errors are not? From rclone's perspective, its job was to copy a file, and it failed to do so. Rclone's expectation is that the dest should always match the source at the end of a sync, or else something went wrong.

I think probably the best solution would be a new flag to allow these kinds of errors to be ignored. But we still have to give rclone some criteria to determine which errors to care about. Do you have suggestions for how it should decide? (What is an example of an error you _would_ want to be fatal?)

> `2024/01/10 08:59:15 ERROR : Local file system at //?/C:/TBD/Test: not deleting files as there were IO errors`
> Is not an error, it's a result of the above one.

This one I am pretty sure can be ignored with the `--ignore-errors` flag. Or rather, the error will still be reported, but it will not be considered fatal and will not prevent the deletions. Let me know if you have a chance to try it.

> My idea using `--suffix datetime` like microtime or `--suffix date` only date. For scenarios that no data loss can be accepted, this flag renames the files unique before move to `--backup-dirN`

It's a great idea, and I think it would be useful for `sync` and `move` as well. `--conflict-suffix` already supports all of [these formats](https://pkg.go.dev/time#pkg-constants).

> Yes that's correct! The idea is seen global - not to a specific sync. purge all where the `--min-age` filter is applicable.
> Then users can plan with a storage budget and doesn't purge manually.

Makes sense. I think if it's global and not specific to a sync, then probably it makes more sense to just schedule a new cron job for `rclone delete [some:dir] --min-age [x]`. I'm not sure I see an advantage to triggering this from `sync` or `bisync`, and I think it could be problematic for a sync to be deleting things from a backup-dir that weren't related to the sync. But that's just my opinion -- feel free to open a new issue for it if you like, and it can be discussed!

Fix patch:
diff --git a/cmd/bisync/cmd.go b/cmd/bisync/cmd.go
index b57bbf1ee8ca6..22a600923efe7 100644
--- a/cmd/bisync/cmd.go
+++ b/cmd/bisync/cmd.go
@@ -51,7 +51,6 @@ type Options struct {
 	Resilient             bool
 	Recover               bool
 	TestFn                TestFunc // test-only option, for mocking errors
-	Retries               int
 	Compare               CompareOpt
 	CompareFlag           string
 	DebugName             string
@@ -118,12 +117,12 @@ func (x *CheckSyncMode) Type() string {
 var Opt Options
 
 func init() {
-	Opt.Retries = 3
 	Opt.MaxLock = 0
 	cmd.Root.AddCommand(commandDefinition)
 	cmdFlags := commandDefinition.Flags()
 	// when adding new flags, remember to also update the rc params:
 	// cmd/bisync/rc.go cmd/bisync/help.go (not docs/content/rc.md)
+	// and the Command line syntax section of docs/content/bisync.md (it doesn't update automatically)
 	flags.BoolVarP(cmdFlags, &Opt.Resync, "resync", "1", Opt.Resync, "Performs the resync run. Equivalent to --resync-mode path1. Consider using --verbose or --dry-run first.", "")
 	flags.FVarP(cmdFlags, &Opt.ResyncMode, "resync-mode", "", "During resync, prefer the version that is: path1, path2, newer, older, larger, smaller (default: path1 if --resync, otherwise none for no resync.)", "")
 	flags.BoolVarP(cmdFlags, &Opt.CheckAccess, "check-access", "", Opt.CheckAccess, makeHelp("Ensure expected {CHECKFILE} files are found on both Path1 and Path2 filesystems, else abort."), "")
@@ -142,7 +141,6 @@ func init() {
 	flags.BoolVarP(cmdFlags, &Opt.IgnoreListingChecksum, "ignore-listing-checksum", "", Opt.IgnoreListingChecksum, "Do not use checksums for listings (add --ignore-checksum to additionally skip post-copy checksum checks)", "")
 	flags.BoolVarP(cmdFlags, &Opt.Resilient, "resilient", "", Opt.Resilient, "Allow future runs to retry after certain less-serious errors, instead of requiring --resync. Use at your own risk!", "")
 	flags.BoolVarP(cmdFlags, &Opt.Recover, "recover", "", Opt.Recover, "Automatically recover from interruptions without requiring --resync.", "")
-	flags.IntVarP(cmdFlags, &Opt.Retries, "retries", "", Opt.Retries, "Retry operations this many times if they fail", "")
 	flags.StringVarP(cmdFlags, &Opt.CompareFlag, "compare", "", Opt.CompareFlag, "Comma-separated list of bisync-specific compare options ex. 'size,modtime,checksum' (default: 'size,modtime')", "")
 	flags.BoolVarP(cmdFlags, &Opt.Compare.NoSlowHash, "no-slow-hash", "", Opt.Compare.NoSlowHash, "Ignore listing checksums only on backends where they are slow", "")
 	flags.BoolVarP(cmdFlags, &Opt.Compare.SlowHashSyncOnly, "slow-hash-sync-only", "", Opt.Compare.SlowHashSyncOnly, "Ignore slow checksums for listings and deltas, but still consider them during sync calls.", "")
diff --git a/cmd/bisync/operations.go b/cmd/bisync/operations.go
index 432ceaee4fe64..4b5a288b13587 100644
--- a/cmd/bisync/operations.go
+++ b/cmd/bisync/operations.go
@@ -127,19 +127,6 @@ func Bisync(ctx context.Context, fs1, fs2 fs.Fs, optArg *Options) (err error) {
 	// Handle SIGINT
 	var finaliseOnce gosync.Once
 
-	// waitFor runs fn() until it returns true or the timeout expires
-	waitFor := func(msg string, totalWait time.Duration, fn func() bool) (ok bool) {
-		const individualWait = 1 * time.Second
-		for i := 0; i < int(totalWait/individualWait); i++ {
-			ok = fn()
-			if ok {
-				return ok
-			}
-			fs.Infof(nil, Color(terminal.YellowFg, "%s: %v"), msg, int(totalWait/individualWait)-i)
-			time.Sleep(individualWait)
-		}
-		return false
-	}
 	finalise := func() {
 		finaliseOnce.Do(func() {
 			if atexit.Signalled() {
@@ -648,6 +635,20 @@ func (b *bisyncRun) debugFn(nametocheck string, fn func()) {
 	}
 }
 
+// waitFor runs fn() until it returns true or the timeout expires
+func waitFor(msg string, totalWait time.Duration, fn func() bool) (ok bool) {
+	const individualWait = 1 * time.Second
+	for i := 0; i < int(totalWait/individualWait); i++ {
+		ok = fn()
+		if ok {
+			return ok
+		}
+		fs.Infof(nil, Color(terminal.YellowFg, "%s: %vs"), msg, int(totalWait/individualWait)-i)
+		time.Sleep(individualWait)
+	}
+	return false
+}
+
 // mainly to make sure tests don't interfere with each other when running more than one
 func resetGlobals() {
 	downloadHash = false
diff --git a/cmd/bisync/queue.go b/cmd/bisync/queue.go
index 8b5cd0801bdeb..f5617ed8aec5c 100644
--- a/cmd/bisync/queue.go
+++ b/cmd/bisync/queue.go
@@ -51,13 +51,15 @@ func (rs *ResultsSlice) has(name string) bool {
 	return false
 }
 
-var logger = operations.NewLoggerOpt()
-var lock mutex.Mutex
-var once mutex.Once
-var ignoreListingChecksum bool
-var ignoreListingModtime bool
-var hashTypes map[string]hash.Type
-var queueCI *fs.ConfigInfo
+var (
+	logger                = operations.NewLoggerOpt()
+	lock                  mutex.Mutex
+	once                  mutex.Once
+	ignoreListingChecksum bool
+	ignoreListingModtime  bool
+	hashTypes             map[string]hash.Type
+	queueCI               *fs.ConfigInfo
+)
 
 // allows us to get the right hashtype during the LoggerFn without knowing whether it's Path1/Path2
 func getHashType(fname string) hash.Type {
@@ -262,10 +264,25 @@ func (b *bisyncRun) fastCopy(ctx context.Context, fsrc, fdst fs.Fs, files bilib.
 }
 
 func (b *bisyncRun) retryFastCopy(ctx context.Context, fsrc, fdst fs.Fs, files bilib.Names, queueName string, results []Results, err error) ([]Results, error) {
-	if err != nil && b.opt.Resilient && !b.InGracefulShutdown && b.opt.Retries > 1 {
-		for tries := 1; tries <= b.opt.Retries; tries++ {
-			fs.Logf(queueName, Color(terminal.YellowFg, "Received error: %v - retrying as --resilient is set. Retry %d/%d"), err, tries, b.opt.Retries)
+	ci := fs.GetConfig(ctx)
+	if err != nil && b.opt.Resilient && !b.InGracefulShutdown && ci.Retries > 1 {
+		for tries := 1; tries <= ci.Retries; tries++ {
+			fs.Logf(queueName, Color(terminal.YellowFg, "Received error: %v - retrying as --resilient is set. Retry %d/%d"), err, tries, ci.Retries)
+			accounting.GlobalStats().ResetErrors()
+			if retryAfter := accounting.GlobalStats().RetryAfter(); !retryAfter.IsZero() {
+				d := time.Until(retryAfter)
+				if d > 0 {
+					fs.Logf(nil, "Received retry after error - sleeping until %s (%v)", retryAfter.Format(time.RFC3339Nano), d)
+					time.Sleep(d)
+				}
+			}
+			if ci.RetriesInterval > 0 {
+				naptime(ci.RetriesInterval)
+			}
 			results, err = b.fastCopy(ctx, fsrc, fdst, files, queueName)
+			if err == nil || b.InGracefulShutdown {
+				return results, err
+			}
 		}
 	}
 	return results, err
@@ -299,7 +316,7 @@ func (b *bisyncRun) syncEmptyDirs(ctx context.Context, dst fs.Fs, candidates bil
 
 		for _, s := range candidatesList {
 			var direrr error
-			if dirsList.has(s) { //make sure it's a dir, not a file
+			if dirsList.has(s) { // make sure it's a dir, not a file
 				r := Results{}
 				r.Name = s
 				r.Size = -1
@@ -358,3 +375,16 @@ func (b *bisyncRun) saveQueue(files bilib.Names, jobName string) error {
 	queueFile := fmt.Sprintf("%s.%s.que", b.basePath, jobName)
 	return files.Save(queueFile)
 }
+
+func naptime(totalWait time.Duration) {
+	expireTime := time.Now().Add(totalWait)
+	fs.Logf(nil, "will retry in %v at %v", totalWait, expireTime.Format("2006-01-02 15:04:05 MST"))
+	for i := 0; time.Until(expireTime) > 0; i++ {
+		if i > 0 && i%10 == 0 {
+			fs.Infof(nil, Color(terminal.Dim, "retrying in %v..."), time.Until(expireTime).Round(1*time.Second))
+		} else {
+			fs.Debugf(nil, Color(terminal.Dim, "retrying in %v..."), time.Until(expireTime).Round(1*time.Second))
+		}
+		time.Sleep(1 * time.Second)
+	}
+}
diff --git a/cmd/cmd.go b/cmd/cmd.go
index 784d785e9308c..65873bfb531c1 100644
--- a/cmd/cmd.go
+++ b/cmd/cmd.go
@@ -47,13 +47,11 @@ import (
 // Globals
 var (
 	// Flags
-	cpuProfile      = flags.StringP("cpuprofile", "", "", "Write cpu profile to file", "Debugging")
-	memProfile      = flags.StringP("memprofile", "", "", "Write memory profile to file", "Debugging")
-	statsInterval   = flags.DurationP("stats", "", time.Minute*1, "Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable)", "Logging")
-	dataRateUnit    = flags.StringP("stats-unit", "", "bytes", "Show data rate in stats as either 'bits' or 'bytes' per second", "Logging")
-	version         bool
-	retries         = flags.IntP("retries", "", 3, "Retry operations this many times if they fail", "Config")
-	retriesInterval = flags.DurationP("retries-sleep", "", 0, "Interval between retrying operations if they fail, e.g. 500ms, 60s, 5m (0 to disable)", "Config")
+	cpuProfile    = flags.StringP("cpuprofile", "", "", "Write cpu profile to file", "Debugging")
+	memProfile    = flags.StringP("memprofile", "", "", "Write memory profile to file", "Debugging")
+	statsInterval = flags.DurationP("stats", "", time.Minute*1, "Interval between printing stats, e.g. 500ms, 60s, 5m (0 to disable)", "Logging")
+	dataRateUnit  = flags.StringP("stats-unit", "", "bytes", "Show data rate in stats as either 'bits' or 'bytes' per second", "Logging")
+	version       bool
 	// Errors
 	errorCommandNotFound    = errors.New("command not found")
 	errorUncategorized      = errors.New("uncategorized error")
@@ -253,7 +251,7 @@ func Run(Retry bool, showStats bool, cmd *cobra.Command, f func() error) {
 		stopStats = StartStats()
 	}
 	SigInfoHandler()
-	for try := 1; try <= *retries; try++ {
+	for try := 1; try <= ci.Retries; try++ {
 		cmdErr = f()
 		cmdErr = fs.CountError(cmdErr)
 		lastErr := accounting.GlobalStats().GetLastError()
@@ -262,7 +260,7 @@ func Run(Retry bool, showStats bool, cmd *cobra.Command, f func() error) {
 		}
 		if !Retry || !accounting.GlobalStats().Errored() {
 			if try > 1 {
-				fs.Errorf(nil, "Attempt %d/%d succeeded", try, *retries)
+				fs.Errorf(nil, "Attempt %d/%d succeeded", try, ci.Retries)
 			}
 			break
 		}
@@ -282,15 +280,15 @@ func Run(Retry bool, showStats bool, cmd *cobra.Command, f func() error) {
 			}
 		}
 		if lastErr != nil {
-			fs.Errorf(nil, "Attempt %d/%d failed with %d errors and: %v", try, *retries, accounting.GlobalStats().GetErrors(), lastErr)
+			fs.Errorf(nil, "Attempt %d/%d failed with %d errors and: %v", try, ci.Retries, accounting.GlobalStats().GetErrors(), lastErr)
 		} else {
-			fs.Errorf(nil, "Attempt %d/%d failed with %d errors", try, *retries, accounting.GlobalStats().GetErrors())
+			fs.Errorf(nil, "Attempt %d/%d failed with %d errors", try, ci.Retries, accounting.GlobalStats().GetErrors())
 		}
-		if try < *retries {
+		if try < ci.Retries {
 			accounting.GlobalStats().ResetErrors()
 		}
-		if *retriesInterval > 0 {
-			time.Sleep(*retriesInterval)
+		if ci.RetriesInterval > 0 {
+			time.Sleep(ci.RetriesInterval)
 		}
 	}
 	stopStats()
@@ -339,7 +337,6 @@ func Run(Retry bool, showStats bool, cmd *cobra.Command, f func() error) {
 		}
 	}
 	resolveExitCode(cmdErr)
-
 }
 
 // CheckArgs checks there are enough arguments and prints a message if not
@@ -554,7 +551,7 @@ func AddBackendFlags() {
 			} else {
 				fs.Errorf(nil, "Not adding duplicate flag --%s", name)
 			}
-			//flag.Hidden = true
+			// flag.Hidden = true
 		}
 	}
 }
diff --git a/docs/content/bisync.md b/docs/content/bisync.md
index 461ac604fc77e..85250985c04a2 100644
--- a/docs/content/bisync.md
+++ b/docs/content/bisync.md
@@ -91,38 +91,36 @@ Positional arguments:
                 Type 'rclone listremotes' for list of configured remotes.
 
 Optional Flags:
-      --check-access            Ensure expected `RCLONE_TEST` files are found on
-                                both Path1 and Path2 filesystems, else abort.
-      --check-filename FILENAME Filename for `--check-access` (default: `RCLONE_TEST`)
-      --check-sync CHOICE       Controls comparison of final listings:
-                                `true | false | only` (default: true)
-                                If set to `only`, bisync will only compare listings
-                                from the last run but skip actual sync.
-      --filters-file PATH       Read filtering patterns from a file
-      --max-delete PERCENT      Safety check on maximum percentage of deleted files allowed.
-                                If exceeded, the bisync run will abort. (default: 50%)
-      --force                   Bypass `--max-delete` safety check and run the sync.
-                                Consider using with `--verbose`
-      --create-empty-src-dirs   Sync creation and deletion of empty directories. 
-                                  (Not compatible with --remove-empty-dirs)
-      --remove-empty-dirs       Remove empty directories at the final cleanup step.
-  -1, --resync                  Performs the resync run.
-                                Warning: Path1 files may overwrite Path2 versions.
-                                Consider using `--verbose` or `--dry-run` first.
-      --ignore-listing-checksum Do not use checksums for listings 
-                                  (add --ignore-checksum to additionally skip post-copy checksum checks)
-      --resilient               Allow future runs to retry after certain less-serious errors, 
-                                  instead of requiring --resync. Use at your own risk!
-      --localtime               Use local time in listings (default: UTC)
-      --no-cleanup              Retain working files (useful for troubleshooting and testing).
-      --workdir PATH            Use custom working directory (useful for testing).
-                                (default: `~/.cache/rclone/bisync`)
-      --backup-dir1 PATH        --backup-dir for Path1. Must be a non-overlapping path on the same remote.
-      --backup-dir2 PATH        --backup-dir for Path2. Must be a non-overlapping path on the same remote.
-  -n, --dry-run                 Go through the motions - No files are copied/deleted.
-  -v, --verbose                 Increases logging verbosity.
-                                May be specified more than once for more details.
-  -h, --help                    help for bisync
+      --backup-dir1 string                   --backup-dir for Path1. Must be a non-overlapping path on the same remote.
+      --backup-dir2 string                   --backup-dir for Path2. Must be a non-overlapping path on the same remote.
+      --check-access                         Ensure expected RCLONE_TEST files are found on both Path1 and Path2 filesystems, else abort.
+      --check-filename string                Filename for --check-access (default: RCLONE_TEST)
+      --check-sync string                    Controls comparison of final listings: true|false|only (default: true) (default "true")
+      --compare string                       Comma-separated list of bisync-specific compare options ex. 'size,modtime,checksum' (default: 'size,modtime')
+      --conflict-loser ConflictLoserAction   Action to take on the loser of a sync conflict (when there is a winner) or on both files (when there is no winner): , num, pathname, delete (default: num)
+      --conflict-resolve string              Automatically resolve conflicts by preferring the version that is: none, path1, path2, newer, older, larger, smaller (default: none) (default "none")
+      --conflict-suffix string               Suffix to use when renaming a --conflict-loser. Can be either one string or two comma-separated strings to assign different suffixes to Path1/Path2. (default: 'conflict')
+      --create-empty-src-dirs                Sync creation and deletion of empty directories. (Not compatible with --remove-empty-dirs)
+      --download-hash                        Compute hash by downloading when otherwise unavailable. (warning: may be slow and use lots of data!)
+      --filters-file string                  Read filtering patterns from a file
+      --force                                Bypass --max-delete safety check and run the sync. Consider using with --verbose
+  -h, --help                                 help for bisync
+      --ignore-listing-checksum              Do not use checksums for listings (add --ignore-checksum to additionally skip post-copy checksum checks)
+      --max-lock Duration                    Consider lock files older than this to be expired (default: 0 (never expire)) (minimum: 2m) (default 0s)
+      --no-cleanup                           Retain working files (useful for troubleshooting and testing).
+      --no-slow-hash                         Ignore listing checksums only on backends where they are slow
+      --recover                              Automatically recover from interruptions without requiring --resync.
+      --remove-empty-dirs                    Remove ALL empty directories at the final cleanup step.
+      --resilient                            Allow future runs to retry after certain less-serious errors, instead of requiring --resync. Use at your own risk!
+  -1, --resync                               Performs the resync run. Equivalent to --resync-mode path1. Consider using --verbose or --dry-run first.
+      --resync-mode string                   During resync, prefer the version that is: path1, path2, newer, older, larger, smaller (default: path1 if --resync, otherwise none for no resync.) (default "none")
+      --retries int                          Retry operations this many times if they fail (requires --resilient). (default 3)
+      --retries-sleep Duration               Interval between retrying operations if they fail, e.g. 500ms, 60s, 5m (0 to disable) (default 0s)
+      --slow-hash-sync-only                  Ignore slow checksums for listings and deltas, but still consider them during sync calls.
+      --workdir string                       Use custom working dir - useful for testing. (default: {WORKDIR})
+      --max-delete PERCENT                   Safety check on maximum percentage of deleted files allowed. If exceeded, the bisync run will abort. (default: 50%)
+  -n, --dry-run                              Go through the motions - No files are copied/deleted.
+  -v, --verbose                              Increases logging verbosity. May be specified more than once for more details.
 ```
 
 Arbitrary rclone flags may be specified on the
@@ -1836,6 +1834,7 @@ instead of of `--size-only`, when `check` is not available.
 * A new `--max-lock` setting allows lock files to automatically renew and expire, for better automatic recovery when a run is interrupted.
 * Bisync now supports auto-resolving sync conflicts and customizing rename behavior with new [`--conflict-resolve`](#conflict-resolve), [`--conflict-loser`](#conflict-loser), and [`--conflict-suffix`](#conflict-suffix) flags.
 * A new [`--resync-mode`](#resync-mode) flag allows more control over which version of a file gets kept during a `--resync`.
+* Bisync now supports [`--retries`](/docs/#retries-int) and [`--retries-sleep`](/docs/#retries-sleep-time) (when [`--resilient`](#resilient) is set.)
 
 ### `v1.64`
 * Fixed an [issue](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=1.%20Dry%20runs%20are%20not%20completely%20dry) 
diff --git a/fs/config.go b/fs/config.go
index 5b1e483dc0279..5deb921fd1042 100644
--- a/fs/config.go
+++ b/fs/config.go
@@ -72,8 +72,10 @@ type ConfigInfo struct {
 	DeleteMode                 DeleteMode
 	MaxDelete                  int64
 	MaxDeleteSize              SizeSuffix
-	TrackRenames               bool   // Track file renames.
-	TrackRenamesStrategy       string // Comma separated list of strategies used to track renames
+	TrackRenames               bool          // Track file renames.
+	TrackRenamesStrategy       string        // Comma separated list of strategies used to track renames
+	Retries                    int           // High-level retries
+	RetriesInterval            time.Duration // --retries-sleep
 	LowLevelRetries            int
 	UpdateOlder                bool // Skip files that are newer on the destination
 	NoGzip                     bool // Disable compression
@@ -171,6 +173,7 @@ func NewConfig() *ConfigInfo {
 	c.DeleteMode = DeleteModeDefault
 	c.MaxDelete = -1
 	c.MaxDeleteSize = SizeSuffix(-1)
+	c.Retries = 3
 	c.LowLevelRetries = 10
 	c.MaxDepth = -1
 	c.DataRateUnit = "bytes"
diff --git a/fs/config/configflags/configflags.go b/fs/config/configflags/configflags.go
index 232af61d5aa7d..ce891f2b4a6ac 100644
--- a/fs/config/configflags/configflags.go
+++ b/fs/config/configflags/configflags.go
@@ -75,6 +75,8 @@ func AddFlags(ci *fs.ConfigInfo, flagSet *pflag.FlagSet) {
 	flags.FVarP(flagSet, &ci.MaxDeleteSize, "max-delete-size", "", "When synchronizing, limit the total size of deletes", "Sync")
 	flags.BoolVarP(flagSet, &ci.TrackRenames, "track-renames", "", ci.TrackRenames, "When synchronizing, track file renames and do a server-side move if possible", "Sync")
 	flags.StringVarP(flagSet, &ci.TrackRenamesStrategy, "track-renames-strategy", "", ci.TrackRenamesStrategy, "Strategies to use when synchronizing using track-renames hash|modtime|leaf", "Sync")
+	flags.IntVarP(flagSet, &ci.Retries, "retries", "", 3, "Retry operations this many times if they fail", "Config")
+	flags.DurationVarP(flagSet, &ci.RetriesInterval, "retries-sleep", "", 0, "Interval between retrying operations if they fail, e.g. 500ms, 60s, 5m (0 to disable)", "Config")
 	flags.IntVarP(flagSet, &ci.LowLevelRetries, "low-level-retries", "", ci.LowLevelRetries, "Number of low level retries to do", "Config")
 	flags.BoolVarP(flagSet, &ci.UpdateOlder, "update", "u", ci.UpdateOlder, "Skip files that are newer on the destination", "Copy")
 	flags.BoolVarP(flagSet, &ci.UseServerModTime, "use-server-modtime", "", ci.UseServerModTime, "Use server modified time instead of object metadata", "Config")


Test patch: