Repository: rclone/rclone (language: go)

Issue:
bisync: recover from exit without `--resync`
It would be great if bisync could recover from an exit (SIGINT, SIGTERM, SIGKILL) without the need of a `--resync`.

e.g. powercut, Ctrl+C, shutdown

This would save the extra work to get it to play nice with everyday usage. Currently, have to run a script to wait for sync(s) to finish then shutdown/restart/etc. In some cases there is nothing that can be done to prevent this such as as system crash or power cut resulting in `--resync` being required. Or sometimes you just want a quick restart due to a kernel/application update or other issue and instead you have to wait or accept the consequences. 

Thanks


Discussion / hints:
Definitely agree this could and should be improved!

(For others reading this: [here's our previous discussion](https://github.com/rclone/rclone/issues/7332#issuecomment-1732881448) about it, with some good background info about why this is a particularly tough challenge for bisync.)

Some possible solution approaches include:

- "autosave" -- save progress state every X seconds (or after every X transfers) throughout a run, so in the event of a crash, the next run can resume from the last autosave. 
Pros: probably the most accurate method
Cons: may be complicated to build, may slow down the sync with lots of writes to disk, may not be practical with a non-local `--workdir` if that is eventually supported

- "replay queues" -- save the queue list at the beginning of each sync, and in the event of an interruption, the next run attempts to finish the prior run by "replaying" it, before starting the new run.
Pros: easier and less expensive than "autosave"
Cons: potentially not robust to changes that occurred after the interrupted run

- "graceful shutdown" -- upon receiving Ctrl+C, gracefully wrap up what we're doing at the earliest opportunity, cleaning up after ourselves to make things easy for next time.
Pros: relatively easy and clean
Cons: only works when we have some warning (won't help with powercut, sudden crash, etc.)

- "diff from last trusted listing" -- similar idea to https://github.com/rclone/rclone/pull/7410/commits/ef4b47a9824754824a05491c3d4c7a2ef97253d4, but revert the whole listing instead of individual files, and re-diff.
Pros: the easiest and perhaps most elegant option
Cons: there's [one particular edge case](https://github.com/rclone/rclone/issues/7332#:~:text=the%20one%20scenario,and%20move%20on.%22) it wouldn't handle as well as the others (not sure how important that is)

One question I have for you is: how important is it to handle the "crash" scenarios in addition to the ones where we have some warning? i.e. would a "graceful shutdown" option go far enough for you?

Another use case to keep in mind here is using a laptop while traveling, and losing internet or needing to close the laptop mid-sync...
Great points personally I believe this would need to be a combination of 3 & 4 e.g. "graceful shutdown" & "diff from last trusted listing"

Of course, graceful shutdown would need to cover all SIG events not just ctrl+c. SIGKILL a forced and immediate shutdown  would then resort to  "diff from last trusted listing" and handle the edge case of conflicts as in #7471.

With graceful shutdown perhaps you could even resume file upload from where it was left off when files have not changed on either side (particularly useful for larger uploads).


> * Cons: there's [one particular edge case](https://github.com/rclone/rclone/issues/7332#:~:text=the%20one%20scenario,and%20move%20on.%22) it wouldn't handle as well as the others (not sure how important that is)

Related to this, here's another tricky situation:
1. File is deleted on Path1
2. Delete is synced successfully during a run that later (ungracefully) aborts
3. Same file is added back on Path2 before the next run (with same timestamp)

With the "diff from last trusted listing" method, it will look like the file should be deleted from Path2, when in fact it should be copied to Path1. (Note that if the timestamp is different in step 3, it arrives at the right answer because it's deleted vs. newer, not deleted vs. unchanged.) Again, kind of an edge case that probably wouldn't happen too often.

Two other things to track here:

- we'll need to make some kind of change to lock files, for the same reasons discussed in #7471 (else one ungraceful shutdown would break `--resilient` forever)
- I'm not currently sure if it's best to bundle this feature into `--resilient` or make a new flag. Reason being: `--resilient` is currently about _retrying_, which is a slight difference in kind from _recovering_. There may be some users who want the retrying features but are not comfortable with recovering due to risks like the edge case described above.

> handle the edge case of conflicts as in https://github.com/rclone/rclone/issues/7471.

#7471 definitely helps here, but unless you're using `--conflict-loser delete` it's still not ideal, as it's really a false conflict... if we had the correct state, we'd know that only one of the sides has changed (and the unchanged side should be overwritten, not renamed).

> With graceful shutdown perhaps you could even resume file upload from where it was left off when files have not changed on either side (particularly useful for larger uploads).

Many (most?) rclone backends don't support partial uploads, so it's probably not possible, but check out #87... if it is someday supported in `sync`, it will be supported in `bisync` too. 
@crocinsocks I did some work on this and have a working proof-of-concept. Would you like to try it and give feedback?

Fix patch:
diff --git a/cmd/bisync/cmd.go b/cmd/bisync/cmd.go
index 03fabc1995314..6e5b95912bca5 100644
--- a/cmd/bisync/cmd.go
+++ b/cmd/bisync/cmd.go
@@ -48,10 +48,13 @@ type Options struct {
 	SaveQueues            bool // save extra debugging files (test only flag)
 	IgnoreListingChecksum bool
 	Resilient             bool
+	Recover               bool
 	TestFn                TestFunc // test-only option, for mocking errors
 	Retries               int
 	Compare               CompareOpt
 	CompareFlag           string
+	DebugName             string
+	MaxLock               time.Duration
 }
 
 // Default values
@@ -110,6 +113,7 @@ var Opt Options
 
 func init() {
 	Opt.Retries = 3
+	Opt.MaxLock = 0
 	cmd.Root.AddCommand(commandDefinition)
 	cmdFlags := commandDefinition.Flags()
 	// when adding new flags, remember to also update the rc params:
@@ -125,15 +129,18 @@ func init() {
 	flags.StringVarP(cmdFlags, &Opt.Workdir, "workdir", "", Opt.Workdir, makeHelp("Use custom working dir - useful for testing. (default: {WORKDIR})"), "")
 	flags.StringVarP(cmdFlags, &Opt.BackupDir1, "backup-dir1", "", Opt.BackupDir1, "--backup-dir for Path1. Must be a non-overlapping path on the same remote.", "")
 	flags.StringVarP(cmdFlags, &Opt.BackupDir2, "backup-dir2", "", Opt.BackupDir2, "--backup-dir for Path2. Must be a non-overlapping path on the same remote.", "")
+	flags.StringVarP(cmdFlags, &Opt.DebugName, "debugname", "", Opt.DebugName, "Debug by tracking one file at various points throughout a bisync run (when -v or -vv)", "")
 	flags.BoolVarP(cmdFlags, &tzLocal, "localtime", "", tzLocal, "Use local time in listings (default: UTC)", "")
 	flags.BoolVarP(cmdFlags, &Opt.NoCleanup, "no-cleanup", "", Opt.NoCleanup, "Retain working files (useful for troubleshooting and testing).", "")
 	flags.BoolVarP(cmdFlags, &Opt.IgnoreListingChecksum, "ignore-listing-checksum", "", Opt.IgnoreListingChecksum, "Do not use checksums for listings (add --ignore-checksum to additionally skip post-copy checksum checks)", "")
 	flags.BoolVarP(cmdFlags, &Opt.Resilient, "resilient", "", Opt.Resilient, "Allow future runs to retry after certain less-serious errors, instead of requiring --resync. Use at your own risk!", "")
+	flags.BoolVarP(cmdFlags, &Opt.Recover, "recover", "", Opt.Recover, "Automatically recover from interruptions without requiring --resync.", "")
 	flags.IntVarP(cmdFlags, &Opt.Retries, "retries", "", Opt.Retries, "Retry operations this many times if they fail", "")
 	flags.StringVarP(cmdFlags, &Opt.CompareFlag, "compare", "", Opt.CompareFlag, "Comma-separated list of bisync-specific compare options ex. 'size,modtime,checksum' (default: 'size,modtime')", "")
 	flags.BoolVarP(cmdFlags, &Opt.Compare.NoSlowHash, "no-slow-hash", "", Opt.Compare.NoSlowHash, "Ignore listing checksums only on backends where they are slow", "")
 	flags.BoolVarP(cmdFlags, &Opt.Compare.SlowHashSyncOnly, "slow-hash-sync-only", "", Opt.Compare.SlowHashSyncOnly, "Ignore slow checksums for listings and deltas, but still consider them during sync calls.", "")
 	flags.BoolVarP(cmdFlags, &Opt.Compare.DownloadHash, "download-hash", "", Opt.Compare.DownloadHash, "Compute hash by downloading when otherwise unavailable. (warning: may be slow and use lots of data!)", "")
+	flags.DurationVarP(cmdFlags, &Opt.MaxLock, "max-lock", "", Opt.MaxLock, "Consider lock files older than this to be expired (default: 0 (never expire)) (minimum: 2m)", "")
 }
 
 // bisync command definition
diff --git a/cmd/bisync/deltas.go b/cmd/bisync/deltas.go
index 9a93bd7b7b848..323d1c799fb70 100644
--- a/cmd/bisync/deltas.go
+++ b/cmd/bisync/deltas.go
@@ -387,9 +387,12 @@ func (b *bisyncRun) applyDeltas(ctx context.Context, ds1, ds2 *deltaSet) (change
 
 				//if files are identical, leave them alone instead of renaming
 				if (dirs1.has(file) || dirs1.has(alias)) && (dirs2.has(file) || dirs2.has(alias)) {
-					fs.Debugf(nil, "This is a directory, not a file. Skipping equality check and will not rename: %s", file)
+					fs.Infof(nil, "This is a directory, not a file. Skipping equality check and will not rename: %s", file)
 					ls1.getPut(file, skippedDirs1)
 					ls2.getPut(file, skippedDirs2)
+					b.debugFn(file, func() {
+						b.debug(file, fmt.Sprintf("deltas dir: %s, ls1 has name?: %v, ls2 has name?: %v", file, ls1.has(b.DebugName), ls2.has(b.DebugName)))
+					})
 				} else {
 					equal := matches.Has(file)
 					if !equal {
@@ -499,7 +502,7 @@ func (b *bisyncRun) applyDeltas(ctx context.Context, ds1, ds2 *deltaSet) (change
 	}
 
 	// Do the batch operation
-	if copy2to1.NotEmpty() {
+	if copy2to1.NotEmpty() && !b.InGracefulShutdown {
 		changes1 = true
 		b.indent("Path2", "Path1", "Do queued copies to")
 		ctx = b.setBackupDir(ctx, 1)
@@ -508,7 +511,7 @@ func (b *bisyncRun) applyDeltas(ctx context.Context, ds1, ds2 *deltaSet) (change
 		// retries, if any
 		results2to1, err = b.retryFastCopy(ctx, b.fs2, b.fs1, copy2to1, "copy2to1", results2to1, err)
 
-		if err != nil {
+		if !b.InGracefulShutdown && err != nil {
 			return
 		}
 
@@ -516,7 +519,7 @@ func (b *bisyncRun) applyDeltas(ctx context.Context, ds1, ds2 *deltaSet) (change
 		b.syncEmptyDirs(ctx, b.fs1, copy2to1, dirs2, &results2to1, "make")
 	}
 
-	if copy1to2.NotEmpty() {
+	if copy1to2.NotEmpty() && !b.InGracefulShutdown {
 		changes2 = true
 		b.indent("Path1", "Path2", "Do queued copies to")
 		ctx = b.setBackupDir(ctx, 2)
@@ -525,7 +528,7 @@ func (b *bisyncRun) applyDeltas(ctx context.Context, ds1, ds2 *deltaSet) (change
 		// retries, if any
 		results1to2, err = b.retryFastCopy(ctx, b.fs1, b.fs2, copy1to2, "copy1to2", results1to2, err)
 
-		if err != nil {
+		if !b.InGracefulShutdown && err != nil {
 			return
 		}
 
@@ -533,7 +536,7 @@ func (b *bisyncRun) applyDeltas(ctx context.Context, ds1, ds2 *deltaSet) (change
 		b.syncEmptyDirs(ctx, b.fs2, copy1to2, dirs1, &results1to2, "make")
 	}
 
-	if delete1.NotEmpty() {
+	if delete1.NotEmpty() && !b.InGracefulShutdown {
 		if err = b.saveQueue(delete1, "delete1"); err != nil {
 			return
 		}
@@ -541,7 +544,7 @@ func (b *bisyncRun) applyDeltas(ctx context.Context, ds1, ds2 *deltaSet) (change
 		b.syncEmptyDirs(ctx, b.fs1, delete1, dirs1, &results2to1, "remove")
 	}
 
-	if delete2.NotEmpty() {
+	if delete2.NotEmpty() && !b.InGracefulShutdown {
 		if err = b.saveQueue(delete2, "delete2"); err != nil {
 			return
 		}
diff --git a/cmd/bisync/listing.go b/cmd/bisync/listing.go
index c037f7162959e..827a2e14ca2cb 100644
--- a/cmd/bisync/listing.go
+++ b/cmd/bisync/listing.go
@@ -17,6 +17,7 @@ import (
 
 	"github.com/rclone/rclone/cmd/bisync/bilib"
 	"github.com/rclone/rclone/fs"
+	"github.com/rclone/rclone/fs/accounting"
 	"github.com/rclone/rclone/fs/filter"
 	"github.com/rclone/rclone/fs/hash"
 	"github.com/rclone/rclone/fs/operations"
@@ -368,6 +369,12 @@ func (b *bisyncRun) replaceCurrentListings() {
 	b.handleErr(b.newListing2, "error replacing Path2 listing", bilib.CopyFileIfExists(b.newListing2, b.listing2), true, true)
 }
 
+// revertToOldListings reverts to the most recent successful listing
+func (b *bisyncRun) revertToOldListings() {
+	b.handleErr(b.listing1, "error reverting to old Path1 listing", bilib.CopyFileIfExists(b.listing1+"-old", b.listing1), true, true)
+	b.handleErr(b.listing2, "error reverting to old Path2 listing", bilib.CopyFileIfExists(b.listing2+"-old", b.listing2), true, true)
+}
+
 func parseHash(str string) (string, string, error) {
 	if str == "-" {
 		return "", "", nil
@@ -498,6 +505,12 @@ func (b *bisyncRun) modifyListing(ctx context.Context, src fs.Fs, dst fs.Fs, res
 		}
 	}
 
+	b.debugFn(b.DebugName, func() {
+		var rs ResultsSlice = results
+		b.debug(b.DebugName, fmt.Sprintf("modifyListing direction: %s, results has name?: %v", direction, rs.has(b.DebugName)))
+		b.debug(b.DebugName, fmt.Sprintf("modifyListing direction: %s, srcList has name?: %v, dstList has name?: %v", direction, srcList.has(b.DebugName), dstList.has(b.DebugName)))
+	})
+
 	srcWinners := newFileList()
 	dstWinners := newFileList()
 	errors := newFileList()
@@ -657,6 +670,55 @@ func (b *bisyncRun) modifyListing(ctx context.Context, src fs.Fs, dst fs.Fs, res
 		b.recheck(ctxRecheck, src, dst, srcList, dstList, is1to2)
 	}
 
+	if b.InGracefulShutdown {
+		var toKeep []string
+		var toRollback []string
+		fs.Debugf(direction, "stats for %s", direction)
+		trs := accounting.Stats(ctx).Transferred()
+		for _, tr := range trs {
+			b.debugFn(tr.Name, func() {
+				prettyprint(tr, tr.Name, fs.LogLevelInfo)
+			})
+			if tr.Error == nil && tr.Bytes > 0 || tr.Size <= 0 {
+				prettyprint(tr, "keeping: "+tr.Name, fs.LogLevelDebug)
+				toKeep = append(toKeep, tr.Name)
+			}
+		}
+		// Dirs (for the unlikely event that the shutdown was triggered post-sync during syncEmptyDirs)
+		for _, r := range results {
+			if r.Origin == "syncEmptyDirs" {
+				if srcWinners.has(r.Name) || dstWinners.has(r.Name) {
+					toKeep = append(toKeep, r.Name)
+					fs.Infof(r.Name, "keeping empty dir")
+				}
+			}
+		}
+		oldSrc, oldDst := b.getOldLists(is1to2)
+		prettyprint(oldSrc.list, "oldSrc", fs.LogLevelDebug)
+		prettyprint(oldDst.list, "oldDst", fs.LogLevelDebug)
+		prettyprint(srcList.list, "srcList", fs.LogLevelDebug)
+		prettyprint(dstList.list, "dstList", fs.LogLevelDebug)
+		combinedList := Concat(oldSrc.list, oldDst.list, srcList.list, dstList.list)
+		for _, f := range combinedList {
+			if !slices.Contains(toKeep, f) && !slices.Contains(toKeep, b.aliases.Alias(f)) && !b.opt.DryRun {
+				toRollback = append(toRollback, f)
+			}
+		}
+		b.prepareRollback(toRollback, srcList, dstList, is1to2)
+		prettyprint(oldSrc.list, "oldSrc", fs.LogLevelDebug)
+		prettyprint(oldDst.list, "oldDst", fs.LogLevelDebug)
+		prettyprint(srcList.list, "srcList", fs.LogLevelDebug)
+		prettyprint(dstList.list, "dstList", fs.LogLevelDebug)
+
+		// clear stats so we only do this once
+		accounting.MaxCompletedTransfers = 0
+		accounting.Stats(ctx).PruneTransfers()
+	}
+
+	if b.DebugName != "" {
+		b.debug(b.DebugName, fmt.Sprintf("%s pre-save srcList has it?: %v", direction, srcList.has(b.DebugName)))
+		b.debug(b.DebugName, fmt.Sprintf("%s pre-save dstList has it?: %v", direction, dstList.has(b.DebugName)))
+	}
 	// update files
 	err = srcList.save(ctx, srcListing)
 	b.handleErr(srcList, "error saving srcList from modifyListing", err, true, true)
@@ -737,8 +799,8 @@ func (b *bisyncRun) recheck(ctxRecheck context.Context, src, dst fs.Fs, srcList,
 		}
 
 		for _, item := range toRollback {
-			rollback(item, oldSrc, srcList)
-			rollback(item, oldDst, dstList)
+			b.rollback(item, oldSrc, srcList)
+			b.rollback(item, oldDst, dstList)
 		}
 	}
 }
@@ -750,10 +812,72 @@ func (b *bisyncRun) getListingNames(is1to2 bool) (srcListing string, dstListing
 	return b.listing2, b.listing1
 }
 
-func rollback(item string, oldList, newList *fileList) {
+func (b *bisyncRun) rollback(item string, oldList, newList *fileList) {
+	alias := b.aliases.Alias(item)
 	if oldList.has(item) {
 		oldList.getPut(item, newList)
+		fs.Debugf(nil, "adding to newlist: %s", item)
+	} else if oldList.has(alias) {
+		oldList.getPut(alias, newList)
+		fs.Debugf(nil, "adding to newlist: %s", alias)
 	} else {
+		fs.Debugf(nil, "removing from newlist: %s (has it?: %v)", item, newList.has(item))
+		prettyprint(newList.list, "newList", fs.LogLevelDebug)
 		newList.remove(item)
+		newList.remove(alias)
+	}
+}
+
+func (b *bisyncRun) prepareRollback(toRollback []string, srcList, dstList *fileList, is1to2 bool) {
+	if len(toRollback) > 0 {
+		oldSrc, oldDst := b.getOldLists(is1to2)
+		if b.critical {
+			return
+		}
+
+		fs.Debugf("new lists", "src: (%v), dest: (%v)", len(srcList.list), len(dstList.list))
+
+		for _, item := range toRollback {
+			b.debugFn(item, func() {
+				b.debug(item, fmt.Sprintf("pre-rollback oldSrc has it?: %v", oldSrc.has(item)))
+				b.debug(item, fmt.Sprintf("pre-rollback oldDst has it?: %v", oldDst.has(item)))
+				b.debug(item, fmt.Sprintf("pre-rollback srcList has it?: %v", srcList.has(item)))
+				b.debug(item, fmt.Sprintf("pre-rollback dstList has it?: %v", dstList.has(item)))
+			})
+			b.rollback(item, oldSrc, srcList)
+			b.rollback(item, oldDst, dstList)
+			b.debugFn(item, func() {
+				b.debug(item, fmt.Sprintf("post-rollback oldSrc has it?: %v", oldSrc.has(item)))
+				b.debug(item, fmt.Sprintf("post-rollback oldDst has it?: %v", oldDst.has(item)))
+				b.debug(item, fmt.Sprintf("post-rollback srcList has it?: %v", srcList.has(item)))
+				b.debug(item, fmt.Sprintf("post-rollback dstList has it?: %v", dstList.has(item)))
+			})
+		}
+	}
+}
+
+func (b *bisyncRun) getOldLists(is1to2 bool) (*fileList, *fileList) {
+	srcListing, dstListing := b.getListingNames(is1to2)
+	oldSrc, err := b.loadListing(srcListing + "-old")
+	b.handleErr(oldSrc, "error loading old src listing", err, true, true)
+	oldDst, err := b.loadListing(dstListing + "-old")
+	b.handleErr(oldDst, "error loading old dst listing", err, true, true)
+	fs.Debugf("get old lists", "is1to2: %v, oldsrc: %s (%v), olddest: %s (%v)", is1to2, srcListing+"-old", len(oldSrc.list), dstListing+"-old", len(oldDst.list))
+	return oldSrc, oldDst
+}
+
+// Concat returns a new slice concatenating the passed in slices.
+func Concat[S ~[]E, E any](ss ...S) S {
+	size := 0
+	for _, s := range ss {
+		size += len(s)
+		if size < 0 {
+			panic("len out of range")
+		}
+	}
+	newslice := slices.Grow[S](nil, size)
+	for _, s := range ss {
+		newslice = append(newslice, s...)
 	}
+	return newslice
 }
diff --git a/cmd/bisync/lockfile.go b/cmd/bisync/lockfile.go
new file mode 100644
index 0000000000000..282fbdd1abf11
--- /dev/null
+++ b/cmd/bisync/lockfile.go
@@ -0,0 +1,154 @@
+package bisync
+
+import (
+	"encoding/json"
+	"fmt"
+	"io"
+	"os"
+	"strconv"
+	"sync"
+	"time"
+
+	"github.com/rclone/rclone/cmd/bisync/bilib"
+	"github.com/rclone/rclone/fs"
+	"github.com/rclone/rclone/lib/terminal"
+)
+
+const basicallyforever = 200 * 365 * 24 * time.Hour
+
+var stopRenewal func()
+
+var data = struct {
+	Session     string
+	PID         string
+	TimeRenewed time.Time
+	TimeExpires time.Time
+}{}
+
+func (b *bisyncRun) setLockFile() error {
+	b.lockFile = ""
+	b.setLockFileExpiration()
+	if !b.opt.DryRun {
+		b.lockFile = b.basePath + ".lck"
+		if bilib.FileExists(b.lockFile) {
+			if !b.lockFileIsExpired() {
+				errTip := Color(terminal.MagentaFg, "Tip: this indicates that another bisync run (of these same paths) either is still running or was interrupted before completion. \n")
+				errTip += Color(terminal.MagentaFg, "If you're SURE you want to override this safety feature, you can delete the lock file with the following command, then run bisync again: \n")
+				errTip += fmt.Sprintf(Color(terminal.HiRedFg, "rclone deletefile \"%s\""), b.lockFile)
+				return fmt.Errorf(Color(terminal.RedFg, "prior lock file found: %s \n")+errTip, Color(terminal.HiYellowFg, b.lockFile))
+			}
+		}
+
+		pidStr := []byte(strconv.Itoa(os.Getpid()))
+		if err = os.WriteFile(b.lockFile, pidStr, bilib.PermSecure); err != nil {
+			return fmt.Errorf(Color(terminal.RedFg, "cannot create lock file: %s: %w"), b.lockFile, err)
+		}
+		fs.Debugf(nil, "Lock file created: %s", b.lockFile)
+		b.renewLockFile()
+		stopRenewal = b.startLockRenewal()
+	}
+	return nil
+}
+
+func (b *bisyncRun) removeLockFile() {
+	if b.lockFile != "" {
+		stopRenewal()
+		errUnlock := os.Remove(b.lockFile)
+		if errUnlock == nil {
+			fs.Debugf(nil, "Lock file removed: %s", b.lockFile)
+		} else if err == nil {
+			err = errUnlock
+		} else {
+			fs.Errorf(nil, "cannot remove lockfile %s: %v", b.lockFile, errUnlock)
+		}
+		b.lockFile = "" // block removing it again
+	}
+}
+
+func (b *bisyncRun) setLockFileExpiration() {
+	if b.opt.MaxLock > 0 && b.opt.MaxLock < 2*time.Minute {
+		fs.Logf(nil, Color(terminal.YellowFg, "--max-lock cannot be shorter than 2 minutes (unless 0.) Changing --max-lock from %v to %v"), b.opt.MaxLock, 2*time.Minute)
+		b.opt.MaxLock = 2 * time.Minute
+	} else if b.opt.MaxLock <= 0 {
+		b.opt.MaxLock = basicallyforever
+	}
+}
+
+func (b *bisyncRun) renewLockFile() {
+	if b.lockFile != "" && bilib.FileExists(b.lockFile) {
+
+		data.Session = b.basePath
+		data.PID = strconv.Itoa(os.Getpid())
+		data.TimeRenewed = time.Now()
+		data.TimeExpires = time.Now().Add(b.opt.MaxLock)
+
+		// save data file
+		df, err := os.Create(b.lockFile)
+		b.handleErr(b.lockFile, "error renewing lock file", err, true, true)
+		b.handleErr(b.lockFile, "error encoding JSON to lock file", json.NewEncoder(df).Encode(data), true, true)
+		b.handleErr(b.lockFile, "error closing lock file", df.Close(), true, true)
+		if b.opt.MaxLock < basicallyforever {
+			fs.Infof(nil, Color(terminal.HiBlueFg, "lock file renewed for %v. New expiration: %v"), b.opt.MaxLock, data.TimeExpires)
+		}
+	}
+}
+
+func (b *bisyncRun) lockFileIsExpired() bool {
+	if b.lockFile != "" && bilib.FileExists(b.lockFile) {
+		rdf, err := os.Open(b.lockFile)
+		b.handleErr(b.lockFile, "error reading lock file", err, true, true)
+		dec := json.NewDecoder(rdf)
+		for {
+			if err := dec.Decode(&data); err == io.EOF {
+				break
+			}
+		}
+		b.handleErr(b.lockFile, "error closing file", rdf.Close(), true, true)
+		if !data.TimeExpires.IsZero() && data.TimeExpires.Before(time.Now()) {
+			fs.Infof(b.lockFile, Color(terminal.GreenFg, "Lock file found, but it expired at %v. Will delete it and proceed."), data.TimeExpires)
+			markFailed(b.listing1) // listing is untrusted so force revert to prior (if --recover) or create new ones (if --resync)
+			markFailed(b.listing2)
+			return true
+		}
+		fs.Infof(b.lockFile, Color(terminal.RedFg, "Valid lock file found. Expires at %v. (%v from now)"), data.TimeExpires, time.Since(data.TimeExpires).Abs().Round(time.Second))
+		prettyprint(data, "Lockfile info", fs.LogLevelInfo)
+	}
+	return false
+}
+
+// StartLockRenewal renews the lockfile every --max-lock minus one minute.
+//
+// It returns a func which should be called to stop the renewal.
+func (b *bisyncRun) startLockRenewal() func() {
+	if b.opt.MaxLock <= 0 || b.opt.MaxLock >= basicallyforever || b.lockFile == "" {
+		return func() {}
+	}
+	stopLockRenewal := make(chan struct{})
+	var wg sync.WaitGroup
+	wg.Add(1)
+	go func() {
+		defer wg.Done()
+		ticker := time.NewTicker(b.opt.MaxLock - time.Minute)
+		for {
+			select {
+			case <-ticker.C:
+				b.renewLockFile()
+			case <-stopLockRenewal:
+				ticker.Stop()
+				return
+			}
+		}
+	}()
+	return func() {
+		close(stopLockRenewal)
+		wg.Wait()
+	}
+}
+
+func markFailed(file string) {
+	failFile := file + "-err"
+	if bilib.FileExists(file) {
+		_ = os.Remove(failFile)
+		_ = os.Rename(file, failFile)
+	}
+}
diff --git a/cmd/bisync/march.go b/cmd/bisync/march.go
index 79784fc4199c5..60263e2d5eb72 100644
--- a/cmd/bisync/march.go
+++ b/cmd/bisync/march.go
@@ -48,6 +48,7 @@ func (b *bisyncRun) makeMarchListing(ctx context.Context) (*fileList, *fileList,
 	if err != nil {
 		b.handleErr("march", "error during march", err, true, true)
 		b.abort = true
+		return ls1, ls2, err
 	}
 
 	// save files
@@ -160,7 +161,7 @@ func (b *bisyncRun) ForObject(o fs.Object, isPath1 bool) {
 	if b.opt.Compare.Modtime {
 		modtime = o.ModTime(marchCtx).In(TZ)
 	}
-	id := ""     // TODO
+	id := ""     // TODO: ID(o)
 	flags := "-" // "-" for a file and "d" for a directory
 	marchLsLock.Lock()
 	ls.put(o.Remote(), o.Size(), modtime, hashVal, id, flags)
@@ -235,3 +236,12 @@ func (b *bisyncRun) findCheckFiles(ctx context.Context) (*fileList, *fileList, e
 
 	return ls1, ls2, err
 }
+
+// ID returns the ID of the Object if known, or "" if not
+func ID(o fs.Object) string {
+	do, ok := o.(fs.IDer)
+	if !ok {
+		return ""
+	}
+	return do.ID()
+}
diff --git a/cmd/bisync/operations.go b/cmd/bisync/operations.go
index 893b9ca867275..7b0fc00337ebb 100644
--- a/cmd/bisync/operations.go
+++ b/cmd/bisync/operations.go
@@ -9,11 +9,13 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
-	"strconv"
+	"strings"
 	gosync "sync"
+	"time"
 
 	"github.com/rclone/rclone/cmd/bisync/bilib"
 	"github.com/rclone/rclone/fs"
+	"github.com/rclone/rclone/fs/accounting"
 	"github.com/rclone/rclone/fs/filter"
 	"github.com/rclone/rclone/fs/operations"
 	"github.com/rclone/rclone/lib/atexit"
@@ -25,21 +27,27 @@ var ErrBisyncAborted = errors.New("bisync aborted")
 
 // bisyncRun keeps bisync runtime state
 type bisyncRun struct {
-	fs1         fs.Fs
-	fs2         fs.Fs
-	abort       bool
-	critical    bool
-	retryable   bool
-	basePath    string
-	workDir     string
-	listing1    string
-	listing2    string
-	newListing1 string
-	newListing2 string
-	aliases     bilib.AliasMap
-	opt         *Options
-	octx        context.Context
-	fctx        context.Context
+	fs1                fs.Fs
+	fs2                fs.Fs
+	abort              bool
+	critical           bool
+	retryable          bool
+	basePath           string
+	workDir            string
+	listing1           string
+	listing2           string
+	newListing1        string
+	newListing2        string
+	aliases            bilib.AliasMap
+	opt                *Options
+	octx               context.Context
+	fctx               context.Context
+	InGracefulShutdown bool
+	CleanupCompleted   bool
+	SyncCI             *fs.ConfigInfo
+	CancelSync         context.CancelFunc
+	DebugName          string
+	lockFile           string
 }
 
 type queues struct {
@@ -58,9 +66,10 @@ func Bisync(ctx context.Context, fs1, fs2 fs.Fs, optArg *Options) (err error) {
 	defer resetGlobals()
 	opt := *optArg // ensure that input is never changed
 	b := &bisyncRun{
-		fs1: fs1,
-		fs2: fs2,
-		opt: &opt,
+		fs1:       fs1,
+		fs2:       fs2,
+		opt:       &opt,
+		DebugName: opt.DebugName,
 	}
 
 	if opt.CheckFilename == "" {
@@ -93,39 +102,62 @@ func Bisync(ctx context.Context, fs1, fs2 fs.Fs, optArg *Options) (err error) {
 	b.aliases = bilib.AliasMap{}
 
 	// Handle lock file
-	lockFile := ""
-	if !opt.DryRun {
-		lockFile = b.basePath + ".lck"
-		if bilib.FileExists(lockFile) {
-			errTip := Color(terminal.MagentaFg, "Tip: this indicates that another bisync run (of these same paths) either is still running or was interrupted before completion. \n")
-			errTip += Color(terminal.MagentaFg, "If you're SURE you want to override this safety feature, you can delete the lock file with the following command, then run bisync again: \n")
-			errTip += fmt.Sprintf(Color(terminal.HiRedFg, "rclone deletefile \"%s\""), lockFile)
-			return fmt.Errorf(Color(terminal.RedFg, "prior lock file found: %s \n")+errTip, Color(terminal.HiYellowFg, lockFile))
-		}
-
-		pidStr := []byte(strconv.Itoa(os.Getpid()))
-		if err = os.WriteFile(lockFile, pidStr, bilib.PermSecure); err != nil {
-			return fmt.Errorf("cannot create lock file: %s: %w", lockFile, err)
-		}
-		fs.Debugf(nil, "Lock file created: %s", lockFile)
+	err = b.setLockFile()
+	if err != nil {
+		return err
 	}
 
 	// Handle SIGINT
 	var finaliseOnce gosync.Once
-	markFailed := func(file string) {
-		failFile := file + "-err"
-		if bilib.FileExists(file) {
-			_ = os.Remove(failFile)
-			_ = os.Rename(file, failFile)
+
+	// waitFor runs fn() until it returns true or the timeout expires
+	waitFor := func(msg string, totalWait time.Duration, fn func() bool) (ok bool) {
+		const individualWait = 1 * time.Second
+		for i := 0; i < int(totalWait/individualWait); i++ {
+			ok = fn()
+			if ok {
+				return ok
+			}
+			fs.Infof(nil, Color(terminal.YellowFg, "%s: %v"), msg, int(totalWait/individualWait)-i)
+			time.Sleep(individualWait)
 		}
+		return false
 	}
 	finalise := func() {
 		finaliseOnce.Do(func() {
 			if atexit.Signalled() {
-				fs.Logf(nil, "Bisync interrupted. Must run --resync to recover.")
-				markFailed(b.listing1)
-				markFailed(b.listing2)
-				_ = os.Remove(lockFile)
+				if b.opt.Resync {
+					fs.Logf(nil, Color(terminal.GreenFg, "No need to gracefully shutdown during --resync (just run it again.)"))
+				} else {
+					fs.Logf(nil, Color(terminal.YellowFg, "Attempting to gracefully shutdown. (Send exit signal again for immediate un-graceful shutdown.)"))
+					b.InGracefulShutdown = true
+					if b.SyncCI != nil {
+						fs.Infof(nil, Color(terminal.YellowFg, "Telling Sync to wrap up early."))
+						b.SyncCI.MaxTransfer = 1
+						b.SyncCI.MaxDuration = 1 * time.Second
+						b.SyncCI.CutoffMode = fs.CutoffModeSoft
+						gracePeriod := 30 * time.Second // TODO: flag to customize this?
+						if !waitFor("Canceling Sync if not done in", gracePeriod, func() bool { return b.CleanupCompleted }) {
+							fs.Logf(nil, Color(terminal.YellowFg, "Canceling sync and cleaning up"))
+							b.CancelSync()
+							waitFor("Aborting Bisync if not done in", 60*time.Second, func() bool { return b.CleanupCompleted })
+						}
+					} else {
+						// we haven't started to sync yet, so we're good.
+						// no need to worry about the listing files, as we haven't overwritten them yet.
+						b.CleanupCompleted = true
+						fs.Logf(nil, Color(terminal.GreenFg, "Graceful shutdown completed successfully."))
+					}
+				}
+				if !b.CleanupCompleted {
+					if !b.opt.Resync {
+						fs.Logf(nil, Color(terminal.HiRedFg, "Graceful shutdown failed."))
+						fs.Logf(nil, Color(terminal.RedFg, "Bisync interrupted. Must run --resync to recover."))
+					}
+					markFailed(b.listing1)
+					markFailed(b.listing2)
+				}
+				b.removeLockFile()
 			}
 		})
 	}
@@ -135,14 +167,16 @@ func Bisync(ctx context.Context, fs1, fs2 fs.Fs, optArg *Options) (err error) {
 	// run bisync
 	err = b.runLocked(ctx)
 
-	if lockFile != "" {
-		errUnlock := os.Remove(lockFile)
-		if errUnlock == nil {
-			fs.Debugf(nil, "Lock file removed: %s", lockFile)
-		} else if err == nil {
-			err = errUnlock
-		} else {
-			fs.Errorf(nil, "cannot remove lockfile %s: %v", lockFile, errUnlock)
+	b.removeLockFile()
+
+	b.CleanupCompleted = true
+	if b.InGracefulShutdown {
+		if err == context.Canceled || err == accounting.ErrorMaxTransferLimitReachedGraceful {
+			err = nil
+			b.critical = false
+		}
+		if err == nil {
+			fs.Logf(nil, Color(terminal.GreenFg, "Graceful shutdown completed successfully."))
 		}
 	}
 
@@ -162,7 +196,7 @@ func Bisync(ctx context.Context, fs1, fs2 fs.Fs, optArg *Options) (err error) {
 		}
 		return ErrBisyncAborted
 	}
-	if b.abort {
+	if b.abort && !b.InGracefulShutdown {
 		fs.Logf(nil, Color(terminal.RedFg, "Bisync aborted. Please try again."))
 	}
 	if err == nil {
@@ -221,21 +255,41 @@ func (b *bisyncRun) runLocked(octx context.Context) (err error) {
 
 	// Check for existence of prior Path1 and Path2 listings
 	if !bilib.FileExists(b.listing1) || !bilib.FileExists(b.listing2) {
-		// On prior critical error abort, the prior listings are renamed to .lst-err to lock out further runs
-		b.critical = true
-		b.retryable = true
-		errTip := Color(terminal.MagentaFg, "Tip: here are the filenames we were looking for. Do they exist? \n")
-		errTip += fmt.Sprintf(Color(terminal.CyanFg, "Path1: %s\n"), Color(terminal.HiBlueFg, b.listing1))
-		errTip += fmt.Sprintf(Color(terminal.CyanFg, "Path2: %s\n"), Color(terminal.HiBlueFg, b.listing2))
-		errTip += Color(terminal.MagentaFg, "Try running this command to inspect the work dir: \n")
-		errTip += fmt.Sprintf(Color(terminal.HiCyanFg, "rclone lsl \"%s\""), b.workDir)
+		if b.opt.Recover && bilib.FileExists(b.listing1+"-old") && bilib.FileExists(b.listing2+"-old") {
+			errTip := fmt.Sprintf(Color(terminal.CyanFg, "Path1: %s\n"), Color(terminal.HiBlueFg, b.listing1))
+			errTip += fmt.Sprintf(Color(terminal.CyanFg, "Path2: %s"), Color(terminal.HiBlueFg, b.listing2))
+			fs.Logf(nil, Color(terminal.YellowFg, "Listings not found. Reverting to prior backup as --recover is set. \n")+errTip)
+			if opt.CheckSync != CheckSyncFalse {
+				// Run CheckSync to ensure old listing is valid (garbage in, garbage out!)
+				fs.Infof(nil, "Validating backup listings for Path1 %s vs Path2 %s", quotePath(path1), quotePath(path2))
+				if err = b.checkSync(b.listing1+"-old", b.listing2+"-old"); err != nil {
+					b.critical = true
+					b.retryable = true
+					return err
+				}
+				fs.Infof(nil, Color(terminal.GreenFg, "Backup listing is valid."))
+			}
+			b.revertToOldListings()
+		} else {
+			// On prior critical error abort, the prior listings are renamed to .lst-err to lock out further runs
+			b.critical = true
+			b.retryable = true
+			errTip := Color(terminal.MagentaFg, "Tip: here are the filenames we were looking for. Do they exist? \n")
+			errTip += fmt.Sprintf(Color(terminal.CyanFg, "Path1: %s\n"), Color(terminal.HiBlueFg, b.listing1))
+			errTip += fmt.Sprintf(Color(terminal.CyanFg, "Path2: %s\n"), Color(terminal.HiBlueFg, b.listing2))
+			errTip += Color(terminal.MagentaFg, "Try running this command to inspect the work dir: \n")
+			errTip += fmt.Sprintf(Color(terminal.HiCyanFg, "rclone lsl \"%s\""), b.workDir)
 
-		return errors.New("cannot find prior Path1 or Path2 listings, likely due to critical error on prior run \n" + errTip)
+			return errors.New("cannot find prior Path1 or Path2 listings, likely due to critical error on prior run \n" + errTip)
+		}
 	}
 
 	fs.Infof(nil, "Building Path1 and Path2 listings")
 	ls1, ls2, err = b.makeMarchListing(fctx)
-	if err != nil {
+	if err != nil || accounting.Stats(fctx).Errored() {
+		fs.Errorf(nil, Color(terminal.RedFg, "There were errors while building listings. Aborting as it is too dangerous to continue."))
+		b.critical = true
+		b.retryable = true
 		return err
 	}
 
@@ -306,31 +360,50 @@ func (b *bisyncRun) runLocked(octx context.Context) (err error) {
 		fs.Infof(nil, "Applying changes")
 		changes1, changes2, results2to1, results1to2, queues, err = b.applyDeltas(octx, ds1, ds2)
 		if err != nil {
-			b.critical = true
-			// b.retryable = true // not sure about this one
-			return err
+			if b.InGracefulShutdown && (err == context.Canceled || err == accounting.ErrorMaxTransferLimitReachedGraceful || strings.Contains(err.Error(), "context canceled")) {
+				fs.Infof(nil, "Ignoring sync error due to Graceful Shutdown: %v", err)
+			} else {
+				b.critical = true
+				// b.retryable = true // not sure about this one
+				return err
+			}
 		}
 	}
 
 	// Clean up and check listings integrity
 	fs.Infof(nil, "Updating listings")
 	var err1, err2 error
+	if b.DebugName != "" {
+		l1, _ := b.loadListing(b.listing1)
+		l2, _ := b.loadListing(b.listing2)
+		newl1, _ := b.loadListing(b.newListing1)
+		newl2, _ := b.loadListing(b.newListing2)
+		b.debug(b.DebugName, fmt.Sprintf("pre-saveOldListings, ls1 has name?: %v, ls2 has name?: %v", l1.has(b.DebugName), l2.has(b.DebugName)))
+		b.debug(b.DebugName, fmt.Sprintf("pre-saveOldListings, newls1 has name?: %v, newls2 has name?: %v", newl1.has(b.DebugName), newl2.has(b.DebugName)))
+	}
 	b.saveOldListings()
 	// save new listings
+	// NOTE: "changes" in this case does not mean this run vs. last run, it means start of this run vs. end of this run.
+	// i.e. whether we can use the March lst-new as this side's lst without modifying it.
 	if noChanges {
 		b.replaceCurrentListings()
 	} else {
-		if changes1 { // 2to1
+		if changes1 || b.InGracefulShutdown { // 2to1
 			err1 = b.modifyListing(fctx, b.fs2, b.fs1, results2to1, queues, false)
 		} else {
 			err1 = bilib.CopyFileIfExists(b.newListing1, b.listing1)
 		}
-		if changes2 { // 1to2
+		if changes2 || b.InGracefulShutdown { // 1to2
 			err2 = b.modifyListing(fctx, b.fs1, b.fs2, results1to2, queues, true)
 		} else {
 			err2 = bilib.CopyFileIfExists(b.newListing2, b.listing2)
 		}
 	}
+	if b.DebugName != "" {
+		l1, _ := b.loadListing(b.listing1)
+		l2, _ := b.loadListing(b.listing2)
+		b.debug(b.DebugName, fmt.Sprintf("post-modifyListing, ls1 has name?: %v, ls2 has name?: %v", l1.has(b.DebugName), l2.has(b.DebugName)))
+	}
 	err = err1
 	if err == nil {
 		err = err2
@@ -609,6 +682,18 @@ func (b *bisyncRun) setBackupDir(ctx context.Context, destPath int) context.Cont
 	return ctx
 }
 
+func (b *bisyncRun) debug(nametocheck, msgiftrue string) {
+	if b.DebugName != "" && b.DebugName == nametocheck {
+		fs.Infof(Color(terminal.MagentaBg, "DEBUGNAME "+b.DebugName), Color(terminal.MagentaBg, msgiftrue))
+	}
+}
+
+func (b *bisyncRun) debugFn(nametocheck string, fn func()) {
+	if b.DebugName != "" && b.DebugName == nametocheck {
+		fn()
+	}
+}
+
 // mainly to make sure tests don't interfere with each other when running more than one
 func resetGlobals() {
 	downloadHash = false
diff --git a/cmd/bisync/queue.go b/cmd/bisync/queue.go
index 6f181e1a9dc50..120b9e9bd2963 100644
--- a/cmd/bisync/queue.go
+++ b/cmd/bisync/queue.go
@@ -11,6 +11,7 @@ import (
 
 	"github.com/rclone/rclone/cmd/bisync/bilib"
 	"github.com/rclone/rclone/fs"
+	"github.com/rclone/rclone/fs/accounting"
 	"github.com/rclone/rclone/fs/filter"
 	"github.com/rclone/rclone/fs/hash"
 	"github.com/rclone/rclone/fs/operations"
@@ -35,6 +36,19 @@ type Results struct {
 	IsWinner bool
 	IsSrc    bool
 	IsDst    bool
+	Origin   string
+}
+
+// ResultsSlice is a slice of Results (obviously)
+type ResultsSlice []Results
+
+func (rs *ResultsSlice) has(name string) bool {
+	for _, r := range *rs {
+		if r.Name == name {
+			return true
+		}
+	}
+	return false
 }
 
 var logger = operations.NewLoggerOpt()
@@ -96,10 +110,11 @@ func WriteResults(ctx context.Context, sigil operations.Sigil, src, dst fs.DirEn
 
 	opt := operations.GetLoggerOpt(ctx)
 	result := Results{
-		Sigil: sigil,
-		Src:   FsPathIfAny(src),
-		Dst:   FsPathIfAny(dst),
-		Err:   err,
+		Sigil:  sigil,
+		Src:    FsPathIfAny(src),
+		Dst:    FsPathIfAny(dst),
+		Err:    err,
+		Origin: "sync",
 	}
 
 	result.Winner = operations.WinningSide(ctx, sigil, src, dst, err)
@@ -202,6 +217,9 @@ func (b *bisyncRun) preCopy(ctx context.Context) context.Context {
 }
 
 func (b *bisyncRun) fastCopy(ctx context.Context, fsrc, fdst fs.Fs, files bilib.Names, queueName string) ([]Results, error) {
+	if b.InGracefulShutdown {
+		return nil, nil
+	}
 	ctx = b.preCopy(ctx)
 	if err := b.saveQueue(files, queueName); err != nil {
 		return nil, err
@@ -220,6 +238,9 @@ func (b *bisyncRun) fastCopy(ctx context.Context, fsrc, fdst fs.Fs, files bilib.
 		}
 	}
 
+	b.SyncCI = fs.GetConfig(ctxCopy)      // allows us to request graceful shutdown
+	accounting.MaxCompletedTransfers = -1 // we need a complete list in the event of graceful shutdown
+	ctxCopy, b.CancelSync = context.WithCancel(ctxCopy)
 	b.testFn()
 	err := sync.Sync(ctxCopy, fdst, fsrc, b.opt.CreateEmptySrcDirs)
 	prettyprint(logger, "logger", fs.LogLevelDebug)
@@ -236,7 +257,7 @@ func (b *bisyncRun) fastCopy(ctx context.Context, fsrc, fdst fs.Fs, files bilib.
 }
 
 func (b *bisyncRun) retryFastCopy(ctx context.Context, fsrc, fdst fs.Fs, files bilib.Names, queueName string, results []Results, err error) ([]Results, error) {
-	if err != nil && b.opt.Resilient && b.opt.Retries > 1 {
+	if err != nil && b.opt.Resilient && !b.InGracefulShutdown && b.opt.Retries > 1 {
 		for tries := 1; tries <= b.opt.Retries; tries++ {
 			fs.Logf(queueName, Color(terminal.YellowFg, "Received error: %v - retrying as --resilient is set. Retry %d/%d"), err, tries, b.opt.Retries)
 			results, err = b.fastCopy(ctx, fsrc, fdst, files, queueName)
@@ -259,6 +280,10 @@ func (b *bisyncRun) resyncDir(ctx context.Context, fsrc, fdst fs.Fs) ([]Results,
 
 // operation should be "make" or "remove"
 func (b *bisyncRun) syncEmptyDirs(ctx context.Context, dst fs.Fs, candidates bilib.Names, dirsList *fileList, results *[]Results, operation string) {
+	if b.InGracefulShutdown {
+		return
+	}
+	fs.Debugf(nil, "syncing empty dirs")
 	if b.opt.CreateEmptySrcDirs && (!b.opt.Resync || operation == "make") {
 
 		candidatesList := candidates.ToList()
@@ -272,10 +297,11 @@ func (b *bisyncRun) syncEmptyDirs(ctx context.Context, dst fs.Fs, candidates bil
 			if dirsList.has(s) { //make sure it's a dir, not a file
 				r := Results{}
 				r.Name = s
-				r.Size = 0
+				r.Size = -1
 				r.Modtime = dirsList.getTime(s).In(time.UTC)
 				r.Flags = "d"
 				r.Err = nil
+				r.Origin = "syncEmptyDirs"
 				r.Winner = operations.Winner{ // note: Obj not set
 					Side: "src",
 					Err:  nil,
diff --git a/docs/content/bisync.md b/docs/content/bisync.md
index 704d992a96f18..84b4a8735524c 100644
--- a/docs/content/bisync.md
+++ b/docs/content/bisync.md
@@ -487,25 +487,99 @@ See also: [Concurrent modifications](#concurrent-modifications), [`--resilient`]
 
 ***Caution: this is an experimental feature. Use at your own risk!***
 
-By default, most errors or interruptions will cause bisync to abort and 
-require [`--resync`](#resync) to recover. This is a safety feature, 
-to prevent bisync from running again until a user checks things out. 
-However, in some cases, bisync can go too far and enforce a lockout when one isn't actually necessary, 
-like for certain less-serious errors that might resolve themselves on the next run. 
-When `--resilient` is specified, bisync tries its best to recover and self-correct, 
-and only requires `--resync` as a last resort when a human's involvement is absolutely necessary. 
-The intended use case is for running bisync as a background process (such as via scheduled [cron](#cron)).
-
-When using `--resilient` mode, bisync will still report the error and abort, 
-however it will not lock out future runs -- allowing the possibility of retrying at the next normally scheduled time, 
-without requiring a `--resync` first. Examples of such retryable errors include 
-access test failures, missing listing files, and filter change detections. 
-These safety features will still prevent the *current* run from proceeding -- 
-the difference is that if conditions have improved by the time of the *next* run, 
-that next run will be allowed to proceed. 
-Certain more serious errors will still enforce a `--resync` lockout, even in `--resilient` mode, to prevent data loss.
-
-Behavior of `--resilient` may change in a future version.
+By default, most errors or interruptions will cause bisync to abort and
+require [`--resync`](#resync) to recover. This is a safety feature,  to prevent
+bisync from running again until a user checks things out.  However, in some
+cases, bisync can go too far and enforce a lockout when one isn't actually
+necessary,  like for certain less-serious errors that might resolve themselves
+on the next run.  When `--resilient` is specified, bisync tries its best to
+recover and self-correct,  and only requires `--resync` as a last resort when a
+human's involvement is absolutely necessary.  The intended use case is for
+running bisync as a background process (such as via scheduled [cron](#cron)).
+
+When using `--resilient` mode, bisync will still report the error and abort,
+however it will not lock out future runs -- allowing the possibility of
+retrying at the next normally scheduled time,  without requiring a `--resync`
+first. Examples of such retryable errors include  access test failures, missing
+listing files, and filter change detections.  These safety features will still
+prevent the *current* run from proceeding --  the difference is that if
+conditions have improved by the time of the *next* run,  that next run will be
+allowed to proceed.  Certain more serious errors will still enforce a
+`--resync` lockout, even in `--resilient` mode, to prevent data loss.
+
+Behavior of `--resilient` may change in a future version. (See also:
+[`--recover`](#recover), [`--max-lock`](#max-lock), [Graceful
+Shutdown](#graceful-shutdown))
+
+### --recover
+
+If `--recover` is set, in the event of a sudden interruption or other
+un-graceful shutdown, bisync will attempt to automatically recover on the next
+run, instead of requiring `--resync`. Bisync is able to recover robustly by
+keeping one "backup" listing at all times, representing the state of both paths
+after the last known successful sync. Bisync can then compare the current state
+with this snapshot to determine which changes it needs to retry. Changes that
+were synced after this snapshot (during the run that was later interrupted)
+will appear to bisync as if they are "new or changed on both sides", but in
+most cases this is not a problem, as bisync will simply do its usual "equality
+check" and learn that no action needs to be taken on these files, since they
+are already identical on both sides.
+
+In the rare event that a file is synced successfully during a run that later
+aborts, and then that same file changes AGAIN before the next run, bisync will
+think it is a sync conflict, and handle it accordingly. (From bisync's
+perspective, the file has changed on both sides since the last trusted sync,
+and the files on either side are not currently identical.) Therefore,
+`--recover` carries with it a slightly increased chance of having conflicts --
+though in practice this is pretty rare, as the conditions required to cause it
+are quite specific. This risk can be reduced by using bisync's ["Graceful
+Shutdown"](#graceful-shutdown) mode (triggered by sending `SIGINT` or
+`Ctrl+C`), when you have the choice, instead of forcing a sudden termination.
+
+`--recover` and `--resilient` are similar, but distinct -- the main difference
+is that `--resilient` is about _retrying_, while `--recover` is about
+_recovering_. Most users will probably want both. `--resilient` allows retrying
+when bisync has chosen to abort itself due to safety features such as failing
+`--check-access` or detecting a filter change. `--resilient` does not cover
+external interruptions such as a user shutting down their computer in the
+middle of a sync -- that is what `--recover` is for.
+
+### --max-lock
+
+Bisync uses [lock files](#lock-file) as a safety feature to prevent
+interference from other bisync runs while it is running. Bisync normally
+removes these lock files at the end of a run, but if bisync is abruptly
+interrupted, these files will be left behind. By default, they will lock out
+all future runs, until the user has a chance to manually check things out and
+remove the lock. As an alternative, `--max-lock` can be used to make them
+automatically expire after a certain period of time, so that future runs are
+not locked out forever, and auto-recovery is possible. `--max-lock` can be any
+duration `2m` or greater (or `0` to disable). If set, lock files older than
+this will be considered "expired", and future runs will be allowed to disregard
+them and proceed. (Note that the `--max-lock` duration must be set by the
+process that left the lock file -- not the later one interpreting it.)
+
+If set, bisync will also "renew" these lock files every `--max-lock minus one
+minute` throughout a run, for extra safety. (For example, with `--max-lock 5m`,
+bisync would renew the lock file (for another 5 minutes) every 4 minutes until
+the run has completed.) In other words, it should not be possible for a lock
+file to pass its expiration time while the process that created it is still
+running -- and you can therefore be reasonably sure that any _expired_ lock
+file you may find was left there by an interrupted run, not one that is still
+running and just taking awhile.
+
+If `--max-lock` is `0` or not set, the default is that lock files will never
+expire, and will block future runs (of these same two bisync paths)
+indefinitely.
+
+For maximum resilience from disruptions, consider setting a relatively short
+duration like `--max-lock 2m` along with [`--resilient`](#resilient) and
+[`--recover`](#recover), and a relatively frequent [cron schedule](#cron). The
+result will be a very robust "set-it-and-forget-it" bisync run that can
+automatically bounce back from almost any interruption it might encounter,
+without requiring the user to get involved and run a `--resync`. (See also:
+[Graceful Shutdown](#graceful-shutdown) mode)
+
 
 ### --backup-dir1 and --backup-dir2
 
@@ -645,7 +719,8 @@ typically at `${HOME}/.cache/rclone/bisync/` on Linux.
 Some errors are considered temporary and re-running the bisync is not blocked.
 The _critical return_ blocks further bisync runs.
 
-See also: [`--resilient`](#resilient)
+See also: [`--resilient`](#resilient), [`--recover`](#recover),
+[`--max-lock`](#max-lock), [Graceful Shutdown](#graceful-shutdown)
 
 ### Lock file
 
@@ -657,6 +732,8 @@ Delete the lock file as part of debugging the situation.
 The lock file effectively blocks follow-on (e.g., scheduled by _cron_) runs
 when the prior invocation is taking a long time.
 The lock file contains _PID_ of the blocking process, which may help in debug.
+Lock files can be set to automatically expire after a certain amount of time,
+using the [`--max-lock`](#max-lock) flag.
 
 **Note**
 that while concurrent bisync runs are allowed, _be very cautious_
@@ -670,6 +747,32 @@ lest there be replicated files, deleted files and general mayhem.
 - `1` for a non-critical failing run (a rerun may be successful),
 - `2` for a critically aborted run (requires a `--resync` to recover).
 
+### Graceful Shutdown
+
+Bisync has a "Graceful Shutdown" mode which is activated by sending `SIGINT` or
+pressing `Ctrl+C` during a run. Once triggered, bisync will use best efforts to
+exit cleanly before the timer runs out. If bisync is in the middle of
+transferring files, it will attempt to cleanly empty its queue by finishing
+what it has started but not taking more. If it cannot do so within 30 seconds,
+it will cancel the in-progress transfers at that point and then give itself a
+maximum of 60 seconds to wrap up, save its state for next time, and exit. With
+the `-vP` flags you will see constant status updates and a final confirmation
+of whether or not the graceful shutdown was successful.
+
+At any point during the "Graceful Shutdown" sequence, a second `SIGINT` or
+`Ctrl+C` will trigger an immediate, un-graceful exit, which will leave things
+in a messier state. Usually a robust recovery will still be possible if using
+[`--recover`](#recover) mode, otherwise you will need to do a `--resync`.
+
+If you plan to use Graceful Shutdown mode, it is recommended to use
+[`--resilient`](#resilient) and [`--recover`](#recover), and it is important to
+NOT use [`--inplace`](/docs/#inplace), otherwise you risk leaving
+partially-written files on one side, which may be confused for real files on
+the next run. Note also that in the event of an abrupt interruption, a [lock
+file](#lock-file) will be left behind to block concurrent runs. You will need
+to delete it before you can proceed with the next run (or wait for it to
+expire on its own, if using `--max-lock`.)
+
 ## Limitations
 
 ### Supported backends
@@ -1498,6 +1601,9 @@ for performance improvements and less [risk of error](https://forum.rclone.org/t
 instead of of `--size-only`, when `check` is not available.
 * Bisync no longer fails to find the correct listing file when configs are overridden with backend-specific flags.
 * Bisync now fully supports comparing based on any combination of size, modtime, and checksum, lifting the prior restriction on backends without modtime support.
+* Bisync now supports a "Graceful Shutdown" mode to cleanly cancel a run early without requiring `--resync`.
+* New `--recover` flag allows robust recovery in the event of interruptions, without requiring `--resync`.
+* A new `--max-lock` setting allows lock files to automatically renew and expire, for better automatic recovery when a run is interrupted.
 
 ### `v1.64`
 * Fixed an [issue](https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=1.%20Dry%20runs%20are%20not%20completely%20dry) 


Test patch: