Repository: rclone/rclone (language: go)

Issue:
sync fails to remove directories on the destination that do not exist on the source
#### What is the problem you are having with rclone?

I have noticed that under some circumstances (in my case, using a crypted remote) `rclone sync` fails to remove directories that should be removed since they do not exist on the source.  In my particular circumstance, it seems like the problem is that `rclone sync` uses something like the `rmdir` command, which (somewhat silently) fails if the directory has any contents left, even broken contents that `rclone` is ignoring.

I think this could be fixed by making the `sync` use `purge` functionality instead of `rmdir` to remove directories that don't exist on the source.

Here is the output from the `rclone sync` command, which claims that it has deleted 23 directories.

```
> rclone sync --progress --bwlimit 1M --links --track-renames --track-renames-strategy size,leaf,modtime --transfers 10 --max-backlog 20 --tpslimit 40 /raid/audio odc:audio
Transferred:              0 B / 0 B, -, 0 B/s, ETA -
Checks:            296345 / 296345, 100%
Deleted:                0 (files), 23 (dirs)
Elapsed time:     13m50.5s
```

However, a subsequent run of the sync without any changes being made still shows it trying to remove the 23 directories it claimed it had removed on the last run.  Note that no warnings or errors were reported at all (which is a separate problem).

So I ran again with `-v` to see what was going on with the directories it thought it was removing and got messages like this:

```
2022-09-12 16:44:10 INFO  : work/new/waiting/Johann_Sebastian_Bach_Canadian_Brass_-_Art_Of_The_Fugue: Removing directory
2022-09-12 16:44:10 INFO  : work/new/waiting/Johann Sebastian Bach; Canadian Brass - Art of the Fugue: Removing directory
```

Note that again, even with `-v`, no warnings or errors are reported with the supposedly removed directories.

So I tried manually listing (with `lsl`) one of the problem directories:

```
> rclone lsl odc:audio/work/new/waiting/Johann_Sebastian_Bach_Canadian_Brass_-_Art_Of_The_Fugue
>
```

Apparently no contents.  So I tried `rmdir` on that directory:

```
> rclone rmdir odc:audio/work/new/waiting/Johann_Sebastian_Bach_Canadian_Brass_-_Art_Of_The_Fugue2022/09/12 16:48:06 ERROR : Attempt 1/3 failed with 1 errors and: folder not empty
2022/09/12 16:48:06 ERROR : Attempt 2/3 failed with 1 errors and: folder not empty
2022/09/12 16:48:06 ERROR : Attempt 3/3 failed with 1 errors and: folder not empty
2022/09/12 16:48:06 Failed to rmdir: folder not empty
```

 So I tried listing the directory again with `lsl -v` and still got no info.  Tried again with `lsl -vv` and finally got some useful info:

```
rclone lsl -vv odc:audio/work/new/waiting/Johann_Sebastian_Bach_Canadian_Brass_-_Art_Of_The_Fugue
2022/09/12 16:50:19 DEBUG : rclone: Version "v1.59.2-beta.6379.603efbfe7" starting with parameters ["rclone" "lsl" "-vv" "odc:audio/work/new/waiting/Johann_Sebastian_Bach_Canadian_Brass_-_Art_Of_The_Fugue"]
2022/09/12 16:50:19 DEBUG : Creating backend with remote "odc:audio/work/new/waiting/Johann_Sebastian_Bach_Canadian_Brass_-_Art_Of_The_Fugue"
2022/09/12 16:50:19 DEBUG : Using config file from "<redacted>"
2022/09/12 16:50:19 DEBUG : Creating backend with remote "od:<redacted>"
2022/09/12 16:50:20 DEBUG : Starting OpenDrive session with ID: <redacted>
2022/09/12 16:50:22 DEBUG : 7lnt06c8j2d8pvtcjv5nav7r5leij3atc8nn371onikjps3jbqk8gscemr6fpeibni2pame6u9hmp32kq50llb81i7vf2sbgu3jifh9k9q542dgnp9b9hira8q0l23luc57me0lm0dreqce9tdmems3es9a7v9nnk82ljvbmmqvf4khe3s4r69ethvqjka3pvccub6bi3hftu623tnjonrtuq12i7atloj8g0oc4538kto2jtmgvjpbtj1bsb2d: Skipping undecryptable file name: not a multiple of blocksize
2022/09/12 16:50:22 DEBUG : m7tnlcgevcrm559nd6mctfirjnmcu6c8esth5esmcba052hqqk5vpav73h5i453353ndsjkbqvbq5n13fpbveqi0gm2j0ljlqdg9l3qv2h2lj9mrrt0mpd5ulebr49svbtbh2jmifsdgu5ha8e16nc6q240qjaufap950lu4b3a3e7dvj5demeqc52d9cp8t5cfj77a0tjbmmjq9m0ta21jqne22qp58l5htbqh4cnjrshrd8g21k5cuptoprum: Skipping undecryptable file name: not a multiple of blocksize
2022/09/12 16:50:22 DEBUG : 2 go routines active
>
```

So it appears there are two files that rclone is ignoring that exist in the directory it claims it is removing, yet it is still not removing the directory when `rclone sync` is used, and is not providing any warning or error messages about the removal failing without `-v` being used.

Attempting to `purge` these directories manually seems to work:

```
> rclone purge -v odc:audio/work/new/waiting/Johann_Sebastian_Bach_Canadian_Brass_-_Art_Of_The_Fugue
>
```

And a repeated purge shows that the directory is actually gone now:

```
> rclone purge -v odc:audio/work/new/waiting/Johann_Sebastian_Bach_Canadian_Brass_-_Art_Of_The_Fugue
2022/09/12 16:58:37 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2022/09/12 16:58:37 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2022/09/12 16:58:37 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2022/09/12 16:58:37 Failed to purge: directory not found
```

#### What is your rclone version (output from `rclone version`)

rclone v1.59.2-beta.6379.603efbfe7
- os/version: debian 11.4 (64 bit)
- os/kernel: 5.10.0-17-amd64 (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.17
- go/linking: dynamic
- go/tags: none

#### Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Debian 11.4

#### Which cloud storage system are you using? (e.g. Google Drive)

Opendrive with crypt remote

<!--- Please keep the note below for others who read your bug report. -->

#### How to use GitHub

* Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue.
* Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
* Subscribe to receive notifications on status change and new comments.



Discussion / hints:
I believe this also points out a problem with `sync` that it allows files that it is "ignoring" to remain existing and un-reported even though `sync` should remove any files that are not on the source, even if they are not readable for some reason.
That's because you are mixing crypt and non crypt.

Fix your remotes so they don't mix..
@animosity22 that is beside the point.  `sync` should make the destination identical to the source, and should delete any files and directories that do not exist on the source, whether they are crypted or not.  In this case, sync is not doing that.
And as an aside, these leftover files were at some point created by `rclone`, as this remote has never been touched as non-crypt.  I haven't been able to track down why yet, but when I do I'll be filing a bug on that as well.  I think it has something to do with exceeding the storable cryptable filename length.
If the file name is too long, you get a message stating such as that's an easy one to produce.

The use case here is at some point, you probably accidentally copied from a non crypt location to a crypt location by mistake.
@animosity22 that is a presumption without any evidence on your part, and I'm stating that that's not the case.  I merely offered the info to dispel your presumption, even though it is off topic to this bug report.
Isn't yours a presumption too as there's no evidence? :)

I'm just sharing what I've seen over reading tons of posts, debug logs, etc. The only time I've seen that use case is by an accidental copy from a non crypted remote to a crypt location in a remote.

That's my "theory" which has no malice, ill will, bad intent or anything else.

Could there be a bug? Sure. Could I be incorrect? Sure. I'm sharing my thoughts based on my experience.
This is not an enhancement, it is a bug.  Please mark it as so, since it clearly a violation of the stated operation of `sync`.  There is an exception made in the documentation for duplicate objects, but that is not the case here (directory vs files with errors).
It's really an enhancement as garbage in there isn't a bug but a flow issue or a bug somewhere else per your message if something else caused it but the label really doesn't matter than much imo so I changed it.

Fix patch:
diff --git a/backend/crypt/crypt.go b/backend/crypt/crypt.go
index 54b1335dc833d..8fdeb7a06edf4 100644
--- a/backend/crypt/crypt.go
+++ b/backend/crypt/crypt.go
@@ -130,6 +130,16 @@ trying to recover an encrypted file with errors and it is desired to
 recover as much of the file as possible.`,
 			Default:  false,
 			Advanced: true,
+		}, {
+			Name: "strict_names",
+			Help: `If set, this will raise an error when crypt comes across a filename that can't be decrypted.
+
+(By default, rclone will just log a NOTICE and continue as normal.)
+This can happen if encrypted and unencrypted files are stored in the same
+directory (which is not recommended.) It may also indicate a more serious
+problem that should be investigated.`,
+			Default:  false,
+			Advanced: true,
 		}, {
 			Name: "filename_encoding",
 			Help: `How to encode the encrypted filename to text string.
@@ -294,6 +304,7 @@ type Options struct {
 	PassBadBlocks           bool   `config:"pass_bad_blocks"`
 	FilenameEncoding        string `config:"filename_encoding"`
 	Suffix                  string `config:"suffix"`
+	StrictNames             bool   `config:"strict_names"`
 }
 
 // Fs represents a wrapped fs.Fs
@@ -328,45 +339,64 @@ func (f *Fs) String() string {
 }
 
 // Encrypt an object file name to entries.
-func (f *Fs) add(entries *fs.DirEntries, obj fs.Object) {
+func (f *Fs) add(entries *fs.DirEntries, obj fs.Object) error {
 	remote := obj.Remote()
 	decryptedRemote, err := f.cipher.DecryptFileName(remote)
 	if err != nil {
-		fs.Debugf(remote, "Skipping undecryptable file name: %v", err)
-		return
+		if f.opt.StrictNames {
+			return fmt.Errorf("%s: undecryptable file name detected: %v", remote, err)
+		}
+		fs.Logf(remote, "Skipping undecryptable file name: %v", err)
+		return nil
 	}
 	if f.opt.ShowMapping {
 		fs.Logf(decryptedRemote, "Encrypts to %q", remote)
 	}
 	*entries = append(*entries, f.newObject(obj))
+	return nil
 }
 
 // Encrypt a directory file name to entries.
-func (f *Fs) addDir(ctx context.Context, entries *fs.DirEntries, dir fs.Directory) {
+func (f *Fs) addDir(ctx context.Context, entries *fs.DirEntries, dir fs.Directory) error {
 	remote := dir.Remote()
 	decryptedRemote, err := f.cipher.DecryptDirName(remote)
 	if err != nil {
-		fs.Debugf(remote, "Skipping undecryptable dir name: %v", err)
-		return
+		if f.opt.StrictNames {
+			return fmt.Errorf("%s: undecryptable dir name detected: %v", remote, err)
+		}
+		fs.Logf(remote, "Skipping undecryptable dir name: %v", err)
+		return nil
 	}
 	if f.opt.ShowMapping {
 		fs.Logf(decryptedRemote, "Encrypts to %q", remote)
 	}
 	*entries = append(*entries, f.newDir(ctx, dir))
+	return nil
 }
 
 // Encrypt some directory entries.  This alters entries returning it as newEntries.
 func (f *Fs) encryptEntries(ctx context.Context, entries fs.DirEntries) (newEntries fs.DirEntries, err error) {
 	newEntries = entries[:0] // in place filter
+	errors := 0
+	var firsterr error
 	for _, entry := range entries {
 		switch x := entry.(type) {
 		case fs.Object:
-			f.add(&newEntries, x)
+			err = f.add(&newEntries, x)
 		case fs.Directory:
-			f.addDir(ctx, &newEntries, x)
+			err = f.addDir(ctx, &newEntries, x)
 		default:
 			return nil, fmt.Errorf("unknown object type %T", entry)
 		}
+		if err != nil {
+			errors++
+			if firsterr == nil {
+				firsterr = err
+			}
+		}
+	}
+	if firsterr != nil {
+		return nil, fmt.Errorf("there were %v undecryptable name errors. first error: %v", errors, firsterr)
 	}
 	return newEntries, nil
 }


Test patch: