Repository: rclone/rclone (language: go)

Issue:
new feature: VFS auto normalization
Immediate goal of this enhancement is to fix macOS `rclone mount` which at the moment does not work properly regardless of mounting method used - macOSFUSE or FUSE-T. Optionally normalized VFS can bring other benefits as well e.g. some less common or future filesystems rclone users can work with and normalization constraints imposed.

The background issue and possible solutions have been discussed on the forum:

https://forum.rclone.org/t/possible-special-character-encoding-issue-on-macos/39048/30

At the moment many cloud storage destinations (and rclone virtual crypt remote) allow to store both NFC and NFD encoded files/directories names. Also rclone allows to upload/download both forms.

This in general works trouble free on Linux and Windows but breaks rclone mount functionality in macOS - depending on mount options either only NFC or only NFD content is visible/accessible - this is [unfortunate](https://eclecticlight.co/2021/05/08/explainer-unicode-normalization-and-apfs/) consequence of Apple design decision to be different than rest of the world and use exclusively NFD in their APIs. We won't change it.

Experimentation and testing, including [attempts](https://github.com/macos-fuse-t/fuse-t/issues/16) to improve situation with FUSE-T (it has now - since v1.0.20 - `-o nfc` option which should help with NFC path) lead me to believe that last missing part of the puzzle is rclone VFS lack of normalization.

I would like to propose to add optional normalization to VFS.

New option:

--vfs-normalize OFF/NFC/NFD - with OFF being default 

When set to NFC or NFD it would apply characters normalization to served content.

Edge case to watch is situation where two names differ only by normalization and coexist in the same folder e.g.:

```
$ rclone lsf crypt:TEST

CONFLICTéééFILE.txt
CONFLICTéééFILE.txt
```

one file is NFC and other is NFD. Only solution I see for such case is use only one name and log ERROR.



Discussion / hints:
I think this should be achievable relatively easily.

All file / dir objects are ultimately made from the directory cache. So we only need to do the normalized comparison in the directory cache searching routine.

We actually do this already for case sensitive / insensitive file systems.

So first we look for an exact match which is a quick hash look up, and then we look for a case insensitive match which involves comparing every item.

https://github.com/rclone/rclone/blob/2c50f26c36e7b18e0fcfe18bc844a2141c9245af/vfs/dir.go#L710-L732

We could do something very similar for the normalization options too. I think we'd want to make a normalization function which does the (NFC, NFD) and/or (lower case) and just run through the loop once. We do exactly this in the sync routines so factoring this out would be sensible.

https://github.com/rclone/rclone/blob/2c50f26c36e7b18e0fcfe18bc844a2141c9245af/fs/march/march.go#L58-L71

> Edge case to watch is situation where two names differ only by normalization and coexist in the same folder

We currently just use the first one in the case vs CASE comparison. That is the easy thing to do. To detect a collision would require scanning the whole directory which would take on average twice as long.

Worth the extra CPU for an ERROR message?

> New option:
> 
> --vfs-normalize OFF/NFC/NFD - with OFF being default

Assuming we are only using the normalize option to find matching file names and not actually returning them to the OS then we can get away with using NFC only and this would match this flag which only does NFC normalization.

      --no-unicode-normalization   Don't normalize unicode characters in filenames

Once thing I am unclear on though - do you think we need to apply that normalization to the file names sent to the OS - is that your intention? So we  receive an NFC file name from Google Drive (say) and send the macOS kernel an NFD file name according to the flag? That will be a bit more work (but not much). I think this might be what the FUSE-T option `-o nfc` does though so maybe isn't necessary?
> Once thing I am unclear on though - do you think we need to apply that normalization to the file names sent to the OS - is that your intention?

Yes this is the most important thing actually and whole purpose of this  - it is about sending to OS normalized names. Today we have a problem that remotes can contain both NFC and NFD names - it works on Linux but breaks macOS. 

> I think this might be what the FUSE-T option `-o nfc` does though so maybe isn't necessary?

This option converts names received from OS (macOS uses NFD) and makes sure that NFS mount receives NFC - so all NFS content is NFC. It also makes NFC content to work with macOS APIs.
It breaks when some content in remote is already NFD - files are not visible/directories are not browsable.

On macOS it happens for example when user uses `rclone copy/sync` - NFD names are saved in remote.

With `--vfs-normalize` to make macOS mount work we could either:

1. `--vfs-normalize NFD` - this is what macOS likes the most I think

REMOTE  ----------------------->  MOUNT
NFC -> RCLONE -> NFD -> FUSE -> NFD <---- rclone normalized NFC->NFD
NFD -> RCLONE -> NFD -> FUSE -> NFD


MOUNT  ----------------------->  REMOTE
NFC -> FUSE -> NFC -> RCLONE -> NFC
NFD -> FUSE -> NFD -> RCLONE -> NFD



2. `--vfs-normalize NFC` and use `-o nfc`

REMOTE  ----------------------->  MOUNT
NFC -> RCLONE -> NFC -> FUSE -> NFD 
NFD -> RCLONE -> NFC -> FUSE -> NFD <---- rclone normalized NFD->NFC


MOUNT  ----------------------->  REMOTE
NFC -> FUSE -> NFC -> RCLONE -> NFC
NFD -> FUSE -> NFC-> RCLONE -> NFC

It is difficult to say which one will behave better on macOS - I think that both would work. But with NFC/NFD and macOS I have learnt that only testing can tell.
The key purpose of this enhancement is to fix `rclone mount` on macOS. 

Today it simply does not work - does not matter with macFUSE or FUSE-T. And is unfixable by `-o modules=iconv,from_code=UTF-8,to_code=UTF-8` or `-o modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC` workaround. It only helps in some cases and creates new issues - like for example that files can not be open in Finder (macOS file browser)

![image](https://github.com/rclone/rclone/assets/15635386/8f61ee97-d18d-432d-8606-9a357b4d0f7c)

I think what I'll do is try the smallest patch that could work first. That would fix the "... can't be found" errors I think but it wouldn't normalize the returns to the OS. This might mean we need the `-o nfc` flag. If that doesn't work I can do the further parts..
Cool. I am happy at all stages to test - I do not need binary build I can happily build myself from `git clone`. I have my test bed - crypt with carefully crafted mix of NFC/NFD files and folders:) so far it breaks everything.

I hoped that fuse-t can handle it fully but does not seem to be possible.
Note:

With `--vfs-normalize` set to NFC or NFD `iconv` should not be applied:

https://github.com/rclone/rclone/blob/fcb912a664de2e4df0e71b63aa3a8a1f956dd495/cmd/cmount/mount.go#L120-L126

It is crude way to actually try to normalize to NFD- but it stopped working fully with advent of APFS FS few years ago and changes in normalization in all macOS - also I suspect not 100% correct way macFUSE handles it. I actually suspect it never worked 100% before neither - but nowadays it is more apparent.

Still should be there for old Apple computers users - this is why  `--vfs-normalize`  should be optional
Sorry to keep pestering... but can it be implemented one day?  I am not skilled enough in go to give it a go.  I do try to earn my karma points on the forum instead:) I do believe that it is missing piece of macOS `rclone mount` to work predictably and consistently - and now it is disaster people using it do not realize.

I managed to convince author of `fuse-t` to make [changes](https://github.com/macos-fuse-t/fuse-t/issues/16) helping to alleviate problems moving forward. But without option to normalize `mount` to NFD on rclone side it is only half measure.
> Sorry to keep pestering... but can it be implemented one day? I am not skilled enough in go to give it a go. I do try to earn my karma points on the forum instead:)

You have lots of karma indeed :-)

I put this on the 1.64 milestone which isn't a guarantee that it will make that release, but it will be on my radar at least!


Just dropping in to say that I would be extremely thankful for having this fixed at some point. It would save a lot of trouble for me and many users in Slovakia, Czech republic and other countries, where basically all words are accented, which makes using `rclone` very problematic.

Fix patch:
diff --git a/cmd/cmount/mount.go b/cmd/cmount/mount.go
index f6d86bdb5c5d9..d6d684d0496c8 100644
--- a/cmd/cmount/mount.go
+++ b/cmd/cmount/mount.go
@@ -116,13 +116,6 @@ func mountOptions(VFS *vfs.VFS, device string, mountpoint string, opt *mountlib.
 	for _, option := range opt.ExtraFlags {
 		options = append(options, option)
 	}
-	if runtime.GOOS == "darwin" {
-		if !findOption("modules=iconv", options) {
-			iconv := "modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC"
-			options = append(options, "-o", iconv)
-			fs.Debugf(nil, "Adding \"-o %s\" for macOS", iconv)
-		}
-	}
 	return options
 }
 
diff --git a/cmd/mountlib/mount.md b/cmd/mountlib/mount.md
index 9d367ca1ec08b..e8ce1c28dc7fd 100644
--- a/cmd/mountlib/mount.md
+++ b/cmd/mountlib/mount.md
@@ -257,7 +257,12 @@ Mounting on macOS can be done either via [built-in NFS server](/commands/rclone_
 FUSE driver utilizing a macOS kernel extension (kext). FUSE-T is an alternative FUSE system
 which "mounts" via an NFSv4 local server.
 
-## NFS mount
+##### Unicode Normalization
+
+It is highly recommended to keep the default of `--no-unicode-normalization=false`
+for all `mount` and `serve` commands on macOS. For details, see [vfs-case-sensitivity](https://rclone.org/commands/rclone_mount/#vfs-case-sensitivity).
+
+#### NFS mount
 
 This method spins up an NFS server using [serve nfs](/commands/rclone_serve_nfs/) command and mounts
 it to the specified mountpoint. If you run this in background mode using |--daemon|, you will need to
@@ -290,15 +295,6 @@ As per the [FUSE-T wiki](https://github.com/macos-fuse-t/fuse-t/wiki#caveats):
 This means that viewing files with various tools, notably macOS Finder, will cause rlcone
 to update the modification time of the file. This may make rclone upload a full new copy
 of the file.
-
-##### Unicode Normalization
-
-Rclone includes flags for unicode normalization with macFUSE that should be updated
-for FUSE-T. See [this forum post](https://forum.rclone.org/t/some-unicode-forms-break-mount-on-macos-with-fuse-t/36403)
-and [FUSE-T issue #16](https://github.com/macos-fuse-t/fuse-t/issues/16). The following
-flag should be added to the `rclone mount` command.
-
-    -o modules=iconv,from_code=UTF-8,to_code=UTF-8
     
 ##### Read Only mounts
 
diff --git a/fs/operations/check.go b/fs/operations/check.go
index 2279b3fe6d3b0..b9fe36aabe2d2 100644
--- a/fs/operations/check.go
+++ b/fs/operations/check.go
@@ -380,10 +380,19 @@ func CheckDownload(ctx context.Context, opt *CheckOpt) error {
 // so that it matches behavior of Check (where it's handled by March)
 func ApplyTransforms(ctx context.Context, s string) string {
 	ci := fs.GetConfig(ctx)
-	if !ci.NoUnicodeNormalization {
+	return ToNormal(s, !ci.NoUnicodeNormalization, ci.IgnoreCaseSync)
+}
+
+// ToNormal normalizes case and unicode form and returns the transformed string.
+// It is similar to ApplyTransforms but does not use a context.
+// If normUnicode == true, s will be transformed to NFC.
+// If normCase == true, s will be transformed to lowercase.
+// If both are true, both transformations will be performed.
+func ToNormal(s string, normUnicode, normCase bool) string {
+	if normUnicode {
 		s = norm.NFC.String(s)
 	}
-	if ci.IgnoreCaseSync {
+	if normCase {
 		s = strings.ToLower(s)
 	}
 	return s
diff --git a/vfs/dir.go b/vfs/dir.go
index 5044ee855d2fb..47ffd8aaa5ad3 100644
--- a/vfs/dir.go
+++ b/vfs/dir.go
@@ -18,6 +18,7 @@ import (
 	"github.com/rclone/rclone/fs/operations"
 	"github.com/rclone/rclone/fs/walk"
 	"github.com/rclone/rclone/vfs/vfscommon"
+	"golang.org/x/text/unicode/norm"
 )
 
 // Dir represents a directory entry
@@ -466,7 +467,6 @@ func (d *Dir) AddVirtual(leaf string, size int64, isDir bool) {
 		node = f
 	}
 	d.addObject(node)
-
 }
 
 // delObject removes an object from the directory
@@ -514,6 +514,35 @@ func (d *Dir) _readDir() error {
 		return err
 	}
 
+	if d.vfs.Opt.BlockNormDupes { // do this only if requested, as it will have a performance hit
+		ci := fs.GetConfig(context.TODO())
+
+		// sort entries such that NFD comes before NFC of same name
+		sort.Slice(entries, func(i, j int) bool {
+			if entries[i] != entries[j] && fs.DirEntryType(entries[i]) == fs.DirEntryType(entries[j]) && norm.NFC.String(entries[i].Remote()) == norm.NFC.String(entries[j].Remote()) {
+				if norm.NFD.IsNormalString(entries[i].Remote()) && !norm.NFD.IsNormalString(entries[j].Remote()) {
+					return true
+				}
+			}
+			return entries.Less(i, j)
+		})
+
+		// detect dupes, remove them from the list and log an error
+		normalizedNames := make(map[string]struct{}, entries.Len())
+		filteredEntries := make(fs.DirEntries, 0)
+		for _, e := range entries {
+			normName := fmt.Sprintf("%s-%T", operations.ToNormal(e.Remote(), !ci.NoUnicodeNormalization, (ci.IgnoreCaseSync || d.vfs.Opt.CaseInsensitive)), e) // include type to track objects and dirs separately
+			_, found := normalizedNames[normName]
+			if found {
+				fs.Errorf(e.Remote(), "duplicate normalized names detected - skipping")
+				continue
+			}
+			normalizedNames[normName] = struct{}{}
+			filteredEntries = append(filteredEntries, e)
+		}
+		entries = filteredEntries
+	}
+
 	err = d._readDirFromEntries(entries, nil, time.Time{})
 	if err != nil {
 		return err
@@ -767,15 +796,18 @@ func (d *Dir) stat(leaf string) (Node, error) {
 	}
 	item, ok := d.items[leaf]
 
-	if !ok && d.vfs.Opt.CaseInsensitive {
-		leafLower := strings.ToLower(leaf)
+	ci := fs.GetConfig(context.TODO())
+	normUnicode := !ci.NoUnicodeNormalization
+	normCase := ci.IgnoreCaseSync || d.vfs.Opt.CaseInsensitive
+	if !ok && (normUnicode || normCase) {
+		leafNormalized := operations.ToNormal(leaf, normUnicode, normCase) // this handles both case and unicode normalization
 		for name, node := range d.items {
-			if strings.ToLower(name) == leafLower {
+			if operations.ToNormal(name, normUnicode, normCase) == leafNormalized {
 				if ok {
-					// duplicate case insensitive match is an error
-					return nil, fmt.Errorf("duplicate filename %q detected with --vfs-case-insensitive set", leaf)
+					// duplicate normalized match is an error
+					return nil, fmt.Errorf("duplicate filename %q detected with case/unicode normalization settings", leaf)
 				}
-				// found a case insensitive match
+				// found a normalized match
 				ok = true
 				item = node
 			}
diff --git a/vfs/vfs.md b/vfs/vfs.md
index dba7afa0a0d40..8d53507646d70 100644
--- a/vfs/vfs.md
+++ b/vfs/vfs.md
@@ -309,6 +309,28 @@ If the flag is not provided on the command line, then its default value depends
 on the operating system where rclone runs: "true" on Windows and macOS, "false"
 otherwise. If the flag is provided without a value, then it is "true".
 
+The `--no-unicode-normalization` flag controls whether a similar "fixup" is
+performed for filenames that differ but are [canonically
+equivalent](https://en.wikipedia.org/wiki/Unicode_equivalence) with respect to
+unicode. Unicode normalization can be particularly helpful for users of macOS,
+which prefers form NFD instead of the NFC used by most other platforms. It is
+therefore highly recommended to keep the default of `false` on macOS, to avoid
+encoding compatibility issues.
+
+In the (probably unlikely) event that a directory has multiple duplicate
+filenames after applying case and unicode normalization, the `--vfs-block-norm-dupes`
+flag allows hiding these duplicates. This comes with a performance tradeoff, as
+rclone will have to scan the entire directory for duplicates when listing a
+directory. For this reason, it is recommended to leave this disabled if not
+needed. However, macOS users may wish to consider using it, as otherwise, if a
+remote directory contains both NFC and NFD versions of the same filename, an odd
+situation will occur: both versions of the file will be visible in the mount,
+and both will appear to be editable, however, editing either version will
+actually result in only the NFD version getting edited under the hood. `--vfs-block-
+norm-dupes` prevents this confusion by detecting this scenario, hiding the
+duplicates, and logging an error, similar to how this is handled in `rclone
+sync`.
+
 ### VFS Disk Options
 
 This flag allows you to manually set the statistics about the filing system.
diff --git a/vfs/vfscommon/options.go b/vfs/vfscommon/options.go
index bb5f3313a4aec..f1e4368474ff8 100644
--- a/vfs/vfscommon/options.go
+++ b/vfs/vfscommon/options.go
@@ -30,6 +30,7 @@ type Options struct {
 	CacheMinFreeSpace  fs.SizeSuffix
 	CachePollInterval  time.Duration
 	CaseInsensitive    bool
+	BlockNormDupes     bool
 	WriteWait          time.Duration // time to wait for in-sequence write
 	ReadWait           time.Duration // time to wait for in-sequence read
 	WriteBack          time.Duration // time to wait before writing back dirty files
diff --git a/vfs/vfsflags/vfsflags.go b/vfs/vfsflags/vfsflags.go
index 83b593d295f5a..1dedd891818b6 100644
--- a/vfs/vfsflags/vfsflags.go
+++ b/vfs/vfsflags/vfsflags.go
@@ -35,6 +35,7 @@ func AddFlags(flagSet *pflag.FlagSet) {
 	flags.FVarP(flagSet, DirPerms, "dir-perms", "", "Directory permissions", "VFS")
 	flags.FVarP(flagSet, FilePerms, "file-perms", "", "File permissions", "VFS")
 	flags.BoolVarP(flagSet, &Opt.CaseInsensitive, "vfs-case-insensitive", "", Opt.CaseInsensitive, "If a file name not found, find a case insensitive match", "VFS")
+	flags.BoolVarP(flagSet, &Opt.BlockNormDupes, "vfs-block-norm-dupes", "", Opt.BlockNormDupes, "If duplicate filenames exist in the same directory (after normalization), log an error and hide the duplicates (may have a performance cost)", "VFS")
 	flags.DurationVarP(flagSet, &Opt.WriteWait, "vfs-write-wait", "", Opt.WriteWait, "Time to wait for in-sequence write before giving error", "VFS")
 	flags.DurationVarP(flagSet, &Opt.ReadWait, "vfs-read-wait", "", Opt.ReadWait, "Time to wait for in-sequence read before seeking", "VFS")
 	flags.DurationVarP(flagSet, &Opt.WriteBack, "vfs-write-back", "", Opt.WriteBack, "Time to writeback files after last use when using cache", "VFS")


Test patch:
diff --git a/vfs/vfs_case_test.go b/vfs/vfs_case_test.go
index 5ba81a169b7c7..7641cf89fac6a 100644
--- a/vfs/vfs_case_test.go
+++ b/vfs/vfs_case_test.go
@@ -5,10 +5,12 @@ import (
 	"os"
 	"testing"
 
+	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fstest"
 	"github.com/rclone/rclone/vfs/vfscommon"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
+	"golang.org/x/text/unicode/norm"
 )
 
 func TestCaseSensitivity(t *testing.T) {
@@ -160,3 +162,36 @@ func assertFileAbsentVFS(t *testing.T, vfs *VFS, name string) {
 	assert.Error(t, err)
 	assert.Equal(t, err, ENOENT)
 }
+
+func TestUnicodeNormalization(t *testing.T) {
+	r := fstest.NewRun(t)
+
+	var (
+		nfc  = norm.NFC.String(norm.NFD.String("測試_Русский___ě_áñ"))
+		nfd  = norm.NFD.String(nfc)
+		both = "normal name with no special characters.txt"
+	)
+
+	// Create test files
+	ctx := context.Background()
+	file1 := r.WriteObject(ctx, both, "data1", t1)
+	file2 := r.WriteObject(ctx, nfc, "data2", t2)
+	r.CheckRemoteItems(t, file1, file2)
+
+	// Create VFS
+	opt := vfscommon.DefaultOpt
+	vfs := New(r.Fremote, &opt)
+	defer cleanupVFS(t, vfs)
+
+	// assert that both files are found under NFD-normalized names
+	assertFileDataVFS(t, vfs, norm.NFD.String(both), "data1")
+	assertFileDataVFS(t, vfs, nfd, "data2")
+
+	// change ci.NoUnicodeNormalization to true and verify that only file1 is found
+	ci := fs.GetConfig(ctx) // need to set the global config here as the *Dir methods don't take a ctx param
+	oldVal := ci.NoUnicodeNormalization
+	defer func() { fs.GetConfig(ctx).NoUnicodeNormalization = oldVal }() // restore the prior value after the test
+	ci.NoUnicodeNormalization = true
+	assertFileDataVFS(t, vfs, norm.NFD.String(both), "data1")
+	assertFileAbsentVFS(t, vfs, nfd)
+}