Repository: rclone/rclone (language: go)

Issue:
nfsmount — not functional on macOS for writing files to OneDrive via Finder
<!--

We understand you are having a problem with rclone; we want to help you with that!

**STOP and READ**
**YOUR POST WILL BE REMOVED IF IT IS LOW QUALITY**:
Please show the effort you've put into solving the problem and please be specific.
People are volunteering their time to help! Low effort posts are not likely to get good answers!

If you think you might have found a bug, try to replicate it with the latest beta (or stable).
The update instructions are available at https://rclone.org/commands/rclone_selfupdate/

If you can still replicate it or just got a question then please use the rclone forum:

    https://forum.rclone.org/

for a quick response instead of filing an issue on this repo.

If nothing else helps, then please fill in the info below which helps us help you.

**DO NOT REDACT** any information except passwords/keys/personal info.

You should use 3 backticks to begin and end your paste to make it readable.

Make sure to include a log obtained with '-vv'.

You can also use '-vv --log-file bug.log' and a service such as https://pastebin.com or https://gist.github.com/

Thank you

The Rclone Developers

-->

**Hello! I'm writing with an active issue that seems to affect `rclone nfsmount` on macOS.**

-----

#### The associated forum post URL from `https://forum.rclone.org`

https://forum.rclone.org/t/rclone-v1-65-0-release/43100/25?u=scuba-tech

#### What is the problem you are having with rclone?

**ALL** attempts to copy files of any size or type, from Finder into OneDrive, via `nfsmount` fail. 

It is not possible to write from Finder or save work. It *is* possible to read/open files.

Simple writing operations like `touch` and `cat > filename` from a shell do write properly into nfsmount.

-----

#### What is your rclone version (output from `rclone version`)

1.65.0 via brew

-----

#### Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

**macOS Sonoma (14.1.1)** // ARM64


-----

#### Which cloud storage system are you using? (e.g. Google Drive)

Microsoft OneDrive (personal family // 6TB)

-----

#### The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`)

`rclone nfsmount --vfs-cache-mode full --filter-from "/Users/<user>/.config/rclone/rclone-filter-rules.txt" OD: ~/MOUNT-OneDrive --verbose`

Notes:

1. I am using `--vfs-cache-mode full` to allow writing with OneDrive. Using none gives an (expected) read-only error message.
2. Here are the contents of my filter rules file:

```
- **/.DS_Store
- **/.hidden*
- **/._*
```

#### A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`)

```
[ERROR] No file for lookup of ._GH010215.mp4
[ERROR] call to 0x106<###>  failed: No such file or directory
INFO  : SdT-Inboxes/GH010215.mp4: vfs cache: removed cache file as file deleted
[ERROR] No file for lookup of ._7F2A7976.JPG
[ERROR] call to 0x106<###>  failed: No such file or directory
[ERROR] err loading attrs for SdT-Inboxes/GH010215.mp4: file does not exist
[ERROR] No file for lookup of ._7F2A7977.JPG
[ERROR] call to 0x106<###>  failed: No such file or directory
[ERROR] err loading attrs for SdT-Inboxes/GH010215.mp4: file does not exist
```

<img width="684" alt="image" src="https://github.com/rclone/rclone/assets/35608185/9d2adedc-09e3-44e5-b4c5-bf5b91602585">


Confirming that the copy into the mount _did fail_, checking with Finder, `ls -hAls`, and even OneDrive webUI... all of these confirm that the file does **not** actually "already exist" as the error message above claims!

I hope this report helps with getting `nfsmount` sorted. :) 

<!--- Please keep the note below for others who read your bug report. -->

#### How to use GitHub

* Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue.
* Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
* Subscribe to receive notifications on status change and new comments.



Discussion / hints:
Including a further --verbose excerpt for context as I believe it is related;  it appears that the filter rules are not respected.

```
NOTICE: NFS Server running at 127.0.0.1:55165
[ERROR] No file for lookup of ._.
[ERROR] call to 0x106<###> failed: No such file or directory
[ERROR] No file for lookup of .hidden
[ERROR] call to 0x106<###> failed: No such file or directory
[ERROR] No file for lookup of .DS_Store
[ERROR] call to 0x106<###> failed: No such file or directory
[...]
[ERROR] call to 0x106<###> failed: No such file or directory
[ERROR] No file for lookup of ._7F2A8214.JPG
[ERROR] call to 0x106<###> failed: No such file or directory
[ERROR] No file for lookup of ._7F2A8215.JPG
[ERROR] call to 0x106<###> failed: No such file or directory
[ERROR] No file for lookup of ._7F2A8196.JPG
[ERROR] call to 0x106<###> failed: No such file or directory
[ERROR] No file for lookup of ._7F2A8216.JPG
[ERROR] call to 0x106<###> failed: No such file or directory
[ERROR] No file for lookup of ._7F2A8197.JPG

[repeat approx 7200x times]
```
+1 I have the exact same issue. @scuba-tech did you figure it out?
> +1 I have the exact same issue. @scuba-tech did you figure it out?

Negative. Presently, this command is non-functional for me. There is no solution that I have found.
Bummer. I guess I will use Mountain Duck now (sadly not Open Source tho)

Fix patch:
diff --git a/cmd/mountlib/mount.md b/cmd/mountlib/mount.md
index 9d367ca1ec08b..6b3ccc6209554 100644
--- a/cmd/mountlib/mount.md
+++ b/cmd/mountlib/mount.md
@@ -263,6 +263,10 @@ This method spins up an NFS server using [serve nfs](/commands/rclone_serve_nfs/
 it to the specified mountpoint. If you run this in background mode using |--daemon|, you will need to
 send SIGTERM signal to the rclone process using |kill| command to stop the mount.
 
+Note that `--nfs-cache-handle-limit` controls the maximum number of cached file handles stored by the `nfsmount` caching handler.
+This should not be set too low or you may experience errors when trying to access files. The default is 1000000,
+but consider lowering this limit if the server's system resource usage causes problems.
+
 #### macFUSE Notes
 
 If installing macFUSE using [dmg packages](https://github.com/osxfuse/osxfuse/releases) from
diff --git a/cmd/nfsmount/nfsmount.go b/cmd/nfsmount/nfsmount.go
index e515e995a8c82..8a1fdea2ed471 100644
--- a/cmd/nfsmount/nfsmount.go
+++ b/cmd/nfsmount/nfsmount.go
@@ -22,7 +22,8 @@ import (
 )
 
 var (
-	sudo = false
+	sudo         = false
+	nfsServerOpt nfs.Options
 )
 
 func init() {
@@ -33,10 +34,11 @@ func init() {
 	mountlib.AddRc(name, mount)
 	cmdFlags := cmd.Flags()
 	flags.BoolVarP(cmdFlags, &sudo, "sudo", "", sudo, "Use sudo to run the mount command as root.", "")
+	nfs.AddFlags(cmdFlags, &nfsServerOpt)
 }
 
 func mount(VFS *vfs.VFS, mountpoint string, opt *mountlib.Options) (asyncerrors <-chan error, unmount func() error, err error) {
-	s, err := nfs.NewServer(context.Background(), VFS, &nfs.Options{})
+	s, err := nfs.NewServer(context.Background(), VFS, &nfsServerOpt)
 	if err != nil {
 		return
 	}
@@ -68,7 +70,7 @@ func mount(VFS *vfs.VFS, mountpoint string, opt *mountlib.Options) (asyncerrors
 	}
 	cmd = append(cmd, "mount")
 	cmd = append(cmd, options...)
-	cmd = append(cmd, "localhost:", mountpoint)
+	cmd = append(cmd, "localhost:"+opt.VolumeName, mountpoint)
 	fs.Debugf(nil, "Running mount command: %q", cmd)
 
 	out, err := exec.Command(cmd[0], cmd[1:]...).CombinedOutput()
@@ -79,6 +81,9 @@ func mount(VFS *vfs.VFS, mountpoint string, opt *mountlib.Options) (asyncerrors
 	}
 	asyncerrors = errChan
 	unmount = func() error {
+		if s.UnmountedExternally {
+			return nil
+		}
 		var umountErr error
 		var out []byte
 		if runtime.GOOS == "darwin" {
@@ -96,5 +101,12 @@ func mount(VFS *vfs.VFS, mountpoint string, opt *mountlib.Options) (asyncerrors
 		}
 		return nil
 	}
+
+	nfs.OnUnmountFunc = func() {
+		s.UnmountedExternally = true
+		errChan <- nil
+		VFS.Shutdown()
+	}
+
 	return
 }
diff --git a/cmd/serve/nfs/filesystem.go b/cmd/serve/nfs/filesystem.go
index 25916e0ddc804..9d3db34d99d7f 100644
--- a/cmd/serve/nfs/filesystem.go
+++ b/cmd/serve/nfs/filesystem.go
@@ -22,27 +22,35 @@ type FS struct {
 
 // ReadDir implements read dir
 func (f *FS) ReadDir(path string) (dir []os.FileInfo, err error) {
+	fs.Debugf("nfs", "ReadDir %v", path)
 	return f.vfs.ReadDir(path)
 }
 
 // Create implements creating new files
 func (f *FS) Create(filename string) (billy.File, error) {
+	fs.Debugf("nfs", "Create %v", filename)
 	return f.vfs.Create(filename)
 }
 
 // Open opens a file
 func (f *FS) Open(filename string) (billy.File, error) {
-	return f.vfs.Open(filename)
+	file, err := f.vfs.Open(filename)
+	fs.Debugf("nfs", "Open %v file: %v err: %v", filename, file, err)
+	return file, err
 }
 
 // OpenFile opens a file
 func (f *FS) OpenFile(filename string, flag int, perm os.FileMode) (billy.File, error) {
-	return f.vfs.OpenFile(filename, flag, perm)
+	file, err := f.vfs.OpenFile(filename, flag, perm)
+	fs.Debugf("nfs", "OpenFile %v flag: %v perm: %v file: %v err: %v", filename, flag, perm, file, err)
+	return file, err
 }
 
 // Stat gets the file stat
 func (f *FS) Stat(filename string) (os.FileInfo, error) {
-	return f.vfs.Stat(filename)
+	node, err := f.vfs.Stat(filename)
+	fs.Debugf("nfs", "Stat %v node: %v err: %v", filename, node, err)
+	return node, err
 }
 
 // Rename renames a file
@@ -85,7 +93,9 @@ func (f *FS) MkdirAll(filename string, perm os.FileMode) error {
 
 // Lstat gets the stats for symlink
 func (f *FS) Lstat(filename string) (os.FileInfo, error) {
-	return f.vfs.Stat(filename)
+	node, err := f.vfs.Stat(filename)
+	fs.Debugf("nfs", "Lstat %v node: %v err: %v", filename, node, err)
+	return node, err
 }
 
 // Symlink is not supported over NFS
diff --git a/cmd/serve/nfs/handler.go b/cmd/serve/nfs/handler.go
index 7d9a8e4d192ac..ed00639afbbd0 100644
--- a/cmd/serve/nfs/handler.go
+++ b/cmd/serve/nfs/handler.go
@@ -5,22 +5,33 @@ package nfs
 
 import (
 	"context"
+	"fmt"
 	"net"
+	"strings"
 
 	"github.com/go-git/go-billy/v5"
+	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/vfs"
 	"github.com/willscott/go-nfs"
 	nfshelper "github.com/willscott/go-nfs/helpers"
 )
 
 // NewBackendAuthHandler creates a handler for the provided filesystem
-func NewBackendAuthHandler(vfs *vfs.VFS) nfs.Handler {
-	return &BackendAuthHandler{vfs}
+func NewBackendAuthHandler(vfs *vfs.VFS, opt *Options) nfs.Handler {
+	handler := &BackendAuthHandler{
+		vfs: vfs,
+		opt: opt,
+	}
+	handler.opt.HandleLimit = handler.opt.Limit()
+	handler.cache = cacheHelper(handler, handler.HandleLimit())
+	return handler
 }
 
 // BackendAuthHandler returns a NFS backing that exposes a given file system in response to all mount requests.
 type BackendAuthHandler struct {
-	vfs *vfs.VFS
+	vfs   *vfs.VFS
+	opt   *Options
+	cache nfs.Handler
 }
 
 // Mount backs Mount RPC Requests, allowing for access control policies.
@@ -50,17 +61,17 @@ func (h *BackendAuthHandler) FSStat(ctx context.Context, f billy.Filesystem, s *
 
 // ToHandle handled by CachingHandler
 func (h *BackendAuthHandler) ToHandle(f billy.Filesystem, s []string) []byte {
-	return []byte{}
+	return h.cache.ToHandle(f, s)
 }
 
 // FromHandle handled by CachingHandler
-func (h *BackendAuthHandler) FromHandle([]byte) (billy.Filesystem, []string, error) {
-	return nil, []string{}, nil
+func (h *BackendAuthHandler) FromHandle(b []byte) (billy.Filesystem, []string, error) {
+	return h.cache.FromHandle(b)
 }
 
 // HandleLimit handled by cachingHandler
 func (h *BackendAuthHandler) HandleLimit() int {
-	return -1
+	return h.opt.HandleLimit
 }
 
 // InvalidateHandle is called on removes or renames
@@ -68,8 +79,152 @@ func (h *BackendAuthHandler) InvalidateHandle(billy.Filesystem, []byte) error {
 	return nil
 }
 
-func newHandler(vfs *vfs.VFS) nfs.Handler {
-	handler := NewBackendAuthHandler(vfs)
-	cacheHelper := nfshelper.NewCachingHandler(handler, 1024)
+func newHandler(vfs *vfs.VFS, opt *Options) nfs.Handler {
+	handler := NewBackendAuthHandler(vfs, opt)
+	nfs.SetLogger(&LogIntercepter{Level: nfs.DebugLevel})
+	return handler
+}
+
+func cacheHelper(handler nfs.Handler, limit int) nfs.Handler {
+	cacheHelper := nfshelper.NewCachingHandler(handler, limit)
 	return cacheHelper
 }
+
+// Limit overrides the --nfs-cache-handle-limit value if out-of-range
+func (o *Options) Limit() int {
+	if o.HandleLimit < 0 {
+		return 1000000
+	}
+	if o.HandleLimit <= 5 {
+		return 5
+	}
+	return o.HandleLimit
+}
+
+// OnUnmountFunc registers a function to call when externally unmounted
+var OnUnmountFunc func()
+
+func onUnmount() {
+	fs.Infof(nil, "unmount detected")
+	if OnUnmountFunc != nil {
+		OnUnmountFunc()
+	}
+}
+
+// LogIntercepter intercepts noisy go-nfs logs and reroutes them to DEBUG
+type LogIntercepter struct {
+	Level nfs.LogLevel
+}
+
+// Intercept intercepts go-nfs logs and calls fs.Debugf instead
+func (l *LogIntercepter) Intercept(args ...interface{}) {
+	args = append([]interface{}{"[NFS DEBUG] "}, args...)
+	argsS := fmt.Sprint(args...)
+	fs.Debugf(nil, "%v", argsS)
+}
+
+// Interceptf intercepts go-nfs logs and calls fs.Debugf instead
+func (l *LogIntercepter) Interceptf(format string, args ...interface{}) {
+	argsS := fmt.Sprint(args...)
+	// bit of a workaround... the real fix is probably https://github.com/willscott/go-nfs/pull/28
+	if strings.Contains(argsS, "mount.Umnt") {
+		onUnmount()
+	}
+
+	fs.Debugf(nil, "[NFS DEBUG] "+format, args...)
+}
+
+// Debug reroutes go-nfs Debug messages to Intercept
+func (l *LogIntercepter) Debug(args ...interface{}) {
+	l.Intercept(args...)
+}
+
+// Debugf reroutes go-nfs Debugf messages to Interceptf
+func (l *LogIntercepter) Debugf(format string, args ...interface{}) {
+	l.Interceptf(format, args...)
+}
+
+// Error reroutes go-nfs Error messages to Intercept
+func (l *LogIntercepter) Error(args ...interface{}) {
+	l.Intercept(args...)
+}
+
+// Errorf reroutes go-nfs Errorf messages to Interceptf
+func (l *LogIntercepter) Errorf(format string, args ...interface{}) {
+	l.Interceptf(format, args...)
+}
+
+// Fatal reroutes go-nfs Fatal messages to Intercept
+func (l *LogIntercepter) Fatal(args ...interface{}) {
+	l.Intercept(args...)
+}
+
+// Fatalf reroutes go-nfs Fatalf messages to Interceptf
+func (l *LogIntercepter) Fatalf(format string, args ...interface{}) {
+	l.Interceptf(format, args...)
+}
+
+// GetLevel returns the nfs.LogLevel
+func (l *LogIntercepter) GetLevel() nfs.LogLevel {
+	return l.Level
+}
+
+// Info reroutes go-nfs Info messages to Intercept
+func (l *LogIntercepter) Info(args ...interface{}) {
+	l.Intercept(args...)
+}
+
+// Infof reroutes go-nfs Infof messages to Interceptf
+func (l *LogIntercepter) Infof(format string, args ...interface{}) {
+	l.Interceptf(format, args...)
+}
+
+// Panic reroutes go-nfs Panic messages to Intercept
+func (l *LogIntercepter) Panic(args ...interface{}) {
+	l.Intercept(args...)
+}
+
+// Panicf reroutes go-nfs Panicf messages to Interceptf
+func (l *LogIntercepter) Panicf(format string, args ...interface{}) {
+	l.Interceptf(format, args...)
+}
+
+// ParseLevel parses the nfs.LogLevel
+func (l *LogIntercepter) ParseLevel(level string) (nfs.LogLevel, error) {
+	return nfs.Log.ParseLevel(level)
+}
+
+// Print reroutes go-nfs Print messages to Intercept
+func (l *LogIntercepter) Print(args ...interface{}) {
+	l.Intercept(args...)
+}
+
+// Printf reroutes go-nfs Printf messages to Intercept
+func (l *LogIntercepter) Printf(format string, args ...interface{}) {
+	l.Interceptf(format, args...)
+}
+
+// SetLevel sets the nfs.LogLevel
+func (l *LogIntercepter) SetLevel(level nfs.LogLevel) {
+	l.Level = level
+}
+
+// Trace reroutes go-nfs Trace messages to Intercept
+func (l *LogIntercepter) Trace(args ...interface{}) {
+	l.Intercept(args...)
+}
+
+// Tracef reroutes go-nfs Tracef messages to Interceptf
+func (l *LogIntercepter) Tracef(format string, args ...interface{}) {
+	l.Interceptf(format, args...)
+}
+
+// Warn reroutes go-nfs Warn messages to Intercept
+func (l *LogIntercepter) Warn(args ...interface{}) {
+	l.Intercept(args...)
+}
+
+// Warnf reroutes go-nfs Warnf messages to Interceptf
+func (l *LogIntercepter) Warnf(format string, args ...interface{}) {
+	l.Interceptf(format, args...)
+}
diff --git a/cmd/serve/nfs/nfs.go b/cmd/serve/nfs/nfs.go
index 7a67f679e4eda..4ef4818787843 100644
--- a/cmd/serve/nfs/nfs.go
+++ b/cmd/serve/nfs/nfs.go
@@ -25,15 +25,17 @@ import (
 
 // Options contains options for the NFS Server
 type Options struct {
-	ListenAddr string // Port to listen on
+	ListenAddr  string // Port to listen on
+	HandleLimit int    // max file handles cached by go-nfs CachingHandler
 }
 
 var opt Options
 
-// AddFlags adds flags for the sftp
+// AddFlags adds flags for serve nfs (and nfsmount)
 func AddFlags(flagSet *pflag.FlagSet, Opt *Options) {
 	rc.AddOption("nfs", &Opt)
 	flags.StringVarP(flagSet, &Opt.ListenAddr, "addr", "", Opt.ListenAddr, "IPaddress:Port or :Port to bind server to", "")
+	flags.IntVarP(flagSet, &Opt.HandleLimit, "nfs-cache-handle-limit", "", 1000000, "max file handles cached simultaneously (min 5)", "")
 }
 
 func init() {
@@ -72,7 +74,9 @@ NFS mount over local network, you need to specify the listening address and port
 
 Modifying files through NFS protocol requires VFS caching. Usually you will need to specify ` + "`--vfs-cache-mode`" + `
 in order to be able to write to the mountpoint (full is recommended). If you don't specify VFS cache mode,
-the mount will be read-only.
+the mount will be read-only. Note also that ` + "`--nfs-cache-handle-limit`" + ` controls the maximum number of cached file handles stored by the caching handler.
+This should not be set too low or you may experience errors when trying to access files. The default is ` + "`1000000`" + `, but consider lowering this limit if
+the server's system resource usage causes problems.
 
 To serve NFS over the network use following command:
 
diff --git a/cmd/serve/nfs/server.go b/cmd/serve/nfs/server.go
index c66e687a901df..14e12526481e4 100644
--- a/cmd/serve/nfs/server.go
+++ b/cmd/serve/nfs/server.go
@@ -16,10 +16,11 @@ import (
 
 // Server contains everything to run the Server
 type Server struct {
-	opt      Options
-	handler  nfs.Handler
-	ctx      context.Context // for global config
-	listener net.Listener
+	opt                 Options
+	handler             nfs.Handler
+	ctx                 context.Context // for global config
+	listener            net.Listener
+	UnmountedExternally bool
 }
 
 // NewServer creates a new server
@@ -36,7 +37,7 @@ func NewServer(ctx context.Context, vfs *vfs.VFS, opt *Options) (s *Server, err
 		ctx: ctx,
 		opt: *opt,
 	}
-	s.handler = newHandler(vfs)
+	s.handler = newHandler(vfs, opt)
 	s.listener, err = net.Listen("tcp", s.opt.ListenAddr)
 	if err != nil {
 		fs.Errorf(nil, "NFS server failed to listen: %v\n", err)


Test patch: