Repository: rclone/rclone (language: go)

Issue:
with --vfs-used-is-size value is calculated and then thrown away
I have ssh access to a 500gb quota'd filesystem that I'm mounting using rclone. `df` on the server reports information about the entire disk. I've been trying to use `--vfs-used-is-size` in combination with `--vfs-disk-space-total-size=XXX` but that's giving me garbage values:

```
$ rclone  mount 500gb:/home/ilias/tmp /home/ilias/foo --log-level=DEBUG --vfs-used-is-size
 --vfs-disk-space-total-size=500G
$ du -h /home/ilias/foo
1.5K	/home/ilias/foo
$ df -h /home/ilias/foo
Filesystem             Size  Used Avail Use% Mounted on
500gb:/home/ilias/tmp  500G -224T  224T    - /home/ilias/foo
```
Relevant debug output:
```
2024/12/02 15:02:33 DEBUG : : >Statfs: stat={Blocks:131072000 Bfree:60049752808 Bavail:60049752808 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
```

I did some digging around, and the problem seems to be that in `vfs/vfs.go:Statfs()` we make an About() call, then properly overwrite `used` and `size` based on the command-line arguments, but keep the `free` value returned by the server. The call to `vfs/vfs.go:fillInMissingSizes()` doesn't change `free` (since it's been populated). Later on `used` (which we just spent so much time calculating) is thrown away and `statvfs(3)` returns the `size` (from `vfs-disk-space-total-size`) and the `free` (which we got from the `About()` call), and `used` is calculated (in userland) based on those.

Essentialy, `vfs-used-is-size` is a very very expensive no-op. Note that the problem always exists, and is independent of the backend and not only when used in combination with `vfs-disk-space-total-size`.

The fix is to ignore the `free` value returned by the server in the About() call, and calculate it based on `size` and `used`.

```
diff --git a/vfs/vfs.go b/vfs/vfs.go
index 9f0fc1f14..5fcfe9c4c 100644
--- a/vfs/vfs.go
+++ b/vfs/vfs.go
@@ -650,6 +650,11 @@ func (vfs *VFS) Statfs() (total, used, free int64) {
                total = int64(vfs.Opt.DiskSpaceTotalSize)
        }
 
+       // if we've calculated the "used", then we should calculate the "free"
+       if vfs.Opt.UsedIsSize {
+               free = -1
+       }
+
        total, used, free = fillInMissingSizes(total, used, free, unknownFreeBytes)
        return
 }
```

which results in:

```
$ du -h /home/ilias/foo
1.5K	/home/ilias/foo
$ df -h /home/ilias/foo
Filesystem             Size  Used Avail Use% Mounted on
500gb:/home/ilias/tmp  500G  4.0K  500G   1% /home/ilias/foo
```

With the debug output:

```
2024/12/02 15:06:08 DEBUG : : >Statfs: stat={Blocks:131072000 Bfree:131071999 Bavail:131071999 Files:1000000000 Ffree:1000000000 Bsize:4096 Namelen:255 Frsize:4096}, err=<nil>
```
<!--- Please keep the note below for others who read your bug report. -->

#### How to use GitHub

* Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue.
* Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
* Subscribe to receive notifications on status change and new comments.



Discussion / hints:
> Essentialy, `vfs-used-is-size` is a very very expensive no-op. Note that the problem always exists, and is independent of the backend and not only when used in combination with `vfs-disk-space-total-size`.

He he! I've always thought of it as very expensive, but it isn't supposed to be a no-op!

Do you want to send a PR with a proposed fix?
> Do you want to send a PR with a proposed fix?

:) just did....
TBH if both `--vfs-used-is-size` and `--vfs-disk-space-total-size=XXX` are specified we *could* skip the `About()` call, since we're essentially ignoring what it returns. But I figured since we're just about to a recursive filetree walk over the network, who cares if we do an unnecessary `About()` call.... I don't know if it's worth the code clutter. (I guess it would depend on whether `About()` is expensive for some backends).
> TBH if both `--vfs-used-is-size` and `--vfs-disk-space-total-size=XXX` are specified we _could_ skip the `About()` call, since we're essentially ignoring what it returns.

Actually I was surprised to see we called About at all - generally this gets used on backends with no About call.

Skipping the About if both  `--vfs-used-is-size` and `--vfs-disk-space-total-size=XXX` are specified seems like a reasonable idea.

> But I figured since we're just about to a recursive filetree walk over the network, who cares if we do an unnecessary `About()` call.... I don't know if it's worth the code clutter. (I guess it would depend on whether `About()` is expensive for some backends).

Well there is that! And no, About is a single call on all backends otherwise it doesn't get implemented.

Fix patch:
diff --git a/vfs/vfs.go b/vfs/vfs.go
index 9f0fc1f14cc70..40d48b5b1b74d 100644
--- a/vfs/vfs.go
+++ b/vfs/vfs.go
@@ -626,6 +626,10 @@ func (vfs *VFS) Statfs() (total, used, free int64) {
 				return nil
 			})
 			vfs.usage.Used = &usedBySizeAlgorithm
+			// if we read a Total size then we should calculate Free from it
+			if vfs.usage.Total != nil {
+				vfs.usage.Free = nil
+			}
 		}
 		vfs.usageTime = time.Now()
 		if err != nil {


Test patch: