Repository: rclone/rclone (go)

Issue:
Breaking behavior change for sync/copy commands in `1.69.0`
#### The associated forum post URL from `https://forum.rclone.org`

None

#### What is the problem you are having with rclone?

I have encountered this problem specifically with `http` remote, but other could be affected too. Given `sync`/`copy` command with specific source filename `1.68.2` used to sync only the single specified file. In `1.69.0` however, it copies entire file's directory instead (or nothing if HTTP server doesn't allow file listing).

Using `git bisect` I have identified the breaking change to introduced by https://github.com/rclone/rclone/commit/9b4b3033da25ec41aabb35b858a765dd734a5f09 (PR: https://github.com/rclone/rclone/pull/7668).

#### What is your rclone version (output from `rclone version`)

```
rclone v1.69.0
- os/version: arch 25.0.0 (64 bit)
- os/kernel: 5.10.230-1-MANJARO (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.4
- go/linking: dynamic
- go/tags: none
```

Also tested with `1.69.0` and `master` (`2a1e28f5f5683eaba22d67a0e253c95b934f8c41`)

#### Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Linux Manjaro, 64 bit

#### Which cloud storage system are you using? (e.g. Google Drive)

HTTP

#### Setup

```sh
mkdir rclone-server
echo 111 > rclone-server/file1.txt
echo 222 > rclone-server/file2.md

# Configure the remote
rclone config create http_sync http url=http://localhost:9999

# Run a server (sever version is probably irrelevant)
rclone serve http ./rclone-server --addr 127.0.0.1:9999 -vv
```

#### The command you were trying to run

```
rclone sync http_sync:/file1.txt ./rclone-sync -vv
```

Weirdly enough, when using `--http-url` argument instead of configured HTTP remote, the behavior stays the same even for the newer versions.
```
rclone sync :http:/file1.txt ./rclone-sync --http-url http://localhost:9999 -vv
````

#### A log from the command with the `-vv` flag [version: `1.69.0`]

```
2025/03/05 23:55:18 DEBUG : rclone: Version "v1.69.0" starting with parameters ["./rclone" "sync" "http_sync:/file1.txt" "./rclone-sync" "-vv"]
2025/03/05 23:55:18 DEBUG : Creating backend with remote "http_sync:/file1.txt"
2025/03/05 23:55:18 DEBUG : Using config file from "/home/janosik/.config/rclone/rclone.conf"
2025/03/05 23:55:18 DEBUG : Assuming path is a file as HEAD response is success (200 OK)
2025/03/05 23:55:18 DEBUG : If path is a directory you must add a trailing '/'
2025/03/05 23:55:18 DEBUG : Root: http://localhost:9999/
2025/03/05 23:55:18 DEBUG : Creating backend with remote "./rclone-sync"
2025/03/05 23:55:18 DEBUG : fs cache: renaming cache item "./rclone-sync" to be canonical "/home/janosik/Downloads/rclone-v1.69.0-linux-amd64/rclone-sync"
2025/03/05 23:55:18 DEBUG : file1.txt: Need to transfer - File not found at Destination
2025/03/05 23:55:18 DEBUG : file2.md: Need to transfer - File not found at Destination
2025/03/05 23:55:18 DEBUG : Local file system at /home/janosik/Downloads/rclone-v1.69.0-linux-amd64/rclone-sync: Waiting for checks to finish
2025/03/05 23:55:18 DEBUG : Local file system at /home/janosik/Downloads/rclone-v1.69.0-linux-amd64/rclone-sync: Waiting for transfers to finish
2025/03/05 23:55:18 DEBUG : file1.txt.f170f1d5.partial: renamed to: file1.txt
2025/03/05 23:55:18 DEBUG : file2.md.d66d5580.partial: renamed to: file2.md
2025/03/05 23:55:18 INFO  : file1.txt: Copied (new)
2025/03/05 23:55:18 INFO  : file2.md: Copied (new)
2025/03/05 23:55:18 DEBUG : Waiting for deletions to finish
2025/03/05 23:55:18 INFO  : 
Transferred:   	        675 B / 675 B, 100%, 0 B/s, ETA -
Transferred:            2 / 2, 100%
Elapsed time:         0.0s
```

#### A log from the command with the `-vv` flag [version: `1.68.2`]

```
2025/03/05 23:53:34 DEBUG : rclone: Version "v1.68.2" starting with parameters ["./rclone" "sync" "http_sync:/file1.txt" "./rclone-sync" "-vv"]
2025/03/05 23:53:34 DEBUG : Creating backend with remote "http_sync:/file1.txt"
2025/03/05 23:53:34 DEBUG : Using config file from "/home/janosik/.config/rclone/rclone.conf"
2025/03/05 23:53:34 DEBUG : Assuming path is a file as HEAD response is success (200 OK)
2025/03/05 23:53:34 DEBUG : If path is a directory you must add a trailing '/'
2025/03/05 23:53:34 DEBUG : Root: http://localhost:9999/
2025/03/05 23:53:34 DEBUG : Creating backend with remote "./rclone-sync"
2025/03/05 23:53:34 DEBUG : fs cache: renaming cache item "./rclone-sync" to be canonical "/home/janosik/Downloads/rclone-v1.68.2-linux-amd64/rclone-sync"
2025/03/05 23:53:34 DEBUG : file1.txt: Need to transfer - File not found at Destination
2025/03/05 23:53:34 DEBUG : file1.txt.f170f1d5.partial: renamed to: file1.txt
2025/03/05 23:53:34 INFO  : file1.txt: Copied (new)
2025/03/05 23:53:34 INFO  : 
Transferred:   	          8 B / 8 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.0s
```

#### Used `bisect-rclone.sh` test script

```sh
rm -rf test-sync || true
rclone sync -vv http_sync:/file1.txt ./test-sync/
test -f test-sync/file1.txt
test ! -f test-sync/file2.md
```

<!--- Please keep the note below for others who read your bug report. -->

#### How to use GitHub

* Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue.
* Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
* Subscribe to receive notifications on status change and new comments.


Pull request:
http: correct root if definitely pointing to a file - fixes #8428
#### What is the purpose of this change?

Fixes #8428 by ensuring that `http` returns the correct root when pointing to a file (as opposed to a dir).

This was formalized in https://github.com/rclone/rclone/commit/c69eb84573c85206ab028eda2987180e049ef2e4 But it appears that we forgot to update `http`, and the `FsRoot` test didn't catch it because we don't currently have an `http` integration test.

#### Was the change discussed in an issue or in the forum before?

- #8428

#### Checklist

- [x] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-new-feature-or-bug-fix).
- [x] I have added tests for all changes in this PR if appropriate.
- [x] I have added documentation for the changes if appropriate.
- [x] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages).
- [x] I'm done, this Pull Request is ready for review :-)


Patch:
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 615dd2231..e6f7bc0be 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -226,6 +226,8 @@ jobs:
 
       - name: Checkout
         uses: actions/checkout@v4
+        with:
+            fetch-depth: 0
 
       - name: Install Go
         id: setup-go
@@ -289,6 +291,10 @@ jobs:
       - name: Scan for vulnerabilities
         run: govulncheck ./...
 
+      - name: Scan edits of autogenerated files
+        run: bin/check_autogenerated_edits.py
+        if: github.event_name == 'pull_request'
+
   android:
     if: inputs.manual || (github.repository == 'rclone/rclone' && (github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name != github.event.pull_request.base.repo.full_name))
     timeout-minutes: 30
diff --git a/backend/http/http.go b/backend/http/http.go
index 4b64ec304..d4b426a7f 100644
--- a/backend/http/http.go
+++ b/backend/http/http.go
@@ -180,7 +180,6 @@ func getFsEndpoint(ctx context.Context, client *http.Client, url string, opt *Op
 	}
 	addHeaders(req, opt)
 	res, err := noRedir.Do(req)
-
 	if err != nil {
 		fs.Debugf(nil, "Assuming path is a file as HEAD request could not be sent: %v", err)
 		return createFileResult()
@@ -249,6 +248,14 @@ func (f *Fs) httpConnection(ctx context.Context, opt *Options) (isFile bool, err
 	f.httpClient = client
 	f.endpoint = u
 	f.endpointURL = u.String()
+
+	if isFile {
+		// Correct root if definitely pointing to a file
+		f.root = path.Dir(f.root)
+		if f.root == "." || f.root == "/" {
+			f.root = ""
+		}
+	}
 	return isFile, nil
 }
 
diff --git a/bin/check_autogenerated_edits.py b/bin/check_autogenerated_edits.py
new file mode 100755
index 000000000..59ad0ff84
--- /dev/null
+++ b/bin/check_autogenerated_edits.py
@@ -0,0 +1,133 @@
+#!/usr/bin/env python3
+"""
+This script checks for unauthorized modifications in autogenerated sections of markdown files.
+It is designed to be used in a GitHub Actions workflow or a local pre-commit hook.
+
+Features:
+- Detects markdown files changed in the last commit.
+- Identifies modified autogenerated sections marked by specific comments.
+- Reports violations using GitHub Actions error messages.
+- Exits with a nonzero status code if unauthorized changes are found.
+
+It currently only checks the last commit.
+"""
+
+import re
+import subprocess
+import sys
+
+def run_git(args):
+    """
+    Run a Git command with the provided arguments and return its output as a string.
+    """
+    return subprocess.run(["git"] + args, stdout=subprocess.PIPE, text=True, check=True).stdout.strip()
+
+def get_changed_files():
+    """
+    Retrieve a list of markdown files that were changed in the last commit.
+    """
+    files = run_git(["diff", "--name-only", "HEAD~1", "HEAD"]).splitlines()
+    return [f for f in files if f.endswith(".md")]
+
+def get_diff(file):
+    """
+    Get the diff of a given file between the last commit and the current version.
+    """
+    return run_git(["diff", "-U0", "HEAD~1", "HEAD", "--", file]).splitlines()
+
+def get_file_content(ref, file):
+    """
+    Retrieve the content of a file from a given Git reference.
+    """
+    try:
+        return run_git(["show", f"{ref}:{file}"]).splitlines()
+    except Exception:
+        return []
+
+def find_regions(lines):
+    """
+    Identify the start and end line numbers of autogenerated regions in a file.
+    """
+    regions = []
+    start = None
+    for i, line in enumerate(lines, 1):
+        if "rem autogenerated options start" in line:
+            start = i
+        elif "rem autogenerated options stop" in line and start is not None:
+            regions.append((start, i))
+            start = None
+    return regions
+
+def in_region(ln, regions):
+    """
+    Check if a given line number falls within an autogenerated region.
+    """
+    return any(start <= ln <= end for start, end in regions)
+
+def show_error(file_name, line, message):
+    """
+    Print an error message in a GitHub Actions-compatible format.
+    """
+    print(f"::error file={file_name},line={line}::{message} at {file_name} line {line}")
+
+def check_file(file):
+    """
+    Check a markdown file for modifications in autogenerated regions.
+    """
+    viol = False
+    new_lines = get_file_content("HEAD", file)
+    old_lines = get_file_content("HEAD~1", file)
+    
+    # Entire autogenerated file check.
+    if any("autogenerated - DO NOT EDIT" in l for l in new_lines[:10]):
+        if get_diff(file):
+            show_error(file, 1, "Autogenerated file modified")
+            return True
+        return False
+    
+    # Partial autogenerated regions.
+    regions_new = find_regions(new_lines)
+    regions_old = find_regions(old_lines)
+    diff = get_diff(file)
+    hunk_re = re.compile(r"^@@ -(\d+),?(\d*) \+(\d+),?(\d*) @@")
+    new_ln = old_ln = None
+    
+    for line in diff:
+        if line.startswith("@@"):
+            m = hunk_re.match(line)
+            if m:
+                old_ln = int(m.group(1))
+                new_ln = int(m.group(3))
+        elif new_ln is None:
+            continue
+        elif line.startswith("+"):
+            if in_region(new_ln, regions_new):
+                show_error(file, new_ln, "Autogenerated region of file modified")
+                viol = True
+            new_ln += 1
+        elif line.startswith("-"):
+            if in_region(old_ln, regions_old):
+                show_error(file, old_ln, "Autogenerated region of file modified")
+                viol = True
+            old_ln += 1
+        else:
+            new_ln += 1
+            old_ln += 1
+    
+    return viol
+
+def main():
+    """
+    Main function that iterates over changed files and checks them for violations.
+    """
+    found = False
+    for f in get_changed_files():
+        if check_file(f):
+            found = True
+    if found:
+        sys.exit(1)
+    print("No unauthorized edits found in autogenerated sections.")
+    sys.exit(0)
+
+if __name__ == "__main__":
+    main()
diff --git a/docs/content/authors.md b/docs/content/authors.md
index 737f7b776..662028431 100644
--- a/docs/content/authors.md
+++ b/docs/content/authors.md
@@ -947,3 +947,4 @@ put them back in again.` >}}
   * Lorenz Brun <lorenz@brun.one>
   * Dave Vasilevsky <djvasi@gmail.com> <dave@vasilevsky.ca>
   * luzpaz <luzpaz@users.noreply.github.com>
+  * jack <9480542+jackusm@users.noreply.github.com>
diff --git a/docs/content/docs.md b/docs/content/docs.md
index 17c909fea..c4c1de48d 100644
--- a/docs/content/docs.md
+++ b/docs/content/docs.md
@@ -1526,6 +1526,24 @@ of the remote which may be desirable.
 Setting this to a negative number will make the backlog as large as
 possible.
 
+### --max-buffer-memory=SIZE {#max-buffer-memory}
+
+If set, don't allocate more than SIZE amount of memory as buffers. If
+not set or set to `0` or `off` this will not limit the amount of memory
+in use.
+
+This includes memory used by buffers created by the `--buffer` flag
+and buffers used by multi-thread transfers.
+
+Most multi-thread transfers do not take additional memory, but some do
+depending on the backend (eg the s3 backend for uploads). This means
+there is a tension between total setting `--transfers` as high as
+possible and memory use.
+
+Setting `--max-buffer-memory` allows the buffer memory to be
+controlled so that it doesn't overwhelm the machine and allows
+`--transfers` to be set large.
+
 ### --max-delete=N ###
 
 This tells rclone not to delete more than N files.  If that limit is
@@ -1785,6 +1803,14 @@ This will work with the `sync`/`copy`/`move` commands and friends
 mount` and `rclone serve` if `--vfs-cache-mode` is set to `writes` or
 above.
 
+Most multi-thread transfers do not take additional memory, but some do
+(for example uploading to s3). In the worst case memory usage can be
+at maximum `--transfers` * `--multi-thread-chunk-size` *
+`--multi-thread-streams` or specifically for the s3 backend
+`--transfers` * `--s3-chunk-size` * `--s3-concurrency`. However you
+can use the the [--max-buffer-memory](/docs/#max-buffer-memory) flag
+to control the maximum memory used here.
+
 **NB** that this **only** works with supported backends as the
 destination but will work with any backend as the source.
 
@@ -2808,6 +2834,7 @@ For the filtering options
   * `--max-size`
   * `--min-age`
   * `--max-age`
+  * `--hash-filter`
   * `--dump filters`
   * `--metadata-include`
   * `--metadata-include-from`
diff --git a/docs/content/faq.md b/docs/content/faq.md
index 050caf8ef..19eafe6dd 100644
--- a/docs/content/faq.md
+++ b/docs/content/faq.md
@@ -239,6 +239,10 @@ memory. There is
 [a workaround for this](https://github.com/rclone/rclone/wiki/Big-syncs-with-millions-of-files)
 which involves a bit of scripting.
 
+From v1.70 rclone also has the [--max-buffer-memory](/docs/#max-buffer-memory)
+flag which helps particularly when multi-thread transfers are using
+too much memory.
+
 ### Rclone changes fullwidth Unicode punctuation marks in file names
 
 For example: On a Windows system, you have a file with name `Test：1.jpg`,
diff --git a/docs/content/filtering.md b/docs/content/filtering.md
index 8c7ce232f..dc881e8bf 100644
--- a/docs/content/filtering.md
+++ b/docs/content/filtering.md
@@ -718,6 +718,98 @@ old or more.
 
 See [the time option docs](/docs/#time-option) for valid formats.
 
+### `--hash-filter` - Deterministically select a subset of files {#hash-filter}
+
+The `--hash-filter` flag enables selecting a deterministic subset of files, useful for:
+
+1. Running large sync operations across multiple machines.
+2. Checking a subset of files for bitrot.
+3. Any other operations where a sample of files is required.
+
+#### Syntax
+
+The flag takes two parameters expressed as a fraction:
+
+```
+--hash-filter K/N
+```
+
+- `N`: The total number of partitions (must be a positive integer).
+- `K`: The specific partition to select (an integer from `0` to `N`).
+
+For example:
+- `--hash-filter 1/3`: Selects the first third of the files.
+- `--hash-filter 2/3` and `--hash-filter 3/3`: Select the second and third partitions, respectively.
+
+Each partition is non-overlapping, ensuring all files are covered without duplication.
+
+#### Random Partition Selection
+
+Use `@` as `K` to randomly select a partition:
+
+```
+--hash-filter @/M
+```
+
+For example, `--hash-filter @/3` will randomly select a number between 0 and 2. This will stay constant across retries.
+
+#### How It Works
+
+- Rclone takes each file's full path, normalizes it to lowercase, and applies Unicode normalization.
+- It then hashes the normalized path into a 64 bit number.
+- The hash result is reduced modulo `N` to assign the file to a partition.
+- If the calculated partition does not match `K` the file is excluded.
+- Other filters may apply if the file is not excluded.
+
+**Important:** Rclone will traverse all directories to apply the filter.
+
+#### Usage Notes
+
+- Safe to use with `rclone sync`; source and destination selections will match.
+- **Do not** use with `--delete-excluded`, as this could delete unselected files.
+- Ignored if `--files-from` is used.
+
+#### Examples
+
+##### Dividing files into 4 partitions
+
+Assuming the current directory contains `file1.jpg` through `file9.jpg`:
+
+```
+$ rclone lsf --hash-filter 0/4 .
+file1.jpg
+file5.jpg
+
+$ rclone lsf --hash-filter 1/4 .
+file3.jpg
+file6.jpg
+file9.jpg
+
+$ rclone lsf --hash-filter 2/4 .
+file2.jpg
+file4.jpg
+
+$ rclone lsf --hash-filter 3/4 .
+file7.jpg
+file8.jpg
+
+$ rclone lsf --hash-filter 4/4 . # the same as --hash-filter 0/4
+file1.jpg
+file5.jpg
+```
+
+##### Syncing the first quarter of files
+
+```
+rclone sync --hash-filter 1/4 source:path destination:path
+```
+
+##### Checking a random 1% of files for integrity
+
+```
+rclone check --download --hash-filter @/100 source:path destination:path
+```
+
 ## Other flags
 
 ### `--delete-excluded` - Delete files on dest excluded from sync
diff --git a/fs/accounting/stats.go b/fs/accounting/stats.go
index 88686e6c5..c50ef77b0 100644
--- a/fs/accounting/stats.go
+++ b/fs/accounting/stats.go
@@ -92,7 +92,9 @@ func NewStats(ctx context.Context) *StatsInfo {
 }
 
 // RemoteStats returns stats for rc
-func (s *StatsInfo) RemoteStats() (out rc.Params, err error) {
+//
+// If short is true then the transfers and checkers won't be added.
+func (s *StatsInfo) RemoteStats(short bool) (out rc.Params, err error) {
 	// NB if adding values here - make sure you update the docs in
 	// stats_groups.go
 
@@ -128,10 +130,10 @@ func (s *StatsInfo) RemoteStats() (out rc.Params, err error) {
 	}
 	s.mu.RUnlock()
 
-	if !s.checking.empty() {
+	if !short && !s.checking.empty() {
 		out["checking"] = s.checking.remotes()
 	}
-	if !s.transferring.empty() {
+	if !short && !s.transferring.empty() {
 		out["transferring"] = s.transferring.rcStats(s.inProgress)
 	}
 	if s.errors > 0 {
@@ -561,7 +563,7 @@ func (s *StatsInfo) Transferred() []TransferSnapshot {
 // Log outputs the StatsInfo to the log
 func (s *StatsInfo) Log() {
 	if s.ci.UseJSONLog {
-		out, _ := s.RemoteStats()
+		out, _ := s.RemoteStats(false)
 		fs.LogLevelPrintf(s.ci.StatsLogLevel, nil, "%v%v\n", s, fs.LogValueHide("stats", out))
 	} else {
 		fs.LogLevelPrintf(s.ci.StatsLogLevel, nil, "%v\n", s)
diff --git a/fs/accounting/stats_groups.go b/fs/accounting/stats_groups.go
index 5f8e44d17..13236aa52 100644
--- a/fs/accounting/stats_groups.go
+++ b/fs/accounting/stats_groups.go
@@ -56,11 +56,13 @@ func rcRemoteStats(ctx context.Context, in rc.Params) (rc.Params, error) {
 	if rc.NotErrParamNotFound(err) {
 		return rc.Params{}, err
 	}
+	short, _ := in.GetBool("short")
+
 	if group != "" {
-		return StatsGroup(ctx, group).RemoteStats()
+		return StatsGroup(ctx, group).RemoteStats(short)
 	}
 
-	return groups.sum(ctx).RemoteStats()
+	return groups.sum(ctx).RemoteStats(short)
 }
 
 func init() {
@@ -78,7 +80,8 @@ returned.
 
 Parameters
 
-- group - name of the stats group (string)
+- group - name of the stats group (string, optional)
+- short - if true will not return the transferring and checking arrays (boolean, optional)
 
 Returns the following values:
 
diff --git a/fs/config.go b/fs/config.go
index e83d2cac1..611552ffa 100644
--- a/fs/config.go
+++ b/fs/config.go
@@ -413,6 +413,11 @@ var ConfigOptionsInfo = Options{{
 	Default: false,
 	Help:    "Use mmap allocator (see docs)",
 	Groups:  "Config",
+}, {
+	Name:    "max_buffer_memory",
+	Default: SizeSuffix(-1),
+	Help:    "If set, don't allocate more than this amount of memory as buffers",
+	Groups:  "Config",
 }, {
 	Name:    "ca_cert",
 	Default: []string{},
@@ -613,6 +618,7 @@ type ConfigInfo struct {
 	ProgressTerminalTitle      bool              `config:"progress_terminal_title"`
 	Cookie                     bool              `config:"use_cookies"`
 	UseMmap                    bool              `config:"use_mmap"`
+	MaxBufferMemory            SizeSuffix        `config:"max_buffer_memory"`
 	CaCert                     []string          `config:"ca_cert"`     // Client Side CA
 	ClientCert                 string            `config:"client_cert"` // Client Side Cert
 	ClientKey                  string            `config:"client_key"`  // Client Side Key
diff --git a/fs/filter/filter.go b/fs/filter/filter.go
index ebe95e37c..0bbf3e69d 100644
--- a/fs/filter/filter.go
+++ b/fs/filter/filter.go
@@ -3,15 +3,20 @@ package filter
 
 import (
 	"context"
+	"crypto/md5"
+	"encoding/binary"
 	"errors"
 	"fmt"
+	"math/rand/v2"
 	"path"
 	"slices"
+	"strconv"
 	"strings"
 	"time"
 
 	"github.com/rclone/rclone/fs"
 	"golang.org/x/sync/errgroup"
+	"golang.org/x/text/unicode/norm"
 )
 
 // This is the globally active filter
@@ -65,6 +70,11 @@ var OptionsInfo = fs.Options{{
 	Default: false,
 	Help:    "Ignore case in filters (case insensitive)",
 	Groups:  "Filter",
+}, {
+	Name:    "hash_filter",
+	Default: "",
+	Help:    "Partition filenames by hash k/n or randomly @/n",
+	Groups:  "Filter",
 }, {
 	Name:     "filter",
 	Default:  []string{},
@@ -141,6 +151,7 @@ type Options struct {
 	MinSize        fs.SizeSuffix `config:"min_size"`
 	MaxSize        fs.SizeSuffix `config:"max_size"`
 	IgnoreCase     bool          `config:"ignore_case"`
+	HashFilter     string        `config:"hash_filter"`
 }
 
 func init() {
@@ -168,6 +179,8 @@ type Filter struct {
 	metaRules   rules
 	files       FilesMap // files if filesFrom
 	dirs        FilesMap // dirs from filesFrom
+	hashFilterN uint64   // if non 0 do hash filtering
+	hashFilterK uint64   // select partition K/N
 }
 
 // NewFilter parses the command line options and creates a Filter
@@ -190,10 +203,17 @@ func NewFilter(opt *Options) (f *Filter, err error) {
 	if f.Opt.MaxAge.IsSet() {
 		f.ModTimeFrom = time.Now().Add(-time.Duration(f.Opt.MaxAge))
 		if !f.ModTimeTo.IsZero() && f.ModTimeTo.Before(f.ModTimeFrom) {
-			fs.Fatalf(nil, "filter: --min-age %q can't be larger than --max-age %q", opt.MinAge, opt.MaxAge)
+			return nil, fmt.Errorf("filter: --min-age %q can't be larger than --max-age %q", opt.MinAge, opt.MaxAge)
 		}
 		fs.Debugf(nil, "--max-age %v to %v", f.Opt.MaxAge, f.ModTimeFrom)
 	}
+	if f.Opt.HashFilter != "" {
+		f.hashFilterK, f.hashFilterN, err = parseHashFilter(f.Opt.HashFilter)
+		if err != nil {
+			return nil, err
+		}
+		fs.Debugf(nil, "Using --hash-filter %d/%d", f.hashFilterK, f.hashFilterN)
+	}
 
 	err = parseRules(&f.Opt.RulesOpt, f.Add, f.Clear)
 	if err != nil {
@@ -243,6 +263,32 @@ func NewFilter(opt *Options) (f *Filter, err error) {
 	return f, nil
 }
 
+// Parse the --hash-filter arguments into k/n
+func parseHashFilter(hashFilter string) (k, n uint64, err error) {
+	slash := strings.IndexRune(hashFilter, '/')
+	if slash < 0 {
+		return 0, 0, fmt.Errorf("filter: --hash-filter: no / found")
+	}
+	kStr, nStr := hashFilter[:slash], hashFilter[slash+1:]
+	n, err = strconv.ParseUint(nStr, 10, 64)
+	if err != nil {
+		return 0, 0, fmt.Errorf("filter: --hash-filter: can't parse N=%q: %v", nStr, err)
+	}
+	if n == 0 {
+		return 0, 0, fmt.Errorf("filter: --hash-filter: N must be greater than 0")
+	}
+	if kStr == "@" {
+		k = rand.Uint64N(n)
+	} else {
+		k, err = strconv.ParseUint(kStr, 10, 64)
+		if err != nil {
+			return 0, 0, fmt.Errorf("filter: --hash-filter: can't parse K=%q: %v", kStr, err)
+		}
+		k %= n
+	}
+	return k, n, nil
+}
+
 func mustNewFilter(opt *Options) *Filter {
 	f, err := NewFilter(opt)
 	if err != nil {
@@ -367,7 +413,8 @@ func (f *Filter) InActive() bool {
 		f.fileRules.len() == 0 &&
 		f.dirRules.len() == 0 &&
 		f.metaRules.len() == 0 &&
-		len(f.Opt.ExcludeFile) == 0)
+		len(f.Opt.ExcludeFile) == 0 &&
+		f.hashFilterN == 0)
 }
 
 // IncludeRemote returns whether this remote passes the filter rules.
@@ -377,6 +424,21 @@ func (f *Filter) IncludeRemote(remote string) bool {
 		_, include := f.files[remote]
 		return include
 	}
+	if f.hashFilterN != 0 {
+		// Normalise the remote first in case we are using a
+		// case insensitive remote or a remote which needs
+		// unicode normalisation. This means all the remotes
+		// which could be normalised together will be in the
+		// same partition.
+		normalized := norm.NFC.String(remote)
+		normalized = strings.ToLower(normalized)
+		hashBytes := md5.Sum([]byte(normalized))
+		hash := binary.LittleEndian.Uint64(hashBytes[:])
+		partition := hash % f.hashFilterN
+		if partition != f.hashFilterK {
+			return false
+		}
+	}
 	return f.fileRules.include(remote)
 }
 
@@ -522,6 +584,12 @@ func (f *Filter) DumpFilters() string {
 	if !f.ModTimeTo.IsZero() {
 		rules = append(rules, fmt.Sprintf("Last-modified date must be equal or less than: %s", f.ModTimeTo.String()))
 	}
+	if f.Opt.MinSize >= 0 {
+		rules = append(rules, fmt.Sprintf("Minimum size is: %s", f.Opt.MinSize.ByteUnit()))
+	}
+	if f.Opt.MaxSize >= 0 {
+		rules = append(rules, fmt.Sprintf("Maximum size is: %s", f.Opt.MaxSize.ByteUnit()))
+	}
 	rules = append(rules, "--- File filter rules ---")
 	for _, rule := range f.fileRules.rules {
 		rules = append(rules, rule.String())
diff --git a/fs/registry.go b/fs/registry.go
index 628a18afe..2df893394 100644
--- a/fs/registry.go
+++ b/fs/registry.go
@@ -259,15 +259,27 @@ func (o *Option) IsDefault() bool {
 // String turns Option into a string
 func (o *Option) String() string {
 	v := o.GetValue()
-	if stringArray, isStringArray := v.([]string); isStringArray {
+	switch x := v.(type) {
+	case []string:
 		// Treat empty string array as empty string
 		// This is to make the default value of the option help nice
-		if len(stringArray) == 0 {
+		if len(x) == 0 {
 			return ""
 		}
 		// Encode string arrays as CSV
 		// The default Go encoding can't be decoded uniquely
-		return CommaSepList(stringArray).String()
+		return CommaSepList(x).String()
+	case SizeSuffix:
+		str := x.String()
+		// Suffix bare numbers with "B" unless they are 0
+		//
+		// This makes sure that fs.SizeSuffix roundtrips through string
+		if len(str) > 0 && str != "0" {
+			if lastDigit := str[len(str)-1]; lastDigit >= '0' && lastDigit <= '9' {
+				str += "B"
+			}
+		}
+		return str
 	}
 	return fmt.Sprint(v)
 }
diff --git a/go.mod b/go.mod
index 0554b4431..093a444e6 100644
--- a/go.mod
+++ b/go.mod
@@ -81,7 +81,7 @@ require (
 	go.etcd.io/bbolt v1.4.0
 	goftp.io/server/v2 v2.0.1
 	golang.org/x/crypto v0.35.0
-	golang.org/x/net v0.35.0
+	golang.org/x/net v0.36.0
 	golang.org/x/oauth2 v0.27.0
 	golang.org/x/sync v0.11.0
 	golang.org/x/sys v0.30.0
diff --git a/go.sum b/go.sum
index 4cb7c8cc8..0fb09b3dd 100644
--- a/go.sum
+++ b/go.sum
@@ -787,8 +787,8 @@ golang.org/x/net v0.15.0/go.mod h1:idbUs1IY1+zTqbi8yxTbhexhEEk5ur9LInksu6HrEpk=
 golang.org/x/net v0.17.0/go.mod h1:NxSsAGuq816PNPmqtQdLE42eU2Fs7NoRIZrHJAlaCOE=
 golang.org/x/net v0.21.0/go.mod h1:bIjVDfnllIU7BJ2DNgfnXvpSvtn8VRwhlsaeUTyUS44=
 golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
-golang.org/x/net v0.35.0 h1:T5GQRQb2y08kTAByq9L4/bz8cipCdA8FbRTXewonqY8=
-golang.org/x/net v0.35.0/go.mod h1:EglIi67kWsHKlRzzVMUD93VMSWGFOMSZgxFjparz1Qk=
+golang.org/x/net v0.36.0 h1:vWF2fRbw4qslQsQzgFqZff+BItCvGFQqKzKIzx1rmoA=
+golang.org/x/net v0.36.0/go.mod h1:bFmbeoIPfrw4sMHNhb4J9f6+tPziuGjq7Jk/38fxi1I=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
 golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
 golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
diff --git a/lib/pool/pool.go b/lib/pool/pool.go
index a6484b7cc..a4a5dc752 100644
--- a/lib/pool/pool.go
+++ b/lib/pool/pool.go
@@ -3,12 +3,14 @@
 package pool
 
 import (
+	"context"
 	"fmt"
 	"sync"
 	"time"
 
 	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/lib/mmap"
+	"golang.org/x/sync/semaphore"
 )
 
 // Pool of internal buffers
@@ -33,6 +35,14 @@ type Pool struct {
 	free         func([]byte) error
 }
 
+// totalMemory is a semaphore used to control total buffer usage of
+// all Pools. It it may be nil in which case the total buffer usage
+// will not be controlled.
+var totalMemory *semaphore.Weighted
+
+// Make sure we initialise the totalMemory semaphore once
+var totalMemoryInit sync.Once
+
 // New makes a buffer pool
 //
 // flushTime is the interval the buffer pools is flushed
@@ -145,6 +155,33 @@ func (bp *Pool) updateMinFill() {
 	}
 }
 
+// acquire mem bytes of memory
+func (bp *Pool) acquire(mem int64) error {
+	ctx := context.Background()
+
+	totalMemoryInit.Do(func() {
+		ci := fs.GetConfig(ctx)
+
+		// Set max buffer memory limiter
+		if ci.MaxBufferMemory > 0 {
+			totalMemory = semaphore.NewWeighted(int64(ci.MaxBufferMemory))
+		}
+	})
+
+	if totalMemory == nil {
+		return nil
+	}
+	return totalMemory.Acquire(ctx, mem)
+}
+
+// release mem bytes of memory
+func (bp *Pool) release(mem int64) {
+	if totalMemory == nil {
+		return
+	}
+	totalMemory.Release(mem)
+}
+
 // Get a buffer from the pool or allocate one
 func (bp *Pool) Get() []byte {
 	bp.mu.Lock()
@@ -156,10 +193,16 @@ func (bp *Pool) Get() []byte {
 			break
 		} else {
 			var err error
-			buf, err = bp.alloc(bp.bufferSize)
+			bp.mu.Unlock()
+			err = bp.acquire(int64(bp.bufferSize))
+			bp.mu.Lock()
 			if err == nil {
-				bp.alloced++
-				break
+				buf, err = bp.alloc(bp.bufferSize)
+				if err == nil {
+					bp.alloced++
+					break
+				}
+				bp.release(int64(bp.bufferSize))
 			}
 			fs.Logf(nil, "Failed to get memory for buffer, waiting for %v: %v", waitTime, err)
 			bp.mu.Unlock()
@@ -179,6 +222,8 @@ func (bp *Pool) freeBuffer(mem []byte) {
 	err := bp.free(mem)
 	if err != nil {
 		fs.Logf(nil, "Failed to free memory: %v", err)
+	} else {
+		bp.release(int64(bp.bufferSize))
 	}
 	bp.alloced--
 }
diff --git a/vfs/vfs.md b/vfs/vfs.md
index 3d0d5622c..dde8f9f30 100644
--- a/vfs/vfs.md
+++ b/vfs/vfs.md
@@ -95,11 +95,11 @@ seconds. If rclone is quit or dies with files that haven't been
 uploaded, these will be uploaded next time rclone is run with the same
 flags.
 
-If using `--vfs-cache-max-size` or `--vfs-cache-min-free-size` note
+If using `--vfs-cache-max-size` or `--vfs-cache-min-free-space` note
 that the cache may exceed these quotas for two reasons. Firstly
 because it is only checked every `--vfs-cache-poll-interval`. Secondly
 because open files cannot be evicted from the cache. When
-`--vfs-cache-max-size` or `--vfs-cache-min-free-size` is exceeded,
+`--vfs-cache-max-size` or `--vfs-cache-min-free-space` is exceeded,
 rclone will attempt to evict the least accessed files from the cache
 first. rclone will start with files that haven't been accessed for the
 longest. This cache flushing strategy is efficient and more relevant


Test patch:
diff --git a/fs/accounting/stats_groups_test.go b/fs/accounting/stats_groups_test.go
index 4a0cd8628..e0d156672 100644
--- a/fs/accounting/stats_groups_test.go
+++ b/fs/accounting/stats_groups_test.go
@@ -2,6 +2,7 @@ package accounting
 
 import (
 	"context"
+	"encoding/json"
 	"fmt"
 	"runtime"
 	"testing"
@@ -10,6 +11,7 @@ import (
 	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fs/fserrors"
 	"github.com/rclone/rclone/fs/rc"
+	"github.com/rclone/rclone/fstest/mockobject"
 	"github.com/rclone/rclone/fstest/testy"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -141,14 +143,35 @@ func TestStatsGroupOperations(t *testing.T) {
 	})
 
 	t.Run("core/stats", func(t *testing.T) {
+		tr := Stats(ctx).NewCheckingTransfer(mockobject.New("core-check"), "deleting")
+		// defer tr.Done(ctx, nil)
+		_ = tr // don't finish the transfer so we don't mess up the other tests
+		tr2 := Stats(ctx).NewTransfer(mockobject.New("core-transfer"), nil)
+		//defer tr2.Done(ctx, nil)
+		_ = tr2 // don't finish the transfer so we don't mess up the other tests
+
 		call := rc.Calls.Get("core/stats")
 		require.NotNil(t, call)
-		gotNoGroup, err := call.Fn(ctx, rc.Params{})
+
+		got, err := call.Fn(ctx, rc.Params{})
 		require.NoError(t, err)
-		gotGroup, err := call.Fn(ctx, rc.Params{"group": "test-group"})
+		assert.Equal(t, int64(42), got["deletes"])
+		data, err := json.Marshal(got["transferring"])
+		require.NoError(t, err)
+		assert.Contains(t, string(data), "core-transfer")
+		data, err = json.Marshal(got["checking"])
+		require.NoError(t, err)
+		assert.Contains(t, string(data), "core-check")
+
+		got, err = call.Fn(ctx, rc.Params{"short": true})
+		require.NoError(t, err)
+		assert.Equal(t, int64(42), got["deletes"])
+		assert.Nil(t, got["transferring"])
+		assert.Nil(t, got["checking"])
+
+		got, err = call.Fn(ctx, rc.Params{"group": "test-group"})
 		require.NoError(t, err)
-		assert.Equal(t, int64(42), gotNoGroup["deletes"])
-		assert.Equal(t, int64(1), gotGroup["deletes"])
+		assert.Equal(t, int64(1), got["deletes"])
 	})
 
 	t.Run("core/transferred", func(t *testing.T) {
diff --git a/fs/accounting/stats_test.go b/fs/accounting/stats_test.go
index a6378d065..6fcc09837 100644
--- a/fs/accounting/stats_test.go
+++ b/fs/accounting/stats_test.go
@@ -265,7 +265,7 @@ func TestRemoteStats(t *testing.T) {
 		}
 		s.AddTransfer(tr1)
 		time.Sleep(time.Millisecond)
-		rs, err := s.RemoteStats()
+		rs, err := s.RemoteStats(false)
 
 		require.NoError(t, err)
 		assert.Equal(t, float64(10), rs["transferTime"])
diff --git a/fs/filter/filter_test.go b/fs/filter/filter_test.go
index 0b5f934da..a5e82d3ab 100644
--- a/fs/filter/filter_test.go
+++ b/fs/filter/filter_test.go
@@ -28,6 +28,38 @@ func TestNewFilterDefault(t *testing.T) {
 	assert.True(t, f.InActive())
 }
 
+func TestParseHashFilter(t *testing.T) {
+	for _, test := range []struct {
+		hashFilter string
+		n          uint64
+		k          uint64
+		err        string
+	}{
+		{hashFilter: "", err: "no / found"},
+		{hashFilter: "17", err: "no / found"},
+		{hashFilter: "-1/2", err: "can't parse K="},
+		{hashFilter: "1/-2", err: "can't parse N="},
+		{hashFilter: "0/0", err: "N must be greater than 0"},
+		{hashFilter: "0/18446744073709551615", k: 0, n: 18446744073709551615},
+		{hashFilter: "0/18446744073709551616", err: "can't parse N="},
+		{hashFilter: "18446744073709551615/1", k: 0, n: 1},
+		{hashFilter: "18446744073709551616/1", err: "can't parse K="},
+		{hashFilter: "1/2", k: 1, n: 2},
+		{hashFilter: "17/3", k: 2, n: 3},
+		{hashFilter: "@/1", k: 0, n: 1},
+	} {
+		gotK, gotN, gotErr := parseHashFilter(test.hashFilter)
+		if test.err != "" {
+			assert.Error(t, gotErr)
+			assert.ErrorContains(t, gotErr, test.err, test.hashFilter)
+		} else {
+			assert.Equal(t, test.k, gotK, test.hashFilter)
+			assert.Equal(t, test.n, gotN, test.hashFilter)
+			assert.NoError(t, gotErr, test.hashFilter)
+		}
+	}
+}
+
 // testFile creates a temp file with the contents
 func testFile(t *testing.T, contents string) string {
 	out, err := os.CreateTemp("", "filter_test")
@@ -180,7 +212,9 @@ func TestNewFilterFullExceptFilesFromOpt(t *testing.T) {
 	assert.Equal(t, f.Opt.MinSize, mins)
 	assert.Equal(t, f.Opt.MaxSize, maxs)
 	got := f.DumpFilters()
-	want := `--- File filter rules ---
+	want := `Minimum size is: 100 KiB
+Maximum size is: 1000 KiB
+--- File filter rules ---
 + (^|/)include1$
 + (^|/)include2$
 + (^|/)include3$
@@ -207,6 +241,7 @@ type includeTest struct {
 }
 
 func testInclude(t *testing.T, f *Filter, tests []includeTest) {
+	t.Helper()
 	for _, test := range tests {
 		got := f.Include(test.in, test.size, time.Unix(test.modTime, 0), nil)
 		assert.Equal(t, test.want, got, fmt.Sprintf("in=%q, size=%v, modTime=%v", test.in, test.size, time.Unix(test.modTime, 0)))
@@ -537,6 +572,42 @@ func TestNewFilterMatchesRegexp(t *testing.T) {
 	assert.False(t, f.InActive())
 }
 
+func TestNewFilterHashFilter(t *testing.T) {
+	const e1 = "filé1.jpg" // one of the unicode E characters
+	const e2 = "filé1.jpg"  // a different unicode E character
+	assert.NotEqual(t, e1, e2)
+	for i := 0; i <= 4; i++ {
+		opt := Opt
+		opt.HashFilter = fmt.Sprintf("%d/4", i)
+		opt.ExcludeRule = []string{"*.bin"}
+		f, err := NewFilter(&opt)
+		require.NoError(t, err)
+		t.Run(opt.HashFilter, func(t *testing.T) {
+			testInclude(t, f, []includeTest{
+				{"file1.jpg", 0, 0, i == 0 || i == 4},
+				{"FILE1.jpg", 0, 0, i == 0 || i == 4},
+				{"file2.jpg", 1, 0, i == 2},
+				{"File2.jpg", 1, 0, i == 2},
+				{"file3.jpg", 2, 0, i == 1},
+				{"file4.jpg", 3, 0, i == 2},
+				{"file5.jpg", 4, 0, i == 0 || i == 4},
+				{"file6.jpg", 5, 0, i == 1},
+				{"file7.jpg", 6, 0, i == 3},
+				{"file8.jpg", 7, 0, i == 3},
+				{"file9.jpg", 7, 0, i == 1},
+				{e1, 0, 0, i == 3},
+				{e2, 0, 0, i == 3},
+				{"hello" + e1, 0, 0, i == 2},
+				{"HELLO" + e2, 0, 0, i == 2},
+				{"hello1" + e1, 0, 0, i == 1},
+				{"Hello1" + e2, 0, 0, i == 1},
+				{"exclude.bin", 8, 0, false},
+			})
+		})
+		assert.False(t, f.InActive())
+	}
+}
+
 type includeTestMetadata struct {
 	in       string
 	metadata fs.Metadata
diff --git a/fs/fs_test.go b/fs/fs_test.go
index e0099e96a..45b16229c 100644
--- a/fs/fs_test.go
+++ b/fs/fs_test.go
@@ -261,6 +261,34 @@ func TestOptionString(t *testing.T) {
 	assert.Equal(t, "true", caseInsensitiveOption.String())
 }
 
+func TestOptionStringStringArray(t *testing.T) {
+	opt := Option{
+		Name:    "string_array",
+		Default: []string(nil),
+	}
+	assert.Equal(t, "", opt.String())
+	opt.Default = []string{}
+	assert.Equal(t, "", opt.String())
+	opt.Default = []string{"a", "b"}
+	assert.Equal(t, "a,b", opt.String())
+	opt.Default = []string{"hello, world!", "goodbye, world!"}
+	assert.Equal(t, `"hello, world!","goodbye, world!"`, opt.String())
+}
+
+func TestOptionStringSizeSuffix(t *testing.T) {
+	opt := Option{
+		Name:    "size_suffix",
+		Default: SizeSuffix(0),
+	}
+	assert.Equal(t, "0", opt.String())
+	opt.Default = SizeSuffix(-1)
+	assert.Equal(t, "off", opt.String())
+	opt.Default = SizeSuffix(100)
+	assert.Equal(t, "100B", opt.String())
+	opt.Default = SizeSuffix(1024)
+	assert.Equal(t, "1Ki", opt.String())
+}
+
 func TestOptionSet(t *testing.T) {
 	o := caseInsensitiveOption
 	assert.Equal(t, true, o.Value)
diff --git a/lib/pool/pool_test.go b/lib/pool/pool_test.go
index 72ac42a8b..23618194a 100644
--- a/lib/pool/pool_test.go
+++ b/lib/pool/pool_test.go
@@ -1,12 +1,15 @@
 package pool
 
 import (
+	"context"
 	"errors"
 	"fmt"
 	"math/rand"
+	"sync"
 	"testing"
 	"time"
 
+	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fstest/testy"
 	"github.com/stretchr/testify/assert"
 )
@@ -225,3 +228,55 @@ func TestPool(t *testing.T) {
 		})
 	}
 }
+
+func TestPoolMaxBufferMemory(t *testing.T) {
+	ctx := context.Background()
+	ci := fs.GetConfig(ctx)
+	ci.MaxBufferMemory = 4 * 4096
+	defer func() {
+		ci.MaxBufferMemory = 0
+		totalMemory = nil
+	}()
+	totalMemoryInit = sync.Once{} // reset the sync.Once as it likely has been used
+	totalMemory = nil
+	bp := New(60*time.Second, 4096, 2, true)
+
+	assert.Equal(t, bp.alloced, 0)
+	assert.Nil(t, totalMemory)
+	buf := bp.Get()
+	assert.NotNil(t, totalMemory)
+	bp.Put(buf)
+	assert.Equal(t, bp.alloced, 1)
+
+	var (
+		wg       sync.WaitGroup
+		mu       sync.Mutex
+		bufs     int
+		maxBufs  int
+		countBuf = func(i int) {
+			mu.Lock()
+			defer mu.Unlock()
+			bufs += i
+			if bufs > maxBufs {
+				maxBufs = bufs
+			}
+		}
+	)
+	for i := 0; i < 20; i++ {
+		wg.Add(1)
+		go func() {
+			defer wg.Done()
+			buf := bp.Get()
+			countBuf(1)
+			time.Sleep(100 * time.Millisecond)
+			bp.Put(buf)
+			countBuf(-1)
+		}()
+	}
+
+	wg.Wait()
+
+	assert.Equal(t, bufs, 0)
+	assert.Equal(t, maxBufs, 4)
+	assert.Equal(t, bp.alloced, 2)
+}