Repository: rclone/rclone
Pull request #4617 (2020-09-20 14:32:50)

mailru: avoid prehashing of large local files
#### What is the purpose of this change?

Mailru backend suffers from problem similar to `chunker` PR https://github.com/rclone/rclone/pull/4614
When a localfs file is uploaded to mailru, the backend requests hashsum from file source.
Local files provide any kind of hashsum but hashing can take long for large files and postpone actual transfer.
Unlike chunker, mailru can use hashsum in advance to completely skip upload (transparent deduplication on server).
However, in case of encrypted or unique files (eg. `restic` dumps) the hashsum check is irrelevant and ineffective.

The mailru backend has config options that control eligibility of this optimization:
`speedup_enable` (can completely disable hashsumming), `speedup_file_patterns` (enable based on file extension), `speedup_max_disk` (disable based on file size), etc.

Due to a bug in backend these options were ignored for local FS files resulting in pre-transfer delays.
This PR fixes backend behavior and enforces above options on local files.

#### Was the change discussed in an issue or in the forum before?

No. It's bugfix/optimization, internal detail of backend implementation.

#### Checklist

- [x] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-pull-request).
- [ ] I have added tests for all changes in this PR if appropriate (**using integration suite**).
- [ ] I have added documentation for the changes if appropriate (**user visible behavior does not change**).
- [x] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages).
- [x] I'm done, this Pull Request is ready for review :-)

Mini integration test passed: https://github.com/ivandeex/rclone/runs/1140319875?check_suite_focus=true

ivandeex: This PR could benefit from **Features.SlowHash**
ncw: Sorry behind in PRs at the moment :-(

I think this is waiting an update from you for SlowHash ?
ivandeex: NP. I'm just busy elsewhere. Will revert to this soon.
ivandeex: Also adding body close after _stream-skip error_ [in Object.Open](https://github.com/rclone/rclone/pull/4607/files#diff-6951211abf5f87eb0e379afacd58000bf97769e16edaa5fb0ba55f2e0db66495R2185) forgotten by PR https://github.com/rclone/rclone/pull/4607
ivandeex: Integration test suite passed:
https://github.com/ivandeex/rclone/runs/1261590695?check_suite_focus=true

I'm in doubt on how to automate this PR's unit test (measure time before actual download? intercept os.Stdout and look for messages?), so here follows the manual procedure:

Download beta version of rclone with this fix applied from https://beta.rclone.org or from my repository https://github.com/ivandeex/rclone/releases/tag/v1.53-mailru-prehash-7:
```
curl -sLo rclone.test https://github.com/ivandeex/rclone/releases/download/v1.53-mailru-prehash-7/rclone-linux-amd64
chmod +x rclone.test
./rclone.test version
```

You need a reasonably large file, which has been already uploaded to mailru by someone (eg. a popular movie or song). I used [Tesla, 2020, 1.4Gb](https://rutracker.org/forum/viewtopic.php?t=5931157).
```
./rclone.test hashsum MailruHash ./tesla.avi
```

Upload your test file to mailru. Since its extension `.avi` is eligible for speedup, rclone will delay 5-10 seconds to calculate hashsum before upload, then sends no data since the file is already present on server:
```
./rclone.test purge TestMailru:test
./rclone.test copy -vvvP ./tesla.avi TestMailru:test

20:43:19 DEBUG : tesla.avi: Need to transfer - File not found at Destination
20:43:19 DEBUG : [test]: new meta server: https://cld-extapi2.datacloudmail.ru/meta/
20:43:24 DEBUG : tesla.avi: File has been put by hash from localfs
20:43:24 DEBUG : tesla.avi: MailruHash = 19a154a2cba475f7c5b4ecfefed9a8e6c80a06f2 OK
20:43:24 INFO  : tesla.avi: Copied (new)
Transferred:             0 / 0 Bytes, -, 0 Bytes/s, ETA -
```

Disable speedup completely and try again, rclone should skip hashing and proceed to upload without delays, the upload will take 20-30 seconds depending on your internet bandwidth:
```
./rclone.test purge TestMailru:test
./rclone.test copy --mailru-speedup-enable=false -vvvP ./tesla.avi TestMailru:test

20:46:05 DEBUG : tesla.avi: Need to transfer - File not found at Destination
20:46:05 DEBUG : [test]: new meta server: https://cld-extapi3.datacloudmail.ru/meta/
20:46:05 DEBUG : [test]: new upload shard: https://cld-upload6.cloud.mail.ru/upload/
20:46:28 DEBUG : tesla.avi: MailruHash = 19a154a2cba475f7c5b4ecfefed9a8e6c80a06f2 OK
20:46:28 INFO  : tesla.avi: Copied (new)
Transferred:        1.456G / 1.456 GBytes, 100%, 64.823 MBytes/s, ETA 0s
```

Limit maximum size of files eligible for speedup and try again, rclone should skip hashing and start upload immediately:
```
./rclone.test purge TestMailru:test
./rclone.test copy --mailru-speedup-max-disk=100M -vvvP ./tesla.avi TestMailru:test

20:54:30 DEBUG : tesla.avi: Need to transfer - File not found at Destination
20:54:30 DEBUG : [test]: new meta server: https://cld-extapi1.datacloudmail.ru/meta/
20:54:31 DEBUG : [test]: new upload shard: https://cld-upload3.cloud.mail.ru/upload/
20:54:53 DEBUG : tesla.avi: MailruHash = 19a154a2cba475f7c5b4ecfefed9a8e6c80a06f2 OK
20:54:53 INFO  : tesla.avi: Copied (new)
Transferred:        1.456G / 1.456 GBytes, 100%, 64.401 MBytes/s, ETA 0s
```

Change file extension so that it's not eligible anymore and try again, rclone should skip long pre-hashing like before:
```
cp ./tesla.avi ./test.bin
./rclone.test purge TestMailru:test
./rclone.test copy -vvvP ./test.bin TestMailru:test

20:50:43 DEBUG : test.bin: Need to transfer - File not found at Destination
20:50:43 DEBUG : [test]: new meta server: https://cld-extapi4.datacloudmail.ru/meta/
20:50:44 DEBUG : [test]: new upload shard: https://cld-upload4.cloud.mail.ru/upload/
20:51:09 DEBUG : test.bin: MailruHash = 19a154a2cba475f7c5b4ecfefed9a8e6c80a06f2 OK
20:51:09 INFO  : test.bin: Copied (new)
Transferred:        1.456G / 1.456 GBytes, 100%, 58.223 MBytes/s, ETA 0s
```

@ncw Please review

Patch:
diff --git a/.github/workflows/build.yml b/.github/workflows/build.yml
index 000cd67d3..507dc9a3d 100644
--- a/.github/workflows/build.yml
+++ b/.github/workflows/build.yml
@@ -107,10 +107,10 @@ jobs:
       - name: Set environment variables
         shell: bash
         run: |
-          echo '::set-env name=GOTAGS::${{ matrix.gotags }}'
-          echo '::set-env name=BUILD_FLAGS::${{ matrix.build_flags }}'
-          if [[ "${{ matrix.goarch }}" != "" ]]; then echo '::set-env name=GOARCH::${{ matrix.goarch }}' ; fi
-          if [[ "${{ matrix.cgo }}" != "" ]]; then echo '::set-env name=CGO_ENABLED::${{ matrix.cgo }}' ; fi
+          echo 'GOTAGS=${{ matrix.gotags }}' >> $GITHUB_ENV
+          echo 'BUILD_FLAGS=${{ matrix.build_flags }}' >> $GITHUB_ENV
+          if [[ "${{ matrix.goarch }}" != "" ]]; then echo 'GOARCH=${{ matrix.goarch }}' >> $GITHUB_ENV ; fi
+          if [[ "${{ matrix.cgo }}" != "" ]]; then echo 'CGO_ENABLED=${{ matrix.cgo }}' >> $GITHUB_ENV ; fi
 
       - name: Install Libraries on Linux
         shell: bash
@@ -124,6 +124,8 @@ jobs:
       - name: Install Libraries on macOS
         shell: bash
         run: |
+          brew untap local/homebrew-openssl  # workaround for https://github.com/actions/virtual-environments/issues/1811
+          brew untap local/homebrew-python2  # workaround for https://github.com/actions/virtual-environments/issues/1811
           brew update
           brew cask install osxfuse
         if: matrix.os == 'macOS-latest'
@@ -133,10 +135,10 @@ jobs:
         run: |
           $ProgressPreference = 'SilentlyContinue'
           choco install -y winfsp zip
-          Write-Host "::set-env name=CPATH::C:\Program Files\WinFsp\inc\fuse;C:\Program Files (x86)\WinFsp\inc\fuse"
+          echo "CPATH=C:\Program Files\WinFsp\inc\fuse;C:\Program Files (x86)\WinFsp\inc\fuse" | Out-File -FilePath $env:GITHUB_ENV -Encoding utf8 -Append
           if ($env:GOARCH -eq "386") {
             choco install -y mingw --forcex86 --force
-            Write-Host "::add-path::C:\\ProgramData\\chocolatey\\lib\\mingw\\tools\\install\\mingw32\\bin"
+            echo "C:\\ProgramData\\chocolatey\\lib\\mingw\\tools\\install\\mingw32\\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append
           }
           # Copy mingw32-make.exe to make.exe so the same command line
           # can be used on Windows as on macOS and Linux
@@ -223,8 +225,8 @@ jobs:
       - name: Set environment variables
         shell: bash
         run: |
-          echo '::set-env name=GOPATH::${{ runner.workspace }}'
-          echo '::add-path::${{ runner.workspace }}/bin'
+          echo 'GOPATH=${{ runner.workspace }}' >> $GITHUB_ENV
+          echo '${{ runner.workspace }}/bin' >> $GITHUB_PATH
 
       - name: Cross-compile rclone
         run: |
diff --git a/backend/jottacloud/jottacloud.go b/backend/jottacloud/jottacloud.go
index 29c307183..9d677c8f8 100644
--- a/backend/jottacloud/jottacloud.go
+++ b/backend/jottacloud/jottacloud.go
@@ -553,7 +553,7 @@ func (f *Fs) setEndpointURL() {
 	if f.opt.Mountpoint == "" {
 		f.opt.Mountpoint = defaultMountpoint
 	}
-	f.endpointURL = urlPathEscape(path.Join(f.user, f.opt.Device, f.opt.Mountpoint))
+	f.endpointURL = path.Join(f.user, f.opt.Device, f.opt.Mountpoint)
 }
 
 // readMetaDataForPath reads the metadata from the path
diff --git a/backend/mailru/mailru.go b/backend/mailru/mailru.go
index 9c6d6433a..1a571feab 100644
--- a/backend/mailru/mailru.go
+++ b/backend/mailru/mailru.go
@@ -102,6 +102,7 @@ func init() {
 This feature is called "speedup" or "put by hash". It is especially efficient
 in case of generally available files like popular books, video or audio clips,
 because files are searched by hash in all accounts of all mailru users.
+It is meaningless and ineffective if source file is unique or encrypted.
 Please note that rclone may need local memory and disk space to calculate
 content hash in advance and decide whether full upload is required.
 Also, if rclone does not know file size in advance (e.g. in case of
@@ -192,7 +193,7 @@ This option must not be used by an ordinary user. It is intended only to
 facilitate remote troubleshooting of backend issues. Strict meaning of
 flags is not documented and not guaranteed to persist between releases.
 Quirks will be removed when the backend grows stable.
-Supported quirks: atomicmkdir binlist`,
+Supported quirks: atomicmkdir binlist unknowndirs`,
 		}, {
 			Name:     config.ConfigEncoding,
 			Help:     config.ConfigEncodingHelp,
@@ -381,6 +382,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 type quirks struct {
 	binlist     bool
 	atomicmkdir bool
+	unknowndirs bool
 }
 
 func (q *quirks) parseQuirks(option string) {
@@ -401,6 +403,9 @@ func (q *quirks) parseQuirks(option string) {
 			// use mkdir as a locking primitive and depend on its atomicity.
 			// Remove this quirk when the above issue is investigated.
 			q.atomicmkdir = true
+		case "unknowndirs":
+			// Accepts unknown resource types as folders.
+			q.unknowndirs = true
 		default:
 			// Ignore unknown flags
 		}
@@ -518,7 +523,7 @@ func (f *Fs) relPath(absPath string) (string, error) {
 	return "", fmt.Errorf("path %q should be under %q", absPath, f.root)
 }
 
-// metaServer ...
+// metaServer returns URL of current meta server
 func (f *Fs) metaServer(ctx context.Context) (string, error) {
 	f.metaMu.Lock()
 	defer f.metaMu.Unlock()
@@ -623,33 +628,56 @@ func (f *Fs) itemToDirEntry(ctx context.Context, item *api.ListItem) (entry fs.D
 	if err != nil {
 		return nil, -1, err
 	}
+
 	mTime := int64(item.Mtime)
 	if mTime < 0 {
 		fs.Debugf(f, "Fixing invalid timestamp %d on mailru file %q", mTime, remote)
 		mTime = 0
 	}
-	switch item.Kind {
-	case "folder":
-		dir := fs.NewDir(remote, time.Unix(mTime, 0)).SetSize(item.Size)
-		dirSize := item.Count.Files + item.Count.Folders
-		return dir, dirSize, nil
+	modTime := time.Unix(mTime, 0)
+
+	isDir, err := f.isDir(item.Kind, remote)
+	if err != nil {
+		return nil, -1, err
+	}
+	if isDir {
+		dir := fs.NewDir(remote, modTime).SetSize(item.Size)
+		return dir, item.Count.Files + item.Count.Folders, nil
+	}
+
+	binHash, err := mrhash.DecodeString(item.Hash)
+	if err != nil {
+		return nil, -1, err
+	}
+	file := &Object{
+		fs:          f,
+		remote:      remote,
+		hasMetaData: true,
+		size:        item.Size,
+		mrHash:      binHash,
+		modTime:     modTime,
+	}
+	return file, -1, nil
+}
+
+// isDir returns true for directories, false for files
+func (f *Fs) isDir(kind, path string) (bool, error) {
+	switch kind {
+	case "":
+		return false, errors.New("empty resource type")
 	case "file":
-		binHash, err := mrhash.DecodeString(item.Hash)
-		if err != nil {
-			return nil, -1, err
-		}
-		file := &Object{
-			fs:          f,
-			remote:      remote,
-			hasMetaData: true,
-			size:        item.Size,
-			mrHash:      binHash,
-			modTime:     time.Unix(mTime, 0),
-		}
-		return file, -1, nil
+		return false, nil
+	case "folder":
+		// fall thru
+	case "camera-upload", "mounted", "shared":
+		fs.Debugf(f, "[%s]: folder has type %q", path, kind)
 	default:
-		return nil, -1, fmt.Errorf("Unknown resource type %q", item.Kind)
+		if !f.quirks.unknowndirs {
+			return false, fmt.Errorf("unknown resource type %q", kind)
+		}
+		fs.Errorf(f, "[%s]: folder has unknown type %q", path, kind)
 	}
+	return true, nil
 }
 
 // List the objects and directories in dir into entries.
@@ -717,7 +745,11 @@ func (f *Fs) listM1(ctx context.Context, dirPath string, offset int, limit int)
 		return nil, err
 	}
 
-	if info.Body.Kind != "folder" {
+	isDir, err := f.isDir(info.Body.Kind, dirPath)
+	if err != nil {
+		return nil, err
+	}
+	if !isDir {
 		return nil, fs.ErrorIsFile
 	}
 
@@ -1570,23 +1602,28 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
 	}
 
 	var (
-		fileBuf    []byte
-		fileHash   []byte
-		newHash    []byte
-		trySpeedup bool
+		fileBuf  []byte
+		fileHash []byte
+		newHash  []byte
+		slowHash bool
+		localSrc bool
 	)
+	if srcObj := fs.UnWrapObjectInfo(src); srcObj != nil {
+		srcFeatures := srcObj.Fs().Features()
+		slowHash = srcFeatures.SlowHash
+		localSrc = srcFeatures.IsLocal
+	}
 
-	// Don't disturb the source if file fits in hash.
-	// Skip an extra speedup request if file fits in hash.
-	if size > mrhash.Size {
-		// Request hash from source.
+	// Try speedup if it's globally enabled but skip extra post
+	// request if file is small and fits in the metadata request
+	trySpeedup := o.fs.opt.SpeedupEnable && size > mrhash.Size
+
+	// Try to get the hash if it's instant
+	if trySpeedup && !slowHash {
 		if srcHash, err := src.Hash(ctx, MrHashType); err == nil && srcHash != "" {
 			fileHash, _ = mrhash.DecodeString(srcHash)
 		}
-
-		// Try speedup if it's globally enabled and source hash is available.
-		trySpeedup = o.fs.opt.SpeedupEnable
-		if trySpeedup && fileHash != nil {
+		if fileHash != nil {
 			if o.putByHash(ctx, fileHash, src, "source") {
 				return nil
 			}
@@ -1595,13 +1632,22 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
 	}
 
 	// Need to calculate hash, check whether file is still eligible for speedup
-	if trySpeedup {
-		trySpeedup = o.fs.eligibleForSpeedup(o.Remote(), size, options...)
+	trySpeedup = trySpeedup && o.fs.eligibleForSpeedup(o.Remote(), size, options...)
+
+	// Attempt to put by hash if file is local and eligible
+	if trySpeedup && localSrc {
+		if srcHash, err := src.Hash(ctx, MrHashType); err == nil && srcHash != "" {
+			fileHash, _ = mrhash.DecodeString(srcHash)
+		}
+		if fileHash != nil && o.putByHash(ctx, fileHash, src, "localfs") {
+			return nil
+		}
+		// If local file hashing has failed, it's pointless to try anymore
+		trySpeedup = false
 	}
 
 	// Attempt to put by calculating hash in memory
 	if trySpeedup && size <= int64(o.fs.opt.SpeedupMaxMem) {
-		//fs.Debugf(o, "attempt to put by hash from memory")
 		fileBuf, err = ioutil.ReadAll(in)
 		if err != nil {
 			return err
@@ -1731,6 +1777,7 @@ func (f *Fs) parseSpeedupPatterns(patternString string) (err error) {
 	return nil
 }
 
+// putByHash is a thin wrapper around addFileMetaData
 func (o *Object) putByHash(ctx context.Context, mrHash []byte, info fs.ObjectInfo, method string) bool {
 	oNew := new(Object)
 	*oNew = *o
@@ -2157,6 +2204,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
 			// Discard the beginning of the data
 			_, err = io.CopyN(ioutil.Discard, wrapStream, start)
 			if err != nil {
+				closeBody(res)
 				return nil, err
 			}
 		}
diff --git a/backend/union/union.go b/backend/union/union.go
index fdb447f57..390e60fe8 100644
--- a/backend/union/union.go
+++ b/backend/union/union.go
@@ -145,11 +145,16 @@ func (f *Fs) Hashes() hash.Set {
 // Mkdir makes the root directory of the Fs object
 func (f *Fs) Mkdir(ctx context.Context, dir string) error {
 	upstreams, err := f.create(ctx, dir)
-	if err == fs.ErrorObjectNotFound && dir != parentDir(dir) {
-		if err := f.Mkdir(ctx, parentDir(dir)); err != nil {
-			return err
+	if err == fs.ErrorObjectNotFound {
+		if dir != parentDir(dir) {
+			if err := f.Mkdir(ctx, parentDir(dir)); err != nil {
+				return err
+			}
+			upstreams, err = f.create(ctx, dir)
+		} else if dir == "" {
+			// If root dirs not created then create them
+			upstreams, err = f.upstreams, nil
 		}
-		upstreams, err = f.create(ctx, dir)
 	}
 	if err != nil {
 		return err
@@ -818,6 +823,7 @@ func NewFs(name, root string, m configmap.Mapper) (fs.Fs, error) {
 	if err != nil {
 		return nil, err
 	}
+	fs.Debugf(f, "actionPolicy = %T, createPolicy = %T, searchPolicy = %T", f.actionPolicy, f.createPolicy, f.searchPolicy)
 	var features = (&fs.Features{
 		CaseInsensitive:         true,
 		DuplicateFiles:          false,
diff --git a/cmd/genautocomplete/genautocomplete_bash.go b/cmd/genautocomplete/genautocomplete_bash.go
index 63660e283..f371d690c 100644
--- a/cmd/genautocomplete/genautocomplete_bash.go
+++ b/cmd/genautocomplete/genautocomplete_bash.go
@@ -2,6 +2,7 @@ package genautocomplete
 
 import (
 	"log"
+	"os"
 
 	"github.com/rclone/rclone/cmd"
 	"github.com/spf13/cobra"
@@ -29,11 +30,20 @@ them directly
 
 If you supply a command line argument the script will be written
 there.
+
+If output_file is "-", then the output will be written to stdout.
 `,
 	Run: func(command *cobra.Command, args []string) {
 		cmd.CheckArgs(0, 1, command, args)
 		out := "/etc/bash_completion.d/rclone"
 		if len(args) > 0 {
+			if args[0] == "-" {
+				err := cmd.Root.GenBashCompletion(os.Stdout)
+				if err != nil {
+					log.Fatal(err)
+				}
+				return
+			}
 			out = args[0]
 		}
 		err := cmd.Root.GenBashCompletionFile(out)
diff --git a/cmd/genautocomplete/genautocomplete_fish.go b/cmd/genautocomplete/genautocomplete_fish.go
index edb327b8b..a60230c40 100644
--- a/cmd/genautocomplete/genautocomplete_fish.go
+++ b/cmd/genautocomplete/genautocomplete_fish.go
@@ -2,6 +2,7 @@ package genautocomplete
 
 import (
 	"log"
+	"os"
 
 	"github.com/rclone/rclone/cmd"
 	"github.com/spf13/cobra"
@@ -29,11 +30,20 @@ them directly
 
 If you supply a command line argument the script will be written
 there.
+
+If output_file is "-", then the output will be written to stdout.
 `,
 	Run: func(command *cobra.Command, args []string) {
 		cmd.CheckArgs(0, 1, command, args)
 		out := "/etc/fish/completions/rclone.fish"
 		if len(args) > 0 {
+			if args[0] == "-" {
+				err := cmd.Root.GenFishCompletion(os.Stdout, true)
+				if err != nil {
+					log.Fatal(err)
+				}
+				return
+			}
 			out = args[0]
 		}
 		err := cmd.Root.GenFishCompletionFile(out, true)
diff --git a/cmd/genautocomplete/genautocomplete_zsh.go b/cmd/genautocomplete/genautocomplete_zsh.go
index 0d635506d..6a2c1c4a0 100644
--- a/cmd/genautocomplete/genautocomplete_zsh.go
+++ b/cmd/genautocomplete/genautocomplete_zsh.go
@@ -30,11 +30,20 @@ them directly
 
 If you supply a command line argument the script will be written
 there.
+
+If output_file is "-", then the output will be written to stdout.
 `,
 	Run: func(command *cobra.Command, args []string) {
 		cmd.CheckArgs(0, 1, command, args)
 		out := "/usr/share/zsh/vendor-completions/_rclone"
 		if len(args) > 0 {
+			if args[0] == "-" {
+				err := cmd.Root.GenZshCompletion(os.Stdout)
+				if err != nil {
+					log.Fatal(err)
+				}
+				return
+			}
 			out = args[0]
 		}
 		outFile, err := os.Create(out)
diff --git a/cmd/mountlib/mount.go b/cmd/mountlib/mount.go
index 2a8324b1c..88ac403ae 100644
--- a/cmd/mountlib/mount.go
+++ b/cmd/mountlib/mount.go
@@ -83,8 +83,8 @@ func AddFlags(flagSet *pflag.FlagSet) {
 	rc.AddOption("mount", &Opt)
 	flags.BoolVarP(flagSet, &Opt.DebugFUSE, "debug-fuse", "", Opt.DebugFUSE, "Debug the FUSE internals - needs -v.")
 	flags.BoolVarP(flagSet, &Opt.AllowNonEmpty, "allow-non-empty", "", Opt.AllowNonEmpty, "Allow mounting over a non-empty directory (not Windows).")
-	flags.BoolVarP(flagSet, &Opt.AllowRoot, "allow-root", "", Opt.AllowRoot, "Allow access to root user.")
-	flags.BoolVarP(flagSet, &Opt.AllowOther, "allow-other", "", Opt.AllowOther, "Allow access to other users.")
+	flags.BoolVarP(flagSet, &Opt.AllowRoot, "allow-root", "", Opt.AllowRoot, "Allow access to root user (not Windows).")
+	flags.BoolVarP(flagSet, &Opt.AllowOther, "allow-other", "", Opt.AllowOther, "Allow access to other users (not Windows).")
 	flags.BoolVarP(flagSet, &Opt.DefaultPermissions, "default-permissions", "", Opt.DefaultPermissions, "Makes kernel enforce access control based on the file mode.")
 	flags.BoolVarP(flagSet, &Opt.WritebackCache, "write-back-cache", "", Opt.WritebackCache, "Makes kernel buffer writes before sending them to rclone. Without this, writethrough caching is used.")
 	flags.FVarP(flagSet, &Opt.MaxReadAhead, "max-read-ahead", "", "The number of bytes that can be prefetched for sequential reads.")
@@ -101,7 +101,7 @@ func AddFlags(flagSet *pflag.FlagSet) {
 	}
 }
 
-// Check is folder is empty
+// Check if folder is empty
 func checkMountEmpty(mountpoint string) error {
 	fp, fpErr := os.Open(mountpoint)
 
@@ -359,15 +359,25 @@ When --vfs-read-chunk-size-limit 500M is specified, the result would be
 				defer cmd.StartStats()()
 			}
 
-			// Skip checkMountEmpty if --allow-non-empty flag is used or if
-			// the Operating System is Windows
-			if !opt.AllowNonEmpty && runtime.GOOS != "windows" {
+
+			// Inform about ignored flags on Windows,
+			// and if not on Windows and not --allow-non-empty flag is used
+			// verify that mountpoint is empty.
+			if runtime.GOOS == "windows" {
+				if opt.AllowNonEmpty {
+					fs.Logf(nil, "--allow-non-empty flag does nothing on Windows")
+				}
+				if opt.AllowRoot {
+					fs.Logf(nil, "--allow-root flag does nothing on Windows")
+				}
+				if opt.AllowOther {
+					fs.Logf(nil, "--allow-other flag does nothing on Windows")
+				}
+			} else if !opt.AllowNonEmpty {
 				err := checkMountEmpty(mountpoint)
 				if err != nil {
 					log.Fatalf("Fatal error: %v", err)
 				}
-			} else if opt.AllowNonEmpty && runtime.GOOS == "windows" {
-				fs.Logf(nil, "--allow-non-empty flag does nothing on Windows")
 			}
 
 			// Work out the volume name, removing special
diff --git a/docs/content/authors.md b/docs/content/authors.md
index b3870a337..fe5db95c1 100644
--- a/docs/content/authors.md
+++ b/docs/content/authors.md
@@ -421,3 +421,4 @@ put them back in again.` >}}
   * Dan Hipschman <dan.hipschman@opendoor.com>
   * Josh Soref <jsoref@users.noreply.github.com>
   * David <david@staron.nl>
+  * Ingo <ingo@hoffmann.cx>
diff --git a/docs/content/changelog.md b/docs/content/changelog.md
index b588bb06a..3b8611fd9 100644
--- a/docs/content/changelog.md
+++ b/docs/content/changelog.md
@@ -5,6 +5,56 @@ description: "Rclone Changelog"
 
 # Changelog
 
+## v1.53.2 - 2020-10-26
+
+[See commits](https://github.com/rclone/rclone/compare/v1.53.1...v1.53.2)
+
+* Bug Fixes
+    * acounting
+        * Fix incorrect speed and transferTime in core/stats (Nick Craig-Wood)
+        * Stabilize display order of transfers on Windows (Nick Craig-Wood)
+    * operations
+        * Fix use of --suffix without --backup-dir (Nick Craig-Wood)
+        * Fix spurious "--checksum is in use but the source and destination have no hashes in common" (Nick Craig-Wood)
+    * build
+        * Work around GitHub actions brew problem (Nick Craig-Wood)
+        * Stop using set-env and set-path in the GitHub actions (Nick Craig-Wood)
+* Mount
+    * mount2: Fix the swapped UID / GID values (Russell Cattelan)
+* VFS
+    * Detect and recover from a file being removed externally from the cache (Nick Craig-Wood)
+    * Fix a deadlock vulnerability in downloaders.Close (Leo Luan)
+    * Fix a race condition in retryFailedResets (Leo Luan)
+    * Fix missed concurrency control between some item operations and reset (Leo Luan)
+    * Add exponential backoff during ENOSPC retries (Leo Luan)
+    * Add a missed update of used cache space (Leo Luan)
+    * Fix --no-modtime to not attempt to set modtimes (as documented) (Nick Craig-Wood)
+* Local
+    * Fix sizes and syncing with --links option on Windows (Nick Craig-Wood)
+* Chunker
+    * Disable ListR to fix missing files on GDrive (workaround) (Ivan Andreev)
+    * Fix upload over crypt (Ivan Andreev)
+* Fichier
+    * Increase maximum file size from 100GB to 300GB (gyutw)
+* Jottacloud
+    * Remove clientSecret from config when upgrading to token based authentication (buengese)
+    * Avoid double url escaping of device/mountpoint (albertony)
+    * Remove DirMove workaround as it's not required anymore - also (buengese)
+* Mailru
+    * Fix uploads after recent changes on server (Ivan Andreev)
+    * Fix range requests after june changes on server (Ivan Andreev)
+    * Fix invalid timestamp on corrupted files (fixes) (Ivan Andreev)
+* Onedrive
+    * Fix disk usage for sharepoint (Nick Craig-Wood)
+* S3
+    * Add missing regions for AWS (Anagh Kumar Baranwal)
+* Seafile
+    * Fix accessing libraries > 2GB on 32 bit systems (Muffin King)
+* SFTP
+    * Always convert the checksum to lower case (buengese)
+* Union
+    * Create root directories if none exist (Nick Craig-Wood)
+
 ## v1.53.1 - 2020-09-13
 
 [See commits](https://github.com/rclone/rclone/compare/v1.53.0...v1.53.1)
diff --git a/docs/content/commands/rclone_genautocomplete_bash.md b/docs/content/commands/rclone_genautocomplete_bash.md
index 72dc7cc21..27928c680 100644
--- a/docs/content/commands/rclone_genautocomplete_bash.md
+++ b/docs/content/commands/rclone_genautocomplete_bash.md
@@ -27,6 +27,7 @@ them directly
 If you supply a command line argument the script will be written
 there.
 
+If output_file is `-`, then the output will be written to stdout.
 
 ```
 rclone genautocomplete bash [output_file] [flags]
diff --git a/docs/content/commands/rclone_genautocomplete_fish.md b/docs/content/commands/rclone_genautocomplete_fish.md
index 8d9a122ed..d9ba9706c 100644
--- a/docs/content/commands/rclone_genautocomplete_fish.md
+++ b/docs/content/commands/rclone_genautocomplete_fish.md
@@ -27,6 +27,7 @@ them directly
 If you supply a command line argument the script will be written
 there.
 
+If output_file is `-`, then the output will be written to stdout.
 
 ```
 rclone genautocomplete fish [output_file] [flags]
diff --git a/docs/content/commands/rclone_genautocomplete_zsh.md b/docs/content/commands/rclone_genautocomplete_zsh.md
index 6816f7dde..e31df0726 100644
--- a/docs/content/commands/rclone_genautocomplete_zsh.md
+++ b/docs/content/commands/rclone_genautocomplete_zsh.md
@@ -27,6 +27,7 @@ them directly
 If you supply a command line argument the script will be written
 there.
 
+If output_file is `-`, then the output will be written to stdout.
 
 ```
 rclone genautocomplete zsh [output_file] [flags]
diff --git a/go.sum b/go.sum
index 4b5ec2d63..e5abaa4a5 100644
--- a/go.sum
+++ b/go.sum
@@ -400,6 +400,7 @@ github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
 github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
 github.com/spf13/viper v1.4.0/go.mod h1:PTJ7Z/lr49W6bUbkmS1V3by4uWynFiR9p7+dSq/yZzE=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
+github.com/stretchr/objx v0.1.1 h1:2vfRuCMp5sSVIDSqO8oNnWJq7mPa6KVP3iPIwFBuy8A=
 github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.2.1/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=


Test patch:
diff --git a/cmd/genautocomplete/genautocomplete_test.go b/cmd/genautocomplete/genautocomplete_test.go
index e97ad1169..ce766c6fd 100644
--- a/cmd/genautocomplete/genautocomplete_test.go
+++ b/cmd/genautocomplete/genautocomplete_test.go
@@ -11,8 +11,10 @@ import (
 func TestCompletionBash(t *testing.T) {
 	tempFile, err := ioutil.TempFile("", "completion_bash")
 	assert.NoError(t, err)
-	defer func() { _ = tempFile.Close() }()
-	defer func() { _ = os.Remove(tempFile.Name()) }()
+	defer func() {
+		_ = tempFile.Close()
+		_ = os.Remove(tempFile.Name())
+	}()
 
 	bashCommandDefinition.Run(bashCommandDefinition, []string{tempFile.Name()})
 
@@ -21,11 +23,32 @@ func TestCompletionBash(t *testing.T) {
 	assert.NotEmpty(t, string(bs))
 }
 
+func TestCompletionBashStdout(t *testing.T) {
+	originalStdout := os.Stdout
+	tempFile, err := ioutil.TempFile("", "completion_zsh")
+	assert.NoError(t, err)
+	defer func() {
+		_ = tempFile.Close()
+		_ = os.Remove(tempFile.Name())
+	}()
+
+	os.Stdout = tempFile
+	defer func() { os.Stdout = originalStdout }()
+
+	bashCommandDefinition.Run(bashCommandDefinition, []string{"-"})
+
+	output, err := ioutil.ReadFile(tempFile.Name())
+	assert.NoError(t, err)
+	assert.NotEmpty(t, string(output))
+}
+
 func TestCompletionZsh(t *testing.T) {
 	tempFile, err := ioutil.TempFile("", "completion_zsh")
 	assert.NoError(t, err)
-	defer func() { _ = tempFile.Close() }()
-	defer func() { _ = os.Remove(tempFile.Name()) }()
+	defer func() {
+		_ = tempFile.Close()
+		_ = os.Remove(tempFile.Name())
+	}()
 
 	zshCommandDefinition.Run(zshCommandDefinition, []string{tempFile.Name()})
 
@@ -34,11 +57,31 @@ func TestCompletionZsh(t *testing.T) {
 	assert.NotEmpty(t, string(bs))
 }
 
+func TestCompletionZshStdout(t *testing.T) {
+	originalStdout := os.Stdout
+	tempFile, err := ioutil.TempFile("", "completion_zsh")
+	assert.NoError(t, err)
+	defer func() {
+		_ = tempFile.Close()
+		_ = os.Remove(tempFile.Name())
+	}()
+
+	os.Stdout = tempFile
+	defer func() { os.Stdout = originalStdout }()
+
+	zshCommandDefinition.Run(zshCommandDefinition, []string{"-"})
+	output, err := ioutil.ReadFile(tempFile.Name())
+	assert.NoError(t, err)
+	assert.NotEmpty(t, string(output))
+}
+
 func TestCompletionFish(t *testing.T) {
 	tempFile, err := ioutil.TempFile("", "completion_fish")
 	assert.NoError(t, err)
-	defer func() { _ = tempFile.Close() }()
-	defer func() { _ = os.Remove(tempFile.Name()) }()
+	defer func() {
+		_ = tempFile.Close()
+		_ = os.Remove(tempFile.Name())
+	}()
 
 	fishCommandDefinition.Run(fishCommandDefinition, []string{tempFile.Name()})
 
@@ -46,3 +89,22 @@ func TestCompletionFish(t *testing.T) {
 	assert.NoError(t, err)
 	assert.NotEmpty(t, string(bs))
 }
+
+func TestCompletionFishStdout(t *testing.T) {
+	originalStdout := os.Stdout
+	tempFile, err := ioutil.TempFile("", "completion_zsh")
+	assert.NoError(t, err)
+	defer func() {
+		_ = tempFile.Close()
+		_ = os.Remove(tempFile.Name())
+	}()
+
+	os.Stdout = tempFile
+	defer func() { os.Stdout = originalStdout }()
+
+	fishCommandDefinition.Run(fishCommandDefinition, []string{"-"})
+
+	output, err := ioutil.ReadFile(tempFile.Name())
+	assert.NoError(t, err)
+	assert.NotEmpty(t, string(output))
+}
diff --git a/fstest/test_all/config.go b/fstest/test_all/config.go
index ab271b58d..9d3440d15 100644
--- a/fstest/test_all/config.go
+++ b/fstest/test_all/config.go
@@ -27,15 +27,16 @@ type Test struct {
 //
 // FIXME make bucket based remotes set sub-dir automatically???
 type Backend struct {
-	Backend  string   // name of the backend directory
-	Remote   string   // name of the test remote
-	FastList bool     // set to test with -fast-list
-	Short    bool     // set to test with -short
-	OneOnly  bool     // set to run only one backend test at once
-	MaxFile  string   // file size limit
-	CleanUp  bool     // when running clean, run cleanup first
-	Ignore   []string // test names to ignore the failure of
-	Tests    []string // paths of tests to run, blank for all
+	Backend     string   // name of the backend directory
+	Remote      string   // name of the test remote
+	FastList    bool     // set to test with -fast-list
+	Short       bool     // set to test with -short
+	OneOnly     bool     // set to run only one backend test at once
+	MaxFile     string   // file size limit
+	CleanUp     bool     // when running clean, run cleanup first
+	Ignore      []string // test names to ignore the failure of
+	Tests       []string // paths of tests to run, blank for all
+	ListRetries int      // -list-retries if > 0
 }
 
 // includeTest returns true if this backend should be included in this
@@ -79,16 +80,17 @@ func (b *Backend) MakeRuns(t *Test) (runs []*Run) {
 			continue
 		}
 		run := &Run{
-			Remote:    b.Remote,
-			Backend:   b.Backend,
-			Path:      t.Path,
-			FastList:  fastlist,
-			Short:     (b.Short && t.Short),
-			NoRetries: t.NoRetries,
-			OneOnly:   b.OneOnly,
-			NoBinary:  t.NoBinary,
-			SizeLimit: int64(maxSize),
-			Ignore:    ignore,
+			Remote:      b.Remote,
+			Backend:     b.Backend,
+			Path:        t.Path,
+			FastList:    fastlist,
+			Short:       (b.Short && t.Short),
+			NoRetries:   t.NoRetries,
+			OneOnly:     b.OneOnly,
+			NoBinary:    t.NoBinary,
+			SizeLimit:   int64(maxSize),
+			Ignore:      ignore,
+			ListRetries: b.ListRetries,
 		}
 		if t.AddBackend {
 			run.Path = path.Join(run.Path, b.Backend)
diff --git a/fstest/test_all/config.yaml b/fstest/test_all/config.yaml
index 9581d3b5e..44a74ff46 100644
--- a/fstest/test_all/config.yaml
+++ b/fstest/test_all/config.yaml
@@ -20,6 +20,7 @@ backends:
  - backend:  "b2"
    remote:   "TestB2:"
    fastlist: true
+   listretries: 5
  - backend:  "crypt"
    remote:   "TestCryptDrive:"
    fastlist: true
@@ -146,12 +147,12 @@ backends:
  #   ignore:
  #     - TestIntegration/FsMkdir/FsPutFiles/FsCopy
  #     - TestIntegration/FsMkdir/FsPutFiles/SetTier
- - backend:  "s3"
-   remote:   "TestS3Ceph:"
-   fastlist: true
-   ignore:
-     - TestIntegration/FsMkdir/FsPutFiles/FsCopy
-     - TestIntegration/FsMkdir/FsPutFiles/SetTier
+ # - backend:  "s3"
+ #   remote:   "TestS3Ceph:"
+ #   fastlist: true
+ #   ignore:
+ #     - TestIntegration/FsMkdir/FsPutFiles/FsCopy
+ #     - TestIntegration/FsMkdir/FsPutFiles/SetTier
  - backend:  "s3"
    remote:   "TestS3Alibaba:"
    fastlist: true
@@ -172,11 +173,11 @@ backends:
  - backend:  "swift"
    remote:   "TestSwift:"
    fastlist: true
- - backend:  "swift"
-   remote:   "TestSwiftCeph:"
-   fastlist: true
-   ignore:
-     - TestIntegration/FsMkdir/FsPutFiles/FsCopy
+ # - backend:  "swift"
+ #   remote:   "TestSwiftCeph:"
+ #   fastlist: true
+ #   ignore:
+ #     - TestIntegration/FsMkdir/FsPutFiles/FsCopy
  - backend:  "yandex"
    remote:   "TestYandex:"
    fastlist: false
diff --git a/fstest/test_all/run.go b/fstest/test_all/run.go
index e4f8b2ee8..89cb96070 100644
--- a/fstest/test_all/run.go
+++ b/fstest/test_all/run.go
@@ -35,16 +35,17 @@ var (
 // if retries are needed.
 type Run struct {
 	// Config
-	Remote    string // name of the test remote
-	Backend   string // name of the backend
-	Path      string // path to the source directory
-	FastList  bool   // add -fast-list to tests
-	Short     bool   // add -short
-	NoRetries bool   // don't retry if set
-	OneOnly   bool   // only run test for this backend at once
-	NoBinary  bool   // set to not build a binary
-	SizeLimit int64  // maximum test file size
-	Ignore    map[string]struct{}
+	Remote      string // name of the test remote
+	Backend     string // name of the backend
+	Path        string // path to the source directory
+	FastList    bool   // add -fast-list to tests
+	Short       bool   // add -short
+	NoRetries   bool   // don't retry if set
+	OneOnly     bool   // only run test for this backend at once
+	NoBinary    bool   // set to not build a binary
+	SizeLimit   int64  // maximum test file size
+	Ignore      map[string]struct{}
+	ListRetries int // -list-retries if > 0
 	// Internals
 	CmdLine     []string
 	CmdString   string
@@ -336,8 +337,12 @@ func (r *Run) Init() {
 		r.CmdLine = []string{"./" + r.BinaryName()}
 	}
 	r.CmdLine = append(r.CmdLine, prefix+"v", prefix+"timeout", timeout.String(), "-remote", r.Remote)
-	if *listRetries > 0 {
-		r.CmdLine = append(r.CmdLine, "-list-retries", fmt.Sprint(*listRetries))
+	listRetries := *listRetries
+	if r.ListRetries > 0 {
+		listRetries = r.ListRetries
+	}
+	if listRetries > 0 {
+		r.CmdLine = append(r.CmdLine, "-list-retries", fmt.Sprint(listRetries))
 	}
 	r.Try = 1
 	if *verbose {