Repository: rclone/rclone
Pull request #5485 (2021-07-26 16:28:28)

copyurl: Update copyurl to honor HTTP filename directive
#### What is the purpose of this change?

Adding `--auto-filename-header`

To set preferred download filenames for HTTP requests, RFC 6226
specifies a 'filename' directive, available within 'Content-Disposition'
header, and suggests both token and quoted string syntaxes should be allowable
for the filename directive. i.e.: either `filename=name.txt` or
`filename="name.txt"` should be acceptable.

See below for details:
https://httpwg.org/specs/rfc6266.html#disposition.parameter.filename
https://httpwg.org/specs/rfc6266.html#advice.generating

`--auto-filename-header` is designed to work in conjunction with `--auto-filename`, honoring the filename directive when retrieving files from a server.

Directives contained within the `Content-Disposition` header are semicolon delimited, and can vary in length and order. Consideration had to be given to parsing entries like `inline; filename="file.txt";` as well as the more straightforward examples above.

#### Was the change discussed in an issue or in the forum before?
I raised it on the forum, got no response, and thought it was an clear enough issue to tackle myself without altering existing functionality. Maybe that was good enough. If not, maybe a rogue PR is enough to get a response. 😉 

https://forum.rclone.org/t/copyurl-auto-filename-could-be-cleverer-and-respect-the-http-content-disposition-optional-directive-filename/25553

#### Checklist

- [x] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-pull-request).
- [x] I have added tests for all changes in this PR if appropriate.
- [x] I have added documentation for the changes if appropriate.
- [x] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages).
- [x] I'm done, this Pull Request is ready for review :-)


jptreen: Fixed linux workflow failure and squashed the commits.
jptreen: Fixed some linting issues.
ncw: > `--auto-filename-header` is designed to work in conjunction with `--auto-filename`, honoring the filename directive when retrieving files from a server.

Is a new flag necessary? Maybe it should always be parsing `Content-Disposition` if `--auto-filename` is set? That is what browsers would do, after all.

What do you think?

> I raised it on the forum, got no response, and thought it was an clear enough issue to tackle myself without altering existing functionality. Maybe that was good enough. If not, maybe a rogue PR is enough to get a response. 

:-) It looks like a sensible idea to me.

I'd be interested to hear what actual use case you have for it.

I'll review the code now.
jptreen: Thanks for the review:

> Is a new flag necessary? Maybe it should always be parsing Content-Disposition if --auto-filename is set? That is what browsers would do, after all.

Through an abundance of caution, I didn't want to tread on any existing behavior, even if that behavior was wrong. I'm happy, however, to aim for browser functionality.

> I'd be interested to hear what actual use case you have for it.

I have to work with a list of temporary web available files, which look like they've been secured with a salted hash, and the filename just isn't in the path, the hash is. If I don't want to download the files as something like `0894635ac5a1128086581d149e963c2dd9a4ea90`, but instead `name_i_would_like.jpg`, I need to be able to parse the headers. This job was done by somebody in a browser.

> In Go a capitalised first letter makes a function externally accessible to the package.

Many thanks for the review. Apologies for my appauling Go code, its *not exactly* a language I'm best familiar with. The reason I exported this function is because I wanted to test it on its own, and I was getting complaints when trying to run tests directly calling a non-exported function.

I'll take a look at your notes, remove the new flag, check out MIME parsing code, and rework this PR when I get the chance.
ncw: > Thanks for the review:
> 
> > Is a new flag necessary? Maybe it should always be parsing Content-Disposition if --auto-filename is set? That is what browsers would do, after all.
> 
> Through an abundance of caution, I didn't want to tread on any existing behavior, even if that behavior was wrong. I'm happy, however, to aim to emulate a generic browser functionality.

I checked the docs for `curl` and `wget` and they both require a flag (`--remote-header-name` and `--content-disposition` respectively).

Reading the curl docs

>        -J, --remote-header-name
>               (HTTP) This option tells the -O, --remote-name option to use the
>               server-specified  Content-Disposition  filename  instead  of ex‐
>               tracting a filename from the URL.
>               [snip]
>               WARNING:  Exercise  judicious  use of this option, especially on
>               Windows. A rogue server could send you the  name  of  a  DLL  or
>               other  file  that could possibly be loaded automatically by Win‐
>               dows or some third party software.

Made me think that we should probably excercise a little care with the filename

The [mozilla docs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition) say (emphasis mine)

> filename
> 
> Is followed by a string containing the original name of the file transmitted. The filename is always optional and **must not be used blindly by the application: path information should be stripped**, and conversion to the server file system rules should be done. This parameter provides mostly indicative information. When used in combination with Content-Disposition: attachment, it is used as the default filename for an eventual "Save As" dialog presented to the user.

So I think your flag is a good idea.

I think we should also sanitise the filename...

I'd do this by converting `\` to `/` (with `strings.Replace`) then running path.Base() on the result to discard the path information as suggested above. This string could still contain characters which can't be represented on the destination filesystem, however the backend should then convert it into something which is safe.

> > I'd be interested to hear what actual use case you have for it.
> 
> I have to work with a list of temporary web available files, which look like they've been secured with a salted hash, and the filename just isn't in the path, the hash is the final segment. If I don't want to download the files as something like `0894635ac5a1128086581d149e963c2dd9a4ea90`, but instead `name_i_would_like.jpg`, I need to be able to parse the headers. This job was previously done by a person with a browser, so it just never came up as an issue.

Thanks for the explanation.

> > In Go a capitalised first letter makes a function externally accessible to the package.
> 
> Many thanks for the review. Apologies for my appauling Go code, its _not exactly_ a language I'm best familiar with. 

That is fine - very happy to help newcomers to Go with the syntax!


> The reason I exported this function is because I wanted to test it on its own, and I was getting complaints when trying to run tests directly calling a non-exported function.

What you want to do is stick the test in `fs/operations/operations_internal_test.go` which is the internal tests - the `operations_test.go` are the external (black box) tests which don't have access to the internal symbols.  (The `package` header is different between those two files, one is `operations` and the other is `operations_test` which is why they are different).
 
> I'll take a look at your notes, remove the new flag, check out MIME parsing code, and rework this PR when I get the chance.

Great! Give me a ping when you are read for another review.

ivandeex: @jptreen 
> we should probably exercise a little care with the filename
> I think we should also sanitize the filename...
> I'd do this by converting `\` to `/` then running path.Base() on the result to discard the path information as suggested above
> you want to stick the test in `fs/operations/operations_internal_test.go`
> ping when you are read for another review.

ivandeex: Unfortunately rclone is short on resources to finish all PRs unfinished by authors.
Marking as draft until we have news from submitter.
buengese: So cleaned up and switched to [mime.ParseMediaType](https://pkg.go.dev/mime#ParseMediaType). I've kept a separate flag for using the content-disposition header that has to be set in addition to `--auto-filename` but renamed it `--header-filename`. 

Patch:
diff --git a/backend/fichier/api.go b/backend/fichier/api.go
index 6a1746265..15c916c73 100644
--- a/backend/fichier/api.go
+++ b/backend/fichier/api.go
@@ -28,25 +28,44 @@ var retryErrorCodes = []int{
 	509, // Bandwidth Limit Exceeded
 }
 
+var errorRegex = regexp.MustCompile(`#\d{1,3}`)
+
+func parseFichierError(err error) int {
+	matches := errorRegex.FindStringSubmatch(err.Error())
+	if len(matches) == 0 {
+		return 0
+	}
+	code, err := strconv.Atoi(matches[0])
+	if err != nil {
+		fs.Debugf(nil, "failed parsing fichier error: %v", err)
+		return 0
+	}
+	return code
+}
+
 // shouldRetry returns a boolean as to whether this resp and err
 // deserve to be retried.  It returns the err as a convenience
 func shouldRetry(ctx context.Context, resp *http.Response, err error) (bool, error) {
 	if fserrors.ContextError(ctx, &err) {
 		return false, err
 	}
-	// Detect this error which the integration tests provoke
-	// error HTTP error 403 (403 Forbidden) returned body: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}"
-	//
-	// https://1fichier.com/api.html
-	//
-	// file/ls.cgi is limited :
+	// 1Fichier uses HTTP error code 403 (Forbidden) for all kinds of errors with
+	// responses looking like this: "{\"message\":\"Flood detected: IP Locked #374\",\"status\":\"KO\"}"
 	//
-	// Warning (can be changed in case of abuses) :
-	// List all files of the account is limited to 1 request per hour.
-	// List folders is limited to 5 000 results and 1 request per folder per 30s.
-	if err != nil && strings.Contains(err.Error(), "Flood detected") {
-		fs.Debugf(nil, "Sleeping for 30 seconds due to: %v", err)
-		time.Sleep(30 * time.Second)
+	// We attempt to parse the actual 1Fichier error code from this body and handle it accordingly
+	// Most importantly #374 (Flood detected: IP locked) which the integration tests provoke
+	// The list below is far from complete and should be expanded if we see any more error codes.
+	if err != nil {
+		switch parseFichierError(err) {
+		case 93:
+			return false, err // No such user
+		case 186:
+			return false, err // IP blocked?
+		case 374:
+			fs.Debugf(nil, "Sleeping for 30 seconds due to: %v", err)
+			time.Sleep(30 * time.Second)
+		default:
+		}
 	}
 	return fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
 }
diff --git a/cmd/copyurl/copyurl.go b/cmd/copyurl/copyurl.go
index e7e31f188..fc3cf38cc 100644
--- a/cmd/copyurl/copyurl.go
+++ b/cmd/copyurl/copyurl.go
@@ -14,16 +14,18 @@ import (
 )
 
 var (
-	autoFilename  = false
-	printFilename = false
-	stdout        = false
-	noClobber     = false
+	autoFilename   = false
+	headerFilename = false
+	printFilename  = false
+	stdout         = false
+	noClobber      = false
 )
 
 func init() {
 	cmd.Root.AddCommand(commandDefinition)
 	cmdFlags := commandDefinition.Flags()
 	flags.BoolVarP(cmdFlags, &autoFilename, "auto-filename", "a", autoFilename, "Get the file name from the URL and use it for destination file path")
+	flags.BoolVarP(cmdFlags, &headerFilename, "header-filename", "", headerFilename, "Get the file name from the Content-Disposition header")
 	flags.BoolVarP(cmdFlags, &printFilename, "print-filename", "p", printFilename, "Print the resulting name from --auto-filename")
 	flags.BoolVarP(cmdFlags, &noClobber, "no-clobber", "", noClobber, "Prevent overwriting file with same name")
 	flags.BoolVarP(cmdFlags, &stdout, "stdout", "", stdout, "Write the output to stdout rather than a file")
@@ -36,10 +38,11 @@ var commandDefinition = &cobra.Command{
 Download a URL's content and copy it to the destination without saving
 it in temporary storage.
 
-Setting ` + "`--auto-filename`" + ` will cause the file name to be retrieved from
-the URL (after any redirections) and used in the destination
-path. With ` + "`--print-filename`" + ` in addition, the resulting file name will
-be printed.
+Setting ` + "`--auto-filename`" + ` will attempt to automatically determine the filename from the URL
+(after any redirections) and used in the destination path. 
+With ` + "`--auto-filename-header`" + ` in 
+addition, if a specific filename is set in HTTP headers, it will be used instead of the name from the URL.
+With ` + "`--print-filename`" + ` in addition, the resulting file name will be printed.
 
 Setting ` + "`--no-clobber`" + ` will prevent overwriting file on the 
 destination if there is one with the same name.
@@ -69,7 +72,7 @@ will cause the output to be written to standard output.
 			if stdout {
 				err = operations.CopyURLToWriter(context.Background(), args[0], os.Stdout)
 			} else {
-				dst, err = operations.CopyURL(context.Background(), fsdst, dstFileName, args[0], autoFilename, noClobber)
+				dst, err = operations.CopyURL(context.Background(), fsdst, dstFileName, args[0], autoFilename, headerFilename, noClobber)
 				if printFilename && err == nil && dst != nil {
 					fmt.Println(dst.Remote())
 				}
diff --git a/cmd/ncdu/ncdu.go b/cmd/ncdu/ncdu.go
index 963ddb628..aea2501eb 100644
--- a/cmd/ncdu/ncdu.go
+++ b/cmd/ncdu/ncdu.go
@@ -1,7 +1,7 @@
 // Package ncdu implements a text based user interface for exploring a remote
 
-//go:build !plan9 && !solaris && !js
-// +build !plan9,!solaris,!js
+//go:build !plan9 && !js
+// +build !plan9,!js
 
 package ncdu
 
@@ -14,8 +14,8 @@ import (
 	"strings"
 
 	"github.com/atotto/clipboard"
+	"github.com/gdamore/tcell/v2/termbox"
 	runewidth "github.com/mattn/go-runewidth"
-	termbox "github.com/nsf/termbox-go"
 	"github.com/rclone/rclone/cmd"
 	"github.com/rclone/rclone/cmd/ncdu/scan"
 	"github.com/rclone/rclone/fs"
@@ -333,10 +333,7 @@ func (u *UI) Draw() error {
 	u.dirListHeight = h - 3
 
 	// Plot
-	err := termbox.Clear(termbox.ColorDefault, termbox.ColorDefault)
-	if err != nil {
-		return fmt.Errorf("failed to clear screen: %w", err)
-	}
+	termbox.Clear(termbox.ColorWhite, termbox.ColorBlack)
 
 	// Header line
 	Linef(0, 0, w, termbox.ColorBlack, termbox.ColorWhite, ' ', "rclone ncdu %s - use the arrow keys to navigate, press ? for help", fs.Version)
@@ -374,8 +371,9 @@ func (u *UI) Draw() error {
 			if err != nil {
 				fg = termbox.ColorRed
 			}
+			const colorLightYellow = termbox.ColorYellow + 8
 			if isSelected {
-				fg = termbox.ColorLightYellow
+				fg = colorLightYellow
 			}
 			bg := termbox.ColorBlack
 			if n == dirPos.entry {
@@ -456,7 +454,7 @@ func (u *UI) Draw() error {
 	if u.showBox {
 		u.Box()
 	}
-	err = termbox.Flush()
+	err := termbox.Flush()
 	if err != nil {
 		return fmt.Errorf("failed to flush screen: %w", err)
 	}
diff --git a/cmd/ncdu/ncdu_unsupported.go b/cmd/ncdu/ncdu_unsupported.go
index 6f66cfb6c..b4a9b22cf 100644
--- a/cmd/ncdu/ncdu_unsupported.go
+++ b/cmd/ncdu/ncdu_unsupported.go
@@ -1,7 +1,7 @@
 // Build for ncdu for unsupported platforms to stop go complaining
 // about "no buildable Go source files "
 
-//go:build plan9 || solaris || js
-// +build plan9 solaris js
+//go:build plan9 || js
+// +build plan9 js
 
 package ncdu
diff --git a/docs/content/commands/rclone_sync.md b/docs/content/commands/rclone_sync.md
index 57e24dd2d..5596240e1 100644
--- a/docs/content/commands/rclone_sync.md
+++ b/docs/content/commands/rclone_sync.md
@@ -28,7 +28,7 @@ errors at any point.  Duplicate objects (files with the same name, on
 those providers that support it) are also not yet handled.
 
 It is always the contents of the directory that is synced, not the
-directory so when source:path is a directory, it's the contents of
+directory itself. So when source:path is a directory, it's the contents of
 source:path that are copied, not the directory name and contents.  See
 extended explanation in the `copy` command above if unsure.
 
diff --git a/fs/operations/operations.go b/fs/operations/operations.go
index c21675443..5eccc44f7 100644
--- a/fs/operations/operations.go
+++ b/fs/operations/operations.go
@@ -11,6 +11,7 @@ import (
 	"fmt"
 	"io"
 	"io/ioutil"
+	"mime"
 	"net/http"
 	"os"
 	"path"
@@ -1757,7 +1758,7 @@ func RcatSize(ctx context.Context, fdst fs.Fs, dstFileName string, in io.ReadClo
 type copyURLFunc func(ctx context.Context, dstFileName string, in io.ReadCloser, size int64, modTime time.Time) (err error)
 
 // copyURLFn copies the data from the url to the function supplied
-func copyURLFn(ctx context.Context, dstFileName string, url string, dstFileNameFromURL bool, fn copyURLFunc) (err error) {
+func copyURLFn(ctx context.Context, dstFileName string, url string, autoFilename, dstFileNameFromHeader bool, fn copyURLFunc) (err error) {
 	client := fshttp.NewClient(ctx)
 	resp, err := client.Get(url)
 	if err != nil {
@@ -1771,7 +1772,17 @@ func copyURLFn(ctx context.Context, dstFileName string, url string, dstFileNameF
 	if err != nil {
 		modTime = time.Now()
 	}
-	if dstFileNameFromURL {
+	if autoFilename {
+		if dstFileNameFromHeader {
+			_, params, err := mime.ParseMediaType(resp.Header.Get("Content-Disposition"))
+			headerFilename := path.Base(strings.Replace(params["filename"], "\\", "/", -1))
+			if err != nil || headerFilename == "" {
+				return fmt.Errorf("copyurl failed: filename not found in the Content-Dispoition header")
+			}
+			fs.Debugf(headerFilename, "filename found in Content-Disposition header.")
+			return fn(ctx, headerFilename, resp.Body, resp.ContentLength, modTime)
+		}
+
 		dstFileName = path.Base(resp.Request.URL.Path)
 		if dstFileName == "." || dstFileName == "/" {
 			return fmt.Errorf("CopyURL failed: file name wasn't found in url")
@@ -1782,9 +1793,9 @@ func copyURLFn(ctx context.Context, dstFileName string, url string, dstFileNameF
 }
 
 // CopyURL copies the data from the url to (fdst, dstFileName)
-func CopyURL(ctx context.Context, fdst fs.Fs, dstFileName string, url string, dstFileNameFromURL bool, noClobber bool) (dst fs.Object, err error) {
+func CopyURL(ctx context.Context, fdst fs.Fs, dstFileName string, url string, autoFilename, dstFileNameFromHeader bool, noClobber bool) (dst fs.Object, err error) {
 
-	err = copyURLFn(ctx, dstFileName, url, dstFileNameFromURL, func(ctx context.Context, dstFileName string, in io.ReadCloser, size int64, modTime time.Time) (err error) {
+	err = copyURLFn(ctx, dstFileName, url, autoFilename, dstFileNameFromHeader, func(ctx context.Context, dstFileName string, in io.ReadCloser, size int64, modTime time.Time) (err error) {
 		if noClobber {
 			_, err = fdst.NewObject(ctx, dstFileName)
 			if err == nil {
@@ -1799,7 +1810,7 @@ func CopyURL(ctx context.Context, fdst fs.Fs, dstFileName string, url string, ds
 
 // CopyURLToWriter copies the data from the url to the io.Writer supplied
 func CopyURLToWriter(ctx context.Context, url string, out io.Writer) (err error) {
-	return copyURLFn(ctx, "", url, false, func(ctx context.Context, dstFileName string, in io.ReadCloser, size int64, modTime time.Time) (err error) {
+	return copyURLFn(ctx, "", url, false, false, func(ctx context.Context, dstFileName string, in io.ReadCloser, size int64, modTime time.Time) (err error) {
 		_, err = io.Copy(out, in)
 		return err
 	})
diff --git a/fs/operations/rc.go b/fs/operations/rc.go
index de77dae09..848b56ca1 100644
--- a/fs/operations/rc.go
+++ b/fs/operations/rc.go
@@ -274,8 +274,9 @@ func rcSingleCommand(ctx context.Context, in rc.Params, name string, noRemote bo
 		}
 		autoFilename, _ := in.GetBool("autoFilename")
 		noClobber, _ := in.GetBool("noClobber")
+		headerFilename, _ := in.GetBool("headerFilename")
 
-		_, err = CopyURL(ctx, f, remote, url, autoFilename, noClobber)
+		_, err = CopyURL(ctx, f, remote, url, autoFilename, headerFilename, noClobber)
 		return nil, err
 	case "uploadfile":
 
diff --git a/go.mod b/go.mod
index 75a242ff7..1fe102fd6 100644
--- a/go.mod
+++ b/go.mod
@@ -24,6 +24,7 @@ require (
 	github.com/dop251/scsu v0.0.0-20220106150536-84ac88021d00
 	github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.4
 	github.com/gabriel-vasile/mimetype v1.4.0
+	github.com/gdamore/tcell/v2 v2.5.1
 	github.com/go-chi/chi/v5 v5.0.7
 	github.com/google/uuid v1.3.0
 	github.com/hanwen/go-fuse/v2 v2.1.0
@@ -38,7 +39,6 @@ require (
 	github.com/mitchellh/go-homedir v1.1.0
 	github.com/ncw/go-acd v0.0.0-20201019170801-fe55f33415b1
 	github.com/ncw/swift/v2 v2.0.1
-	github.com/nsf/termbox-go v1.1.1
 	github.com/patrickmn/go-cache v2.1.0+incompatible
 	github.com/pkg/sftp v1.13.5-0.20211228200725-31aac3e1878d
 	github.com/pmezard/go-difflib v1.0.0
diff --git a/go.sum b/go.sum
index 8d64ccdfa..e957043f8 100644
--- a/go.sum
+++ b/go.sum
@@ -191,6 +191,10 @@ github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWo
 github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
 github.com/gabriel-vasile/mimetype v1.4.0 h1:Cn9dkdYsMIu56tGho+fqzh7XmvY2YyGU0FnbhiOsEro=
 github.com/gabriel-vasile/mimetype v1.4.0/go.mod h1:fA8fi6KUiG7MgQQ+mEWotXoEOvmxRtOJlERCzSmRvr8=
+github.com/gdamore/encoding v1.0.0 h1:+7OoQ1Bc6eTm5niUzBa0Ctsh6JbMW6Ra+YNuAtDBdko=
+github.com/gdamore/encoding v1.0.0/go.mod h1:alR0ol34c49FCSBLjhosxzcPHQbf2trDkoo5dl+VrEg=
+github.com/gdamore/tcell/v2 v2.5.1 h1:zc3LPdpK184lBW7syF2a5C6MV827KmErk9jGVnmsl/I=
+github.com/gdamore/tcell/v2 v2.5.1/go.mod h1:wSkrPaXoiIWZqW/g7Px4xc79di6FTcpB8tvaKJ6uGBo=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/gliderlabs/ssh v0.1.1/go.mod h1:U7qILu1NlMHj9FlMhZLlkCdDnU1DBEAqr0aevW3Awn0=
 github.com/glycerine/go-unsnap-stream v0.0.0-20180323001048-9f0cb55181dd/go.mod h1:/20jfyN9Y5QPEAprSgKAUr+glWDY39ZiUEAYOEv5dsE=
@@ -403,6 +407,8 @@ github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
 github.com/kylelemons/godebug v0.0.0-20170820004349-d65d576e9348 h1:MtvEpTB6LX3vkb4ax0b5D2DHbNAUsen0Gx5wZoq3lV4=
 github.com/kylelemons/godebug v0.0.0-20170820004349-d65d576e9348/go.mod h1:B69LEHPfb2qLo0BaaOLcbitczOKLWTsrBG9LczfCD4k=
 github.com/lucas-clemente/quic-go v0.25.0/go.mod h1:YtzP8bxRVCBlO77yRanE264+fY/T2U9ZlW1AaHOsMOg=
+github.com/lucasb-eyer/go-colorful v1.2.0 h1:1nnpGOrhyZZuNyfu1QjKiUICQ74+3FNCN69Aj6K7nkY=
+github.com/lucasb-eyer/go-colorful v1.2.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
 github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0 h1:6E+4a0GO5zZEnZ81pIr0yLvtUWk2if982qA3F3QD6H4=
 github.com/lufia/plan9stats v0.0.0-20211012122336-39d0f177ccd0/go.mod h1:zJYVVT2jmtg6P3p1VtQj7WsuWi/y4VnjVBn7F8KPB3I=
 github.com/lunixbochs/vtclean v1.0.0/go.mod h1:pHhQNgMf3btfWnGBVipUOjRYhoOsdGqdm/+2c2E2WMI=
@@ -418,7 +424,6 @@ github.com/mattn/go-ieproxy v0.0.1 h1:qiyop7gCflfhwCzGyeT0gro3sF9AIg9HU98JORTkqf
 github.com/mattn/go-ieproxy v0.0.1/go.mod h1:pYabZ6IHcRpFh7vIaLfK7rdcWgFEb3SFJ6/gNWuh88E=
 github.com/mattn/go-isatty v0.0.14 h1:yVuAays6BHfxijgZPzw+3Zlu5yQgKGP2/hcQbHb7S9Y=
 github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94=
-github.com/mattn/go-runewidth v0.0.9/go.mod h1:H031xJmbD/WCDINGzjvQ9THkh0rPKHF+m2gUSrubnMI=
 github.com/mattn/go-runewidth v0.0.13 h1:lTGmDsbAYt5DmK6OnoV7EuIF1wEIFAcxld6ypU4OSgU=
 github.com/mattn/go-runewidth v0.0.13/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
 github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0jegS5sx/RkqARlsWZ6pIwiU=
@@ -444,8 +449,6 @@ github.com/ncw/swift/v2 v2.0.1/go.mod h1:z0A9RVdYPjNjXVo2pDOPxZ4eu3oarO1P91fTItc
 github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86/go.mod h1:kHJEU3ofeGjhHklVoIGuVj85JJwZ6kWPaJwCIxgnFmo=
 github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab/go.mod h1:Qr6/a/Q4r9LP1IltGz7tA7iOK1WonHEYhu1HRBA7ZiM=
 github.com/nfnt/resize v0.0.0-20180221191011-83c6a9932646/go.mod h1:jpp1/29i3P1S/RLdc7JQKbRpFeM1dOBd8T9ki5s+AY8=
-github.com/nsf/termbox-go v1.1.1 h1:nksUPLCb73Q++DwbYUBEglYBRPZyoXJdrj5L+TkjyZY=
-github.com/nsf/termbox-go v1.1.1/go.mod h1:T0cTdVuOwf7pHQNtfhnEbzHbcNyCEcVU4YPpouCbVxo=
 github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
 github.com/nxadm/tail v1.4.8 h1:nPr65rt6Y5JFSKQO7qToXr7pePgD6Gwiw05lkbyAQTE=
 github.com/nxadm/tail v1.4.8/go.mod h1:+ncqLTQzXmGhMZNUePPaPqPvBxHAIsmXswZKocGu+AU=
@@ -877,12 +880,14 @@ golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20220128215802-99c3d69c2c27/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220209214540-3681064d5158/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220227234510-4e6760a101f9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220318055525-2edf467146b5/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220328115105-d36c6a25d886/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220412211240-33da011f77ad/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220502124256-b6088ccd6cba/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a h1:dGzPydgVsqGcTRVwiLJ1jVbufYwmzD3LfVPLKsKg+0k=
 golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
+golang.org/x/term v0.0.0-20201210144234-2321bbc49cbf/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
 golang.org/x/term v0.0.0-20220526004731-065cf7ba2467 h1:CBpWXWQpIRjzmkkA+M7q9Fqnwd2mZr3AFqexg8YTfoM=
 golang.org/x/term v0.0.0-20220526004731-065cf7ba2467/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=


Test patch:
diff --git a/fs/operations/operations_test.go b/fs/operations/operations_test.go
index 10475562a..04b26e8b8 100644
--- a/fs/operations/operations_test.go
+++ b/fs/operations/operations_test.go
@@ -729,41 +729,58 @@ func TestCopyURL(t *testing.T) {
 
 	// check when reading from regular HTTP server
 	status := 0
+	nameHeader := false
+	headerFilename := "headerfilename.txt"
 	handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
 		if status != 0 {
 			http.Error(w, "an error ocurred", status)
 		}
+		if nameHeader {
+			w.Header().Set("Content-Disposition", `attachment; filename="folder\`+headerFilename+`"`)
+		}
 		_, err := w.Write([]byte(contents))
 		assert.NoError(t, err)
 	})
 	ts := httptest.NewServer(handler)
 	defer ts.Close()
 
-	o, err := operations.CopyURL(ctx, r.Fremote, "file1", ts.URL, false, false)
+	o, err := operations.CopyURL(ctx, r.Fremote, "file1", ts.URL, false, false, false)
 	require.NoError(t, err)
 	assert.Equal(t, int64(len(contents)), o.Size())
 
 	fstest.CheckListingWithPrecision(t, r.Fremote, []fstest.Item{file1}, nil, fs.ModTimeNotSupported)
 
 	// Check file clobbering
-	_, err = operations.CopyURL(ctx, r.Fremote, "file1", ts.URL, false, true)
+	_, err = operations.CopyURL(ctx, r.Fremote, "file1", ts.URL, false, false, true)
 	require.Error(t, err)
 
 	// Check auto file naming
 	status = 0
 	urlFileName := "filename.txt"
-	o, err = operations.CopyURL(ctx, r.Fremote, "", ts.URL+"/"+urlFileName, true, false)
+	o, err = operations.CopyURL(ctx, r.Fremote, "", ts.URL+"/"+urlFileName, true, false, false)
 	require.NoError(t, err)
 	assert.Equal(t, int64(len(contents)), o.Size())
 	assert.Equal(t, urlFileName, o.Remote())
 
+	// Check header file naming
+	nameHeader = true
+	o, err = operations.CopyURL(ctx, r.Fremote, "", ts.URL, true, true, false)
+	require.NoError(t, err)
+	assert.Equal(t, int64(len(contents)), o.Size())
+	assert.Equal(t, headerFilename, o.Remote())
+
 	// Check auto file naming when url without file name
-	_, err = operations.CopyURL(ctx, r.Fremote, "file1", ts.URL, true, false)
+	_, err = operations.CopyURL(ctx, r.Fremote, "file1", ts.URL, true, false, false)
+	require.Error(t, err)
+
+	// Check header file naming without header set
+	nameHeader = false
+	_, err = operations.CopyURL(ctx, r.Fremote, "file1", ts.URL, true, true, false)
 	require.Error(t, err)
 
 	// Check an error is returned for a 404
 	status = http.StatusNotFound
-	o, err = operations.CopyURL(ctx, r.Fremote, "file1", ts.URL, false, false)
+	o, err = operations.CopyURL(ctx, r.Fremote, "file1", ts.URL, false, false, false)
 	require.Error(t, err)
 	assert.Contains(t, err.Error(), "Not Found")
 	assert.Nil(t, o)
@@ -776,10 +793,10 @@ func TestCopyURL(t *testing.T) {
 	tss := httptest.NewTLSServer(handler)
 	defer tss.Close()
 
-	o, err = operations.CopyURL(ctx, r.Fremote, "file2", tss.URL, false, false)
+	o, err = operations.CopyURL(ctx, r.Fremote, "file2", tss.URL, false, false, false)
 	require.NoError(t, err)
 	assert.Equal(t, int64(len(contents)), o.Size())
-	fstest.CheckListingWithPrecision(t, r.Fremote, []fstest.Item{file1, file2, fstest.NewItem(urlFileName, contents, t1)}, nil, fs.ModTimeNotSupported)
+	fstest.CheckListingWithPrecision(t, r.Fremote, []fstest.Item{file1, file2, fstest.NewItem(urlFileName, contents, t1), fstest.NewItem(headerFilename, contents, t1)}, nil, fs.ModTimeNotSupported)
 }
 
 func TestCopyURLToWriter(t *testing.T) {
diff --git a/fstest/test_all/clean.go b/fstest/test_all/clean.go
index c4d22a8a4..c12a4839a 100644
--- a/fstest/test_all/clean.go
+++ b/fstest/test_all/clean.go
@@ -9,6 +9,7 @@ import (
 	"regexp"
 
 	"github.com/rclone/rclone/fs"
+	"github.com/rclone/rclone/fs/fspath"
 	"github.com/rclone/rclone/fs/list"
 	"github.com/rclone/rclone/fs/operations"
 )
@@ -39,7 +40,7 @@ func cleanFs(ctx context.Context, remote string, cleanup bool) error {
 	}
 	err = entries.ForDirError(func(dir fs.Directory) error {
 		dirPath := dir.Remote()
-		fullPath := remote + dirPath
+		fullPath := fspath.JoinRootPath(remote, dirPath)
 		if MatchTestRemote.MatchString(dirPath) {
 			if *dryRun {
 				log.Printf("Not Purging %s - -dry-run", fullPath)
diff --git a/fstest/test_all/config.yaml b/fstest/test_all/config.yaml
index 3588cdddc..68f17c2dd 100644
--- a/fstest/test_all/config.yaml
+++ b/fstest/test_all/config.yaml
@@ -141,6 +141,7 @@ backends:
    fastlist: true
    tests:
      - backend
+   extratime: 2.0
  - backend:  "jottacloud"
    remote:   "TestJottacloud:"
    fastlist: true