Repository: rclone/rclone
Pull request #7404 (2023-11-03 19:49:28)

Listremotes filtering
#### What is the purpose of this change?

The main purpose is to add support for filtering the output from `listremotes` command, as discussed in the forum. Some additional, somewhat related, features were also added, namely ordering and json output. During extended testing of this command, some issues and inconsistencies within the fs/filter and fs/config libs, which the listremotes command depends on, were discovered and improved.

Now to the details: First the improvements to listremotes, as these are the most user-facing part, then later some details regarding the changes in fs libs - although these commits comes first in the PR as some of they create a foundation for some of the listremote improvements.

Improvements to listremotes:
- Including source of the remote definition in the `--long` output format, i.e. if it was defined in environment variable or in config file. The complete remote configuration may contain options from both environment and config, but this shows where it is "defined" - where the name and type comes from. This follows the documented [preference](https://rclone.org/docs/#precedence), so if it is defined in both places its the one from environment that will be shown.
  ```
  $ rclone listremotes --long
  temp_box: dropbox    environment
  temp_jfs: jottacloud environment
  box:      dropbox    file
  box_1:    dropbox    file
  box_2:    dropbox    file
  jfs:      jottacloud file
  gd:       drive      file
  box_1b:   dropbox    file
  ```
- Optional flag `--type` to list only remotes of a specific type:
  ```
  $ rclone listremotes --long --type dropbox
  temp_box: dropbox environment
  box:      dropbox file
  box_1:    dropbox file
  box_2:    dropbox file
  box_1b:   dropbox file
  ```
- Optional flag `--source` to list only remotes from a specific source:
  ```
  $ rclone listremotes --type dropbox --source environment
  temp_box: dropbox environment
  ```
- Search/filter on name. A query string can be specified as an optional (unnamed) argument. I did considered implementing it as a simple string "binary" matching, or full blown regular expression, but I find the glob syntax quite convenient, flexible while easy to use - easier to understand than regex for less technical users at least. I ended up re-using rclone's [filtering syntax](https://rclone.org/filtering/): This is convenient in that it uses a quite easy-to-use glob pattern by default, while allowing for some more advanced features and even regular expressions for those who want it:
  ```
  $ rclone listremotes box
  
  $ rclone listremotes box*
  box:
  box_1:
  box_2:
  box_1b:
  $ rclone listremotes *box
  temp_box:
  $ rclone listremotes box{{(_\d+)?}} --type dropbox
  box:
  box_1:
  box_2:
  ```
- Sorting. Option flag `--order-by` can be used to sort on any of the fields (`name`, `type` and `source`) in any direction (`ascending` or `asc`, and `descending` or `desc`), and they can be combined with the syntax: `field[=order],field[=order],...` (order is optional).
  ```
  $ rclone listremotes --long --order-by type,source=desc,name
  gd:       drive      file
  box:      dropbox    file
  box_1:    dropbox    file
  box_1b:   dropbox    file
  box_2:    dropbox    file
  temp_box: dropbox    environment
  jfs:      jottacloud file
  temp_jfs: jottacloud environment
  ```
- Machine-readable output with option `--json`, also for completeness with other commands such as `size` and `about`. It produces a valid array json output in format similar to `lsjson`, and can be combined with filtering and other options (except `--long`):
  ```
  $ rclone listremotes --json --type jottacloud
  [
  {"name":"temp_jfs","type":"jottacloud","source":"environment"},
  {"name":"jfs","type":"jottacloud","source":"file"}
  ]
  ```

Improvements in fs/filter:
- To implement the glob-filtering I had to tweak the `fs.GlobToRegexp` function a little, so that we can tell it to not treat `/` as path separator, and to not make `*` stop consuming them, and not support `**` as additional symbol that will consume them, etc.. As can be seen from the examples above, the `rclone listremotes box` does not give any result as it would with the simplest binary matching alternative, so this is sort of a tradeoff between the absolute simplest syntax and flexibility.
- For consistency I updated the `help flags` command to use the same logic. It used plain regex before, and my reasoning was that we should try to avoid using different filtering syntax for different command, as that will not be easy to remember for users. The main difference is that a glob pattern is implicitly anchored. E.g. previously you could write `rclone help flags dropbox` and it would return any flags with this word in its name or description. But now you will have to write `rclone help flags *dropbox*` to get the same result. That is a less convenient, so a bit unfortunate actually. Maybe in an attempt at compensating - I added some more flexibility to the flags filtering:
  - You can now add flag `--name` to make the filter match the *name* of the flag only. E.g. `rclone help flags *password* --name` will only match flags with "password" in their name, it will not include `--sftp-pass` even if it mentions "password" in its description.
  - You can now specify name of a flag group with `--group`, e.g. `rclone help flags --group sync` or  `rclone help flags --group backend`. This uses (case insensitive) string matching by the way, requires complete match on the group name. Contradicting myself a bit here, I guess. Could have used the same pattern matching also here, but that complicated flags and parameter passing code, and there are only a few short group names so I figured there is not much to loose from having to spell out the entire name. The same is true for the new `--type` and `--source` flags in `listremote` by the way.
- **Note:** This part I'm least certain about. Is the existing filter syntax for help flags so well established, and easy to use, that it is bad to change it? If uncertain, I could split this out to a separate PR. In any case, do you agree with me that the glob syntax is clever, and useful enough that we want to keep it for the listremotes command (and possibly use it elsewhere as well)?

Improvements in fs/config:
- Issues:
  - It is possible to create nameless remotes with environment variable, like this:
    ```
    $ set RCLONE_CONFIG__TYPE=sftp
    $ rclone listremotes --long
    : sftp
    ```
  - If you define a remote with same name in environment as in file, it is listed twice:
    ```
    $ type rclone.conf
    [box]
    type = dropbox
    
    [box_1]
    type = dropbox
    
    [box_1b]
    type = dropbox
    
    $ set RCLONE_CONFIG_BOX_1B_TYPE=sftp
    $ rclone listremotes --long
    box:    dropbox
    box_1:  dropbox
    box_1b: dropbox
    box_1b: dropbox
    ```
    And even worse; notice that the duplicate remote is listed with same type - the type from config file. So it does not actually reflect the truth very well. And another point to this is that it chooses the remote type from config file over the environment, which is opposite of what the documentation says - and what rclone uses in "real" operations:
    ```
    $ rclone lsd box_1b: -vvv
    DEBUG : Setting type="sftp" for "box_1b" from environment variable RCLONE_CONFIG_WORK_BOX_1B_TYPE
    ```
- Explanations:
  - While looking into the issues described above, I found the some parts of the fs/config a bit confusing, misleading and wrong (in my opinion): Function `config.FileSections` did not only return the sections, i.e. remotes, from the config file - but also included any configured as environment variables. Same with `config.FileGet`, which retrieves value of options. Misleading function names, at least. But also; while collecting remote names from environment it appends them to the list even if there is one with same name in configuration file, which could be considered a feature to inform users - except as we see above `listremotes` does not present it with its correct type so its a bug there regardless what we want it to show. Which brings me to next point; when collecting remotes from environment, remember these are defined by the `_TYPE` environment variable specifying the type of remote, the type information is ignored and when using `listremotes` command it uses `config.FileGet` to get the value of the type property. But this function reads the configuration file first, and only if not found there considers environment variable - which is the wrong way around. Together this leads to the issue above.
- Improvements:
  - Make fs/config functions with prefix `File` consider the configuration file only, consistent with its name. Then make new "convenience" functions `GetRemotes` and `GetValues` which also considers environment (there were already a couple of similar functions).
  - The new `GetRemotes` returns not only the remote names, but also the type and the "source" (if it was found from file or environment). This ensures we make use of the type information from the `_TYPE` environment variable when that is used to define a remote (name) as well, avoiding the extra call but more importantly avoiding inconsistency (although the next point should also fix this).
  - When considering options set in environment variables, make sure these take precedence over configuration file.

#### Was the change discussed in an issue or in the forum before?

https://forum.rclone.org/t/suggestion-separate-crypt-and-non-crypt-remotes/42581

#### Checklist

- [x] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-new-feature-or-bug-fix).
- [x] I have added tests for all changes in this PR if appropriate.
- [x] I have added documentation for the changes if appropriate.
- [x] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages).
- [x] I'm done, this Pull Request is ready for review :-)


ncw: Hi @albertony - I don't know why I didn't merge this - it looks great!

Can you rebase it please and I'll try to get it in v1.68 - thank you :-)
albertony: ✅ Rebased

Patch:
diff --git a/backend/pikpak/pikpak.go b/backend/pikpak/pikpak.go
index a18fe573f..1a5d82a33 100644
--- a/backend/pikpak/pikpak.go
+++ b/backend/pikpak/pikpak.go
@@ -37,10 +37,11 @@ import (
 	"sync"
 	"time"
 
-	"github.com/aws/aws-sdk-go/aws"
-	"github.com/aws/aws-sdk-go/aws/credentials"
-	"github.com/aws/aws-sdk-go/aws/session"
-	"github.com/aws/aws-sdk-go/service/s3/s3manager"
+	"github.com/aws/aws-sdk-go-v2/aws"
+	awsconfig "github.com/aws/aws-sdk-go-v2/config"
+	"github.com/aws/aws-sdk-go-v2/credentials"
+	"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
+	"github.com/aws/aws-sdk-go-v2/service/s3"
 	"github.com/rclone/rclone/backend/pikpak/api"
 	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fs/accounting"
@@ -70,8 +71,8 @@ const (
 	taskWaitTime                = 500 * time.Millisecond
 	decayConstant               = 2 // bigger for slower decay, exponential
 	rootURL                     = "https://api-drive.mypikpak.com"
-	minChunkSize                = fs.SizeSuffix(s3manager.MinUploadPartSize)
-	defaultUploadConcurrency    = s3manager.DefaultUploadConcurrency
+	minChunkSize                = fs.SizeSuffix(manager.MinUploadPartSize)
+	defaultUploadConcurrency    = manager.DefaultUploadConcurrency
 )
 
 // Globals
@@ -1190,32 +1191,33 @@ func (f *Fs) uploadByForm(ctx context.Context, in io.Reader, name string, size i
 
 func (f *Fs) uploadByResumable(ctx context.Context, in io.Reader, name string, size int64, resumable *api.Resumable) (err error) {
 	p := resumable.Params
-	endpoint := strings.Join(strings.Split(p.Endpoint, ".")[1:], ".") // "mypikpak.com"
 
-	cfg := &aws.Config{
-		Credentials: credentials.NewStaticCredentials(p.AccessKeyID, p.AccessKeySecret, p.SecurityToken),
-		Region:      aws.String("pikpak"),
-		Endpoint:    &endpoint,
-	}
-	sess, err := session.NewSession(cfg)
+	// Create a credentials provider
+	creds := credentials.NewStaticCredentialsProvider(p.AccessKeyID, p.AccessKeySecret, p.SecurityToken)
+
+	cfg, err := awsconfig.LoadDefaultConfig(ctx,
+		awsconfig.WithCredentialsProvider(creds),
+		awsconfig.WithRegion("pikpak"))
 	if err != nil {
 		return
 	}
-	partSize := chunksize.Calculator(name, size, s3manager.MaxUploadParts, f.opt.ChunkSize)
 
-	// Create an uploader with the session and custom options
-	uploader := s3manager.NewUploader(sess, func(u *s3manager.Uploader) {
+	client := s3.NewFromConfig(cfg, func(o *s3.Options) {
+		o.BaseEndpoint = aws.String("https://mypikpak.com/")
+	})
+	partSize := chunksize.Calculator(name, size, int(manager.MaxUploadParts), f.opt.ChunkSize)
+
+	// Create an uploader with custom options
+	uploader := manager.NewUploader(client, func(u *manager.Uploader) {
 		u.PartSize = int64(partSize)
 		u.Concurrency = f.opt.UploadConcurrency
 	})
-	// Upload input parameters
-	uParams := &s3manager.UploadInput{
+	// Perform an upload
+	_, err = uploader.Upload(ctx, &s3.PutObjectInput{
 		Bucket: &p.Bucket,
 		Key:    &p.Key,
 		Body:   in,
-	}
-	// Perform an upload
-	_, err = uploader.UploadWithContext(ctx, uParams)
+	})
 	return
 }
 
diff --git a/backend/s3/gen_setfrom.go b/backend/s3/gen_setfrom.go
index 9dd037086..4748b4da0 100644
--- a/backend/s3/gen_setfrom.go
+++ b/backend/s3/gen_setfrom.go
@@ -13,7 +13,8 @@ import (
 	"reflect"
 	"strings"
 
-	"github.com/aws/aws-sdk-go/service/s3"
+	"github.com/aws/aws-sdk-go-v2/service/s3"
+	"github.com/aws/aws-sdk-go-v2/service/s3/types"
 )
 
 // flags
@@ -82,15 +83,18 @@ func main() {
 
 package s3
 
-import "github.com/aws/aws-sdk-go/service/s3"
+import (
+	"github.com/aws/aws-sdk-go-v2/service/s3"
+	"github.com/aws/aws-sdk-go-v2/service/s3/types"
+)
 `)
 
 	genSetFrom(new(s3.ListObjectsInput), new(s3.ListObjectsV2Input))
 	genSetFrom(new(s3.ListObjectsV2Output), new(s3.ListObjectsOutput))
 	genSetFrom(new(s3.ListObjectVersionsInput), new(s3.ListObjectsV2Input))
-	genSetFrom(new(s3.ObjectVersion), new(s3.DeleteMarkerEntry))
+	genSetFrom(new(types.ObjectVersion), new(types.DeleteMarkerEntry))
 	genSetFrom(new(s3.ListObjectsV2Output), new(s3.ListObjectVersionsOutput))
-	genSetFrom(new(s3.Object), new(s3.ObjectVersion))
+	genSetFrom(new(types.Object), new(types.ObjectVersion))
 	genSetFrom(new(s3.CreateMultipartUploadInput), new(s3.HeadObjectOutput))
 	genSetFrom(new(s3.CreateMultipartUploadInput), new(s3.CopyObjectInput))
 	genSetFrom(new(s3.UploadPartCopyInput), new(s3.CopyObjectInput))
diff --git a/backend/s3/s3.go b/backend/s3/s3.go
index f2d75a1ae..474fe6bcb 100644
--- a/backend/s3/s3.go
+++ b/backend/s3/s3.go
@@ -4,7 +4,6 @@ package s3
 //go:generate go run gen_setfrom.go -o setfrom.go
 
 import (
-	"bytes"
 	"context"
 	"crypto/md5"
 	"crypto/tls"
@@ -26,19 +25,18 @@ import (
 	"sync"
 	"time"
 
-	"github.com/aws/aws-sdk-go/service/s3/s3manager"
-
-	"github.com/aws/aws-sdk-go/aws"
-	"github.com/aws/aws-sdk-go/aws/awserr"
-	"github.com/aws/aws-sdk-go/aws/corehandlers"
-	"github.com/aws/aws-sdk-go/aws/credentials"
-	"github.com/aws/aws-sdk-go/aws/credentials/ec2rolecreds"
-	"github.com/aws/aws-sdk-go/aws/defaults"
-	"github.com/aws/aws-sdk-go/aws/ec2metadata"
-	"github.com/aws/aws-sdk-go/aws/endpoints"
-	"github.com/aws/aws-sdk-go/aws/request"
-	"github.com/aws/aws-sdk-go/aws/session"
-	"github.com/aws/aws-sdk-go/service/s3"
+	"github.com/aws/aws-sdk-go-v2/aws"
+	v4signer "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
+	awsconfig "github.com/aws/aws-sdk-go-v2/config"
+	"github.com/aws/aws-sdk-go-v2/credentials"
+	"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
+	"github.com/aws/aws-sdk-go-v2/service/s3"
+	"github.com/aws/aws-sdk-go-v2/service/s3/types"
+	"github.com/aws/smithy-go"
+	"github.com/aws/smithy-go/logging"
+	"github.com/aws/smithy-go/middleware"
+	smithyhttp "github.com/aws/smithy-go/transport/http"
+
 	"github.com/ncw/swift/v2"
 	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fs/accounting"
@@ -2440,6 +2438,18 @@ object ending with "/", to persist the folder.
 			Name: "use_multipart_etag",
 			Help: `Whether to use ETag in multipart uploads for verification
 
+This should be true, false or left unset to use the default for the provider.
+`,
+			Default:  fs.Tristate{},
+			Advanced: true,
+		}, {
+			Name: "use_unsigned_payload",
+			Help: `Whether to use an unsigned payload in PutObject
+
+Rclone has to avoid the AWS SDK seeking the body when calling
+PutObject. The AWS provider can add checksums in the trailer to avoid
+seeking but other providers can't.
+
 This should be true, false or left unset to use the default for the provider.
 `,
 			Default:  fs.Tristate{},
@@ -2553,9 +2563,10 @@ In this case, you might want to try disabling this option.
 			Default:  false,
 		}, {
 			Name:     "sts_endpoint",
-			Help:     "Endpoint for STS.\n\nLeave blank if using AWS to use the default endpoint for the region.",
+			Help:     "Endpoint for STS (deprecated).\n\nLeave blank if using AWS to use the default endpoint for the region.",
 			Provider: "AWS",
 			Advanced: true,
+			Hide:     fs.OptionHideBoth,
 		}, {
 			Name: "use_already_exists",
 			Help: strings.ReplaceAll(`Set if rclone should report BucketAlreadyExists errors on bucket creation.
@@ -2595,6 +2606,27 @@ knows about - please make a bug report if not.
 `,
 			Default:  fs.Tristate{},
 			Advanced: true,
+		}, {
+			Name: "sdk_log_mode",
+			Help: strings.ReplaceAll(`Set to debug the SDK
+
+This can be set to a comma separated list of the following functions:
+
+- |Signing|
+- |Retries|
+- |Request|
+- |RequestWithBody|
+- |Response|
+- |ResponseWithBody|
+- |DeprecatedUsage|
+- |RequestEventMessage|
+- |ResponseEventMessage|
+
+Use |Off| to disable and |All| to set all log levels. You will need to
+use |-vv| to see the debug level logs.
+`, "|", "`"),
+			Default:  sdkLogMode(0),
+			Advanced: true,
 		},
 		}})
 }
@@ -2614,6 +2646,27 @@ const (
 	maxExpireDuration   = fs.Duration(7 * 24 * time.Hour) // max expiry is 1 week
 )
 
+type (
+	sdkLogMode        = fs.Bits[sdkLogModeChoices]
+	sdkLogModeChoices struct{}
+)
+
+func (sdkLogModeChoices) Choices() []fs.BitsChoicesInfo {
+	return []fs.BitsChoicesInfo{
+		{Bit: uint64(0), Name: "Off"},
+		{Bit: uint64(aws.LogSigning), Name: "Signing"},
+		{Bit: uint64(aws.LogRetries), Name: "Retries"},
+		{Bit: uint64(aws.LogRequest), Name: "Request"},
+		{Bit: uint64(aws.LogRequestWithBody), Name: "RequestWithBody"},
+		{Bit: uint64(aws.LogResponse), Name: "Response"},
+		{Bit: uint64(aws.LogResponseWithBody), Name: "ResponseWithBody"},
+		{Bit: uint64(aws.LogDeprecatedUsage), Name: "DeprecatedUsage"},
+		{Bit: uint64(aws.LogRequestEventMessage), Name: "RequestEventMessage"},
+		{Bit: uint64(aws.LogResponseEventMessage), Name: "ResponseEventMessage"},
+		{Bit: math.MaxUint64, Name: "All"},
+	}
+}
+
 // globals
 var (
 	errNotWithVersionAt = errors.New("can't modify or delete files in --s3-version-at mode")
@@ -2704,7 +2757,7 @@ type Options struct {
 	V2Auth                bool                 `config:"v2_auth"`
 	UseAccelerateEndpoint bool                 `config:"use_accelerate_endpoint"`
 	LeavePartsOnError     bool                 `config:"leave_parts_on_error"`
-	ListChunk             int64                `config:"list_chunk"`
+	ListChunk             int32                `config:"list_chunk"`
 	ListVersion           int                  `config:"list_version"`
 	ListURLEncode         fs.Tristate          `config:"list_url_encode"`
 	NoCheckBucket         bool                 `config:"no_check_bucket"`
@@ -2725,25 +2778,26 @@ type Options struct {
 	NoSystemMetadata      bool                 `config:"no_system_metadata"`
 	UseAlreadyExists      fs.Tristate          `config:"use_already_exists"`
 	UseMultipartUploads   fs.Tristate          `config:"use_multipart_uploads"`
+	UseUnsignedPayload    fs.Tristate          `config:"use_unsigned_payload"`
+	SDKLogMode            sdkLogMode           `config:"sdk_log_mode"`
 }
 
 // Fs represents a remote s3 server
 type Fs struct {
-	name           string           // the name of the remote
-	root           string           // root of the bucket - ignore all objects above this
-	opt            Options          // parsed options
-	ci             *fs.ConfigInfo   // global config
-	ctx            context.Context  // global context for reading config
-	features       *fs.Features     // optional features
-	c              *s3.S3           // the connection to the s3 server
-	ses            *session.Session // the s3 session
-	rootBucket     string           // bucket part of root (if any)
-	rootDirectory  string           // directory part of root (if any)
-	cache          *bucket.Cache    // cache for bucket creation status
-	pacer          *fs.Pacer        // To pace the API calls
-	srv            *http.Client     // a plain http client
-	srvRest        *rest.Client     // the rest connection to the server
-	etagIsNotMD5   bool             // if set ETags are not MD5s
+	name           string          // the name of the remote
+	root           string          // root of the bucket - ignore all objects above this
+	opt            Options         // parsed options
+	ci             *fs.ConfigInfo  // global config
+	ctx            context.Context // global context for reading config
+	features       *fs.Features    // optional features
+	c              *s3.Client      // the connection to the s3 server
+	rootBucket     string          // bucket part of root (if any)
+	rootDirectory  string          // directory part of root (if any)
+	cache          *bucket.Cache   // cache for bucket creation status
+	pacer          *fs.Pacer       // To pace the API calls
+	srv            *http.Client    // a plain http client
+	srvRest        *rest.Client    // the rest connection to the server
+	etagIsNotMD5   bool            // if set ETags are not MD5s
 	versioningMu   sync.Mutex
 	versioning     fs.Tristate // if set bucket is using versions
 	warnCompressed sync.Once   // warn once about compressed files
@@ -2772,6 +2826,24 @@ type Object struct {
 	contentLanguage    *string // Content-Language: header
 }
 
+// safely dereference the pointer, returning a zero T if nil
+func deref[T any](p *T) T {
+	if p == nil {
+		var zero T
+		return zero
+	}
+	return *p
+}
+
+// gets an http status code from err or returns -1
+func getHTTPStatusCode(err error) int {
+	var httpErr interface{ HTTPStatusCode() int }
+	if errors.As(err, &httpErr) {
+		return httpErr.HTTPStatusCode()
+	}
+	return -1
+}
+
 // ------------------------------------------------------------
 
 // Name of the remote (as passed into NewFs)
@@ -2815,33 +2887,35 @@ func (f *Fs) shouldRetry(ctx context.Context, err error) (bool, error) {
 	if fserrors.ContextError(ctx, &err) {
 		return false, err
 	}
+	// https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md#error-handling
 	// If this is an awserr object, try and extract more useful information to determine if we should retry
-	if awsError, ok := err.(awserr.Error); ok {
+	var awsError smithy.APIError
+	if errors.As(err, &awsError) {
 		// Simple case, check the original embedded error in case it's generically retryable
-		if fserrors.ShouldRetry(awsError.OrigErr()) {
+		if fserrors.ShouldRetry(awsError) {
 			return true, err
 		}
 		// If it is a timeout then we want to retry that
-		if awsError.Code() == "RequestTimeout" {
+		if awsError.ErrorCode() == "RequestTimeout" {
 			return true, err
 		}
-		// Failing that, if it's a RequestFailure it's probably got an http status code we can check
-		if reqErr, ok := err.(awserr.RequestFailure); ok {
-			// 301 if wrong region for bucket - can only update if running from a bucket
-			if f.rootBucket != "" {
-				if reqErr.StatusCode() == http.StatusMovedPermanently {
-					urfbErr := f.updateRegionForBucket(ctx, f.rootBucket)
-					if urfbErr != nil {
-						fs.Errorf(f, "Failed to update region for bucket: %v", urfbErr)
-						return false, err
-					}
-					return true, err
+	}
+	// Check http status code if available
+	if httpStatusCode := getHTTPStatusCode(err); httpStatusCode > 0 {
+		// 301 if wrong region for bucket - can only update if running from a bucket
+		if f.rootBucket != "" {
+			if httpStatusCode == http.StatusMovedPermanently {
+				urfbErr := f.updateRegionForBucket(ctx, f.rootBucket)
+				if urfbErr != nil {
+					fs.Errorf(f, "Failed to update region for bucket: %v", urfbErr)
+					return false, err
 				}
+				return true, err
 			}
-			for _, e := range retryErrorCodes {
-				if reqErr.StatusCode() == e {
-					return true, err
-				}
+		}
+		for _, e := range retryErrorCodes {
+			if httpStatusCode == e {
+				return true, err
 			}
 		}
 	}
@@ -2890,164 +2964,190 @@ func getClient(ctx context.Context, opt *Options) *http.Client {
 	}
 }
 
-// Default name resolver
-var defaultResolver = endpoints.DefaultResolver()
-
-// resolve (service, region) to endpoint
+// Google Cloud Storage alters the Accept-Encoding header, which
+// breaks the v2 request signature
 //
-// Used to set endpoint for s3 services and not for other services
-type resolver map[string]string
-
-// Add a service to the resolver, ignoring empty urls
-func (r resolver) addService(service, url string) {
-	if url == "" {
-		return
-	}
-	if !strings.HasPrefix(url, "http") {
-		url = "https://" + url
-	}
-	r[service] = url
-}
-
-// EndpointFor return the endpoint for s3 if set or the default if not
-func (r resolver) EndpointFor(service, region string, opts ...func(*endpoints.Options)) (endpoints.ResolvedEndpoint, error) {
-	fs.Debugf(nil, "Resolving service %q region %q", service, region)
-	url, ok := r[service]
-	if ok {
-		return endpoints.ResolvedEndpoint{
-			URL:           url,
-			SigningRegion: region,
-		}, nil
-	}
-	return defaultResolver.EndpointFor(service, region, opts...)
-}
+// It also doesn't like the x-id URL parameter SDKv2 puts in so we
+// remove that too.
+//
+// See https://github.com/aws/aws-sdk-go-v2/issues/1816.
+// Adapted from: https://github.com/aws/aws-sdk-go-v2/issues/1816#issuecomment-1927281540
+func fixupGCS(o *s3.Options) {
+	type ignoredHeadersKey struct{}
+	headers := []string{"Accept-Encoding"}
+
+	fixup := middleware.FinalizeMiddlewareFunc(
+		"FixupGCS",
+		func(ctx context.Context, in middleware.FinalizeInput, next middleware.FinalizeHandler) (out middleware.FinalizeOutput, metadata middleware.Metadata, err error) {
+			req, ok := in.Request.(*smithyhttp.Request)
+			if !ok {
+				return out, metadata, fmt.Errorf("fixupGCS: unexpected request middleware type %T", in.Request)
+			}
 
-// s3Connection makes a connection to s3
-func s3Connection(ctx context.Context, opt *Options, client *http.Client) (*s3.S3, *session.Session, error) {
-	ci := fs.GetConfig(ctx)
-	// Make the auth
-	v := credentials.Value{
-		AccessKeyID:     opt.AccessKeyID,
-		SecretAccessKey: opt.SecretAccessKey,
-		SessionToken:    opt.SessionToken,
-	}
+			// Delete headers from being signed - will restore later
+			ignored := make(map[string]string, len(headers))
+			for _, h := range headers {
+				ignored[h] = req.Header.Get(h)
+				req.Header.Del(h)
+			}
 
-	lowTimeoutClient := &http.Client{Timeout: 1 * time.Second} // low timeout to ec2 metadata service
+			// Remove x-id because Google doesn't like them
+			if query := req.URL.Query(); query.Has("x-id") {
+				query.Del("x-id")
+				req.URL.RawQuery = query.Encode()
+			}
 
-	def := defaults.Get()
-	def.Config.HTTPClient = lowTimeoutClient
+			// Store ignored on context
+			ctx = middleware.WithStackValue(ctx, ignoredHeadersKey{}, ignored)
 
-	// start a new AWS session
-	awsSession, err := session.NewSession()
-	if err != nil {
-		return nil, nil, fmt.Errorf("NewSession: %w", err)
-	}
+			return next.HandleFinalize(ctx, in)
+		},
+	)
 
-	// first provider to supply a credential set "wins"
-	providers := []credentials.Provider{
-		// use static credentials if they're present (checked by provider)
-		&credentials.StaticProvider{Value: v},
+	// Restore headers if necessary
+	restore := middleware.FinalizeMiddlewareFunc(
+		"FixupGCSRestoreHeaders",
+		func(ctx context.Context, in middleware.FinalizeInput, next middleware.FinalizeHandler) (out middleware.FinalizeOutput, metadata middleware.Metadata, err error) {
+			req, ok := in.Request.(*smithyhttp.Request)
+			if !ok {
+				return out, metadata, fmt.Errorf("fixupGCS: unexpected request middleware type %T", in.Request)
+			}
 
-		// * Access Key ID:     AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY
-		// * Secret Access Key: AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY
-		&credentials.EnvProvider{},
+			// Restore ignored from ctx
+			ignored, _ := middleware.GetStackValue(ctx, ignoredHeadersKey{}).(map[string]string)
+			for k, v := range ignored {
+				req.Header.Set(k, v)
+			}
 
-		// A SharedCredentialsProvider retrieves credentials
-		// from the current user's home directory.  It checks
-		// AWS_SHARED_CREDENTIALS_FILE and AWS_PROFILE too.
-		&credentials.SharedCredentialsProvider{
-			Filename: opt.SharedCredentialsFile, // If empty will look for "AWS_SHARED_CREDENTIALS_FILE" env variable.
-			Profile:  opt.Profile,               // If empty will look gor "AWS_PROFILE" env var or "default" if not set.
+			return next.HandleFinalize(ctx, in)
 		},
+	)
+
+	o.APIOptions = append(o.APIOptions, func(stack *middleware.Stack) error {
+		if err := stack.Finalize.Insert(fixup, "Signing", middleware.Before); err != nil {
+			return err
+		}
+		if err := stack.Finalize.Insert(restore, "Signing", middleware.After); err != nil {
+			return err
+		}
+		return nil
+	})
+}
 
-		// Pick up IAM role if we're in an ECS task
-		defaults.RemoteCredProvider(*def.Config, def.Handlers),
+// A logger for the S3 SDK
+type s3logger struct{}
 
-		// Pick up IAM role in case we're on EC2
-		&ec2rolecreds.EC2RoleProvider{
-			Client: ec2metadata.New(awsSession, &aws.Config{
-				HTTPClient: lowTimeoutClient,
-			}),
-			ExpiryWindow: 3 * time.Minute,
-		},
+// Logf is expected to support the standard fmt package "verbs".
+func (s3logger) Logf(classification logging.Classification, format string, v ...interface{}) {
+	switch classification {
+	default:
+	case logging.Debug:
+		fs.Debugf("S3 SDK", format, v...)
+	case logging.Warn:
+		fs.Infof("S3 SDK", format, v...)
 	}
-	cred := credentials.NewChainCredentials(providers)
+}
 
-	switch {
-	case opt.EnvAuth:
-		// No need for empty checks if "env_auth" is true
-	case v.AccessKeyID == "" && v.SecretAccessKey == "":
-		// if no access key/secret and iam is explicitly disabled then fall back to anon interaction
-		cred = credentials.AnonymousCredentials
-		fs.Debugf(nil, "Using anonymous credentials - did you mean to set env_auth=true?")
-	case v.AccessKeyID == "":
-		return nil, nil, errors.New("access_key_id not found")
-	case v.SecretAccessKey == "":
-		return nil, nil, errors.New("secret_access_key not found")
+// s3Connection makes a connection to s3
+func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Client *s3.Client, err error) {
+	ci := fs.GetConfig(ctx)
+	var awsConfig aws.Config
+
+	// Try to fill in the config from the environment if env_auth=true
+	if opt.EnvAuth {
+		configOpts := []func(*awsconfig.LoadOptions) error{}
+		// Set the name of the profile if supplied
+		if opt.Profile != "" {
+			configOpts = append(configOpts, awsconfig.WithSharedConfigProfile(opt.Profile))
+		}
+		// Set the shared config file if supplied
+		if opt.SharedCredentialsFile != "" {
+			configOpts = append(configOpts, awsconfig.WithSharedConfigFiles([]string{opt.SharedCredentialsFile}))
+		}
+		awsConfig, err = awsconfig.LoadDefaultConfig(ctx, configOpts...)
+		if err != nil {
+			return nil, fmt.Errorf("couldn't load configuration with env_auth=true: %w", err)
+		}
+	} else {
+		switch {
+		case opt.AccessKeyID == "" && opt.SecretAccessKey == "":
+			// if no access key/secret and iam is explicitly disabled then fall back to anon interaction
+			awsConfig.Credentials = aws.AnonymousCredentials{}
+			fs.Debugf(nil, "Using anonymous credentials - did you mean to set env_auth=true?")
+		case opt.AccessKeyID == "":
+			return nil, errors.New("access_key_id not found")
+		case opt.SecretAccessKey == "":
+			return nil, errors.New("secret_access_key not found")
+		default:
+			// Make the static auth
+			v := aws.Credentials{
+				AccessKeyID:     opt.AccessKeyID,
+				SecretAccessKey: opt.SecretAccessKey,
+				SessionToken:    opt.SessionToken,
+			}
+			awsConfig.Credentials = &credentials.StaticCredentialsProvider{Value: v}
+		}
 	}
 
 	if opt.Region == "" {
 		opt.Region = "us-east-1"
 	}
 	setQuirks(opt)
-	awsConfig := aws.NewConfig().
-		WithMaxRetries(ci.LowLevelRetries).
-		WithCredentials(cred).
-		WithHTTPClient(client).
-		WithS3ForcePathStyle(opt.ForcePathStyle).
-		WithS3UseAccelerate(opt.UseAccelerateEndpoint).
-		WithS3UsEast1RegionalEndpoint(endpoints.RegionalS3UsEast1Endpoint)
+	awsConfig.RetryMaxAttempts = ci.LowLevelRetries
+	awsConfig.HTTPClient = client
+
+	options := []func(*s3.Options){}
+	options = append(options, func(s3Opt *s3.Options) {
+		s3Opt.UsePathStyle = opt.ForcePathStyle
+		s3Opt.UseAccelerate = opt.UseAccelerateEndpoint
+		// FIXME maybe this should be a tristate so can default to DualStackEndpointStateUnset?
+		if opt.UseDualStack {
+			s3Opt.EndpointOptions.UseDualStackEndpoint = aws.DualStackEndpointStateEnabled
+		} else {
+			s3Opt.EndpointOptions.UseDualStackEndpoint = aws.DualStackEndpointStateDisabled
+		}
+		// FIXME not ported from SDK v1 - not sure what this does
+		// s3Opt.UsEast1RegionalEndpoint = endpoints.RegionalS3UsEast1Endpoint
+	})
 
 	if opt.Region != "" {
-		awsConfig.WithRegion(opt.Region)
-	}
-	if opt.Endpoint != "" || opt.STSEndpoint != "" {
-		// If endpoints are set, override the relevant services only
-		r := make(resolver)
-		r.addService("s3", opt.Endpoint)
-		r.addService("sts", opt.STSEndpoint)
-		awsConfig.WithEndpointResolver(r)
+		awsConfig.Region = opt.Region
 	}
-	if opt.UseDualStack {
-		awsConfig.UseDualStackEndpoint = endpoints.DualStackEndpointStateEnabled
-	}
-
-	// awsConfig.WithLogLevel(aws.LogDebugWithSigning)
-	awsSessionOpts := session.Options{
-		Config: *awsConfig,
+	if opt.STSEndpoint != "" {
+		// FIXME not sure if anyone is using this
+		// Haven't figured out how to do it with the v2 SDK
+		return nil, errors.New("--s3-sts-endpoint is no longer supported with the v2 SDK - please make an issue")
 	}
-	if opt.EnvAuth && opt.AccessKeyID == "" && opt.SecretAccessKey == "" {
-		// Enable loading config options from ~/.aws/config (selected by AWS_PROFILE env)
-		awsSessionOpts.SharedConfigState = session.SharedConfigEnable
-		// Set the name of the profile if supplied
-		awsSessionOpts.Profile = opt.Profile
-		// Set the shared config file if supplied
-		if opt.SharedCredentialsFile != "" {
-			awsSessionOpts.SharedConfigFiles = []string{opt.SharedCredentialsFile}
+	if opt.Endpoint != "" {
+		if !strings.HasPrefix(opt.Endpoint, "http") {
+			opt.Endpoint = "https://" + opt.Endpoint
 		}
-		// The session constructor (aws/session/mergeConfigSrcs) will only use the user's preferred credential source
-		// (from the shared config file) if the passed-in Options.Config.Credentials is nil.
-		awsSessionOpts.Config.Credentials = nil
-	}
-	ses, err := session.NewSessionWithOptions(awsSessionOpts)
-	if err != nil {
-		return nil, nil, err
+		options = append(options, func(s3Opt *s3.Options) {
+			s3Opt.BaseEndpoint = &opt.Endpoint
+		})
 	}
-	c := s3.New(ses)
+
 	if opt.V2Auth || opt.Region == "other-v2-signature" {
 		fs.Debugf(nil, "Using v2 auth")
-		signer := func(req *request.Request) {
-			// Ignore AnonymousCredentials object
-			if req.Config.Credentials == credentials.AnonymousCredentials {
-				return
-			}
-			sign(v.AccessKeyID, v.SecretAccessKey, req.HTTPRequest)
-		}
-		c.Handlers.Sign.Clear()
-		c.Handlers.Sign.PushBackNamed(corehandlers.BuildContentLengthHandler)
-		c.Handlers.Sign.PushBack(signer)
+		options = append(options, func(s3Opt *s3.Options) {
+			s3Opt.HTTPSignerV4 = &v2Signer{opt: opt}
+		})
+	}
+
+	if opt.Provider == "GCS" {
+		options = append(options, func(o *s3.Options) {
+			fixupGCS(o)
+		})
 	}
-	return c, ses, nil
+
+	// Enable SDK logging if requested
+	if opt.SDKLogMode != 0 {
+		awsConfig.ClientLogMode = aws.ClientLogMode(opt.SDKLogMode)
+		awsConfig.Logger = s3logger{}
+	}
+
+	c := s3.NewFromConfig(awsConfig, options...)
+	return c, nil
 }
 
 func checkUploadChunkSize(cs fs.SizeSuffix) error {
@@ -3091,6 +3191,9 @@ func (f *Fs) setUploadCutoff(cs fs.SizeSuffix) (old fs.SizeSuffix, err error) {
 }
 
 func (f *Fs) setCopyCutoff(cs fs.SizeSuffix) (old fs.SizeSuffix, err error) {
+	if f.opt.CopyCutoff == math.MaxInt64 {
+		return f.opt.CopyCutoff, fmt.Errorf("--s3-copy-cutoff not supported: %w", fs.ErrorNotImplemented)
+	}
 	err = checkUploadChunkSize(cs)
 	if err == nil {
 		old, f.opt.CopyCutoff = f.opt.CopyCutoff, cs
@@ -3147,11 +3250,13 @@ func setQuirks(opt *Options) {
 		mightGzip             = true // assume all providers might use content encoding gzip until proven otherwise
 		useAlreadyExists      = true // Set if provider returns AlreadyOwnedByYou or no error if you try to remake your own bucket
 		useMultipartUploads   = true // Set if provider supports multipart uploads
+		useUnsignedPayload    = true // Do we need to use unsigned payloads to avoid seeking in PutObject
 	)
 	switch opt.Provider {
 	case "AWS":
 		// No quirks
-		mightGzip = false // Never auto gzips objects
+		mightGzip = false          // Never auto gzips objects
+		useUnsignedPayload = false // AWS has trailer support which means it adds checksums in the trailer without seeking
 	case "Alibaba":
 		useMultipartEtag = false // Alibaba seems to calculate multipart Etags differently from AWS
 		useAlreadyExists = true  // returns 200 OK
@@ -3234,6 +3339,10 @@ func setQuirks(opt *Options) {
 		useMultipartEtag = false
 		useAlreadyExists = false
 		// useMultipartUploads = false - set this manually
+		// rclone serve doesn't support multi-part server side copy:
+		// See: https://github.com/rclone/rclone/issues/7454
+		// So make cutoff very large which it does support
+		opt.CopyCutoff = math.MaxInt64
 	case "Scaleway":
 		// Scaleway can only have 1000 parts in an upload
 		if opt.MaxUploadParts > 1000 {
@@ -3347,6 +3456,11 @@ func setQuirks(opt *Options) {
 		opt.UploadCutoff = math.MaxInt64
 	}
 
+	// Set the correct use multipart uploads if not manually set
+	if !opt.UseUnsignedPayload.Valid {
+		opt.UseUnsignedPayload.Valid = true
+		opt.UseUnsignedPayload.Value = useUnsignedPayload
+	}
 }
 
 // setRoot changes the root of the Fs
@@ -3355,14 +3469,6 @@ func (f *Fs) setRoot(root string) {
 	f.rootBucket, f.rootDirectory = bucket.Split(f.root)
 }
 
-// return a pointer to the string if non empty or nil if it is empty
-func stringPointerOrNil(s string) *string {
-	if s == "" {
-		return nil
-	}
-	return &s
-}
-
 // NewFs constructs an Fs from the path, bucket:path
 func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, error) {
 	// Parse config into Options struct
@@ -3398,6 +3504,9 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 			return nil, fmt.Errorf("s3: Could not decode sse_customer_key_base64: %w", err)
 		}
 		opt.SSECustomerKey = string(decoded)
+	} else {
+		// Encode the raw key as base64
+		opt.SSECustomerKeyBase64 = base64.StdEncoding.EncodeToString([]byte(opt.SSECustomerKey))
 	}
 	if opt.SSECustomerKey != "" && opt.SSECustomerKeyMD5 == "" {
 		// calculate CustomerKeyMD5 if not supplied
@@ -3405,7 +3514,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 		opt.SSECustomerKeyMD5 = base64.StdEncoding.EncodeToString(md5sumBinary[:])
 	}
 	srv := getClient(ctx, opt)
-	c, ses, err := s3Connection(ctx, opt, srv)
+	c, err := s3Connection(ctx, opt, srv)
 	if err != nil {
 		return nil, err
 	}
@@ -3423,7 +3532,6 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 		ci:      ci,
 		ctx:     ctx,
 		c:       c,
-		ses:     ses,
 		pacer:   pc,
 		cache:   bucket.NewCache(),
 		srv:     srv,
@@ -3490,7 +3598,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 // This is needed to find versioned objects from their paths.
 //
 // It may return info == nil and err == nil if a HEAD would be more appropriate
-func (f *Fs) getMetaDataListing(ctx context.Context, wantRemote string) (info *s3.Object, versionID *string, err error) {
+func (f *Fs) getMetaDataListing(ctx context.Context, wantRemote string) (info *types.Object, versionID *string, err error) {
 	bucket, bucketPath := f.split(wantRemote)
 
 	// Strip the version string off if using versions
@@ -3512,7 +3620,7 @@ func (f *Fs) getMetaDataListing(ctx context.Context, wantRemote string) (info *s
 		findFile:     true,
 		versionAt:    f.opt.VersionAt,
 		hidden:       f.opt.VersionDeleted,
-	}, func(gotRemote string, object *s3.Object, objectVersionID *string, isDirectory bool) error {
+	}, func(gotRemote string, object *types.Object, objectVersionID *string, isDirectory bool) error {
 		if isDirectory {
 			return nil
 		}
@@ -3535,6 +3643,14 @@ func (f *Fs) getMetaDataListing(ctx context.Context, wantRemote string) (info *s
 	return info, versionID, nil
 }
 
+// stringClone clones the string s into new memory. This is useful to
+// stop us keeping references to small strings carved out of large XML
+// responses.
+func stringClone(s string) *string {
+	var sNew = strings.Clone(s)
+	return &sNew
+}
+
 // stringClonePointer clones the string pointed to by sp into new
 // memory. This is useful to stop us keeping references to small
 // strings carved out of large XML responses.
@@ -3542,14 +3658,14 @@ func stringClonePointer(sp *string) *string {
 	if sp == nil {
 		return nil
 	}
-	var s = *sp
+	var s = strings.Clone(*sp)
 	return &s
 }
 
 // Return an Object from a path
 //
 // If it can't be found it returns the error ErrorObjectNotFound.
-func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *s3.Object, versionID *string) (obj fs.Object, err error) {
+func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *types.Object, versionID *string) (obj fs.Object, err error) {
 	o := &Object{
 		fs:     f,
 		remote: remote,
@@ -3569,9 +3685,9 @@ func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *s3.Obje
 		} else {
 			o.lastModified = *info.LastModified
 		}
-		o.setMD5FromEtag(aws.StringValue(info.ETag))
-		o.bytes = aws.Int64Value(info.Size)
-		o.storageClass = stringClonePointer(info.StorageClass)
+		o.setMD5FromEtag(deref(info.ETag))
+		o.bytes = deref(info.Size)
+		o.storageClass = stringClone(string(info.StorageClass))
 		o.versionID = stringClonePointer(versionID)
 		// If is delete marker, show that metadata has been read as there is none to read
 		if info.Size == isDeleteMarker {
@@ -3594,9 +3710,7 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
 
 // Gets the bucket location
 func (f *Fs) getBucketLocation(ctx context.Context, bucket string) (string, error) {
-	region, err := s3manager.GetBucketRegion(ctx, f.ses, bucket, "", func(r *request.Request) {
-		r.Config.S3ForcePathStyle = aws.Bool(f.opt.ForcePathStyle)
-	})
+	region, err := manager.GetBucketRegion(ctx, f.c, bucket)
 	if err != nil {
 		return "", err
 	}
@@ -3610,22 +3724,21 @@ func (f *Fs) updateRegionForBucket(ctx context.Context, bucket string) error {
 	if err != nil {
 		return fmt.Errorf("reading bucket location failed: %w", err)
 	}
-	if aws.StringValue(f.c.Config.Endpoint) != "" {
+	if f.opt.Endpoint != "" {
 		return fmt.Errorf("can't set region to %q as endpoint is set", region)
 	}
-	if aws.StringValue(f.c.Config.Region) == region {
+	if f.opt.Region == region {
 		return fmt.Errorf("region is already %q - not updating", region)
 	}
 
 	// Make a new session with the new region
 	oldRegion := f.opt.Region
 	f.opt.Region = region
-	c, ses, err := s3Connection(f.ctx, &f.opt, f.srv)
+	c, err := s3Connection(f.ctx, &f.opt, f.srv)
 	if err != nil {
 		return fmt.Errorf("creating new session failed: %w", err)
 	}
 	f.c = c
-	f.ses = ses
 
 	fs.Logf(f, "Switched region to %q from %q", region, oldRegion)
 	return nil
@@ -3656,14 +3769,14 @@ func (f *Fs) newV1List(req *s3.ListObjectsV2Input) bucketLister {
 
 // List a bucket with V1 listing
 func (ls *v1List) List(ctx context.Context) (resp *s3.ListObjectsV2Output, versionIDs []*string, err error) {
-	respv1, err := ls.f.c.ListObjectsWithContext(ctx, &ls.req)
+	respv1, err := ls.f.c.ListObjects(ctx, &ls.req)
 	if err != nil {
 		return nil, nil, err
 	}
 
 	// Set up the request for next time
 	ls.req.Marker = respv1.NextMarker
-	if aws.BoolValue(respv1.IsTruncated) && ls.req.Marker == nil {
+	if deref(respv1.IsTruncated) && ls.req.Marker == nil {
 		if len(respv1.Contents) == 0 {
 			return nil, nil, errors.New("s3 protocol error: received listing v1 with IsTruncated set, no NextMarker and no Contents")
 		}
@@ -3673,7 +3786,7 @@ func (ls *v1List) List(ctx context.Context) (resp *s3.ListObjectsV2Output, versi
 	}
 
 	// If we are URL encoding then must decode the marker
-	if ls.req.Marker != nil && ls.req.EncodingType != nil {
+	if ls.req.Marker != nil && ls.req.EncodingType == types.EncodingTypeUrl {
 		*ls.req.Marker, err = url.QueryUnescape(*ls.req.Marker)
 		if err != nil {
 			return nil, nil, fmt.Errorf("failed to URL decode Marker %q: %w", *ls.req.Marker, err)
@@ -3691,9 +3804,9 @@ func (ls *v1List) List(ctx context.Context) (resp *s3.ListObjectsV2Output, versi
 // URL Encode the listings
 func (ls *v1List) URLEncodeListings(encode bool) {
 	if encode {
-		ls.req.EncodingType = aws.String(s3.EncodingTypeUrl)
+		ls.req.EncodingType = types.EncodingTypeUrl
 	} else {
-		ls.req.EncodingType = nil
+		ls.req.EncodingType = types.EncodingType("")
 	}
 }
 
@@ -3713,11 +3826,11 @@ func (f *Fs) newV2List(req *s3.ListObjectsV2Input) bucketLister {
 
 // Do a V2 listing
 func (ls *v2List) List(ctx context.Context) (resp *s3.ListObjectsV2Output, versionIDs []*string, err error) {
-	resp, err = ls.f.c.ListObjectsV2WithContext(ctx, &ls.req)
+	resp, err = ls.f.c.ListObjectsV2(ctx, &ls.req)
 	if err != nil {
 		return nil, nil, err
 	}
-	if aws.BoolValue(resp.IsTruncated) && (resp.NextContinuationToken == nil || *resp.NextContinuationToken == "") {
+	if deref(resp.IsTruncated) && (resp.NextContinuationToken == nil || *resp.NextContinuationToken == "") {
 		return nil, nil, errors.New("s3 protocol error: received listing v2 with IsTruncated set and no NextContinuationToken. Should you be using `--s3-list-version 1`?")
 	}
 	ls.req.ContinuationToken = resp.NextContinuationToken
@@ -3727,9 +3840,9 @@ func (ls *v2List) List(ctx context.Context) (resp *s3.ListObjectsV2Output, versi
 // URL Encode the listings
 func (ls *v2List) URLEncodeListings(encode bool) {
 	if encode {
-		ls.req.EncodingType = aws.String(s3.EncodingTypeUrl)
+		ls.req.EncodingType = types.EncodingTypeUrl
 	} else {
-		ls.req.EncodingType = nil
+		ls.req.EncodingType = types.EncodingType("")
 	}
 }
 
@@ -3757,13 +3870,13 @@ func (f *Fs) newVersionsList(req *s3.ListObjectsV2Input, hidden bool, versionAt
 	return l
 }
 
-// Any s3.Object or s3.ObjectVersion with this as their Size are delete markers
+// Any types.Object or types.ObjectVersion with this as their Size are delete markers
 var isDeleteMarker = new(int64)
 
-// Compare two s3.ObjectVersions, sorted alphabetically by key with
+// Compare two types.ObjectVersions, sorted alphabetically by key with
 // the newest first if the Keys match or the one with IsLatest set if
 // everything matches.
-func versionLess(a, b *s3.ObjectVersion) bool {
+func versionLess(a, b *types.ObjectVersion) bool {
 	if a == nil || a.Key == nil || a.LastModified == nil {
 		return true
 	}
@@ -3783,7 +3896,7 @@ func versionLess(a, b *s3.ObjectVersion) bool {
 	if dt < 0 {
 		return false
 	}
-	if aws.BoolValue(a.IsLatest) {
+	if deref(a.IsLatest) {
 		return true
 	}
 	return false
@@ -3796,17 +3909,17 @@ func versionLess(a, b *s3.ObjectVersion) bool {
 // so we need to merge them back again
 //
 // We do this by converting the s3.DeleteEntry into
-// s3.ObjectVersion with Size = isDeleteMarker to tell them apart
+// types.ObjectVersion with Size = isDeleteMarker to tell them apart
 //
 // We then merge them back into the Versions in the correct order
-func mergeDeleteMarkers(oldVersions []*s3.ObjectVersion, deleteMarkers []*s3.DeleteMarkerEntry) (newVersions []*s3.ObjectVersion) {
-	newVersions = make([]*s3.ObjectVersion, 0, len(oldVersions)+len(deleteMarkers))
+func mergeDeleteMarkers(oldVersions []types.ObjectVersion, deleteMarkers []types.DeleteMarkerEntry) (newVersions []types.ObjectVersion) {
+	newVersions = make([]types.ObjectVersion, 0, len(oldVersions)+len(deleteMarkers))
 	for _, deleteMarker := range deleteMarkers {
-		var obj = new(s3.ObjectVersion)
+		var obj types.ObjectVersion
 		//structs.SetFrom(obj, deleteMarker)
-		setFrom_s3ObjectVersion_s3DeleteMarkerEntry(obj, deleteMarker)
+		setFrom_typesObjectVersion_typesDeleteMarkerEntry(&obj, &deleteMarker)
 		obj.Size = isDeleteMarker
-		for len(oldVersions) > 0 && versionLess(oldVersions[0], obj) {
+		for len(oldVersions) > 0 && versionLess(&oldVersions[0], &obj) {
 			newVersions = append(newVersions, oldVersions[0])
 			oldVersions = oldVersions[1:]
 		}
@@ -3819,7 +3932,7 @@ func mergeDeleteMarkers(oldVersions []*s3.ObjectVersion, deleteMarkers []*s3.Del
 
 // List a bucket with versions
 func (ls *versionsList) List(ctx context.Context) (resp *s3.ListObjectsV2Output, versionIDs []*string, err error) {
-	respVersions, err := ls.f.c.ListObjectVersionsWithContext(ctx, &ls.req)
+	respVersions, err := ls.f.c.ListObjectVersions(ctx, &ls.req)
 	if err != nil {
 		return nil, nil, err
 	}
@@ -3827,12 +3940,12 @@ func (ls *versionsList) List(ctx context.Context) (resp *s3.ListObjectsV2Output,
 	// Set up the request for next time
 	ls.req.KeyMarker = respVersions.NextKeyMarker
 	ls.req.VersionIdMarker = respVersions.NextVersionIdMarker
-	if aws.BoolValue(respVersions.IsTruncated) && ls.req.KeyMarker == nil {
+	if deref(respVersions.IsTruncated) && ls.req.KeyMarker == nil {
 		return nil, nil, errors.New("s3 protocol error: received versions listing with IsTruncated set with no NextKeyMarker")
 	}
 
 	// If we are URL encoding then must decode the marker
-	if ls.req.KeyMarker != nil && ls.req.EncodingType != nil {
+	if ls.req.KeyMarker != nil && ls.req.EncodingType == types.EncodingTypeUrl {
 		*ls.req.KeyMarker, err = url.QueryUnescape(*ls.req.KeyMarker)
 		if err != nil {
 			return nil, nil, fmt.Errorf("failed to URL decode KeyMarker %q: %w", *ls.req.KeyMarker, err)
@@ -3844,16 +3957,16 @@ func (ls *versionsList) List(ctx context.Context) (resp *s3.ListObjectsV2Output,
 	//structs.SetFrom(resp, respVersions)
 	setFrom_s3ListObjectsV2Output_s3ListObjectVersionsOutput(resp, respVersions)
 
-	// Merge in delete Markers as s3.ObjectVersion if we need them
+	// Merge in delete Markers as types.ObjectVersion if we need them
 	if ls.hidden || ls.usingVersionAt {
 		respVersions.Versions = mergeDeleteMarkers(respVersions.Versions, respVersions.DeleteMarkers)
 	}
 
-	// Convert the Versions and the DeleteMarkers into an array of s3.Object
+	// Convert the Versions and the DeleteMarkers into an array of types.Object
 	//
 	// These are returned in the order that they are stored with the most recent first.
 	// With the annoyance that the Versions and DeleteMarkers are split into two
-	objs := make([]*s3.Object, 0, len(respVersions.Versions))
+	objs := make([]types.Object, 0, len(respVersions.Versions))
 	for _, objVersion := range respVersions.Versions {
 		if ls.usingVersionAt {
 			if objVersion.LastModified.After(ls.versionAt) {
@@ -3870,11 +3983,11 @@ func (ls *versionsList) List(ctx context.Context) (resp *s3.ListObjectsV2Output,
 		if !ls.hidden && objVersion.Size == isDeleteMarker {
 			continue
 		}
-		var obj = new(s3.Object)
+		var obj types.Object
 		//structs.SetFrom(obj, objVersion)
-		setFrom_s3Object_s3ObjectVersion(obj, objVersion)
+		setFrom_typesObject_typesObjectVersion(&obj, &objVersion)
 		// Adjust the file names
-		if !ls.usingVersionAt && (!aws.BoolValue(objVersion.IsLatest) || objVersion.Size == isDeleteMarker) {
+		if !ls.usingVersionAt && (!deref(objVersion.IsLatest) || objVersion.Size == isDeleteMarker) {
 			if obj.Key != nil && objVersion.LastModified != nil {
 				*obj.Key = version.Add(*obj.Key, *objVersion.LastModified)
 			}
@@ -3890,14 +4003,14 @@ func (ls *versionsList) List(ctx context.Context) (resp *s3.ListObjectsV2Output,
 // URL Encode the listings
 func (ls *versionsList) URLEncodeListings(encode bool) {
 	if encode {
-		ls.req.EncodingType = aws.String(s3.EncodingTypeUrl)
+		ls.req.EncodingType = types.EncodingTypeUrl
 	} else {
-		ls.req.EncodingType = nil
+		ls.req.EncodingType = types.EncodingType("")
 	}
 }
 
 // listFn is called from list to handle an object.
-type listFn func(remote string, object *s3.Object, versionID *string, isDirectory bool) error
+type listFn func(remote string, object *types.Object, versionID *string, isDirectory bool) error
 
 // errEndList is a sentinel used to end the list iteration now.
 // listFn should return it to end the iteration with no errors.
@@ -3956,11 +4069,10 @@ func (f *Fs) list(ctx context.Context, opt listOpt, fn listFn) error {
 		MaxKeys:   &f.opt.ListChunk,
 	}
 	if opt.restoreStatus {
-		restoreStatus := "RestoreStatus"
-		req.OptionalObjectAttributes = []*string{&restoreStatus}
+		req.OptionalObjectAttributes = []types.OptionalObjectAttributes{types.OptionalObjectAttributesRestoreStatus}
 	}
 	if f.opt.RequesterPays {
-		req.RequestPayer = aws.String(s3.RequestPayerRequester)
+		req.RequestPayer = types.RequestPayerRequester
 	}
 	var listBucket bucketLister
 	switch {
@@ -3977,37 +4089,31 @@ func (f *Fs) list(ctx context.Context, opt listOpt, fn listFn) error {
 		var err error
 		var versionIDs []*string
 		err = f.pacer.Call(func() (bool, error) {
+
 			listBucket.URLEncodeListings(urlEncodeListings)
 			resp, versionIDs, err = listBucket.List(ctx)
 			if err != nil && !urlEncodeListings {
-				if awsErr, ok := err.(awserr.RequestFailure); ok {
-					if origErr := awsErr.OrigErr(); origErr != nil {
-						if _, ok := origErr.(*xml.SyntaxError); ok {
-							// Retry the listing with URL encoding as there were characters that XML can't encode
-							urlEncodeListings = true
-							fs.Debugf(f, "Retrying listing because of characters which can't be XML encoded")
-							return true, err
-						}
-					}
+				var xmlErr *xml.SyntaxError
+				if errors.As(err, &xmlErr) {
+					// Retry the listing with URL encoding as there were characters that XML can't encode
+					urlEncodeListings = true
+					fs.Debugf(f, "Retrying listing because of characters which can't be XML encoded")
+					return true, err
 				}
 			}
 			return f.shouldRetry(ctx, err)
 		})
 		if err != nil {
-			if awsErr, ok := err.(awserr.RequestFailure); ok {
-				if awsErr.StatusCode() == http.StatusNotFound {
-					err = fs.ErrorDirNotFound
-				}
+			if getHTTPStatusCode(err) == http.StatusNotFound {
+				err = fs.ErrorDirNotFound
 			}
 			if f.rootBucket == "" {
 				// if listing from the root ignore wrong region requests returning
 				// empty directory
-				if reqErr, ok := err.(awserr.RequestFailure); ok {
-					// 301 if wrong region for bucket
-					if reqErr.StatusCode() == http.StatusMovedPermanently {
-						fs.Errorf(f, "Can't change region for bucket %q with no bucket specified", opt.bucket)
-						return nil
-					}
+				// 301 if wrong region for bucket
+				if getHTTPStatusCode(err) == http.StatusMovedPermanently {
+					fs.Errorf(f, "Can't change region for bucket %q with no bucket specified", opt.bucket)
+					return nil
 				}
 			}
 			return err
@@ -4037,7 +4143,7 @@ func (f *Fs) list(ctx context.Context, opt listOpt, fn listFn) error {
 					remote = bucket.Join(opt.bucket, remote)
 				}
 				remote = strings.TrimSuffix(remote, "/")
-				err = fn(remote, &s3.Object{Key: &remote}, nil, true)
+				err = fn(remote, &types.Object{Key: &remote}, nil, true)
 				if err != nil {
 					if err == errEndList {
 						return nil
@@ -4048,11 +4154,11 @@ func (f *Fs) list(ctx context.Context, opt listOpt, fn listFn) error {
 		}
 		foundItems += len(resp.Contents)
 		for i, object := range resp.Contents {
-			remote := aws.StringValue(object.Key)
+			remote := deref(object.Key)
 			if urlEncodeListings {
 				remote, err = url.QueryUnescape(remote)
 				if err != nil {
-					fs.Logf(f, "failed to URL decode %q in listing: %v", aws.StringValue(object.Key), err)
+					fs.Logf(f, "failed to URL decode %q in listing: %v", deref(object.Key), err)
 					continue
 				}
 			}
@@ -4083,9 +4189,9 @@ func (f *Fs) list(ctx context.Context, opt listOpt, fn listFn) error {
 				remote = bucket.Join(opt.bucket, remote)
 			}
 			if versionIDs != nil {
-				err = fn(remote, object, versionIDs[i], isDirectory)
+				err = fn(remote, &object, versionIDs[i], isDirectory)
 			} else {
-				err = fn(remote, object, nil, isDirectory)
+				err = fn(remote, &object, nil, isDirectory)
 			}
 			if err != nil {
 				if err == errEndList {
@@ -4094,7 +4200,7 @@ func (f *Fs) list(ctx context.Context, opt listOpt, fn listFn) error {
 				return err
 			}
 		}
-		if !aws.BoolValue(resp.IsTruncated) {
+		if !deref(resp.IsTruncated) {
 			break
 		}
 	}
@@ -4116,7 +4222,7 @@ func (f *Fs) list(ctx context.Context, opt listOpt, fn listFn) error {
 }
 
 // Convert a list item into a DirEntry
-func (f *Fs) itemToDirEntry(ctx context.Context, remote string, object *s3.Object, versionID *string, isDirectory bool) (fs.DirEntry, error) {
+func (f *Fs) itemToDirEntry(ctx context.Context, remote string, object *types.Object, versionID *string, isDirectory bool) (fs.DirEntry, error) {
 	if isDirectory {
 		size := int64(0)
 		if object.Size != nil {
@@ -4143,7 +4249,7 @@ func (f *Fs) listDir(ctx context.Context, bucket, directory, prefix string, addB
 		withVersions: f.opt.Versions,
 		versionAt:    f.opt.VersionAt,
 		hidden:       f.opt.VersionDeleted,
-	}, func(remote string, object *s3.Object, versionID *string, isDirectory bool) error {
+	}, func(remote string, object *types.Object, versionID *string, isDirectory bool) error {
 		entry, err := f.itemToDirEntry(ctx, remote, object, versionID, isDirectory)
 		if err != nil {
 			return err
@@ -4166,16 +4272,16 @@ func (f *Fs) listBuckets(ctx context.Context) (entries fs.DirEntries, err error)
 	req := s3.ListBucketsInput{}
 	var resp *s3.ListBucketsOutput
 	err = f.pacer.Call(func() (bool, error) {
-		resp, err = f.c.ListBucketsWithContext(ctx, &req)
+		resp, err = f.c.ListBuckets(ctx, &req)
 		return f.shouldRetry(ctx, err)
 	})
 	if err != nil {
 		return nil, err
 	}
 	for _, bucket := range resp.Buckets {
-		bucketName := f.opt.Enc.ToStandardName(aws.StringValue(bucket.Name))
+		bucketName := f.opt.Enc.ToStandardName(deref(bucket.Name))
 		f.cache.MarkOK(bucketName)
-		d := fs.NewDir(bucketName, aws.TimeValue(bucket.CreationDate))
+		d := fs.NewDir(bucketName, deref(bucket.CreationDate))
 		entries = append(entries, d)
 	}
 	return entries, nil
@@ -4230,7 +4336,7 @@ func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (
 			withVersions: f.opt.Versions,
 			versionAt:    f.opt.VersionAt,
 			hidden:       f.opt.VersionDeleted,
-		}, func(remote string, object *s3.Object, versionID *string, isDirectory bool) error {
+		}, func(remote string, object *types.Object, versionID *string, isDirectory bool) error {
 			entry, err := f.itemToDirEntry(ctx, remote, object, versionID, isDirectory)
 			if err != nil {
 				return err
@@ -4290,16 +4396,14 @@ func (f *Fs) bucketExists(ctx context.Context, bucket string) (bool, error) {
 		Bucket: &bucket,
 	}
 	err := f.pacer.Call(func() (bool, error) {
-		_, err := f.c.HeadBucketWithContext(ctx, &req)
+		_, err := f.c.HeadBucket(ctx, &req)
 		return f.shouldRetry(ctx, err)
 	})
 	if err == nil {
 		return true, nil
 	}
-	if err, ok := err.(awserr.RequestFailure); ok {
-		if err.StatusCode() == http.StatusNotFound {
-			return false, nil
-		}
+	if getHTTPStatusCode(err) == http.StatusNotFound {
+		return false, nil
 	}
 	return false, err
 }
@@ -4378,22 +4482,23 @@ func (f *Fs) makeBucket(ctx context.Context, bucket string) error {
 	return f.cache.Create(bucket, func() error {
 		req := s3.CreateBucketInput{
 			Bucket: &bucket,
-			ACL:    stringPointerOrNil(f.opt.BucketACL),
+			ACL:    types.BucketCannedACL(f.opt.BucketACL),
 		}
 		if f.opt.LocationConstraint != "" {
-			req.CreateBucketConfiguration = &s3.CreateBucketConfiguration{
-				LocationConstraint: &f.opt.LocationConstraint,
+			req.CreateBucketConfiguration = &types.CreateBucketConfiguration{
+				LocationConstraint: types.BucketLocationConstraint(f.opt.LocationConstraint),
 			}
 		}
 		err := f.pacer.Call(func() (bool, error) {
-			_, err := f.c.CreateBucketWithContext(ctx, &req)
+			_, err := f.c.CreateBucket(ctx, &req)
 			return f.shouldRetry(ctx, err)
 		})
 		if err == nil {
 			fs.Infof(f, "Bucket %q created with ACL %q", bucket, f.opt.BucketACL)
 		}
-		if awsErr, ok := err.(awserr.Error); ok {
-			switch awsErr.Code() {
+		var awsErr smithy.APIError
+		if errors.As(err, &awsErr) {
+			switch awsErr.ErrorCode() {
 			case "BucketAlreadyOwnedByYou":
 				err = nil
 			case "BucketAlreadyExists", "BucketNameUnavailable":
@@ -4437,7 +4542,7 @@ func (f *Fs) Rmdir(ctx context.Context, dir string) error {
 			Bucket: &bucket,
 		}
 		err := f.pacer.Call(func() (bool, error) {
-			_, err := f.c.DeleteBucketWithContext(ctx, &req)
+			_, err := f.c.DeleteBucket(ctx, &req)
 			return f.shouldRetry(ctx, err)
 		})
 		if err == nil {
@@ -4464,7 +4569,7 @@ func pathEscape(s string) string {
 // method
 func (f *Fs) copy(ctx context.Context, req *s3.CopyObjectInput, dstBucket, dstPath, srcBucket, srcPath string, src *Object) error {
 	req.Bucket = &dstBucket
-	req.ACL = stringPointerOrNil(f.opt.ACL)
+	req.ACL = types.ObjectCannedACL(f.opt.ACL)
 	req.Key = &dstPath
 	source := pathEscape(bucket.Join(srcBucket, srcPath))
 	if src.versionID != nil {
@@ -4472,18 +4577,18 @@ func (f *Fs) copy(ctx context.Context, req *s3.CopyObjectInput, dstBucket, dstPa
 	}
 	req.CopySource = &source
 	if f.opt.RequesterPays {
-		req.RequestPayer = aws.String(s3.RequestPayerRequester)
+		req.RequestPayer = types.RequestPayerRequester
 	}
 	if f.opt.ServerSideEncryption != "" {
-		req.ServerSideEncryption = &f.opt.ServerSideEncryption
+		req.ServerSideEncryption = types.ServerSideEncryption(f.opt.ServerSideEncryption)
 	}
 	if f.opt.SSECustomerAlgorithm != "" {
 		req.SSECustomerAlgorithm = &f.opt.SSECustomerAlgorithm
 		req.CopySourceSSECustomerAlgorithm = &f.opt.SSECustomerAlgorithm
 	}
-	if f.opt.SSECustomerKey != "" {
-		req.SSECustomerKey = &f.opt.SSECustomerKey
-		req.CopySourceSSECustomerKey = &f.opt.SSECustomerKey
+	if f.opt.SSECustomerKeyBase64 != "" {
+		req.SSECustomerKey = &f.opt.SSECustomerKeyBase64
+		req.CopySourceSSECustomerKey = &f.opt.SSECustomerKeyBase64
 	}
 	if f.opt.SSECustomerKeyMD5 != "" {
 		req.SSECustomerKeyMD5 = &f.opt.SSECustomerKeyMD5
@@ -4492,15 +4597,15 @@ func (f *Fs) copy(ctx context.Context, req *s3.CopyObjectInput, dstBucket, dstPa
 	if f.opt.SSEKMSKeyID != "" {
 		req.SSEKMSKeyId = &f.opt.SSEKMSKeyID
 	}
-	if req.StorageClass == nil && f.opt.StorageClass != "" {
-		req.StorageClass = &f.opt.StorageClass
+	if req.StorageClass == types.StorageClass("") && f.opt.StorageClass != "" {
+		req.StorageClass = types.StorageClass(f.opt.StorageClass)
 	}
 
 	if src.bytes >= int64(f.opt.CopyCutoff) {
 		return f.copyMultipart(ctx, req, dstBucket, dstPath, srcBucket, srcPath, src)
 	}
 	return f.pacer.Call(func() (bool, error) {
-		_, err := f.c.CopyObjectWithContext(ctx, req)
+		_, err := f.c.CopyObject(ctx, req)
 		return f.shouldRetry(ctx, err)
 	})
 }
@@ -4532,7 +4637,7 @@ func (f *Fs) copyMultipart(ctx context.Context, copyReq *s3.CopyObjectInput, dst
 
 	// If copy metadata was set then set the Metadata to that read
 	// from the head request
-	if aws.StringValue(copyReq.MetadataDirective) == s3.MetadataDirectiveCopy {
+	if copyReq.MetadataDirective == types.MetadataDirectiveCopy {
 		copyReq.Metadata = info.Metadata
 	}
 
@@ -4546,7 +4651,7 @@ func (f *Fs) copyMultipart(ctx context.Context, copyReq *s3.CopyObjectInput, dst
 	var cout *s3.CreateMultipartUploadOutput
 	if err := f.pacer.Call(func() (bool, error) {
 		var err error
-		cout, err = f.c.CreateMultipartUploadWithContext(ctx, req)
+		cout, err = f.c.CreateMultipartUpload(ctx, req)
 		return f.shouldRetry(ctx, err)
 	}); err != nil {
 		return err
@@ -4557,7 +4662,7 @@ func (f *Fs) copyMultipart(ctx context.Context, copyReq *s3.CopyObjectInput, dst
 		// Try to abort the upload, but ignore the error.
 		fs.Debugf(src, "Cancelling multipart copy")
 		_ = f.pacer.Call(func() (bool, error) {
-			_, err := f.c.AbortMultipartUploadWithContext(context.Background(), &s3.AbortMultipartUploadInput{
+			_, err := f.c.AbortMultipartUpload(context.Background(), &s3.AbortMultipartUploadInput{
 				Bucket:       &dstBucket,
 				Key:          &dstPath,
 				UploadId:     uid,
@@ -4574,11 +4679,11 @@ func (f *Fs) copyMultipart(ctx context.Context, copyReq *s3.CopyObjectInput, dst
 	fs.Debugf(src, "Starting  multipart copy with %d parts", numParts)
 
 	var (
-		parts   = make([]*s3.CompletedPart, numParts)
+		parts   = make([]types.CompletedPart, numParts)
 		g, gCtx = errgroup.WithContext(ctx)
 	)
 	g.SetLimit(f.opt.UploadConcurrency)
-	for partNum := int64(1); partNum <= numParts; partNum++ {
+	for partNum := int32(1); int64(partNum) <= numParts; partNum++ {
 		// Fail fast, in case an errgroup managed function returns an error
 		// gCtx is cancelled. There is no point in uploading all the other parts.
 		if gCtx.Err() != nil {
@@ -4594,15 +4699,15 @@ func (f *Fs) copyMultipart(ctx context.Context, copyReq *s3.CopyObjectInput, dst
 			uploadPartReq.Key = &dstPath
 			uploadPartReq.PartNumber = &partNum
 			uploadPartReq.UploadId = uid
-			uploadPartReq.CopySourceRange = aws.String(calculateRange(partSize, partNum-1, numParts, srcSize))
+			uploadPartReq.CopySourceRange = aws.String(calculateRange(partSize, int64(partNum-1), numParts, srcSize))
 			err := f.pacer.Call(func() (bool, error) {
-				uout, err = f.c.UploadPartCopyWithContext(gCtx, uploadPartReq)
+				uout, err = f.c.UploadPartCopy(gCtx, uploadPartReq)
 				return f.shouldRetry(gCtx, err)
 			})
 			if err != nil {
 				return err
 			}
-			parts[partNum-1] = &s3.CompletedPart{
+			parts[partNum-1] = types.CompletedPart{
 				PartNumber: &partNum,
 				ETag:       uout.CopyPartResult.ETag,
 			}
@@ -4616,10 +4721,10 @@ func (f *Fs) copyMultipart(ctx context.Context, copyReq *s3.CopyObjectInput, dst
 	}
 
 	return f.pacer.Call(func() (bool, error) {
-		_, err := f.c.CompleteMultipartUploadWithContext(ctx, &s3.CompleteMultipartUploadInput{
+		_, err := f.c.CompleteMultipartUpload(ctx, &s3.CompleteMultipartUploadInput{
 			Bucket: &dstBucket,
 			Key:    &dstPath,
-			MultipartUpload: &s3.CompletedMultipartUpload{
+			MultipartUpload: &types.CompletedMultipartUpload{
 				Parts: parts,
 			},
 			RequestPayer: req.RequestPayer,
@@ -4655,7 +4760,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
 
 	srcBucket, srcPath := srcObj.split()
 	req := s3.CopyObjectInput{
-		MetadataDirective: aws.String(s3.MetadataDirectiveCopy),
+		MetadataDirective: types.MetadataDirectiveCopy,
 	}
 
 	// Update the metadata if it is in use
@@ -4665,7 +4770,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
 			return nil, fmt.Errorf("failed to prepare upload: %w", err)
 		}
 		setFrom_s3CopyObjectInput_s3PutObjectInput(&req, ui.req)
-		req.MetadataDirective = aws.String(s3.MetadataDirectiveReplace)
+		req.MetadataDirective = types.MetadataDirectiveReplace
 	}
 
 	err = f.copy(ctx, &req, dstBucket, dstPath, srcBucket, srcPath, srcObj)
@@ -4694,14 +4799,16 @@ func (f *Fs) PublicLink(ctx context.Context, remote string, expire fs.Duration,
 		fs.Logf(f, "Public Link: Reducing expiry to %v as %v is greater than the max time allowed", maxExpireDuration, expire)
 		expire = maxExpireDuration
 	}
-	bucket, bucketPath := o.split()
-	httpReq, _ := f.c.GetObjectRequest(&s3.GetObjectInput{
+	bucket, bucketPath := f.split(remote)
+	httpReq, err := s3.NewPresignClient(f.c).PresignGetObject(ctx, &s3.GetObjectInput{
 		Bucket:    &bucket,
 		Key:       &bucketPath,
 		VersionId: o.versionID,
-	})
-
-	return httpReq.Presign(time.Duration(expire))
+	}, s3.WithPresignExpires(time.Duration(expire)))
+	if err != nil {
+		return "", err
+	}
+	return httpReq.URL, nil
 }
 
 var commandHelp = []fs.CommandHelp{{
@@ -4899,18 +5006,19 @@ func (f *Fs) Command(ctx context.Context, name string, arg []string, opt map[str
 		req := s3.RestoreObjectInput{
 			//Bucket:         &f.rootBucket,
 			//Key:            &encodedDirectory,
-			RestoreRequest: &s3.RestoreRequest{},
+			RestoreRequest: &types.RestoreRequest{},
 		}
 		if lifetime := opt["lifetime"]; lifetime != "" {
 			ilifetime, err := strconv.ParseInt(lifetime, 10, 64)
+			ilifetime32 := int32(ilifetime)
 			if err != nil {
 				return nil, fmt.Errorf("bad lifetime: %w", err)
 			}
-			req.RestoreRequest.Days = &ilifetime
+			req.RestoreRequest.Days = &ilifetime32
 		}
 		if priority := opt["priority"]; priority != "" {
-			req.RestoreRequest.GlacierJobParameters = &s3.GlacierJobParameters{
-				Tier: &priority,
+			req.RestoreRequest.GlacierJobParameters = &types.GlacierJobParameters{
+				Tier: types.Tier(priority),
 			}
 		}
 		if description := opt["description"]; description != "" {
@@ -4950,7 +5058,7 @@ func (f *Fs) Command(ctx context.Context, name string, arg []string, opt map[str
 			reqCopy.Key = &bucketPath
 			reqCopy.VersionId = o.versionID
 			err = f.pacer.Call(func() (bool, error) {
-				_, err = f.c.RestoreObject(&reqCopy)
+				_, err = f.c.RestoreObject(ctx, &reqCopy)
 				return f.shouldRetry(ctx, err)
 			})
 			if err != nil {
@@ -4985,12 +5093,11 @@ func (f *Fs) Command(ctx context.Context, name string, arg []string, opt map[str
 		if err != nil {
 			return nil, fmt.Errorf("reading config: %w", err)
 		}
-		c, ses, err := s3Connection(f.ctx, &newOpt, f.srv)
+		c, err := s3Connection(f.ctx, &newOpt, f.srv)
 		if err != nil {
 			return nil, fmt.Errorf("updating session: %w", err)
 		}
 		f.c = c
-		f.ses = ses
 		f.opt = newOpt
 		keys := []string{}
 		for k := range opt {
@@ -5007,8 +5114,8 @@ func (f *Fs) Command(ctx context.Context, name string, arg []string, opt map[str
 type restoreStatusOut struct {
 	Remote        string
 	VersionID     *string
-	RestoreStatus *s3.RestoreStatus
-	StorageClass  *string
+	RestoreStatus *types.RestoreStatus
+	StorageClass  types.ObjectStorageClass
 }
 
 // Recursively enumerate the current fs to find objects with a restore status
@@ -5026,7 +5133,7 @@ func (f *Fs) restoreStatus(ctx context.Context, all bool) (out []restoreStatusOu
 		versionAt:     f.opt.VersionAt,
 		hidden:        f.opt.VersionDeleted,
 		restoreStatus: true,
-	}, func(remote string, object *s3.Object, versionID *string, isDirectory bool) error {
+	}, func(remote string, object *types.Object, versionID *string, isDirectory bool) error {
 		entry, err := f.itemToDirEntry(ctx, remote, object, versionID, isDirectory)
 		if err != nil {
 			return err
@@ -5056,12 +5163,12 @@ func (f *Fs) restoreStatus(ctx context.Context, all bool) (out []restoreStatusOu
 // Note that rather lazily we treat key as a prefix so it matches
 // directories and objects. This could surprise the user if they ask
 // for "dir" and it returns "dirKey"
-func (f *Fs) listMultipartUploads(ctx context.Context, bucket, key string) (uploads []*s3.MultipartUpload, err error) {
+func (f *Fs) listMultipartUploads(ctx context.Context, bucket, key string) (uploads []types.MultipartUpload, err error) {
 	var (
 		keyMarker      *string
 		uploadIDMarker *string
 	)
-	uploads = []*s3.MultipartUpload{}
+	uploads = []types.MultipartUpload{}
 	for {
 		req := s3.ListMultipartUploadsInput{
 			Bucket:         &bucket,
@@ -5072,14 +5179,14 @@ func (f *Fs) listMultipartUploads(ctx context.Context, bucket, key string) (uplo
 		}
 		var resp *s3.ListMultipartUploadsOutput
 		err = f.pacer.Call(func() (bool, error) {
-			resp, err = f.c.ListMultipartUploads(&req)
+			resp, err = f.c.ListMultipartUploads(ctx, &req)
 			return f.shouldRetry(ctx, err)
 		})
 		if err != nil {
 			return nil, fmt.Errorf("list multipart uploads bucket %q key %q: %w", bucket, key, err)
 		}
 		uploads = append(uploads, resp.Uploads...)
-		if !aws.BoolValue(resp.IsTruncated) {
+		if !deref(resp.IsTruncated) {
 			break
 		}
 		keyMarker = resp.NextKeyMarker
@@ -5088,8 +5195,8 @@ func (f *Fs) listMultipartUploads(ctx context.Context, bucket, key string) (uplo
 	return uploads, nil
 }
 
-func (f *Fs) listMultipartUploadsAll(ctx context.Context) (uploadsMap map[string][]*s3.MultipartUpload, err error) {
-	uploadsMap = make(map[string][]*s3.MultipartUpload)
+func (f *Fs) listMultipartUploadsAll(ctx context.Context) (uploadsMap map[string][]types.MultipartUpload, err error) {
+	uploadsMap = make(map[string][]types.MultipartUpload)
 	bucket, directory := f.split("")
 	if bucket != "" {
 		uploads, err := f.listMultipartUploads(ctx, bucket, directory)
@@ -5116,7 +5223,7 @@ func (f *Fs) listMultipartUploadsAll(ctx context.Context) (uploadsMap map[string
 }
 
 // cleanUpBucket removes all pending multipart uploads for a given bucket over the age of maxAge
-func (f *Fs) cleanUpBucket(ctx context.Context, bucket string, maxAge time.Duration, uploads []*s3.MultipartUpload) (err error) {
+func (f *Fs) cleanUpBucket(ctx context.Context, bucket string, maxAge time.Duration, uploads []types.MultipartUpload) (err error) {
 	fs.Infof(f, "cleaning bucket %q of pending multipart uploads older than %v", bucket, maxAge)
 	for _, upload := range uploads {
 		if upload.Initiated != nil && upload.Key != nil && upload.UploadId != nil {
@@ -5132,7 +5239,7 @@ func (f *Fs) cleanUpBucket(ctx context.Context, bucket string, maxAge time.Durat
 					UploadId: upload.UploadId,
 					Key:      upload.Key,
 				}
-				_, abortErr := f.c.AbortMultipartUpload(&req)
+				_, abortErr := f.c.AbortMultipartUpload(ctx, &req)
 				if abortErr != nil {
 					err = fmt.Errorf("failed to remove %s: %w", what, abortErr)
 					fs.Errorf(f, "%v", err)
@@ -5177,7 +5284,7 @@ func (f *Fs) isVersioned(ctx context.Context) bool {
 // Pass no arguments to get, or pass "Enabled" or "Suspended"
 //
 // Updates f.versioning
-func (f *Fs) setGetVersioning(ctx context.Context, arg ...string) (status string, err error) {
+func (f *Fs) setGetVersioning(ctx context.Context, arg ...string) (status types.BucketVersioningStatus, err error) {
 	if len(arg) > 1 {
 		return "", errors.New("too many arguments")
 	}
@@ -5185,19 +5292,19 @@ func (f *Fs) setGetVersioning(ctx context.Context, arg ...string) (status string
 		return "", errors.New("need a bucket")
 	}
 	if len(arg) == 1 {
-		var versioning = s3.VersioningConfiguration{
-			Status: aws.String(arg[0]),
+		var versioning = types.VersioningConfiguration{
+			Status: types.BucketVersioningStatus(arg[0]),
 		}
 		// Disabled is indicated by the parameter missing
-		if *versioning.Status == "Disabled" {
-			versioning.Status = aws.String("")
+		if versioning.Status == types.BucketVersioningStatus("Disabled") {
+			versioning.Status = types.BucketVersioningStatus("")
 		}
 		req := s3.PutBucketVersioningInput{
 			Bucket:                  &f.rootBucket,
 			VersioningConfiguration: &versioning,
 		}
 		err := f.pacer.Call(func() (bool, error) {
-			_, err = f.c.PutBucketVersioningWithContext(ctx, &req)
+			_, err = f.c.PutBucketVersioning(ctx, &req)
 			return f.shouldRetry(ctx, err)
 		})
 		if err != nil {
@@ -5209,7 +5316,7 @@ func (f *Fs) setGetVersioning(ctx context.Context, arg ...string) (status string
 	}
 	var resp *s3.GetBucketVersioningOutput
 	err = f.pacer.Call(func() (bool, error) {
-		resp, err = f.c.GetBucketVersioningWithContext(ctx, &req)
+		resp, err = f.c.GetBucketVersioning(ctx, &req)
 		return f.shouldRetry(ctx, err)
 	})
 	f.versioning.Valid = true
@@ -5218,11 +5325,11 @@ func (f *Fs) setGetVersioning(ctx context.Context, arg ...string) (status string
 		fs.Errorf(f, "Failed to read versioning status, assuming unversioned: %v", err)
 		return "", err
 	}
-	if resp.Status == nil {
+	if len(resp.Status) == 0 {
 		return "Unversioned", err
 	}
 	f.versioning.Value = true
-	return *resp.Status, err
+	return resp.Status, err
 }
 
 // CleanUp removes all pending multipart uploads older than 24 hours
@@ -5276,7 +5383,7 @@ func (f *Fs) purge(ctx context.Context, dir string, oldOnly bool) error {
 		withVersions:  versioned,
 		hidden:        true,
 		noSkipMarkers: true,
-	}, func(remote string, object *s3.Object, versionID *string, isDirectory bool) error {
+	}, func(remote string, object *types.Object, versionID *string, isDirectory bool) error {
 		if isDirectory {
 			return nil
 		}
@@ -5296,16 +5403,16 @@ func (f *Fs) purge(ctx context.Context, dir string, oldOnly bool) error {
 		if oldOnly && isCurrentVersion {
 			// Check current version of the file
 			if object.Size == isDeleteMarker {
-				fs.Debugf(remote, "Deleting current version (id %q) as it is a delete marker", aws.StringValue(versionID))
+				fs.Debugf(remote, "Deleting current version (id %q) as it is a delete marker", deref(versionID))
 				delChan <- oi
 			} else {
-				fs.Debugf(remote, "Not deleting current version %q", aws.StringValue(versionID))
+				fs.Debugf(remote, "Not deleting current version %q", deref(versionID))
 			}
 		} else {
 			if object.Size == isDeleteMarker {
-				fs.Debugf(remote, "Deleting delete marker (id %q)", aws.StringValue(versionID))
+				fs.Debugf(remote, "Deleting delete marker (id %q)", deref(versionID))
 			} else {
-				fs.Debugf(remote, "Deleting (id %q)", aws.StringValue(versionID))
+				fs.Debugf(remote, "Deleting (id %q)", deref(versionID))
 			}
 			delChan <- oi
 		}
@@ -5408,27 +5515,25 @@ func (o *Object) headObject(ctx context.Context) (resp *s3.HeadObjectOutput, err
 
 func (f *Fs) headObject(ctx context.Context, req *s3.HeadObjectInput) (resp *s3.HeadObjectOutput, err error) {
 	if f.opt.RequesterPays {
-		req.RequestPayer = aws.String(s3.RequestPayerRequester)
+		req.RequestPayer = types.RequestPayerRequester
 	}
 	if f.opt.SSECustomerAlgorithm != "" {
 		req.SSECustomerAlgorithm = &f.opt.SSECustomerAlgorithm
 	}
-	if f.opt.SSECustomerKey != "" {
-		req.SSECustomerKey = &f.opt.SSECustomerKey
+	if f.opt.SSECustomerKeyBase64 != "" {
+		req.SSECustomerKey = &f.opt.SSECustomerKeyBase64
 	}
 	if f.opt.SSECustomerKeyMD5 != "" {
 		req.SSECustomerKeyMD5 = &f.opt.SSECustomerKeyMD5
 	}
 	err = f.pacer.Call(func() (bool, error) {
 		var err error
-		resp, err = f.c.HeadObjectWithContext(ctx, req)
+		resp, err = f.c.HeadObject(ctx, req)
 		return f.shouldRetry(ctx, err)
 	})
 	if err != nil {
-		if awsErr, ok := err.(awserr.RequestFailure); ok {
-			if awsErr.StatusCode() == http.StatusNotFound || awsErr.StatusCode() == http.StatusMethodNotAllowed {
-				return nil, fs.ErrorObjectNotFound
-			}
+		if statusCode := getHTTPStatusCode(err); statusCode == http.StatusNotFound || statusCode == http.StatusMethodNotAllowed {
+			return nil, fs.ErrorObjectNotFound
 		}
 		return nil, err
 	}
@@ -5454,25 +5559,19 @@ func (o *Object) readMetaData(ctx context.Context) (err error) {
 	return nil
 }
 
-// Convert S3 metadata with pointers into a map[string]string
-// while lowercasing the keys
-func s3MetadataToMap(s3Meta map[string]*string) map[string]string {
+// Convert S3 metadata into a map[string]string while lowercasing the
+// keys
+func s3MetadataToMap(s3Meta map[string]string) map[string]string {
 	meta := make(map[string]string, len(s3Meta))
 	for k, v := range s3Meta {
-		if v != nil {
-			meta[strings.ToLower(k)] = *v
-		}
+		meta[strings.ToLower(k)] = v
 	}
 	return meta
 }
 
 // Convert our metadata back into S3 metadata
-func mapToS3Metadata(meta map[string]string) map[string]*string {
-	s3Meta := make(map[string]*string, len(meta))
-	for k, v := range meta {
-		s3Meta[k] = aws.String(v)
-	}
-	return s3Meta
+func mapToS3Metadata(meta map[string]string) map[string]string {
+	return meta
 }
 
 func (o *Object) setMetaData(resp *s3.HeadObjectOutput) {
@@ -5481,7 +5580,7 @@ func (o *Object) setMetaData(resp *s3.HeadObjectOutput) {
 	if resp.ContentLength != nil {
 		o.bytes = *resp.ContentLength
 	}
-	o.setMD5FromEtag(aws.StringValue(resp.ETag))
+	o.setMD5FromEtag(deref(resp.ETag))
 	o.meta = s3MetadataToMap(resp.Metadata)
 	// Read MD5 from metadata if present
 	if md5sumBase64, ok := o.meta[metaMD5Hash]; ok {
@@ -5508,17 +5607,17 @@ func (o *Object) setMetaData(resp *s3.HeadObjectOutput) {
 			o.lastModified = *resp.LastModified
 		}
 	}
-	o.mimeType = aws.StringValue(resp.ContentType)
+	o.mimeType = deref(resp.ContentType)
 
 	// Set system metadata
-	o.storageClass = resp.StorageClass
+	o.storageClass = (*string)(&resp.StorageClass)
 	o.cacheControl = resp.CacheControl
 	o.contentDisposition = resp.ContentDisposition
 	o.contentEncoding = resp.ContentEncoding
 	o.contentLanguage = resp.ContentLanguage
 
 	// If decompressing then size and md5sum are unknown
-	if o.fs.opt.Decompress && aws.StringValue(o.contentEncoding) == "gzip" {
+	if o.fs.opt.Decompress && deref(o.contentEncoding) == "gzip" {
 		o.bytes = -1
 		o.md5 = ""
 	}
@@ -5569,10 +5668,10 @@ func (o *Object) SetModTime(ctx context.Context, modTime time.Time) error {
 	req := s3.CopyObjectInput{
 		ContentType:       aws.String(fs.MimeType(ctx, o)), // Guess the content type
 		Metadata:          mapToS3Metadata(o.meta),
-		MetadataDirective: aws.String(s3.MetadataDirectiveReplace), // replace metadata with that passed in
+		MetadataDirective: types.MetadataDirectiveReplace, // replace metadata with that passed in
 	}
 	if o.fs.opt.RequesterPays {
-		req.RequestPayer = aws.String(s3.RequestPayerRequester)
+		req.RequestPayer = types.RequestPayerRequester
 	}
 	return o.fs.copy(ctx, &req, bucket, bucketPath, bucket, bucketPath, o)
 }
@@ -5608,12 +5707,12 @@ func (o *Object) downloadFromURL(ctx context.Context, bucketPath string, options
 		fs.Debugf(o, "Failed to parse last modified from string %s, %v", resp.Header.Get("Last-Modified"), err)
 	}
 
-	metaData := make(map[string]*string)
+	metaData := make(map[string]string)
 	for key, value := range resp.Header {
 		key = strings.ToLower(key)
 		if strings.HasPrefix(key, "x-amz-meta-") {
 			metaKey := strings.TrimPrefix(key, "x-amz-meta-")
-			metaData[metaKey] = &value[0]
+			metaData[metaKey] = value[0]
 		}
 	}
 
@@ -5635,7 +5734,7 @@ func (o *Object) downloadFromURL(ctx context.Context, bucketPath string, options
 		ContentEncoding:    header("Content-Encoding"),
 		ContentLanguage:    header("Content-Language"),
 		ContentType:        header("Content-Type"),
-		StorageClass:       header("X-Amz-Storage-Class"),
+		StorageClass:       types.StorageClass(*header("X-Amz-Storage-Class")),
 	}
 	o.setMetaData(&head)
 	return resp.Body, err
@@ -5655,24 +5754,29 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
 		VersionId: o.versionID,
 	}
 	if o.fs.opt.RequesterPays {
-		req.RequestPayer = aws.String(s3.RequestPayerRequester)
+		req.RequestPayer = types.RequestPayerRequester
 	}
 	if o.fs.opt.SSECustomerAlgorithm != "" {
 		req.SSECustomerAlgorithm = &o.fs.opt.SSECustomerAlgorithm
 	}
-	if o.fs.opt.SSECustomerKey != "" {
-		req.SSECustomerKey = &o.fs.opt.SSECustomerKey
+	if o.fs.opt.SSECustomerKeyBase64 != "" {
+		req.SSECustomerKey = &o.fs.opt.SSECustomerKeyBase64
 	}
 	if o.fs.opt.SSECustomerKeyMD5 != "" {
 		req.SSECustomerKeyMD5 = &o.fs.opt.SSECustomerKeyMD5
 	}
-	httpReq, resp := o.fs.c.GetObjectRequest(&req)
+	// httpReq, err := s3.NewPresignClient(o.fs.c).PresignGetObject(ctx, &req)
+	// if err != nil {
+	// 	return nil, err
+	// }
 	fs.FixRangeOption(options, o.bytes)
 
+	var APIOptions []func(*middleware.Stack) error
+
 	// Override the automatic decompression in the transport to
 	// download compressed files as-is
 	if o.fs.opt.UseAcceptEncodingGzip.Value {
-		httpReq.HTTPRequest.Header.Set("Accept-Encoding", "gzip")
+		APIOptions = append(APIOptions, smithyhttp.AddHeaderValue("Accept-Encoding", "gzip"))
 	}
 
 	for _, option := range options {
@@ -5682,21 +5786,23 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
 			req.Range = &value
 		case *fs.HTTPOption:
 			key, value := option.Header()
-			httpReq.HTTPRequest.Header.Add(key, value)
+			APIOptions = append(APIOptions, smithyhttp.AddHeaderValue(key, value))
 		default:
 			if option.Mandatory() {
 				fs.Logf(o, "Unsupported mandatory option: %v", option)
 			}
 		}
 	}
+
+	var resp *s3.GetObjectOutput
 	err = o.fs.pacer.Call(func() (bool, error) {
 		var err error
-		httpReq.HTTPRequest = httpReq.HTTPRequest.WithContext(ctx)
-		err = httpReq.Send()
+		resp, err = o.fs.c.GetObject(ctx, &req, s3.WithAPIOptions(APIOptions...))
 		return o.fs.shouldRetry(ctx, err)
 	})
-	if err, ok := err.(awserr.RequestFailure); ok {
-		if err.Code() == "InvalidObjectState" {
+	var awsError smithy.APIError
+	if errors.As(err, &awsError) {
+		if awsError.ErrorCode() == "InvalidObjectState" {
 			return nil, fmt.Errorf("Object in GLACIER, restore first: bucket=%q, key=%q", bucket, bucketPath)
 		}
 	}
@@ -5727,7 +5833,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
 	o.setMetaData(&head)
 
 	// Decompress body if necessary
-	if aws.StringValue(resp.ContentEncoding) == "gzip" {
+	if deref(resp.ContentEncoding) == "gzip" {
 		if o.fs.opt.Decompress || (resp.ContentLength == nil && o.fs.opt.MightGzip.Value) {
 			return readers.NewGzipReader(resp.Body)
 		}
@@ -5751,7 +5857,7 @@ type s3ChunkWriter struct {
 	uploadID             *string
 	multiPartUploadInput *s3.CreateMultipartUploadInput
 	completedPartsMu     sync.Mutex
-	completedParts       []*s3.CompletedPart
+	completedParts       []types.CompletedPart
 	eTag                 string
 	versionID            string
 	md5sMu               sync.Mutex
@@ -5804,7 +5910,7 @@ func (f *Fs) OpenChunkWriter(ctx context.Context, remote string, src fs.ObjectIn
 
 	var mOut *s3.CreateMultipartUploadOutput
 	err = f.pacer.Call(func() (bool, error) {
-		mOut, err = f.c.CreateMultipartUploadWithContext(ctx, &mReq)
+		mOut, err = f.c.CreateMultipartUpload(ctx, &mReq)
 		if err == nil {
 			if mOut == nil {
 				err = fserrors.RetryErrorf("internal error: no info from multipart upload")
@@ -5826,7 +5932,7 @@ func (f *Fs) OpenChunkWriter(ctx context.Context, remote string, src fs.ObjectIn
 		key:                  mOut.Key,
 		uploadID:             mOut.UploadId,
 		multiPartUploadInput: &mReq,
-		completedParts:       make([]*s3.CompletedPart, 0),
+		completedParts:       make([]types.CompletedPart, 0),
 		ui:                   ui,
 		o:                    o,
 	}
@@ -5840,10 +5946,10 @@ func (f *Fs) OpenChunkWriter(ctx context.Context, remote string, src fs.ObjectIn
 }
 
 // add a part number and etag to the completed parts
-func (w *s3ChunkWriter) addCompletedPart(partNum *int64, eTag *string) {
+func (w *s3ChunkWriter) addCompletedPart(partNum *int32, eTag *string) {
 	w.completedPartsMu.Lock()
 	defer w.completedPartsMu.Unlock()
-	w.completedParts = append(w.completedParts, &s3.CompletedPart{
+	w.completedParts = append(w.completedParts, types.CompletedPart{
 		PartNumber: partNum,
 		ETag:       eTag,
 	})
@@ -5891,7 +5997,7 @@ func (w *s3ChunkWriter) WriteChunk(ctx context.Context, chunkNumber int, reader
 	md5sum := base64.StdEncoding.EncodeToString(md5sumBinary[:])
 
 	// S3 requires 1 <= PartNumber <= 10000
-	s3PartNumber := aws.Int64(int64(chunkNumber + 1))
+	s3PartNumber := aws.Int32(int32(chunkNumber + 1))
 	uploadPartReq := &s3.UploadPartInput{
 		Body:                 reader,
 		Bucket:               w.bucket,
@@ -5912,7 +6018,7 @@ func (w *s3ChunkWriter) WriteChunk(ctx context.Context, chunkNumber int, reader
 		if err != nil {
 			return false, err
 		}
-		uout, err = w.f.c.UploadPartWithContext(ctx, uploadPartReq)
+		uout, err = w.f.c.UploadPart(ctx, uploadPartReq)
 		if err != nil {
 			if chunkNumber <= 8 {
 				return w.f.shouldRetry(ctx, err)
@@ -5935,7 +6041,7 @@ func (w *s3ChunkWriter) WriteChunk(ctx context.Context, chunkNumber int, reader
 // Abort the multipart upload
 func (w *s3ChunkWriter) Abort(ctx context.Context) error {
 	err := w.f.pacer.Call(func() (bool, error) {
-		_, err := w.f.c.AbortMultipartUploadWithContext(context.Background(), &s3.AbortMultipartUploadInput{
+		_, err := w.f.c.AbortMultipartUpload(context.Background(), &s3.AbortMultipartUploadInput{
 			Bucket:       w.bucket,
 			Key:          w.key,
 			UploadId:     w.uploadID,
@@ -5958,10 +6064,10 @@ func (w *s3ChunkWriter) Close(ctx context.Context) (err error) {
 	})
 	var resp *s3.CompleteMultipartUploadOutput
 	err = w.f.pacer.Call(func() (bool, error) {
-		resp, err = w.f.c.CompleteMultipartUploadWithContext(ctx, &s3.CompleteMultipartUploadInput{
+		resp, err = w.f.c.CompleteMultipartUpload(ctx, &s3.CompleteMultipartUploadInput{
 			Bucket: w.bucket,
 			Key:    w.key,
-			MultipartUpload: &s3.CompletedMultipartUpload{
+			MultipartUpload: &types.CompletedMultipartUpload{
 				Parts: w.completedParts,
 			},
 			RequestPayer: w.multiPartUploadInput.RequestPayer,
@@ -6003,62 +6109,31 @@ func (o *Object) uploadMultipart(ctx context.Context, src fs.ObjectInfo, in io.R
 	return wantETag, gotETag, versionID, s3cw.ui, nil
 }
 
-// unWrapAwsError unwraps AWS errors, looking for a non AWS error
-//
-// It returns true if one was found and the error, or false and the
-// error passed in.
-func unWrapAwsError(err error) (found bool, outErr error) {
-	if awsErr, ok := err.(awserr.Error); ok {
-		var origErrs []error
-		if batchErr, ok := awsErr.(awserr.BatchedErrors); ok {
-			origErrs = batchErr.OrigErrs()
-		} else {
-			origErrs = []error{awsErr.OrigErr()}
-		}
-		for _, origErr := range origErrs {
-			found, newErr := unWrapAwsError(origErr)
-			if found {
-				return found, newErr
-			}
-		}
-		return false, err
-	}
-	return true, err
-}
-
 // Upload a single part using PutObject
 func (o *Object) uploadSinglepartPutObject(ctx context.Context, req *s3.PutObjectInput, size int64, in io.Reader) (etag string, lastModified time.Time, versionID *string, err error) {
-	r, resp := o.fs.c.PutObjectRequest(req)
-	if req.ContentLength != nil && *req.ContentLength == 0 {
-		// Can't upload zero length files like this for some reason
-		r.Body = bytes.NewReader([]byte{})
-	} else {
-		r.SetStreamingBody(io.NopCloser(in))
-	}
-	r.SetContext(ctx)
-	r.HTTPRequest.Header.Set("X-Amz-Content-Sha256", "UNSIGNED-PAYLOAD")
-
+	req.Body = io.NopCloser(in)
+	var options = []func(*s3.Options){}
+	if o.fs.opt.UseUnsignedPayload.Value {
+		options = append(options, s3.WithAPIOptions(
+			// avoids operation error S3: PutObject, failed to compute payload hash: failed to seek body to start, request stream is not seekable
+			v4signer.SwapComputePayloadSHA256ForUnsignedPayloadMiddleware,
+		))
+	}
+	// Can't retry single part uploads as only have an io.Reader
+	options = append(options, func(s3opt *s3.Options) {
+		s3opt.RetryMaxAttempts = 1
+	})
+	var resp *s3.PutObjectOutput
 	err = o.fs.pacer.CallNoRetry(func() (bool, error) {
-		err := r.Send()
+		resp, err = o.fs.c.PutObject(ctx, req, options...)
 		return o.fs.shouldRetry(ctx, err)
 	})
 	if err != nil {
-		// Return the underlying error if we have a
-		// Serialization or RequestError error if possible
-		//
-		// These errors are synthesized locally in the SDK
-		// (not returned from the server) and we'd rather have
-		// the underlying error if there is one.
-		if do, ok := err.(awserr.Error); ok && (do.Code() == request.ErrCodeSerialization || do.Code() == request.ErrCodeRequestError) {
-			if found, newErr := unWrapAwsError(err); found {
-				err = newErr
-			}
-		}
 		return etag, lastModified, nil, err
 	}
 	lastModified = time.Now()
 	if resp != nil {
-		etag = aws.StringValue(resp.ETag)
+		etag = deref(resp.ETag)
 		versionID = resp.VersionId
 	}
 	return etag, lastModified, versionID, nil
@@ -6066,35 +6141,25 @@ func (o *Object) uploadSinglepartPutObject(ctx context.Context, req *s3.PutObjec
 
 // Upload a single part using a presigned request
 func (o *Object) uploadSinglepartPresignedRequest(ctx context.Context, req *s3.PutObjectInput, size int64, in io.Reader) (etag string, lastModified time.Time, versionID *string, err error) {
-	// Create the request
-	putObj, _ := o.fs.c.PutObjectRequest(req)
-
-	// Sign it so we can upload using a presigned request.
-	//
-	// Note the SDK didn't used to support streaming to
-	// PutObject so we used this work-around.
-	url, headers, err := putObj.PresignRequest(15 * time.Minute)
+	// Create the presigned request
+	putReq, err := s3.NewPresignClient(o.fs.c).PresignPutObject(ctx, req, s3.WithPresignExpires(15*time.Minute))
 	if err != nil {
 		return etag, lastModified, nil, fmt.Errorf("s3 upload: sign request: %w", err)
 	}
 
-	if o.fs.opt.V2Auth && headers == nil {
-		headers = putObj.HTTPRequest.Header
-	}
-
 	// Set request to nil if empty so as not to make chunked encoding
 	if size == 0 {
 		in = nil
 	}
 
 	// create the vanilla http request
-	httpReq, err := http.NewRequestWithContext(ctx, "PUT", url, in)
+	httpReq, err := http.NewRequestWithContext(ctx, "PUT", putReq.URL, in)
 	if err != nil {
 		return etag, lastModified, nil, fmt.Errorf("s3 upload: new request: %w", err)
 	}
 
 	// set the headers we signed and the length
-	httpReq.Header = headers
+	httpReq.Header = putReq.SignedHeader
 	httpReq.ContentLength = size
 
 	var resp *http.Response
@@ -6152,7 +6217,7 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [
 
 	ui.req = &s3.PutObjectInput{
 		Bucket: &bucket,
-		ACL:    stringPointerOrNil(o.fs.opt.ACL),
+		ACL:    types.ObjectCannedACL(o.fs.opt.ACL),
 		Key:    &bucketPath,
 	}
 
@@ -6161,13 +6226,13 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [
 	if err != nil {
 		return ui, fmt.Errorf("failed to read metadata from source object: %w", err)
 	}
-	ui.req.Metadata = make(map[string]*string, len(meta)+2)
+	ui.req.Metadata = make(map[string]string, len(meta)+2)
 	// merge metadata into request and user metadata
 	for k, v := range meta {
 		pv := aws.String(v)
 		k = strings.ToLower(k)
 		if o.fs.opt.NoSystemMetadata {
-			ui.req.Metadata[k] = pv
+			ui.req.Metadata[k] = v
 			continue
 		}
 		switch k {
@@ -6195,14 +6260,14 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [
 			}
 		case "btime":
 			// write as metadata since we can't set it
-			ui.req.Metadata[k] = pv
+			ui.req.Metadata[k] = v
 		default:
-			ui.req.Metadata[k] = pv
+			ui.req.Metadata[k] = v
 		}
 	}
 
 	// Set the mtime in the meta data
-	ui.req.Metadata[metaMtime] = aws.String(swift.TimeToFloatString(modTime))
+	ui.req.Metadata[metaMtime] = swift.TimeToFloatString(modTime)
 
 	// read the md5sum if available
 	// - for non multipart
@@ -6224,7 +6289,7 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [
 					// - a multipart upload
 					// - the Etag is not an MD5, eg when using SSE/SSE-C
 					// provided checksums aren't disabled
-					ui.req.Metadata[metaMD5Hash] = &md5sumBase64
+					ui.req.Metadata[metaMD5Hash] = md5sumBase64
 				}
 			}
 		}
@@ -6241,16 +6306,16 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [
 		ui.req.ContentMD5 = &md5sumBase64
 	}
 	if o.fs.opt.RequesterPays {
-		ui.req.RequestPayer = aws.String(s3.RequestPayerRequester)
+		ui.req.RequestPayer = types.RequestPayerRequester
 	}
 	if o.fs.opt.ServerSideEncryption != "" {
-		ui.req.ServerSideEncryption = &o.fs.opt.ServerSideEncryption
+		ui.req.ServerSideEncryption = types.ServerSideEncryption(o.fs.opt.ServerSideEncryption)
 	}
 	if o.fs.opt.SSECustomerAlgorithm != "" {
 		ui.req.SSECustomerAlgorithm = &o.fs.opt.SSECustomerAlgorithm
 	}
-	if o.fs.opt.SSECustomerKey != "" {
-		ui.req.SSECustomerKey = &o.fs.opt.SSECustomerKey
+	if o.fs.opt.SSECustomerKeyBase64 != "" {
+		ui.req.SSECustomerKey = &o.fs.opt.SSECustomerKeyBase64
 	}
 	if o.fs.opt.SSECustomerKeyMD5 != "" {
 		ui.req.SSECustomerKeyMD5 = &o.fs.opt.SSECustomerKeyMD5
@@ -6259,7 +6324,7 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [
 		ui.req.SSEKMSKeyId = &o.fs.opt.SSEKMSKeyID
 	}
 	if o.fs.opt.StorageClass != "" {
-		ui.req.StorageClass = &o.fs.opt.StorageClass
+		ui.req.StorageClass = types.StorageClass(o.fs.opt.StorageClass)
 	}
 	// Apply upload options
 	for _, option := range options {
@@ -6284,7 +6349,7 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [
 			const amzMetaPrefix = "x-amz-meta-"
 			if strings.HasPrefix(lowerKey, amzMetaPrefix) {
 				metaKey := lowerKey[len(amzMetaPrefix):]
-				ui.req.Metadata[metaKey] = aws.String(value)
+				ui.req.Metadata[metaKey] = value
 			} else {
 				fs.Errorf(o, "Don't know how to set key %q on upload", key)
 			}
@@ -6296,11 +6361,8 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [
 		if !httpguts.ValidHeaderFieldName(key) {
 			fs.Errorf(o, "Dropping invalid metadata key %q", key)
 			delete(ui.req.Metadata, key)
-		} else if value == nil {
-			fs.Errorf(o, "Dropping nil metadata value for key %q", key)
-			delete(ui.req.Metadata, key)
-		} else if !httpguts.ValidHeaderFieldValue(*value) {
-			fs.Errorf(o, "Dropping invalid metadata value %q for key %q", *value, key)
+		} else if !httpguts.ValidHeaderFieldValue(value) {
+			fs.Errorf(o, "Dropping invalid metadata value %q for key %q", value, key)
 			delete(ui.req.Metadata, key)
 		}
 	}
@@ -6398,10 +6460,10 @@ func (o *Object) Remove(ctx context.Context) error {
 		VersionId: o.versionID,
 	}
 	if o.fs.opt.RequesterPays {
-		req.RequestPayer = aws.String(s3.RequestPayerRequester)
+		req.RequestPayer = types.RequestPayerRequester
 	}
 	err := o.fs.pacer.Call(func() (bool, error) {
-		_, err := o.fs.c.DeleteObjectWithContext(ctx, &req)
+		_, err := o.fs.c.DeleteObject(ctx, &req)
 		return o.fs.shouldRetry(ctx, err)
 	})
 	return err
@@ -6423,8 +6485,8 @@ func (o *Object) SetTier(tier string) (err error) {
 	tier = strings.ToUpper(tier)
 	bucket, bucketPath := o.split()
 	req := s3.CopyObjectInput{
-		MetadataDirective: aws.String(s3.MetadataDirectiveCopy),
-		StorageClass:      aws.String(tier),
+		MetadataDirective: types.MetadataDirectiveCopy,
+		StorageClass:      types.StorageClass(tier),
 	}
 	err = o.fs.copy(ctx, &req, bucket, bucketPath, bucket, bucketPath, o)
 	if err != nil {
diff --git a/backend/s3/setfrom.go b/backend/s3/setfrom.go
index 27eb2aafe..439e8b6b0 100644
--- a/backend/s3/setfrom.go
+++ b/backend/s3/setfrom.go
@@ -2,7 +2,10 @@
 
 package s3
 
-import "github.com/aws/aws-sdk-go/service/s3"
+import (
+	"github.com/aws/aws-sdk-go-v2/service/s3"
+	"github.com/aws/aws-sdk-go-v2/service/s3/types"
+)
 
 // setFrom_s3ListObjectsInput_s3ListObjectsV2Input copies matching elements from a to b
 func setFrom_s3ListObjectsInput_s3ListObjectsV2Input(a *s3.ListObjectsInput, b *s3.ListObjectsV2Input) {
@@ -27,6 +30,7 @@ func setFrom_s3ListObjectsV2Output_s3ListObjectsOutput(a *s3.ListObjectsV2Output
 	a.Name = b.Name
 	a.Prefix = b.Prefix
 	a.RequestCharged = b.RequestCharged
+	a.ResultMetadata = b.ResultMetadata
 }
 
 // setFrom_s3ListObjectVersionsInput_s3ListObjectsV2Input copies matching elements from a to b
@@ -41,8 +45,8 @@ func setFrom_s3ListObjectVersionsInput_s3ListObjectsV2Input(a *s3.ListObjectVers
 	a.RequestPayer = b.RequestPayer
 }
 
-// setFrom_s3ObjectVersion_s3DeleteMarkerEntry copies matching elements from a to b
-func setFrom_s3ObjectVersion_s3DeleteMarkerEntry(a *s3.ObjectVersion, b *s3.DeleteMarkerEntry) {
+// setFrom_typesObjectVersion_typesDeleteMarkerEntry copies matching elements from a to b
+func setFrom_typesObjectVersion_typesDeleteMarkerEntry(a *types.ObjectVersion, b *types.DeleteMarkerEntry) {
 	a.IsLatest = b.IsLatest
 	a.Key = b.Key
 	a.LastModified = b.LastModified
@@ -60,10 +64,11 @@ func setFrom_s3ListObjectsV2Output_s3ListObjectVersionsOutput(a *s3.ListObjectsV
 	a.Name = b.Name
 	a.Prefix = b.Prefix
 	a.RequestCharged = b.RequestCharged
+	a.ResultMetadata = b.ResultMetadata
 }
 
-// setFrom_s3Object_s3ObjectVersion copies matching elements from a to b
-func setFrom_s3Object_s3ObjectVersion(a *s3.Object, b *s3.ObjectVersion) {
+// setFrom_typesObject_typesObjectVersion copies matching elements from a to b
+func setFrom_typesObject_typesObjectVersion(a *types.Object, b *types.ObjectVersion) {
 	a.ChecksumAlgorithm = b.ChecksumAlgorithm
 	a.ETag = b.ETag
 	a.Key = b.Key
@@ -71,7 +76,6 @@ func setFrom_s3Object_s3ObjectVersion(a *s3.Object, b *s3.ObjectVersion) {
 	a.Owner = b.Owner
 	a.RestoreStatus = b.RestoreStatus
 	a.Size = b.Size
-	a.StorageClass = b.StorageClass
 }
 
 // setFrom_s3CreateMultipartUploadInput_s3HeadObjectOutput copies matching elements from a to b
@@ -82,6 +86,7 @@ func setFrom_s3CreateMultipartUploadInput_s3HeadObjectOutput(a *s3.CreateMultipa
 	a.ContentEncoding = b.ContentEncoding
 	a.ContentLanguage = b.ContentLanguage
 	a.ContentType = b.ContentType
+	a.Expires = b.Expires
 	a.Metadata = b.Metadata
 	a.ObjectLockLegalHoldStatus = b.ObjectLockLegalHoldStatus
 	a.ObjectLockMode = b.ObjectLockMode
@@ -96,8 +101,9 @@ func setFrom_s3CreateMultipartUploadInput_s3HeadObjectOutput(a *s3.CreateMultipa
 
 // setFrom_s3CreateMultipartUploadInput_s3CopyObjectInput copies matching elements from a to b
 func setFrom_s3CreateMultipartUploadInput_s3CopyObjectInput(a *s3.CreateMultipartUploadInput, b *s3.CopyObjectInput) {
-	a.ACL = b.ACL
 	a.Bucket = b.Bucket
+	a.Key = b.Key
+	a.ACL = b.ACL
 	a.BucketKeyEnabled = b.BucketKeyEnabled
 	a.CacheControl = b.CacheControl
 	a.ChecksumAlgorithm = b.ChecksumAlgorithm
@@ -111,7 +117,6 @@ func setFrom_s3CreateMultipartUploadInput_s3CopyObjectInput(a *s3.CreateMultipar
 	a.GrantRead = b.GrantRead
 	a.GrantReadACP = b.GrantReadACP
 	a.GrantWriteACP = b.GrantWriteACP
-	a.Key = b.Key
 	a.Metadata = b.Metadata
 	a.ObjectLockLegalHoldStatus = b.ObjectLockLegalHoldStatus
 	a.ObjectLockMode = b.ObjectLockMode
@@ -132,6 +137,7 @@ func setFrom_s3CreateMultipartUploadInput_s3CopyObjectInput(a *s3.CreateMultipar
 func setFrom_s3UploadPartCopyInput_s3CopyObjectInput(a *s3.UploadPartCopyInput, b *s3.CopyObjectInput) {
 	a.Bucket = b.Bucket
 	a.CopySource = b.CopySource
+	a.Key = b.Key
 	a.CopySourceIfMatch = b.CopySourceIfMatch
 	a.CopySourceIfModifiedSince = b.CopySourceIfModifiedSince
 	a.CopySourceIfNoneMatch = b.CopySourceIfNoneMatch
@@ -141,7 +147,6 @@ func setFrom_s3UploadPartCopyInput_s3CopyObjectInput(a *s3.UploadPartCopyInput,
 	a.CopySourceSSECustomerKeyMD5 = b.CopySourceSSECustomerKeyMD5
 	a.ExpectedBucketOwner = b.ExpectedBucketOwner
 	a.ExpectedSourceBucketOwner = b.ExpectedSourceBucketOwner
-	a.Key = b.Key
 	a.RequestPayer = b.RequestPayer
 	a.SSECustomerAlgorithm = b.SSECustomerAlgorithm
 	a.SSECustomerKey = b.SSECustomerKey
@@ -166,6 +171,7 @@ func setFrom_s3HeadObjectOutput_s3GetObjectOutput(a *s3.HeadObjectOutput, b *s3.
 	a.ETag = b.ETag
 	a.Expiration = b.Expiration
 	a.Expires = b.Expires
+	a.ExpiresString = b.ExpiresString
 	a.LastModified = b.LastModified
 	a.Metadata = b.Metadata
 	a.MissingMeta = b.MissingMeta
@@ -183,12 +189,14 @@ func setFrom_s3HeadObjectOutput_s3GetObjectOutput(a *s3.HeadObjectOutput, b *s3.
 	a.StorageClass = b.StorageClass
 	a.VersionId = b.VersionId
 	a.WebsiteRedirectLocation = b.WebsiteRedirectLocation
+	a.ResultMetadata = b.ResultMetadata
 }
 
 // setFrom_s3CreateMultipartUploadInput_s3PutObjectInput copies matching elements from a to b
 func setFrom_s3CreateMultipartUploadInput_s3PutObjectInput(a *s3.CreateMultipartUploadInput, b *s3.PutObjectInput) {
-	a.ACL = b.ACL
 	a.Bucket = b.Bucket
+	a.Key = b.Key
+	a.ACL = b.ACL
 	a.BucketKeyEnabled = b.BucketKeyEnabled
 	a.CacheControl = b.CacheControl
 	a.ChecksumAlgorithm = b.ChecksumAlgorithm
@@ -202,7 +210,6 @@ func setFrom_s3CreateMultipartUploadInput_s3PutObjectInput(a *s3.CreateMultipart
 	a.GrantRead = b.GrantRead
 	a.GrantReadACP = b.GrantReadACP
 	a.GrantWriteACP = b.GrantWriteACP
-	a.Key = b.Key
 	a.Metadata = b.Metadata
 	a.ObjectLockLegalHoldStatus = b.ObjectLockLegalHoldStatus
 	a.ObjectLockMode = b.ObjectLockMode
@@ -232,6 +239,7 @@ func setFrom_s3HeadObjectOutput_s3PutObjectInput(a *s3.HeadObjectOutput, b *s3.P
 	a.ContentLanguage = b.ContentLanguage
 	a.ContentLength = b.ContentLength
 	a.ContentType = b.ContentType
+	a.Expires = b.Expires
 	a.Metadata = b.Metadata
 	a.ObjectLockLegalHoldStatus = b.ObjectLockLegalHoldStatus
 	a.ObjectLockMode = b.ObjectLockMode
@@ -246,8 +254,9 @@ func setFrom_s3HeadObjectOutput_s3PutObjectInput(a *s3.HeadObjectOutput, b *s3.P
 
 // setFrom_s3CopyObjectInput_s3PutObjectInput copies matching elements from a to b
 func setFrom_s3CopyObjectInput_s3PutObjectInput(a *s3.CopyObjectInput, b *s3.PutObjectInput) {
-	a.ACL = b.ACL
 	a.Bucket = b.Bucket
+	a.Key = b.Key
+	a.ACL = b.ACL
 	a.BucketKeyEnabled = b.BucketKeyEnabled
 	a.CacheControl = b.CacheControl
 	a.ChecksumAlgorithm = b.ChecksumAlgorithm
@@ -261,7 +270,6 @@ func setFrom_s3CopyObjectInput_s3PutObjectInput(a *s3.CopyObjectInput, b *s3.Put
 	a.GrantRead = b.GrantRead
 	a.GrantReadACP = b.GrantReadACP
 	a.GrantWriteACP = b.GrantWriteACP
-	a.Key = b.Key
 	a.Metadata = b.Metadata
 	a.ObjectLockLegalHoldStatus = b.ObjectLockLegalHoldStatus
 	a.ObjectLockMode = b.ObjectLockMode
diff --git a/backend/s3/v2sign.go b/backend/s3/v2sign.go
index 12e41f0b6..45b1bf7f3 100644
--- a/backend/s3/v2sign.go
+++ b/backend/s3/v2sign.go
@@ -3,6 +3,7 @@
 package s3
 
 import (
+	"context"
 	"crypto/hmac"
 	"crypto/sha1"
 	"encoding/base64"
@@ -10,6 +11,9 @@ import (
 	"sort"
 	"strings"
 	"time"
+
+	"github.com/aws/aws-sdk-go-v2/aws"
+	v4signer "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
 )
 
 // URL parameters that need to be added to the signature
@@ -36,10 +40,17 @@ var s3ParamsToSign = map[string]struct{}{
 	"response-content-encoding":    {},
 }
 
-// sign signs requests using v2 auth
+// Implement HTTPSignerV4 interface
+type v2Signer struct {
+	opt *Options
+}
+
+// SignHTTP signs requests using v2 auth.
+//
+// Cobbled together from goamz and aws-sdk-go.
 //
-// Cobbled together from goamz and aws-sdk-go
-func sign(AccessKey, SecretKey string, req *http.Request) {
+// Bodged up to compile with AWS SDK v2
+func (v2 *v2Signer) SignHTTP(ctx context.Context, credentials aws.Credentials, req *http.Request, payloadHash string, service string, region string, signingTime time.Time, optFns ...func(*v4signer.SignerOptions)) error {
 	// Set date
 	date := time.Now().UTC().Format(time.RFC1123)
 	req.Header.Set("Date", date)
@@ -107,11 +118,12 @@ func sign(AccessKey, SecretKey string, req *http.Request) {
 
 	// Make signature
 	payload := req.Method + "\n" + md5 + "\n" + contentType + "\n" + date + "\n" + joinedHeadersToSign + uri
-	hash := hmac.New(sha1.New, []byte(SecretKey))
+	hash := hmac.New(sha1.New, []byte(v2.opt.SecretAccessKey))
 	_, _ = hash.Write([]byte(payload))
 	signature := make([]byte, base64.StdEncoding.EncodedLen(hash.Size()))
 	base64.StdEncoding.Encode(signature, hash.Sum(nil))
 
 	// Set signature in request
-	req.Header.Set("Authorization", "AWS "+AccessKey+":"+string(signature))
+	req.Header.Set("Authorization", "AWS "+v2.opt.AccessKeyID+":"+string(signature))
+	return nil
 }
diff --git a/bin/cross-compile.go b/bin/cross-compile.go
index a89bac327..8779eff02 100644
--- a/bin/cross-compile.go
+++ b/bin/cross-compile.go
@@ -73,7 +73,7 @@ var osarches = []string{
 	"plan9/386",
 	"plan9/amd64",
 	"solaris/amd64",
-	"js/wasm",
+	// "js/wasm", // Rclone is too big for js/wasm until https://github.com/golang/go/issues/64856 is fixed
 }
 
 // Special environment flags for a given arch
diff --git a/cmd/genautocomplete/genautocomplete.go b/cmd/genautocomplete/genautocomplete.go
index 029b381d3..908d25274 100644
--- a/cmd/genautocomplete/genautocomplete.go
+++ b/cmd/genautocomplete/genautocomplete.go
@@ -1,4 +1,4 @@
-// Package genautocomplete provides the genautocomplete command.
+// Package genautocomplete provides the completion command.
 package genautocomplete
 
 import (
diff --git a/cmd/genautocomplete/genautocomplete_bash.go b/cmd/genautocomplete/genautocomplete_bash.go
index 2b8d22b49..6c976d569 100644
--- a/cmd/genautocomplete/genautocomplete_bash.go
+++ b/cmd/genautocomplete/genautocomplete_bash.go
@@ -20,7 +20,7 @@ Generates a bash shell autocompletion script for rclone.
 
 By default, when run without any arguments, 
 
-    rclone genautocomplete bash
+    rclone completion bash
 
 the generated script will be written to
 
diff --git a/cmd/genautocomplete/genautocomplete_fish.go b/cmd/genautocomplete/genautocomplete_fish.go
index bafe15ffa..c904e4cef 100644
--- a/cmd/genautocomplete/genautocomplete_fish.go
+++ b/cmd/genautocomplete/genautocomplete_fish.go
@@ -21,7 +21,7 @@ Generates a fish autocompletion script for rclone.
 This writes to /etc/fish/completions/rclone.fish by default so will
 probably need to be run with sudo or as root, e.g.
 
-    sudo rclone genautocomplete fish
+    sudo rclone completion fish
 
 Logout and login again to use the autocompletion scripts, or source
 them directly
diff --git a/cmd/genautocomplete/genautocomplete_zsh.go b/cmd/genautocomplete/genautocomplete_zsh.go
index 6cc352a65..1cd2c6401 100644
--- a/cmd/genautocomplete/genautocomplete_zsh.go
+++ b/cmd/genautocomplete/genautocomplete_zsh.go
@@ -21,7 +21,7 @@ Generates a zsh autocompletion script for rclone.
 This writes to /usr/share/zsh/vendor-completions/_rclone by default so will
 probably need to be run with sudo or as root, e.g.
 
-    sudo rclone genautocomplete zsh
+    sudo rclone completion zsh
 
 Logout and login again to use the autocompletion scripts, or source
 them directly
diff --git a/cmd/help.go b/cmd/help.go
index 5d2dbce73..e6106d2f7 100644
--- a/cmd/help.go
+++ b/cmd/help.go
@@ -12,6 +12,7 @@ import (
 	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fs/config/configflags"
 	"github.com/rclone/rclone/fs/config/flags"
+	"github.com/rclone/rclone/fs/filter"
 	"github.com/rclone/rclone/fs/filter/filterflags"
 	"github.com/rclone/rclone/fs/log/logflags"
 	"github.com/rclone/rclone/fs/rc/rcflags"
@@ -58,19 +59,25 @@ var helpCommand = &cobra.Command{
 }
 
 // to filter the flags with
-var flagsRe *regexp.Regexp
+var (
+	filterFlagsGroup     string
+	filterFlagsRe        *regexp.Regexp
+	filterFlagsNamesOnly bool
+)
 
 // Show the flags
 var helpFlags = &cobra.Command{
-	Use:   "flags [<regexp to match>]",
+	Use:   "flags [<filter>]",
 	Short: "Show the global flags for rclone",
 	Run: func(command *cobra.Command, args []string) {
+		command.Flags()
 		if len(args) > 0 {
-			re, err := regexp.Compile(`(?i)` + args[0])
+			re, err := filter.GlobStringToRegexp(args[0], false)
 			if err != nil {
-				log.Fatalf("Failed to compile flags regexp: %v", err)
+				log.Fatalf("Invalid flag filter: %v", err)
 			}
-			flagsRe = re
+			fs.Debugf(nil, "Flag filter: %s", re.String())
+			filterFlagsRe = re
 		}
 		if GeneratingDocs {
 			Root.SetUsageTemplate(docFlagsTemplate)
@@ -157,7 +164,7 @@ func setupRootCommand(rootCmd *cobra.Command) {
 				fs.Errorf(nil, "Flag --%s is unknown", flag.Name)
 			}
 		})
-		groups := flags.All.Filter(flagsRe).Include(cmd.Annotations["groups"])
+		groups := flags.All.Filter(filterFlagsGroup, filterFlagsRe, filterFlagsNamesOnly).Include(cmd.Annotations["groups"])
 		return groups.Groups
 	})
 	rootCmd.SetUsageTemplate(usageTemplate)
@@ -169,6 +176,9 @@ func setupRootCommand(rootCmd *cobra.Command) {
 
 	rootCmd.AddCommand(helpCommand)
 	helpCommand.AddCommand(helpFlags)
+	helpFlagsFlags := helpFlags.Flags()
+	flags.StringVarP(helpFlagsFlags, &filterFlagsGroup, "group", "", "", "Only include flags from specific group", "")
+	flags.BoolVarP(helpFlagsFlags, &filterFlagsNamesOnly, "name", "", false, "Apply filter only on flag names", "")
 	helpCommand.AddCommand(helpBackends)
 	helpCommand.AddCommand(helpBackend)
 
diff --git a/cmd/listremotes/listremotes.go b/cmd/listremotes/listremotes.go
index 68dd291c6..642306ab8 100644
--- a/cmd/listremotes/listremotes.go
+++ b/cmd/listremotes/listremotes.go
@@ -2,60 +2,236 @@
 package ls
 
 import (
+	"encoding/json"
 	"fmt"
+	"os"
+	"regexp"
 	"sort"
+	"strings"
 
 	"github.com/rclone/rclone/cmd"
+	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fs/config"
 	"github.com/rclone/rclone/fs/config/flags"
+	"github.com/rclone/rclone/fs/filter"
 	"github.com/spf13/cobra"
 )
 
-// Globals
 var (
-	listLong bool
+	listLong          bool
+	jsonOutput        bool
+	filterName        string
+	filterType        string
+	filterSource      string
+	filterDescription string
+	orderBy           string
 )
 
 func init() {
 	cmd.Root.AddCommand(commandDefinition)
 	cmdFlags := commandDefinition.Flags()
-	flags.BoolVarP(cmdFlags, &listLong, "long", "", listLong, "Show the type and the description as well as names", "")
+	flags.BoolVarP(cmdFlags, &listLong, "long", "", false, "Show type, source and description in addition to name", "")
+	flags.StringVarP(cmdFlags, &filterName, "name", "", "", "Filter remotes by name", "")
+	flags.StringVarP(cmdFlags, &filterType, "type", "", "", "Filter remotes by type", "")
+	flags.StringVarP(cmdFlags, &filterSource, "source", "", "", "filter remotes by source", "")
+	flags.StringVarP(cmdFlags, &filterDescription, "description", "", "", "filter remotes by description", "")
+	flags.StringVarP(cmdFlags, &orderBy, "order-by", "", "", "Instructions on how to order the result, e.g. 'type,name=descending'", "")
+	flags.BoolVarP(cmdFlags, &jsonOutput, "json", "", false, "Format output as JSON", "")
+}
+
+// lessFn compares to remotes for order by
+type lessFn func(a, b config.Remote) bool
+
+// newLess returns a function for comparing remotes based on an order by string
+func newLess(orderBy string) (less lessFn, err error) {
+	if orderBy == "" {
+		return nil, nil
+	}
+	parts := strings.Split(strings.ToLower(orderBy), ",")
+	n := len(parts)
+	for i := n - 1; i >= 0; i-- {
+		fieldAndDirection := strings.SplitN(parts[i], "=", 2)
+
+		descending := false
+		if len(fieldAndDirection) > 1 {
+			switch fieldAndDirection[1] {
+			case "ascending", "asc":
+			case "descending", "desc":
+				descending = true
+			default:
+				return nil, fmt.Errorf("unknown --order-by direction %q", fieldAndDirection[1])
+			}
+		}
+
+		var field func(o config.Remote) string
+		switch fieldAndDirection[0] {
+		case "name":
+			field = func(o config.Remote) string {
+				return o.Name
+			}
+		case "type":
+			field = func(o config.Remote) string {
+				return o.Type
+			}
+		case "source":
+			field = func(o config.Remote) string {
+				return o.Source
+			}
+		case "description":
+			field = func(o config.Remote) string {
+				return o.Description
+			}
+		default:
+			return nil, fmt.Errorf("unknown --order-by field %q", fieldAndDirection[0])
+		}
+
+		var thisLess lessFn
+		if descending {
+			thisLess = func(a, b config.Remote) bool {
+				return field(a) > field(b)
+			}
+		} else {
+			thisLess = func(a, b config.Remote) bool {
+				return field(a) < field(b)
+			}
+		}
+
+		if i == n-1 {
+			less = thisLess
+		} else {
+			nextLess := less
+			less = func(a, b config.Remote) bool {
+				if field(a) == field(b) {
+					return nextLess(a, b)
+				}
+				return thisLess(a, b)
+			}
+		}
+	}
+	return less, nil
 }
 
 var commandDefinition = &cobra.Command{
-	Use:   "listremotes",
+	Use:   "listremotes [<filter>]",
 	Short: `List all the remotes in the config file and defined in environment variables.`,
 	Long: `
-rclone listremotes lists all the available remotes from the config file.
+rclone listremotes lists all the available remotes from the config file,
+or the remotes matching an optional filter.
+
+Prints the result in human-readable format by default, and as a simple list of
+remote names, or if used with flag ` + "`--long`" + ` a tabular format including
+all attributes of the remotes: name, type, source and description. Using flag
+` + "`--json`" + ` produces machine-readable output instead, which always includes
+all attributes.
 
-When used with the ` + "`--long`" + ` flag it lists the types and the descriptions too.
+Result can be filtered by a filter argument which applies to all attributes,
+and/or filter flags specific for each attribute. The values must be specified
+according to regular rclone filtering pattern syntax.
 `,
 	Annotations: map[string]string{
 		"versionIntroduced": "v1.34",
 	},
-	Run: func(command *cobra.Command, args []string) {
-		cmd.CheckArgs(0, 0, command, args)
-		remotes := config.FileSections()
-		sort.Strings(remotes)
-		maxlen := 1
-		maxlentype := 1
+	RunE: func(command *cobra.Command, args []string) error {
+		cmd.CheckArgs(0, 1, command, args)
+		var filterDefault string
+		if len(args) > 0 {
+			filterDefault = args[0]
+		}
+		filters := make(map[string]*regexp.Regexp)
+		for k, v := range map[string]string{
+			"all":         filterDefault,
+			"name":        filterName,
+			"type":        filterType,
+			"source":      filterSource,
+			"description": filterDescription,
+		} {
+			if v != "" {
+				filterRe, err := filter.GlobStringToRegexp(v, false)
+				if err != nil {
+					return fmt.Errorf("invalid %s filter argument: %w", k, err)
+				}
+				fs.Debugf(nil, "Filter for %s: %s", k, filterRe.String())
+				filters[k] = filterRe
+			}
+		}
+		remotes := config.GetRemotes()
+		maxName := 0
+		maxType := 0
+		maxSource := 0
+		i := 0
 		for _, remote := range remotes {
-			if len(remote) > maxlen {
-				maxlen = len(remote)
+			include := true
+			for k, v := range filters {
+				if k == "all" && !(v.MatchString(remote.Name) || v.MatchString(remote.Type) || v.MatchString(remote.Source) || v.MatchString(remote.Description)) {
+					include = false
+				} else if k == "name" && !v.MatchString(remote.Name) {
+					include = false
+				} else if k == "type" && !v.MatchString(remote.Type) {
+					include = false
+				} else if k == "source" && !v.MatchString(remote.Source) {
+					include = false
+				} else if k == "description" && !v.MatchString(remote.Description) {
+					include = false
+				}
 			}
-			t := config.FileGet(remote, "type")
-			if len(t) > maxlentype {
-				maxlentype = len(t)
+			if include {
+				if len(remote.Name) > maxName {
+					maxName = len(remote.Name)
+				}
+				if len(remote.Type) > maxType {
+					maxType = len(remote.Type)
+				}
+				if len(remote.Source) > maxSource {
+					maxSource = len(remote.Source)
+				}
+				remotes[i] = remote
+				i++
 			}
 		}
-		for _, remote := range remotes {
-			if listLong {
-				remoteType := config.FileGet(remote, "type")
-				description := config.FileGet(remote, "description")
-				fmt.Printf("%-*s %-*s %s\n", maxlen+1, remote+":", maxlentype+1, remoteType, description)
-			} else {
-				fmt.Printf("%s:\n", remote)
+		remotes = remotes[:i]
+
+		less, err := newLess(orderBy)
+		if err != nil {
+			return err
+		}
+		if less != nil {
+			sliceLessFn := func(i, j int) bool {
+				return less(remotes[i], remotes[j])
+			}
+			sort.SliceStable(remotes, sliceLessFn)
+		}
+
+		if jsonOutput {
+			fmt.Println("[")
+			first := true
+			for _, remote := range remotes {
+				out, err := json.Marshal(remote)
+				if err != nil {
+					return fmt.Errorf("failed to marshal remote object: %w", err)
+				}
+				if first {
+					first = false
+				} else {
+					fmt.Print(",\n")
+				}
+				_, err = os.Stdout.Write(out)
+				if err != nil {
+					return fmt.Errorf("failed to write to output: %w", err)
+				}
+			}
+			if !first {
+				fmt.Println()
+			}
+			fmt.Println("]")
+		} else if listLong {
+			for _, remote := range remotes {
+				fmt.Printf("%-*s %-*s %-*s %s\n", maxName+1, remote.Name+":", maxType, remote.Type, maxSource, remote.Source, remote.Description)
+			}
+		} else {
+			for _, remote := range remotes {
+				fmt.Printf("%s:\n", remote.Name)
 			}
 		}
+		return nil
 	},
 }
diff --git a/docs/content/commands/rclone_completion.md b/docs/content/commands/rclone_completion.md
index 36aa16742..32394e0a1 100644
--- a/docs/content/commands/rclone_completion.md
+++ b/docs/content/commands/rclone_completion.md
@@ -1,6 +1,8 @@
 ---
 title: "rclone completion"
 description: "Output completion script for a given shell."
+aliases:
+  - /commands/rclone_genautocomplete/
 versionIntroduced: v1.33
 # autogenerated - DO NOT EDIT, instead edit the source code in cmd/completion/ and as part of making a release run "make commanddocs"
 ---
diff --git a/docs/content/commands/rclone_completion_bash.md b/docs/content/commands/rclone_completion_bash.md
index 6a2ad8a7a..4b1cf3fb3 100644
--- a/docs/content/commands/rclone_completion_bash.md
+++ b/docs/content/commands/rclone_completion_bash.md
@@ -1,6 +1,8 @@
 ---
 title: "rclone completion bash"
 description: "Output bash completion script for rclone."
+aliases:
+  - /commands/rclone_genautocomplete_bash/
 # autogenerated - DO NOT EDIT, instead edit the source code in cmd/completion/bash/ and as part of making a release run "make commanddocs"
 ---
 # rclone completion bash
diff --git a/docs/content/commands/rclone_completion_fish.md b/docs/content/commands/rclone_completion_fish.md
index 2eff18b7d..6b5ed8b64 100644
--- a/docs/content/commands/rclone_completion_fish.md
+++ b/docs/content/commands/rclone_completion_fish.md
@@ -1,6 +1,8 @@
 ---
 title: "rclone completion fish"
 description: "Output fish completion script for rclone."
+aliases:
+  - /commands/rclone_genautocomplete_fish/
 # autogenerated - DO NOT EDIT, instead edit the source code in cmd/completion/fish/ and as part of making a release run "make commanddocs"
 ---
 # rclone completion fish
diff --git a/docs/content/commands/rclone_completion_powershell.md b/docs/content/commands/rclone_completion_powershell.md
index cadfe617d..57b5a0615 100644
--- a/docs/content/commands/rclone_completion_powershell.md
+++ b/docs/content/commands/rclone_completion_powershell.md
@@ -1,6 +1,8 @@
 ---
 title: "rclone completion powershell"
 description: "Output powershell completion script for rclone."
+aliases:
+  - /commands/rclone_genautocomplete_powershell/
 # autogenerated - DO NOT EDIT, instead edit the source code in cmd/completion/powershell/ and as part of making a release run "make commanddocs"
 ---
 # rclone completion powershell
diff --git a/docs/content/commands/rclone_completion_zsh.md b/docs/content/commands/rclone_completion_zsh.md
index bec42f4c6..eb23fa995 100644
--- a/docs/content/commands/rclone_completion_zsh.md
+++ b/docs/content/commands/rclone_completion_zsh.md
@@ -1,6 +1,8 @@
 ---
 title: "rclone completion zsh"
 description: "Output zsh completion script for rclone."
+aliases:
+  - /commands/rclone_genautocomplete_zsh/
 # autogenerated - DO NOT EDIT, instead edit the source code in cmd/completion/zsh/ and as part of making a release run "make commanddocs"
 ---
 # rclone completion zsh
diff --git a/docs/content/commands/rclone_genautocomplete.md b/docs/content/commands/rclone_genautocomplete.md
deleted file mode 100644
index d9174320f..000000000
--- a/docs/content/commands/rclone_genautocomplete.md
+++ /dev/null
@@ -1,34 +0,0 @@
----
-title: "rclone genautocomplete"
-description: "Output completion script for a given shell."
-slug: rclone_genautocomplete
-url: /commands/rclone_genautocomplete/
-versionIntroduced: v1.33
-# autogenerated - DO NOT EDIT, instead edit the source code in cmd/genautocomplete/ and as part of making a release run "make commanddocs"
----
-# rclone genautocomplete
-
-Output completion script for a given shell.
-
-# Synopsis
-
-
-Generates a shell completion script for rclone.
-Run with `--help` to list the supported shells.
-
-
-# Options
-
-```
-  -h, --help   help for genautocomplete
-```
-
-See the [global flags page](/flags/) for global options not listed here.
-
-# SEE ALSO
-
-* [rclone](/commands/rclone/)	 - Show help for rclone commands, flags and backends.
-* [rclone genautocomplete bash](/commands/rclone_genautocomplete_bash/)	 - Output bash completion script for rclone.
-* [rclone genautocomplete fish](/commands/rclone_genautocomplete_fish/)	 - Output fish completion script for rclone.
-* [rclone genautocomplete zsh](/commands/rclone_genautocomplete_zsh/)	 - Output zsh completion script for rclone.
-
diff --git a/docs/content/commands/rclone_genautocomplete_bash.md b/docs/content/commands/rclone_genautocomplete_bash.md
deleted file mode 100644
index 8def9b4a9..000000000
--- a/docs/content/commands/rclone_genautocomplete_bash.md
+++ /dev/null
@@ -1,48 +0,0 @@
----
-title: "rclone genautocomplete bash"
-description: "Output bash completion script for rclone."
-slug: rclone_genautocomplete_bash
-url: /commands/rclone_genautocomplete_bash/
-# autogenerated - DO NOT EDIT, instead edit the source code in cmd/genautocomplete/bash/ and as part of making a release run "make commanddocs"
----
-# rclone genautocomplete bash
-
-Output bash completion script for rclone.
-
-# Synopsis
-
-
-Generates a bash shell autocompletion script for rclone.
-
-This writes to /etc/bash_completion.d/rclone by default so will
-probably need to be run with sudo or as root, e.g.
-
-    sudo rclone genautocomplete bash
-
-Logout and login again to use the autocompletion scripts, or source
-them directly
-
-    . /etc/bash_completion
-
-If you supply a command line argument the script will be written
-there.
-
-If output_file is "-", then the output will be written to stdout.
-
-
-```
-rclone genautocomplete bash [output_file] [flags]
-```
-
-# Options
-
-```
-  -h, --help   help for bash
-```
-
-See the [global flags page](/flags/) for global options not listed here.
-
-# SEE ALSO
-
-* [rclone genautocomplete](/commands/rclone_genautocomplete/)	 - Output completion script for a given shell.
-
diff --git a/docs/content/commands/rclone_genautocomplete_fish.md b/docs/content/commands/rclone_genautocomplete_fish.md
deleted file mode 100644
index db124f3ba..000000000
--- a/docs/content/commands/rclone_genautocomplete_fish.md
+++ /dev/null
@@ -1,48 +0,0 @@
----
-title: "rclone genautocomplete fish"
-description: "Output fish completion script for rclone."
-slug: rclone_genautocomplete_fish
-url: /commands/rclone_genautocomplete_fish/
-# autogenerated - DO NOT EDIT, instead edit the source code in cmd/genautocomplete/fish/ and as part of making a release run "make commanddocs"
----
-# rclone genautocomplete fish
-
-Output fish completion script for rclone.
-
-# Synopsis
-
-
-Generates a fish autocompletion script for rclone.
-
-This writes to /etc/fish/completions/rclone.fish by default so will
-probably need to be run with sudo or as root, e.g.
-
-    sudo rclone genautocomplete fish
-
-Logout and login again to use the autocompletion scripts, or source
-them directly
-
-    . /etc/fish/completions/rclone.fish
-
-If you supply a command line argument the script will be written
-there.
-
-If output_file is "-", then the output will be written to stdout.
-
-
-```
-rclone genautocomplete fish [output_file] [flags]
-```
-
-# Options
-
-```
-  -h, --help   help for fish
-```
-
-See the [global flags page](/flags/) for global options not listed here.
-
-# SEE ALSO
-
-* [rclone genautocomplete](/commands/rclone_genautocomplete/)	 - Output completion script for a given shell.
-
diff --git a/docs/content/commands/rclone_genautocomplete_zsh.md b/docs/content/commands/rclone_genautocomplete_zsh.md
deleted file mode 100644
index 33ca51342..000000000
--- a/docs/content/commands/rclone_genautocomplete_zsh.md
+++ /dev/null
@@ -1,48 +0,0 @@
----
-title: "rclone genautocomplete zsh"
-description: "Output zsh completion script for rclone."
-slug: rclone_genautocomplete_zsh
-url: /commands/rclone_genautocomplete_zsh/
-# autogenerated - DO NOT EDIT, instead edit the source code in cmd/genautocomplete/zsh/ and as part of making a release run "make commanddocs"
----
-# rclone genautocomplete zsh
-
-Output zsh completion script for rclone.
-
-# Synopsis
-
-
-Generates a zsh autocompletion script for rclone.
-
-This writes to /usr/share/zsh/vendor-completions/_rclone by default so will
-probably need to be run with sudo or as root, e.g.
-
-    sudo rclone genautocomplete zsh
-
-Logout and login again to use the autocompletion scripts, or source
-them directly
-
-    autoload -U compinit && compinit
-
-If you supply a command line argument the script will be written
-there.
-
-If output_file is "-", then the output will be written to stdout.
-
-
-```
-rclone genautocomplete zsh [output_file] [flags]
-```
-
-# Options
-
-```
-  -h, --help   help for zsh
-```
-
-See the [global flags page](/flags/) for global options not listed here.
-
-# SEE ALSO
-
-* [rclone genautocomplete](/commands/rclone_genautocomplete/)	 - Output completion script for a given shell.
-
diff --git a/docs/content/docs.md b/docs/content/docs.md
index c4ef41783..861ee963b 100644
--- a/docs/content/docs.md
+++ b/docs/content/docs.md
@@ -153,7 +153,7 @@ The main rclone commands with most used first
 * [rclone authorize](/commands/rclone_authorize/)	- Remote authorization.
 * [rclone cat](/commands/rclone_cat/)		- Concatenate any files and send them to stdout.
 * [rclone copyto](/commands/rclone_copyto/)	- Copy files from source to dest, skipping already copied.
-* [rclone genautocomplete](/commands/rclone_genautocomplete/)	- Output shell completion scripts for rclone.
+* [rclone completion](/commands/rclone_completion/)	- Output shell completion scripts for rclone.
 * [rclone gendocs](/commands/rclone_gendocs/)	- Output markdown docs for rclone to the directory supplied.
 * [rclone listremotes](/commands/rclone_listremotes/)	- List all the remotes in the config file.
 * [rclone mount](/commands/rclone_mount/)	- Mount the remote as a mountpoint.
@@ -2608,7 +2608,7 @@ a configuration file, you can avoid it being loaded by overriding the
 location, e.g. with one of the documented special values for
 memory-only configuration. Since only backend options can be stored
 in configuration files, this is normally unnecessary for commands
-that do not operate on backends, e.g. `genautocomplete`. However,
+that do not operate on backends, e.g. `completion`. However,
 it will be relevant for commands that do operate on backends in
 general, but are used without referencing a stored remote, e.g.
 listing local filesystem paths, or
diff --git a/docs/content/s3.md b/docs/content/s3.md
index c4ccbab89..6f9aef1b9 100644
--- a/docs/content/s3.md
+++ b/docs/content/s3.md
@@ -571,15 +571,21 @@ The different authentication methods are tried in this order:
      - Session Token: `AWS_SESSION_TOKEN` (optional)
    - Or, use a [named profile](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html):
      - Profile files are standard files used by AWS CLI tools
-     - By default it will use the profile in your home directory (e.g. `~/.aws/credentials` on unix based systems) file and the "default" profile, to change set these environment variables:
-         - `AWS_SHARED_CREDENTIALS_FILE` to control which file.
-         - `AWS_PROFILE` to control which profile to use.
+     - By default it will use the profile in your home directory (e.g. `~/.aws/credentials` on unix based systems) file and the "default" profile, to change set these environment variables or config keys:
+         - `AWS_SHARED_CREDENTIALS_FILE` to control which file or the `shared_credentials_file` config key.
+         - `AWS_PROFILE` to control which profile to use or the `profile` config key.
    - Or, run `rclone` in an ECS task with an IAM role (AWS only).
    - Or, run `rclone` on an EC2 instance with an IAM role (AWS only).
    - Or, run `rclone` in an EKS pod with an IAM role that is associated with a service account (AWS only).
+   - Or, use [process credentials](https://docs.aws.amazon.com/sdkref/latest/guide/feature-process-credentials.html) to read config from an external program.
+
+With `env_auth = true` rclone (which uses the SDK for Go v2) should support
+[all authentication methods](https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html)
+that the `aws` CLI tool does and the other AWS SDKs.
 
 If none of these option actually end up providing `rclone` with AWS
-credentials then S3 interaction will be non-authenticated (see below).
+credentials then S3 interaction will be non-authenticated (see the
+[anonymous access](#anonymous-access) section for more info).
 
 ### S3 Permissions
 
@@ -2171,7 +2177,7 @@ It doesn't return anything.
 
 {{< rem autogenerated options stop >}}
 
-### Anonymous access to public buckets
+### Anonymous access to public buckets {#anonymous-access}
 
 If you want to use rclone to access a public bucket, configure with a
 blank `access_key_id` and `secret_access_key`.  Your config should end
@@ -2181,15 +2187,6 @@ up looking like this:
 [anons3]
 type = s3
 provider = AWS
-env_auth = false
-access_key_id =
-secret_access_key =
-region = us-east-1
-endpoint =
-location_constraint =
-acl = private
-server_side_encryption =
-storage_class =
 ```
 
 Then use it as normal with the name of the public bucket, e.g.
@@ -2198,6 +2195,10 @@ Then use it as normal with the name of the public bucket, e.g.
 
 You will be able to list and copy data but not upload it.
 
+You can also do this entirely on the command line
+
+    rclone lsd :s3,provider=AWS:1000genomes
+
 ## Providers
 
 ### AWS S3
@@ -2387,7 +2388,14 @@ acl = private
 Now run `rclone lsf r2:` to see your buckets and `rclone lsf
 r2:bucket` to look within a bucket.
 
-For R2 tokens with the "Object Read & Write" permission, you may also need to add `no_check_bucket = true` for object uploads to work correctly.
+For R2 tokens with the "Object Read & Write" permission, you may also
+need to add `no_check_bucket = true` for object uploads to work
+correctly.
+
+Note that Cloudflare decompresses files uploaded with
+`Content-Encoding: gzip` by default which is a deviation from what AWS
+does. If this is causing a problem then upload the files with
+`--header-upload "Cache-Control: no-transform"`
 
 ### Dreamhost
 
diff --git a/fs/config/config.go b/fs/config/config.go
index c4a56f096..be0a2a914 100644
--- a/fs/config/config.go
+++ b/fs/config/config.go
@@ -121,7 +121,7 @@ var (
 
 func init() {
 	// Set the function pointers up in fs
-	fs.ConfigFileGet = FileGetFlag
+	fs.ConfigFileGet = FileGetValue
 	fs.ConfigFileSet = SetValueAndSave
 	fs.ConfigFileHasSection = func(section string) bool {
 		return LoadedData().HasSection(section)
@@ -351,6 +351,9 @@ func Data() Storage {
 	return data
 }
 
+// ErrorConfigFileNotFound is returned when the config file is not found
+var ErrorConfigFileNotFound = errors.New("config file not found")
+
 // LoadedData ensures the config file storage is loaded and returns it
 func LoadedData() Storage {
 	if !dataLoaded {
@@ -375,9 +378,6 @@ func LoadedData() Storage {
 	return data
 }
 
-// ErrorConfigFileNotFound is returned when the config file is not found
-var ErrorConfigFileNotFound = errors.New("config file not found")
-
 // SaveConfig calling function which saves configuration file.
 // if SaveConfig returns error trying again after sleep.
 func SaveConfig() {
@@ -394,25 +394,116 @@ func SaveConfig() {
 	fs.Errorf(nil, "Failed to save config after %d tries: %v", ci.LowLevelRetries, err)
 }
 
+// FileSections returns the sections in the config file
+func FileSections() []string {
+	return LoadedData().GetSectionList()
+}
+
+// FileGetValue gets the config key under section returning the
+// the value and true if found and or ("", false) otherwise
+func FileGetValue(section, key string) (string, bool) {
+	return LoadedData().GetValue(section, key)
+}
+
+// FileSetValue sets the key in section to value.
+// It doesn't save the config file.
+func FileSetValue(section, key, value string) {
+	LoadedData().SetValue(section, key, value)
+}
+
+// FileDeleteKey deletes the config key in the config file.
+// It returns true if the key was deleted,
+// or returns false if the section or key didn't exist.
+func FileDeleteKey(section, key string) bool {
+	return LoadedData().DeleteKey(section, key)
+}
+
+// GetValue gets the value for a config key from environment
+// or config file under section returning the default if not set.
+//
+// Emulates the preference documented and normally used by rclone via
+// configmap, which means environment variables before config file.
+func GetValue(remote, key string) string {
+	envKey := fs.ConfigToEnv(remote, key)
+	value, found := os.LookupEnv(envKey)
+	if found {
+		return value
+	}
+	value, _ = LoadedData().GetValue(remote, key)
+	return value
+}
+
 // SetValueAndSave sets the key to the value and saves just that
 // value in the config file.  It loads the old config file in from
 // disk first and overwrites the given value only.
-func SetValueAndSave(name, key, value string) error {
+func SetValueAndSave(remote, key, value string) error {
 	// Set the value in config in case we fail to reload it
-	LoadedData().SetValue(name, key, value)
+	FileSetValue(remote, key, value)
 	// Save it again
 	SaveConfig()
 	return nil
 }
 
-// getWithDefault gets key out of section name returning defaultValue if not
-// found.
-func getWithDefault(name, key, defaultValue string) string {
-	value, found := LoadedData().GetValue(name, key)
-	if !found {
-		return defaultValue
+// Remote defines a remote with a name, type, source and description
+type Remote struct {
+	Name        string `json:"name"`
+	Type        string `json:"type"`
+	Source      string `json:"source"`
+	Description string `json:"description"`
+}
+
+var remoteEnvRe = regexp.MustCompile(`^RCLONE_CONFIG_(.+?)_TYPE=(.+)$`)
+
+// GetRemotes returns the list of remotes defined in environment and config file.
+//
+// Emulates the preference documented and normally used by rclone via
+// configmap, which means environment variables before config file.
+func GetRemotes() []Remote {
+	var remotes []Remote
+	for _, item := range os.Environ() {
+		matches := remoteEnvRe.FindStringSubmatch(item)
+		if len(matches) == 3 {
+			remotes = append(remotes, Remote{
+				Name:   strings.ToLower(matches[1]),
+				Type:   strings.ToLower(matches[2]),
+				Source: "environment",
+			})
+		}
 	}
-	return value
+	remoteExists := func(name string) bool {
+		for _, remote := range remotes {
+			if name == remote.Name {
+				return true
+			}
+		}
+		return false
+	}
+	sections := LoadedData().GetSectionList()
+	for _, section := range sections {
+		if !remoteExists(section) {
+			typeValue, found := LoadedData().GetValue(section, "type")
+			if found {
+				description, _ := LoadedData().GetValue(section, "description")
+				remotes = append(remotes, Remote{
+					Name:        section,
+					Type:        typeValue,
+					Source:      "file",
+					Description: description,
+				})
+			}
+		}
+	}
+	return remotes
+}
+
+// GetRemoteNames returns the names of remotes defined in environment and config file.
+func GetRemoteNames() []string {
+	remotes := GetRemotes()
+	var remoteNames []string
+	for _, remote := range remotes {
+		remoteNames = append(remoteNames, remote.Name)
+	}
+	return remoteNames
 }
 
 // UpdateRemoteOpt configures the remote update
@@ -449,7 +540,7 @@ func updateRemote(ctx context.Context, name string, keyValues rc.Params, opt Upd
 		ctx = suppressConfirm(ctx)
 	}
 
-	fsType := FileGet(name, "type")
+	fsType := GetValue(name, "type")
 	if fsType == "" {
 		return nil, errors.New("couldn't find type field in config")
 	}
@@ -601,62 +692,11 @@ func fsOption() *fs.Option {
 	return o
 }
 
-// FileGetFlag gets the config key under section returning the
-// the value and true if found and or ("", false) otherwise
-func FileGetFlag(section, key string) (string, bool) {
-	return LoadedData().GetValue(section, key)
-}
-
-// FileGet gets the config key under section returning the default if not set.
-//
-// It looks up defaults in the environment if they are present
-func FileGet(section, key string) string {
-	var defaultVal string
-	envKey := fs.ConfigToEnv(section, key)
-	newValue, found := os.LookupEnv(envKey)
-	if found {
-		defaultVal = newValue
-	}
-	return getWithDefault(section, key, defaultVal)
-}
-
-// FileSet sets the key in section to value.  It doesn't save
-// the config file.
-func FileSet(section, key, value string) {
-	if value != "" {
-		LoadedData().SetValue(section, key, value)
-	} else {
-		FileDeleteKey(section, key)
-	}
-}
-
-// FileDeleteKey deletes the config key in the config file.
-// It returns true if the key was deleted,
-// or returns false if the section or key didn't exist.
-func FileDeleteKey(section, key string) bool {
-	return LoadedData().DeleteKey(section, key)
-}
-
-var matchEnv = regexp.MustCompile(`^RCLONE_CONFIG_(.*?)_TYPE=.*$`)
-
-// FileSections returns the sections in the config file
-// including any defined by environment variables.
-func FileSections() []string {
-	sections := LoadedData().GetSectionList()
-	for _, item := range os.Environ() {
-		matches := matchEnv.FindStringSubmatch(item)
-		if len(matches) == 2 {
-			sections = append(sections, strings.ToLower(matches[1]))
-		}
-	}
-	return sections
-}
-
 // DumpRcRemote dumps the config for a single remote
 func DumpRcRemote(name string) (dump rc.Params) {
 	params := rc.Params{}
 	for _, key := range LoadedData().GetKeyList(name) {
-		params[key] = FileGet(name, key)
+		params[key] = GetValue(name, key)
 	}
 	return params
 }
diff --git a/fs/config/flags/flags.go b/fs/config/flags/flags.go
index 25183fa0f..10b1fd77f 100644
--- a/fs/config/flags/flags.go
+++ b/fs/config/flags/flags.go
@@ -50,16 +50,17 @@ func (gs *Groups) NewGroup(name, help string) *Group {
 }
 
 // Filter makes a copy of groups filtered by flagsRe
-func (gs *Groups) Filter(flagsRe *regexp.Regexp) *Groups {
+func (gs *Groups) Filter(group string, filterRe *regexp.Regexp, filterNamesOnly bool) *Groups {
 	newGs := NewGroups()
 	for _, g := range gs.Groups {
-		newG := newGs.NewGroup(g.Name, g.Help)
-		g.Flags.VisitAll(func(f *pflag.Flag) {
-			matched := flagsRe == nil || flagsRe.MatchString(f.Name) || flagsRe.MatchString(f.Usage)
-			if matched {
-				newG.Flags.AddFlag(f)
-			}
-		})
+		if group == "" || strings.EqualFold(g.Name, group) {
+			newG := newGs.NewGroup(g.Name, g.Help)
+			g.Flags.VisitAll(func(f *pflag.Flag) {
+				if filterRe == nil || filterRe.MatchString(f.Name) || (!filterNamesOnly && filterRe.MatchString(f.Usage)) {
+					newG.Flags.AddFlag(f)
+				}
+			})
+		}
 	}
 	return newGs
 }
diff --git a/fs/config/rc.go b/fs/config/rc.go
index ba75b8f7b..2d7a26809 100644
--- a/fs/config/rc.go
+++ b/fs/config/rc.go
@@ -74,9 +74,9 @@ See the [listremotes](/commands/rclone_listremotes/) command for more informatio
 // Return the a list of remotes in the config file
 // including any defined by environment variables.
 func rcListRemotes(ctx context.Context, in rc.Params) (out rc.Params, err error) {
-	remotes := FileSections()
+	remoteNames := GetRemoteNames()
 	out = rc.Params{
-		"remotes": remotes,
+		"remotes": remoteNames,
 	}
 	return out, nil
 }
diff --git a/fs/config/ui.go b/fs/config/ui.go
index 7c99871b2..ff8acda0f 100644
--- a/fs/config/ui.go
+++ b/fs/config/ui.go
@@ -280,7 +280,7 @@ func ShowRemotes() {
 	fmt.Printf("%-20s %s\n", "Name", "Type")
 	fmt.Printf("%-20s %s\n", "====", "====")
 	for _, remote := range remotes {
-		fmt.Printf("%-20s %s\n", remote, FileGet(remote, "type"))
+		fmt.Printf("%-20s %s\n", remote, GetValue(remote, "type"))
 	}
 }
 
@@ -295,7 +295,7 @@ func ChooseRemote() string {
 // mustFindByName finds the RegInfo for the remote name passed in or
 // exits with a fatal error.
 func mustFindByName(name string) *fs.RegInfo {
-	fsType := FileGet(name, "type")
+	fsType := GetValue(name, "type")
 	if fsType == "" {
 		log.Fatalf("Couldn't find type of fs for %q", name)
 	}
@@ -305,7 +305,7 @@ func mustFindByName(name string) *fs.RegInfo {
 // findByName finds the RegInfo for the remote name passed in or
 // returns an error
 func findByName(name string) (*fs.RegInfo, error) {
-	fsType := FileGet(name, "type")
+	fsType := GetValue(name, "type")
 	if fsType == "" {
 		return nil, fmt.Errorf("couldn't find type of fs for %q", name)
 	}
@@ -333,7 +333,7 @@ func printRemoteOptions(name string, prefix string, sep string, redacted bool) {
 				}
 			}
 		}
-		value := FileGet(name, key)
+		value := GetValue(name, key)
 		if redacted && (isSensitive || isPassword) && value != "" {
 			fmt.Printf("%s%s%sXXX\n", prefix, key, sep)
 		} else if isPassword && value != "" {
@@ -613,7 +613,7 @@ func copyRemote(name string) string {
 	newName := NewRemoteName()
 	// Copy the keys
 	for _, key := range LoadedData().GetKeyList(name) {
-		value := getWithDefault(name, key, "")
+		value, _ := FileGetValue(name, key)
 		LoadedData().SetValue(newName, key, value)
 	}
 	return newName
diff --git a/fs/filter/filter.go b/fs/filter/filter.go
index 408165080..61c1a2556 100644
--- a/fs/filter/filter.go
+++ b/fs/filter/filter.go
@@ -258,7 +258,7 @@ func (f *Filter) addDirGlobs(Include bool, glob string) error {
 		if dirGlob == "/" {
 			continue
 		}
-		dirRe, err := GlobToRegexp(dirGlob, f.Opt.IgnoreCase)
+		dirRe, err := GlobPathToRegexp(dirGlob, f.Opt.IgnoreCase)
 		if err != nil {
 			return err
 		}
@@ -278,7 +278,7 @@ func (f *Filter) Add(Include bool, glob string) error {
 	if strings.Contains(glob, "**") {
 		isDirRule, isFileRule = true, true
 	}
-	re, err := GlobToRegexp(glob, f.Opt.IgnoreCase)
+	re, err := GlobPathToRegexp(glob, f.Opt.IgnoreCase)
 	if err != nil {
 		return err
 	}
diff --git a/fs/filter/glob.go b/fs/filter/glob.go
index 1ad53ccdb..de48da6d7 100644
--- a/fs/filter/glob.go
+++ b/fs/filter/glob.go
@@ -11,30 +11,59 @@ import (
 	"github.com/rclone/rclone/fs"
 )
 
-// GlobToRegexp converts an rsync style glob to a regexp
+// GlobPathToRegexp converts an rsync style glob path to a regexp
+func GlobPathToRegexp(glob string, ignoreCase bool) (*regexp.Regexp, error) {
+	return globToRegexp(glob, true, true, ignoreCase)
+}
+
+// GlobStringToRegexp converts an rsync style glob string to a regexp
+func GlobStringToRegexp(glob string, ignoreCase bool) (*regexp.Regexp, error) {
+	return globToRegexp(glob, false, true, ignoreCase)
+}
+
+// globToRegexp converts an rsync style glob to a regexp
 //
-// documented in filtering.md
-func GlobToRegexp(glob string, ignoreCase bool) (*regexp.Regexp, error) {
+// Set pathMode true for matching of path/file names, e.g.
+// special treatment of path separator `/` and double asterisk `**`,
+// see filtering.md for details.
+//
+// Set addAnchors true to add start of string `^` and end of string `$` anchors.
+func globToRegexp(glob string, pathMode bool, addAnchors bool, ignoreCase bool) (*regexp.Regexp, error) {
 	var re bytes.Buffer
 	if ignoreCase {
 		_, _ = re.WriteString("(?i)")
 	}
-	if strings.HasPrefix(glob, "/") {
-		glob = glob[1:]
-		_ = re.WriteByte('^')
-	} else {
-		_, _ = re.WriteString("(^|/)")
+	if addAnchors {
+		if pathMode {
+			if strings.HasPrefix(glob, "/") {
+				glob = glob[1:]
+				_ = re.WriteByte('^')
+			} else {
+				_, _ = re.WriteString("(^|/)")
+			}
+		} else {
+			_, _ = re.WriteString("^")
+		}
 	}
 	consecutiveStars := 0
 	insertStars := func() error {
 		if consecutiveStars > 0 {
-			switch consecutiveStars {
-			case 1:
-				_, _ = re.WriteString(`[^/]*`)
-			case 2:
-				_, _ = re.WriteString(`.*`)
-			default:
-				return fmt.Errorf("too many stars in %q", glob)
+			if pathMode {
+				switch consecutiveStars {
+				case 1:
+					_, _ = re.WriteString(`[^/]*`)
+				case 2:
+					_, _ = re.WriteString(`.*`)
+				default:
+					return fmt.Errorf("too many stars in %q", glob)
+				}
+			} else {
+				switch consecutiveStars {
+				case 1:
+					_, _ = re.WriteString(`.*`)
+				default:
+					return fmt.Errorf("too many stars in %q", glob)
+				}
 			}
 		}
 		consecutiveStars = 0
@@ -102,7 +131,11 @@ func GlobToRegexp(glob string, ignoreCase bool) (*regexp.Regexp, error) {
 		case '*':
 			consecutiveStars++
 		case '?':
-			_, _ = re.WriteString(`[^/]`)
+			if pathMode {
+				_, _ = re.WriteString(`[^/]`)
+			} else {
+				_, _ = re.WriteString(`.`)
+			}
 		case '[':
 			_, _ = re.WriteRune(c)
 			inBrackets++
@@ -152,7 +185,9 @@ func GlobToRegexp(glob string, ignoreCase bool) (*regexp.Regexp, error) {
 	if inRegexp {
 		return nil, fmt.Errorf("mismatched '{{' and '}}' in glob %q", glob)
 	}
-	_ = re.WriteByte('$')
+	if addAnchors {
+		_ = re.WriteByte('$')
+	}
 	result, err := regexp.Compile(re.String())
 	if err != nil {
 		return nil, fmt.Errorf("bad glob pattern %q (regexp %q): %w", glob, re.String(), err)
diff --git a/fs/filter/rules.go b/fs/filter/rules.go
index dce5851a6..cffca0d20 100644
--- a/fs/filter/rules.go
+++ b/fs/filter/rules.go
@@ -67,7 +67,7 @@ func (rs *rules) add(Include bool, re *regexp.Regexp) {
 
 // Add adds a filter rule with include or exclude status indicated
 func (rs *rules) Add(Include bool, glob string) error {
-	re, err := GlobToRegexp(glob, false /* f.Opt.IgnoreCase */)
+	re, err := GlobPathToRegexp(glob, false /* f.Opt.IgnoreCase */)
 	if err != nil {
 		return err
 	}
diff --git a/fs/rc/rcserver/rcserver.go b/fs/rc/rcserver/rcserver.go
index 7e20213ab..88f82a484 100644
--- a/fs/rc/rcserver/rcserver.go
+++ b/fs/rc/rcserver/rcserver.go
@@ -308,14 +308,14 @@ func (s *Server) handleOptions(w http.ResponseWriter, r *http.Request, path stri
 }
 
 func (s *Server) serveRoot(w http.ResponseWriter, r *http.Request) {
-	remotes := config.FileSections()
-	sort.Strings(remotes)
+	remoteNames := config.GetRemoteNames()
+	sort.Strings(remoteNames)
 	directory := serve.NewDirectory("", s.server.HTMLTemplate())
 	directory.Name = "List of all rclone remotes."
 	q := url.Values{}
-	for _, remote := range remotes {
-		q.Set("fs", remote)
-		directory.AddHTMLEntry("["+remote+":]", true, -1, time.Time{})
+	for _, remoteName := range remoteNames {
+		q.Set("fs", remoteName)
+		directory.AddHTMLEntry("["+remoteName+":]", true, -1, time.Time{})
 	}
 	sortParm := r.URL.Query().Get("sort")
 	orderParm := r.URL.Query().Get("order")
diff --git a/go.mod b/go.mod
index 7c1623df0..45963cfcd 100644
--- a/go.mod
+++ b/go.mod
@@ -17,7 +17,12 @@ require (
 	github.com/anacrolix/dms v1.7.1
 	github.com/anacrolix/log v0.15.2
 	github.com/atotto/clipboard v0.1.4
-	github.com/aws/aws-sdk-go v1.54.19
+	github.com/aws/aws-sdk-go-v2 v1.30.3
+	github.com/aws/aws-sdk-go-v2/config v1.27.27
+	github.com/aws/aws-sdk-go-v2/credentials v1.17.27
+	github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.10
+	github.com/aws/aws-sdk-go-v2/service/s3 v1.58.3
+	github.com/aws/smithy-go v1.20.3
 	github.com/buengese/sgzip v0.1.1
 	github.com/cloudsoda/go-smb2 v0.0.0-20231124195312-f3ec8ae2c891
 	github.com/colinmarc/hdfs/v2 v2.4.0
@@ -52,7 +57,7 @@ require (
 	github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
 	github.com/prometheus/client_golang v1.19.1
 	github.com/putdotio/go-putio/putio v0.0.0-20200123120452-16d982cac2b8
-	github.com/rclone/gofakes3 v0.0.3-0.20240716093803-d6abc178be56
+	github.com/rclone/gofakes3 v0.0.3-0.20240807151802-e80146f8de87
 	github.com/rfjakob/eme v1.1.2
 	github.com/rivo/uniseg v0.4.7
 	github.com/rogpeppe/go-internal v1.12.0
@@ -99,6 +104,19 @@ require (
 	github.com/akavel/rsrc v0.10.2 // indirect
 	github.com/anacrolix/generics v0.0.1 // indirect
 	github.com/andybalholm/cascadia v1.3.2 // indirect
+	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.3 // indirect
+	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.11 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.15 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.15 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.15 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.3 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.17 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.17 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.15 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sso v1.22.4 // indirect
+	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.26.4 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sts v1.30.3 // indirect
 	github.com/beorn7/perks v1.0.1 // indirect
 	github.com/bradenaw/juniper v0.15.2 // indirect
 	github.com/calebcase/tmpfile v1.0.3 // indirect
@@ -139,7 +157,6 @@ require (
 	github.com/jcmturner/gofork v1.7.6 // indirect
 	github.com/jcmturner/goidentity/v6 v6.0.1 // indirect
 	github.com/jcmturner/rpc/v2 v2.0.3 // indirect
-	github.com/jmespath/go-jmespath v0.4.0 // indirect
 	github.com/jtolio/noiseconn v0.0.0-20231127013910-f6d9ecbf1de7 // indirect
 	github.com/klauspost/cpuid/v2 v2.2.8 // indirect
 	github.com/kr/fs v0.1.0 // indirect
diff --git a/go.sum b/go.sum
index f0aaa8f62..21a995234 100644
--- a/go.sum
+++ b/go.sum
@@ -102,8 +102,44 @@ github.com/andybalholm/cascadia v1.3.2 h1:3Xi6Dw5lHF15JtdcmAHD3i1+T8plmv7BQ/nsVi
 github.com/andybalholm/cascadia v1.3.2/go.mod h1:7gtRlve5FxPPgIgX36uWBX58OdBsSS6lUvCFb+h7KvU=
 github.com/atotto/clipboard v0.1.4 h1:EH0zSVneZPSuFR11BlR9YppQTVDbh5+16AmcJi4g1z4=
 github.com/atotto/clipboard v0.1.4/go.mod h1:ZY9tmq7sm5xIbd9bOK4onWV4S6X0u6GY7Vn0Yu86PYI=
-github.com/aws/aws-sdk-go v1.54.19 h1:tyWV+07jagrNiCcGRzRhdtVjQs7Vy41NwsuOcl0IbVI=
-github.com/aws/aws-sdk-go v1.54.19/go.mod h1:eRwEWoyTWFMVYVQzKMNHWP5/RV4xIUGMQfXQHfHkpNU=
+github.com/aws/aws-sdk-go-v2 v1.30.3 h1:jUeBtG0Ih+ZIFH0F4UkmL9w3cSpaMv9tYYDbzILP8dY=
+github.com/aws/aws-sdk-go-v2 v1.30.3/go.mod h1:nIQjQVp5sfpQcTc9mPSr1B0PaWK5ByX9MOoDadSN4lc=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.3 h1:tW1/Rkad38LA15X4UQtjXZXNKsCgkshC3EbmcUmghTg=
+github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.3/go.mod h1:UbnqO+zjqk3uIt9yCACHJ9IVNhyhOCnYk8yA19SAWrM=
+github.com/aws/aws-sdk-go-v2/config v1.27.27 h1:HdqgGt1OAP0HkEDDShEl0oSYa9ZZBSOmKpdpsDMdO90=
+github.com/aws/aws-sdk-go-v2/config v1.27.27/go.mod h1:MVYamCg76dFNINkZFu4n4RjDixhVr51HLj4ErWzrVwg=
+github.com/aws/aws-sdk-go-v2/credentials v1.17.27 h1:2raNba6gr2IfA0eqqiP2XiQ0UVOpGPgDSi0I9iAP+UI=
+github.com/aws/aws-sdk-go-v2/credentials v1.17.27/go.mod h1:gniiwbGahQByxan6YjQUMcW4Aov6bLC3m+evgcoN4r4=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.11 h1:KreluoV8FZDEtI6Co2xuNk/UqI9iwMrOx/87PBNIKqw=
+github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.11/go.mod h1:SeSUYBLsMYFoRvHE0Tjvn7kbxaUhl75CJi1sbfhMxkU=
+github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.10 h1:zeN9UtUlA6FTx0vFSayxSX32HDw73Yb6Hh2izDSFxXY=
+github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.10/go.mod h1:3HKuexPDcwLWPaqpW2UR/9n8N/u/3CKcGAzSs8p8u8g=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.15 h1:SoNJ4RlFEQEbtDcCEt+QG56MY4fm4W8rYirAmq+/DdU=
+github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.15/go.mod h1:U9ke74k1n2bf+RIgoX1SXFed1HLs51OgUSs+Ph0KJP8=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.15 h1:C6WHdGnTDIYETAm5iErQUiVNsclNx9qbJVPIt03B6bI=
+github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.15/go.mod h1:ZQLZqhcu+JhSrA9/NXRm8SkDvsycE+JkV3WGY41e+IM=
+github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 h1:hT8rVHwugYE2lEfdFE0QWVo81lF7jMrYJVDWI+f+VxU=
+github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0/go.mod h1:8tu/lYfQfFe6IGnaOdrpVgEL2IrrDOf6/m9RQum4NkY=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.15 h1:Z5r7SycxmSllHYmaAZPpmN8GviDrSGhMS6bldqtXZPw=
+github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.15/go.mod h1:CetW7bDE00QoGEmPUoZuRog07SGVAUVW6LFpNP0YfIg=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.3 h1:dT3MqvGhSoaIhRseqw2I0yH81l7wiR2vjs57O51EAm8=
+github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.3/go.mod h1:GlAeCkHwugxdHaueRr4nhPuY+WW+gR8UjlcqzPr1SPI=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.17 h1:YPYe6ZmvUfDDDELqEKtAd6bo8zxhkm+XEFEzQisqUIE=
+github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.17/go.mod h1:oBtcnYua/CgzCWYN7NZ5j7PotFDaFSUjCYVTtfyn7vw=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.17 h1:HGErhhrxZlQ044RiM+WdoZxp0p+EGM62y3L6pwA4olE=
+github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.17/go.mod h1:RkZEx4l0EHYDJpWppMJ3nD9wZJAa8/0lq9aVC+r2UII=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.15 h1:246A4lSTXWJw/rmlQI+TT2OcqeDMKBdyjEQrafMaQdA=
+github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.15/go.mod h1:haVfg3761/WF7YPuJOER2MP0k4UAXyHaLclKXB6usDg=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.58.3 h1:hT8ZAZRIfqBqHbzKTII+CIiY8G2oC9OpLedkZ51DWl8=
+github.com/aws/aws-sdk-go-v2/service/s3 v1.58.3/go.mod h1:Lcxzg5rojyVPU/0eFwLtcyTaek/6Mtic5B1gJo7e/zE=
+github.com/aws/aws-sdk-go-v2/service/sso v1.22.4 h1:BXx0ZIxvrJdSgSvKTZ+yRBeSqqgPM89VPlulEcl37tM=
+github.com/aws/aws-sdk-go-v2/service/sso v1.22.4/go.mod h1:ooyCOXjvJEsUw7x+ZDHeISPMhtwI3ZCB7ggFMcFfWLU=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.26.4 h1:yiwVzJW2ZxZTurVbYWA7QOrAaCYQR72t0wrSBfoesUE=
+github.com/aws/aws-sdk-go-v2/service/ssooidc v1.26.4/go.mod h1:0oxfLkpz3rQ/CHlx5hB7H69YUpFiI1tql6Q6Ne+1bCw=
+github.com/aws/aws-sdk-go-v2/service/sts v1.30.3 h1:ZsDKRLXGWHk8WdtyYMoGNO7bTudrvuKpDKgMVRlepGE=
+github.com/aws/aws-sdk-go-v2/service/sts v1.30.3/go.mod h1:zwySh8fpFyXp9yOr/KVzxOl8SRqgf/IDw5aUt9UKFcQ=
+github.com/aws/smithy-go v1.20.3 h1:ryHwveWzPV5BIof6fyDvor6V3iUL7nTfiTKXHiW05nE=
+github.com/aws/smithy-go v1.20.3/go.mod h1:krry+ya/rV9RDcV/Q16kpu6ypI4K2czasz0NC3qS14E=
 github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
 github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
 github.com/bradenaw/juniper v0.15.2 h1:0JdjBGEF2jP1pOxmlNIrPhAoQN7Ng5IMAY5D0PHMW4U=
@@ -335,10 +371,6 @@ github.com/jcmturner/rpc/v2 v2.0.3/go.mod h1:VUJYCIDm3PVOEHw8sgt091/20OJjskO/YJk
 github.com/jlaffaye/ftp v0.0.0-20190624084859-c1312a7102bf/go.mod h1:lli8NYPQOFy3O++YmYbqVgOcQ1JPCwdOy+5zSjKJ9qY=
 github.com/jlaffaye/ftp v0.2.0 h1:lXNvW7cBu7R/68bknOX3MrRIIqZ61zELs1P2RAiA3lg=
 github.com/jlaffaye/ftp v0.2.0/go.mod h1:is2Ds5qkhceAPy2xD6RLI6hmp/qysSoymZ+Z2uTnspI=
-github.com/jmespath/go-jmespath v0.4.0 h1:BEgLn5cpjn8UN1mAw4NjwDrS35OdebyEtFe+9YPoQUg=
-github.com/jmespath/go-jmespath v0.4.0/go.mod h1:T8mJZnbsbmF+m6zOOFylbeCJqk5+pHWvzYPziyZiYoo=
-github.com/jmespath/go-jmespath/internal/testify v1.5.1 h1:shLQSRRSCCPj3f2gpwzGwWFoC7ycTf1rcQZHOlsJ6N8=
-github.com/jmespath/go-jmespath/internal/testify v1.5.1/go.mod h1:L3OGu8Wl2/fWfCI6z80xFu9LTZmf1ZRjMHUOPmWr69U=
 github.com/josephspurrier/goversioninfo v1.4.0 h1:Puhl12NSHUSALHSuzYwPYQkqa2E1+7SrtAPJorKK0C8=
 github.com/josephspurrier/goversioninfo v1.4.0/go.mod h1:JWzv5rKQr+MmW+LvM412ToT/IkYDZjaclF2pKDss8IY=
 github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
@@ -458,8 +490,8 @@ github.com/quic-go/quic-go v0.40.1 h1:X3AGzUNFs0jVuO3esAGnTfvdgvL4fq655WaOi1snv1
 github.com/quic-go/quic-go v0.40.1/go.mod h1:PeN7kuVJ4xZbxSv/4OX6S1USOX8MJvydwpTx31vx60c=
 github.com/rasky/go-xdr v0.0.0-20170124162913-1a41d1a06c93 h1:UVArwN/wkKjMVhh2EQGC0tEc1+FqiLlvYXY5mQ2f8Wg=
 github.com/rasky/go-xdr v0.0.0-20170124162913-1a41d1a06c93/go.mod h1:Nfe4efndBz4TibWycNE+lqyJZiMX4ycx+QKV8Ta0f/o=
-github.com/rclone/gofakes3 v0.0.3-0.20240716093803-d6abc178be56 h1:JmCt3EsTnlZrg/PHIyZqvKDRvBCde/rmThAQFliE9bU=
-github.com/rclone/gofakes3 v0.0.3-0.20240716093803-d6abc178be56/go.mod h1:L0VIBE0mT6ArN/5dfHsJm3UjqCpi5B/cdN+qWDNh7ko=
+github.com/rclone/gofakes3 v0.0.3-0.20240807151802-e80146f8de87 h1:0YRo2aYhE+SCZsjWYMFe8zLD18xieXy7wQ8M9Ywcr/g=
+github.com/rclone/gofakes3 v0.0.3-0.20240807151802-e80146f8de87/go.mod h1:z7+o2VUwitO0WuVHReQlOW9jZ03LpeJ0PUFSULyTIds=
 github.com/relvacode/iso8601 v1.3.0 h1:HguUjsGpIMh/zsTczGN3DVJFxTU/GX+MMmzcKoMO7ko=
 github.com/relvacode/iso8601 v1.3.0/go.mod h1:FlNp+jz+TXpyRqgmM7tnzHHzBnz776kmAH2h3sZCn0I=
 github.com/rfjakob/eme v1.1.2 h1:SxziR8msSOElPayZNFfQw4Tjx/Sbaeeh3eRvrHVMUs4=
@@ -950,7 +982,6 @@ gopkg.in/validator.v2 v2.0.1 h1:xF0KWyGWXm/LM2G1TrEjqOu4pa6coO9AlWSf3msVfDY=
 gopkg.in/validator.v2 v2.0.1/go.mod h1:lIUZBlB3Im4s/eYp39Ry/wkR02yOPhZ9IwIRBjuPuG8=
 gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v2 v2.2.8/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v2 v2.4.0 h1:D8xgwECY7CYvx+Y2n4sBz93Jn9JRvxdiyyo8CTfuKaY=
 gopkg.in/yaml.v2 v2.4.0/go.mod h1:RDklbk79AGWmwhnvt/jBztapEOGDOx6ZbXqjP6csGnQ=


Test patch:
diff --git a/backend/alias/alias_internal_test.go b/backend/alias/alias_internal_test.go
index fa0956b6d..18312eb02 100644
--- a/backend/alias/alias_internal_test.go
+++ b/backend/alias/alias_internal_test.go
@@ -23,8 +23,8 @@ func prepare(t *testing.T, root string) {
 	configfile.Install()
 
 	// Configure the remote
-	config.FileSet(remoteName, "type", "alias")
-	config.FileSet(remoteName, "remote", root)
+	config.FileSetValue(remoteName, "type", "alias")
+	config.FileSetValue(remoteName, "remote", root)
 }
 
 func TestNewFS(t *testing.T) {
diff --git a/backend/cache/cache_internal_test.go b/backend/cache/cache_internal_test.go
index 6467e4cd9..e1baaa155 100644
--- a/backend/cache/cache_internal_test.go
+++ b/backend/cache/cache_internal_test.go
@@ -850,8 +850,8 @@ func (r *run) encryptRemoteIfNeeded(t *testing.T, remote string) string {
 func (r *run) newCacheFs(t *testing.T, remote, id string, needRemote, purge bool, flags map[string]string) (fs.Fs, *cache.Persistent) {
 	fstest.Initialise()
 	remoteExists := false
-	for _, s := range config.FileSections() {
-		if s == remote {
+	for _, s := range config.GetRemotes() {
+		if s.Name == remote {
 			remoteExists = true
 		}
 	}
@@ -875,12 +875,12 @@ func (r *run) newCacheFs(t *testing.T, remote, id string, needRemote, purge bool
 	cacheRemote := remote
 	if !remoteExists {
 		localRemote := remote + "-local"
-		config.FileSet(localRemote, "type", "local")
-		config.FileSet(localRemote, "nounc", "true")
+		config.FileSetValue(localRemote, "type", "local")
+		config.FileSetValue(localRemote, "nounc", "true")
 		m.Set("type", "cache")
 		m.Set("remote", localRemote+":"+filepath.Join(os.TempDir(), localRemote))
 	} else {
-		remoteType := config.FileGet(remote, "type")
+		remoteType := config.GetValue(remote, "type")
 		if remoteType == "" {
 			t.Skipf("skipped due to invalid remote type for %v", remote)
 			return nil, nil
@@ -891,14 +891,14 @@ func (r *run) newCacheFs(t *testing.T, remote, id string, needRemote, purge bool
 				m.Set("password", cryptPassword1)
 				m.Set("password2", cryptPassword2)
 			}
-			remoteRemote := config.FileGet(remote, "remote")
+			remoteRemote := config.GetValue(remote, "remote")
 			if remoteRemote == "" {
 				t.Skipf("skipped due to invalid remote wrapper for %v", remote)
 				return nil, nil
 			}
 			remoteRemoteParts := strings.Split(remoteRemote, ":")
 			remoteWrapping := remoteRemoteParts[0]
-			remoteType := config.FileGet(remoteWrapping, "type")
+			remoteType := config.GetValue(remoteWrapping, "type")
 			if remoteType != "cache" {
 				t.Skipf("skipped due to invalid remote type for %v: '%v'", remoteWrapping, remoteType)
 				return nil, nil
diff --git a/backend/s3/s3_internal_test.go b/backend/s3/s3_internal_test.go
index 5ac299d2a..86e23e436 100644
--- a/backend/s3/s3_internal_test.go
+++ b/backend/s3/s3_internal_test.go
@@ -5,15 +5,17 @@ import (
 	"compress/gzip"
 	"context"
 	"crypto/md5"
+	"errors"
 	"fmt"
 	"path"
 	"strings"
 	"testing"
 	"time"
 
-	"github.com/aws/aws-sdk-go/aws"
-	"github.com/aws/aws-sdk-go/aws/awserr"
-	"github.com/aws/aws-sdk-go/service/s3"
+	"github.com/aws/aws-sdk-go-v2/aws"
+	"github.com/aws/aws-sdk-go-v2/service/s3"
+	"github.com/aws/aws-sdk-go-v2/service/s3/types"
+	"github.com/aws/smithy-go"
 	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fs/cache"
 	"github.com/rclone/rclone/fs/hash"
@@ -58,6 +60,16 @@ func (f *Fs) InternalTestMetadata(t *testing.T) {
 		// "tier" - read only
 		// "btime" - read only
 	}
+	// Cloudflare insists on decompressing `Content-Encoding: gzip` unless
+	// `Cache-Control: no-transform` is supplied. This is a deviation from
+	// AWS but we fudge the tests here rather than breaking peoples
+	// expectations of what Cloudflare does.
+	//
+	// This can always be overridden by using
+	// `--header-upload "Cache-Control: no-transform"`
+	if f.opt.Provider == "Cloudflare" {
+		metadata["cache-control"] = "no-transform"
+	}
 	obj := fstests.PutTestContentsMetadata(ctx, t, f, &item, true, contents, true, "text/html", metadata)
 	defer func() {
 		assert.NoError(t, obj.Remove(ctx))
@@ -131,20 +143,20 @@ func TestVersionLess(t *testing.T) {
 	t1 := fstest.Time("2022-01-21T12:00:00+01:00")
 	t2 := fstest.Time("2022-01-21T12:00:01+01:00")
 	for n, test := range []struct {
-		a, b *s3.ObjectVersion
+		a, b *types.ObjectVersion
 		want bool
 	}{
 		{a: nil, b: nil, want: true},
-		{a: &s3.ObjectVersion{Key: &key1, LastModified: &t1}, b: nil, want: false},
-		{a: nil, b: &s3.ObjectVersion{Key: &key1, LastModified: &t1}, want: true},
-		{a: &s3.ObjectVersion{Key: &key1, LastModified: &t1}, b: &s3.ObjectVersion{Key: &key1, LastModified: &t1}, want: false},
-		{a: &s3.ObjectVersion{Key: &key1, LastModified: &t1}, b: &s3.ObjectVersion{Key: &key1, LastModified: &t2}, want: false},
-		{a: &s3.ObjectVersion{Key: &key1, LastModified: &t2}, b: &s3.ObjectVersion{Key: &key1, LastModified: &t1}, want: true},
-		{a: &s3.ObjectVersion{Key: &key1, LastModified: &t1}, b: &s3.ObjectVersion{Key: &key2, LastModified: &t1}, want: true},
-		{a: &s3.ObjectVersion{Key: &key2, LastModified: &t1}, b: &s3.ObjectVersion{Key: &key1, LastModified: &t1}, want: false},
-		{a: &s3.ObjectVersion{Key: &key1, LastModified: &t1, IsLatest: aws.Bool(false)}, b: &s3.ObjectVersion{Key: &key1, LastModified: &t1}, want: false},
-		{a: &s3.ObjectVersion{Key: &key1, LastModified: &t1, IsLatest: aws.Bool(true)}, b: &s3.ObjectVersion{Key: &key1, LastModified: &t1}, want: true},
-		{a: &s3.ObjectVersion{Key: &key1, LastModified: &t1, IsLatest: aws.Bool(false)}, b: &s3.ObjectVersion{Key: &key1, LastModified: &t1, IsLatest: aws.Bool(true)}, want: false},
+		{a: &types.ObjectVersion{Key: &key1, LastModified: &t1}, b: nil, want: false},
+		{a: nil, b: &types.ObjectVersion{Key: &key1, LastModified: &t1}, want: true},
+		{a: &types.ObjectVersion{Key: &key1, LastModified: &t1}, b: &types.ObjectVersion{Key: &key1, LastModified: &t1}, want: false},
+		{a: &types.ObjectVersion{Key: &key1, LastModified: &t1}, b: &types.ObjectVersion{Key: &key1, LastModified: &t2}, want: false},
+		{a: &types.ObjectVersion{Key: &key1, LastModified: &t2}, b: &types.ObjectVersion{Key: &key1, LastModified: &t1}, want: true},
+		{a: &types.ObjectVersion{Key: &key1, LastModified: &t1}, b: &types.ObjectVersion{Key: &key2, LastModified: &t1}, want: true},
+		{a: &types.ObjectVersion{Key: &key2, LastModified: &t1}, b: &types.ObjectVersion{Key: &key1, LastModified: &t1}, want: false},
+		{a: &types.ObjectVersion{Key: &key1, LastModified: &t1, IsLatest: aws.Bool(false)}, b: &types.ObjectVersion{Key: &key1, LastModified: &t1}, want: false},
+		{a: &types.ObjectVersion{Key: &key1, LastModified: &t1, IsLatest: aws.Bool(true)}, b: &types.ObjectVersion{Key: &key1, LastModified: &t1}, want: true},
+		{a: &types.ObjectVersion{Key: &key1, LastModified: &t1, IsLatest: aws.Bool(false)}, b: &types.ObjectVersion{Key: &key1, LastModified: &t1, IsLatest: aws.Bool(true)}, want: false},
 	} {
 		got := versionLess(test.a, test.b)
 		assert.Equal(t, test.want, got, fmt.Sprintf("%d: %+v", n, test))
@@ -157,24 +169,24 @@ func TestMergeDeleteMarkers(t *testing.T) {
 	t1 := fstest.Time("2022-01-21T12:00:00+01:00")
 	t2 := fstest.Time("2022-01-21T12:00:01+01:00")
 	for n, test := range []struct {
-		versions []*s3.ObjectVersion
-		markers  []*s3.DeleteMarkerEntry
-		want     []*s3.ObjectVersion
+		versions []types.ObjectVersion
+		markers  []types.DeleteMarkerEntry
+		want     []types.ObjectVersion
 	}{
 		{
-			versions: []*s3.ObjectVersion{},
-			markers:  []*s3.DeleteMarkerEntry{},
-			want:     []*s3.ObjectVersion{},
+			versions: []types.ObjectVersion{},
+			markers:  []types.DeleteMarkerEntry{},
+			want:     []types.ObjectVersion{},
 		},
 		{
-			versions: []*s3.ObjectVersion{
+			versions: []types.ObjectVersion{
 				{
 					Key:          &key1,
 					LastModified: &t1,
 				},
 			},
-			markers: []*s3.DeleteMarkerEntry{},
-			want: []*s3.ObjectVersion{
+			markers: []types.DeleteMarkerEntry{},
+			want: []types.ObjectVersion{
 				{
 					Key:          &key1,
 					LastModified: &t1,
@@ -182,14 +194,14 @@ func TestMergeDeleteMarkers(t *testing.T) {
 			},
 		},
 		{
-			versions: []*s3.ObjectVersion{},
-			markers: []*s3.DeleteMarkerEntry{
+			versions: []types.ObjectVersion{},
+			markers: []types.DeleteMarkerEntry{
 				{
 					Key:          &key1,
 					LastModified: &t1,
 				},
 			},
-			want: []*s3.ObjectVersion{
+			want: []types.ObjectVersion{
 				{
 					Key:          &key1,
 					LastModified: &t1,
@@ -198,7 +210,7 @@ func TestMergeDeleteMarkers(t *testing.T) {
 			},
 		},
 		{
-			versions: []*s3.ObjectVersion{
+			versions: []types.ObjectVersion{
 				{
 					Key:          &key1,
 					LastModified: &t2,
@@ -208,13 +220,13 @@ func TestMergeDeleteMarkers(t *testing.T) {
 					LastModified: &t2,
 				},
 			},
-			markers: []*s3.DeleteMarkerEntry{
+			markers: []types.DeleteMarkerEntry{
 				{
 					Key:          &key1,
 					LastModified: &t1,
 				},
 			},
-			want: []*s3.ObjectVersion{
+			want: []types.ObjectVersion{
 				{
 					Key:          &key1,
 					LastModified: &t2,
@@ -399,22 +411,23 @@ func (f *Fs) InternalTestVersions(t *testing.T) {
 		// quirk is set correctly
 		req := s3.CreateBucketInput{
 			Bucket: &f.rootBucket,
-			ACL:    stringPointerOrNil(f.opt.BucketACL),
+			ACL:    types.BucketCannedACL(f.opt.BucketACL),
 		}
 		if f.opt.LocationConstraint != "" {
-			req.CreateBucketConfiguration = &s3.CreateBucketConfiguration{
-				LocationConstraint: &f.opt.LocationConstraint,
+			req.CreateBucketConfiguration = &types.CreateBucketConfiguration{
+				LocationConstraint: types.BucketLocationConstraint(f.opt.LocationConstraint),
 			}
 		}
 		err := f.pacer.Call(func() (bool, error) {
-			_, err := f.c.CreateBucketWithContext(ctx, &req)
+			_, err := f.c.CreateBucket(ctx, &req)
 			return f.shouldRetry(ctx, err)
 		})
 		var errString string
+		var awsError smithy.APIError
 		if err == nil {
 			errString = "No Error"
-		} else if awsErr, ok := err.(awserr.Error); ok {
-			errString = awsErr.Code()
+		} else if errors.As(err, &awsError) {
+			errString = awsError.ErrorCode()
 		} else {
 			assert.Fail(t, "Unknown error %T %v", err, err)
 		}
diff --git a/backend/s3/s3_test.go b/backend/s3/s3_test.go
index 7e6e55360..e0499ba66 100644
--- a/backend/s3/s3_test.go
+++ b/backend/s3/s3_test.go
@@ -4,12 +4,14 @@ package s3
 import (
 	"context"
 	"net/http"
-	"strings"
 	"testing"
 
+	"github.com/aws/aws-sdk-go-v2/aws"
 	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fstest"
 	"github.com/rclone/rclone/fstest/fstests"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 )
 
 func SetupS3Test(t *testing.T) (context.Context, *Options, *http.Client) {
@@ -54,20 +56,16 @@ func TestAWSDualStackOption(t *testing.T) {
 		// test enabled
 		ctx, opt, client := SetupS3Test(t)
 		opt.UseDualStack = true
-		s3Conn, _, _ := s3Connection(ctx, opt, client)
-		if !strings.Contains(s3Conn.Endpoint, "dualstack") {
-			t.Errorf("dualstack failed got: %s, wanted: dualstack", s3Conn.Endpoint)
-			t.Fail()
-		}
+		s3Conn, err := s3Connection(ctx, opt, client)
+		require.NoError(t, err)
+		assert.Equal(t, aws.DualStackEndpointStateEnabled, s3Conn.Options().EndpointOptions.UseDualStackEndpoint)
 	}
 	{
 		// test default case
 		ctx, opt, client := SetupS3Test(t)
-		s3Conn, _, _ := s3Connection(ctx, opt, client)
-		if strings.Contains(s3Conn.Endpoint, "dualstack") {
-			t.Errorf("dualstack failed got: %s, NOT wanted: dualstack", s3Conn.Endpoint)
-			t.Fail()
-		}
+		s3Conn, err := s3Connection(ctx, opt, client)
+		require.NoError(t, err)
+		assert.Equal(t, aws.DualStackEndpointStateDisabled, s3Conn.Options().EndpointOptions.UseDualStackEndpoint)
 	}
 }
 
diff --git a/cmd/bisync/bisync_test.go b/cmd/bisync/bisync_test.go
index dd9f91ee3..5321eab9a 100644
--- a/cmd/bisync/bisync_test.go
+++ b/cmd/bisync/bisync_test.go
@@ -1304,7 +1304,7 @@ func touchFiles(ctx context.Context, dateStr string, f fs.Fs, dir, glob string)
 		return files, fmt.Errorf("invalid date %q: %w", dateStr, err)
 	}
 
-	matcher, firstErr := filter.GlobToRegexp(glob, false)
+	matcher, firstErr := filter.GlobPathToRegexp(glob, false)
 	if firstErr != nil {
 		return files, fmt.Errorf("invalid glob %q", glob)
 	}
diff --git a/fs/config/rc_test.go b/fs/config/rc_test.go
index 891b9bcbc..aeed3f513 100644
--- a/fs/config/rc_test.go
+++ b/fs/config/rc_test.go
@@ -33,8 +33,8 @@ func TestRc(t *testing.T) {
 	out, err := call.Fn(ctx, in)
 	require.NoError(t, err)
 	require.Nil(t, out)
-	assert.Equal(t, "local", config.FileGet(testName, "type"))
-	assert.Equal(t, "sausage", config.FileGet(testName, "test_key"))
+	assert.Equal(t, "local", config.GetValue(testName, "type"))
+	assert.Equal(t, "sausage", config.GetValue(testName, "test_key"))
 
 	// The sub tests rely on the remote created above but they can
 	// all be run independently
@@ -102,9 +102,9 @@ func TestRc(t *testing.T) {
 		require.NoError(t, err)
 		assert.Nil(t, out)
 
-		assert.Equal(t, "local", config.FileGet(testName, "type"))
-		assert.Equal(t, "rutabaga", config.FileGet(testName, "test_key"))
-		assert.Equal(t, "cabbage", config.FileGet(testName, "test_key2"))
+		assert.Equal(t, "local", config.GetValue(testName, "type"))
+		assert.Equal(t, "rutabaga", config.GetValue(testName, "test_key"))
+		assert.Equal(t, "cabbage", config.GetValue(testName, "test_key2"))
 	})
 
 	t.Run("Password", func(t *testing.T) {
@@ -122,9 +122,9 @@ func TestRc(t *testing.T) {
 		require.NoError(t, err)
 		assert.Nil(t, out)
 
-		assert.Equal(t, "local", config.FileGet(testName, "type"))
-		assert.Equal(t, "rutabaga", obscure.MustReveal(config.FileGet(testName, "test_key")))
-		assert.Equal(t, pw2, obscure.MustReveal(config.FileGet(testName, "test_key2")))
+		assert.Equal(t, "local", config.GetValue(testName, "type"))
+		assert.Equal(t, "rutabaga", obscure.MustReveal(config.GetValue(testName, "test_key")))
+		assert.Equal(t, pw2, obscure.MustReveal(config.GetValue(testName, "test_key2")))
 	})
 
 	// Delete the test remote
@@ -136,8 +136,8 @@ func TestRc(t *testing.T) {
 	out, err = call.Fn(context.Background(), in)
 	require.NoError(t, err)
 	assert.Nil(t, out)
-	assert.Equal(t, "", config.FileGet(testName, "type"))
-	assert.Equal(t, "", config.FileGet(testName, "test_key"))
+	assert.Equal(t, "", config.GetValue(testName, "type"))
+	assert.Equal(t, "", config.GetValue(testName, "test_key"))
 }
 
 func TestRcProviders(t *testing.T) {
diff --git a/fs/config/ui_test.go b/fs/config/ui_test.go
index 172921c9b..f24f2891f 100644
--- a/fs/config/ui_test.go
+++ b/fs/config/ui_test.go
@@ -110,9 +110,9 @@ func TestCRUD(t *testing.T) {
 	require.NoError(t, config.NewRemote(ctx, "test"))
 
 	assert.Equal(t, []string{"test"}, config.Data().GetSectionList())
-	assert.Equal(t, "config_test_remote", config.FileGet("test", "type"))
-	assert.Equal(t, "true", config.FileGet("test", "bool"))
-	assert.Equal(t, "secret", obscure.MustReveal(config.FileGet("test", "pass")))
+	assert.Equal(t, "config_test_remote", config.GetValue("test", "type"))
+	assert.Equal(t, "true", config.GetValue("test", "bool"))
+	assert.Equal(t, "secret", obscure.MustReveal(config.GetValue("test", "pass")))
 
 	// normal rename, test → asdf
 	config.ReadLine = makeReadLine([]string{
@@ -123,9 +123,9 @@ func TestCRUD(t *testing.T) {
 	config.RenameRemote("test")
 
 	assert.Equal(t, []string{"asdf"}, config.Data().GetSectionList())
-	assert.Equal(t, "config_test_remote", config.FileGet("asdf", "type"))
-	assert.Equal(t, "true", config.FileGet("asdf", "bool"))
-	assert.Equal(t, "secret", obscure.MustReveal(config.FileGet("asdf", "pass")))
+	assert.Equal(t, "config_test_remote", config.GetValue("asdf", "type"))
+	assert.Equal(t, "true", config.GetValue("asdf", "bool"))
+	assert.Equal(t, "secret", obscure.MustReveal(config.GetValue("asdf", "pass")))
 
 	// delete remote
 	config.DeleteRemote("asdf")
@@ -152,8 +152,8 @@ func TestChooseOption(t *testing.T) {
 	}
 	require.NoError(t, config.NewRemote(ctx, "test"))
 
-	assert.Equal(t, "", config.FileGet("test", "bool")) // this is the default now
-	assert.Equal(t, "not very random password", obscure.MustReveal(config.FileGet("test", "pass")))
+	assert.Equal(t, "", config.GetValue("test", "bool")) // this is the default now
+	assert.Equal(t, "not very random password", obscure.MustReveal(config.GetValue("test", "pass")))
 
 	// script for creating remote
 	config.ReadLine = makeReadLine([]string{
@@ -164,8 +164,8 @@ func TestChooseOption(t *testing.T) {
 	})
 	require.NoError(t, config.NewRemote(ctx, "test"))
 
-	assert.Equal(t, "true", config.FileGet("test", "bool"))
-	assert.Equal(t, "", config.FileGet("test", "pass"))
+	assert.Equal(t, "true", config.GetValue("test", "bool"))
+	assert.Equal(t, "", config.GetValue("test", "pass"))
 }
 
 func TestNewRemoteName(t *testing.T) {
@@ -212,9 +212,9 @@ func TestCreateUpdatePasswordRemote(t *testing.T) {
 				require.NoError(t, err)
 
 				assert.Equal(t, []string{"test2"}, config.Data().GetSectionList())
-				assert.Equal(t, "config_test_remote", config.FileGet("test2", "type"))
-				assert.Equal(t, "true", config.FileGet("test2", "bool"))
-				gotPw := config.FileGet("test2", "pass")
+				assert.Equal(t, "config_test_remote", config.GetValue("test2", "type"))
+				assert.Equal(t, "true", config.GetValue("test2", "bool"))
+				gotPw := config.GetValue("test2", "pass")
 				if !noObscure {
 					gotPw = obscure.MustReveal(gotPw)
 				}
@@ -229,9 +229,9 @@ func TestCreateUpdatePasswordRemote(t *testing.T) {
 				require.NoError(t, err)
 
 				assert.Equal(t, []string{"test2"}, config.Data().GetSectionList())
-				assert.Equal(t, "config_test_remote", config.FileGet("test2", "type"))
-				assert.Equal(t, "false", config.FileGet("test2", "bool"))
-				gotPw = config.FileGet("test2", "pass")
+				assert.Equal(t, "config_test_remote", config.GetValue("test2", "type"))
+				assert.Equal(t, "false", config.GetValue("test2", "bool"))
+				gotPw = config.GetValue("test2", "pass")
 				if doObscure {
 					gotPw = obscure.MustReveal(gotPw)
 				}
@@ -242,9 +242,9 @@ func TestCreateUpdatePasswordRemote(t *testing.T) {
 				}))
 
 				assert.Equal(t, []string{"test2"}, config.Data().GetSectionList())
-				assert.Equal(t, "config_test_remote", config.FileGet("test2", "type"))
-				assert.Equal(t, "false", config.FileGet("test2", "bool"))
-				assert.Equal(t, "potato3", obscure.MustReveal(config.FileGet("test2", "pass")))
+				assert.Equal(t, "config_test_remote", config.GetValue("test2", "type"))
+				assert.Equal(t, "false", config.GetValue("test2", "bool"))
+				assert.Equal(t, "potato3", obscure.MustReveal(config.GetValue("test2", "pass")))
 			})
 		}
 	}
@@ -280,10 +280,10 @@ func TestDefaultRequired(t *testing.T) {
 	require.NoError(t, config.NewRemote(ctx, "test"))
 
 	assert.Equal(t, []string{"test"}, config.Data().GetSectionList())
-	assert.Equal(t, "config_test_remote", config.FileGet("test", "type"))
-	assert.Equal(t, "111", config.FileGet("test", "string_required"))
-	assert.Equal(t, "222", config.FileGet("test", "string_default"))
-	assert.Equal(t, "333", config.FileGet("test", "string_required_default"))
+	assert.Equal(t, "config_test_remote", config.GetValue("test", "type"))
+	assert.Equal(t, "111", config.GetValue("test", "string_required"))
+	assert.Equal(t, "222", config.GetValue("test", "string_default"))
+	assert.Equal(t, "333", config.GetValue("test", "string_required_default"))
 
 	// delete remote
 	config.DeleteRemote("test")
@@ -301,10 +301,10 @@ func TestDefaultRequired(t *testing.T) {
 	require.NoError(t, config.NewRemote(ctx, "test"))
 
 	assert.Equal(t, []string{"test"}, config.Data().GetSectionList())
-	assert.Equal(t, "config_test_remote", config.FileGet("test", "type"))
-	assert.Equal(t, "111", config.FileGet("test", "string_required"))
-	assert.Equal(t, "", config.FileGet("test", "string_default"))
-	assert.Equal(t, "", config.FileGet("test", "string_required_default"))
+	assert.Equal(t, "config_test_remote", config.GetValue("test", "type"))
+	assert.Equal(t, "111", config.GetValue("test", "string_required"))
+	assert.Equal(t, "", config.GetValue("test", "string_default"))
+	assert.Equal(t, "", config.GetValue("test", "string_required_default"))
 }
 
 func TestMultipleChoice(t *testing.T) {
@@ -383,11 +383,11 @@ func TestMultipleChoice(t *testing.T) {
 	require.NoError(t, config.NewRemote(ctx, "test"))
 
 	assert.Equal(t, []string{"test"}, config.Data().GetSectionList())
-	assert.Equal(t, "config_test_remote", config.FileGet("test", "type"))
-	assert.Equal(t, "CCC", config.FileGet("test", "multiple_choice"))
-	assert.Equal(t, "CCC", config.FileGet("test", "multiple_choice_required"))
-	assert.Equal(t, "CCC", config.FileGet("test", "multiple_choice_default"))
-	assert.Equal(t, "CCC", config.FileGet("test", "multiple_choice_required_default"))
+	assert.Equal(t, "config_test_remote", config.GetValue("test", "type"))
+	assert.Equal(t, "CCC", config.GetValue("test", "multiple_choice"))
+	assert.Equal(t, "CCC", config.GetValue("test", "multiple_choice_required"))
+	assert.Equal(t, "CCC", config.GetValue("test", "multiple_choice_default"))
+	assert.Equal(t, "CCC", config.GetValue("test", "multiple_choice_required_default"))
 
 	// delete remote
 	config.DeleteRemote("test")
@@ -405,11 +405,11 @@ func TestMultipleChoice(t *testing.T) {
 	require.NoError(t, config.NewRemote(ctx, "test"))
 
 	assert.Equal(t, []string{"test"}, config.Data().GetSectionList())
-	assert.Equal(t, "config_test_remote", config.FileGet("test", "type"))
-	assert.Equal(t, "XXX", config.FileGet("test", "multiple_choice"))
-	assert.Equal(t, "XXX", config.FileGet("test", "multiple_choice_required"))
-	assert.Equal(t, "XXX", config.FileGet("test", "multiple_choice_default"))
-	assert.Equal(t, "XXX", config.FileGet("test", "multiple_choice_required_default"))
+	assert.Equal(t, "config_test_remote", config.GetValue("test", "type"))
+	assert.Equal(t, "XXX", config.GetValue("test", "multiple_choice"))
+	assert.Equal(t, "XXX", config.GetValue("test", "multiple_choice_required"))
+	assert.Equal(t, "XXX", config.GetValue("test", "multiple_choice_default"))
+	assert.Equal(t, "XXX", config.GetValue("test", "multiple_choice_required_default"))
 
 	// delete remote
 	config.DeleteRemote("test")
@@ -428,11 +428,11 @@ func TestMultipleChoice(t *testing.T) {
 	require.NoError(t, config.NewRemote(ctx, "test"))
 
 	assert.Equal(t, []string{"test"}, config.Data().GetSectionList())
-	assert.Equal(t, "config_test_remote", config.FileGet("test", "type"))
-	assert.Equal(t, "", config.FileGet("test", "multiple_choice"))
-	assert.Equal(t, "XXX", config.FileGet("test", "multiple_choice_required"))
-	assert.Equal(t, "", config.FileGet("test", "multiple_choice_default"))
-	assert.Equal(t, "", config.FileGet("test", "multiple_choice_required_default"))
+	assert.Equal(t, "config_test_remote", config.GetValue("test", "type"))
+	assert.Equal(t, "", config.GetValue("test", "multiple_choice"))
+	assert.Equal(t, "XXX", config.GetValue("test", "multiple_choice_required"))
+	assert.Equal(t, "", config.GetValue("test", "multiple_choice_default"))
+	assert.Equal(t, "", config.GetValue("test", "multiple_choice_required_default"))
 }
 
 func TestMultipleChoiceExclusive(t *testing.T) {
@@ -483,9 +483,9 @@ func TestMultipleChoiceExclusive(t *testing.T) {
 	require.NoError(t, config.NewRemote(ctx, "test"))
 
 	assert.Equal(t, []string{"test"}, config.Data().GetSectionList())
-	assert.Equal(t, "config_test_remote", config.FileGet("test", "type"))
-	assert.Equal(t, "", config.FileGet("test", "multiple_choice_exclusive"))
-	assert.Equal(t, "", config.FileGet("test", "multiple_choice_exclusive_default"))
+	assert.Equal(t, "config_test_remote", config.GetValue("test", "type"))
+	assert.Equal(t, "", config.GetValue("test", "multiple_choice_exclusive"))
+	assert.Equal(t, "", config.GetValue("test", "multiple_choice_exclusive_default"))
 }
 
 func TestMultipleChoiceExclusiveRequired(t *testing.T) {
@@ -539,7 +539,7 @@ func TestMultipleChoiceExclusiveRequired(t *testing.T) {
 	require.NoError(t, config.NewRemote(ctx, "test"))
 
 	assert.Equal(t, []string{"test"}, config.Data().GetSectionList())
-	assert.Equal(t, "config_test_remote", config.FileGet("test", "type"))
-	assert.Equal(t, "CCC", config.FileGet("test", "multiple_choice_exclusive_required"))
-	assert.Equal(t, "", config.FileGet("test", "multiple_choice_exclusive_required_default"))
+	assert.Equal(t, "config_test_remote", config.GetValue("test", "type"))
+	assert.Equal(t, "CCC", config.GetValue("test", "multiple_choice_exclusive_required"))
+	assert.Equal(t, "", config.GetValue("test", "multiple_choice_exclusive_required_default"))
 }
diff --git a/fs/filter/glob_test.go b/fs/filter/glob_test.go
index 5946a5a0a..04a07581d 100644
--- a/fs/filter/glob_test.go
+++ b/fs/filter/glob_test.go
@@ -7,7 +7,129 @@ import (
 	"github.com/stretchr/testify/require"
 )
 
-func TestGlobToRegexp(t *testing.T) {
+func TestGlobStringToRegexp(t *testing.T) {
+	for _, test := range []struct {
+		in    string
+		want  string
+		error string
+	}{
+		{``, `^$`, ``},
+		{`potato`, `^potato$`, ``},
+		{`potato,sausage`, `^potato,sausage$`, ``},
+		{`/potato`, `^/potato$`, ``},
+		{`potato?sausage`, `^potato.sausage$`, ``},
+		{`potat[oa]`, `^potat[oa]$`, ``},
+		{`potat[a-z]or`, `^potat[a-z]or$`, ``},
+		{`potat[[:alpha:]]or`, `^potat[[:alpha:]]or$`, ``},
+		{`'.' '+' '(' ')' '|' '^' '$'`, `^'\.' '\+' '\(' '\)' '\|' '\^' '\$'$`, ``},
+		{`*.jpg`, `^.*\.jpg$`, ``},
+		{`a{b,c,d}e`, `^a(b|c|d)e$`, ``},
+		{`potato**`, ``, `too many stars`},
+		{`potato**sausage`, ``, `too many stars`},
+		{`*.p[lm]`, `^.*\.p[lm]$`, ``},
+		{`[\[\]]`, `^[\[\]]$`, ``},
+		{`***potato`, ``, `too many stars`},
+		{`***`, ``, `too many stars`},
+		{`ab]c`, ``, `mismatched ']'`},
+		{`ab[c`, ``, `mismatched '[' and ']'`},
+		{`ab{x{cd`, ``, `can't nest`},
+		{`ab{}}cd`, ``, `mismatched '{' and '}'`},
+		{`ab}c`, ``, `mismatched '{' and '}'`},
+		{`ab{c`, ``, `mismatched '{' and '}'`},
+		{`*.{jpg,png,gif}`, `^.*\.(jpg|png|gif)$`, ``},
+		{`[a--b]`, ``, `bad glob pattern`},
+		{`a\*b`, `^a\*b$`, ``},
+		{`a\\b`, `^a\\b$`, ``},
+		{`a{{.*}}b`, `^a(.*)b$`, ``},
+		{`a{{.*}`, ``, `mismatched '{{' and '}}'`},
+		{`{{regexp}}`, `^(regexp)$`, ``},
+		{`\{{{regexp}}`, `^\{(regexp)$`, ``},
+		{`/{{regexp}}`, `^/(regexp)$`, ``},
+		{`/{{\d{8}}}`, `^/(\d{8})$`, ``},
+		{`/{{\}}}`, `^/(\})$`, ``},
+		{`{{(?i)regexp}}`, `^((?i)regexp)$`, ``},
+	} {
+		for _, ignoreCase := range []bool{false, true} {
+			gotRe, err := GlobStringToRegexp(test.in, ignoreCase)
+			if test.error == "" {
+				require.NoError(t, err, test.in)
+				prefix := ""
+				if ignoreCase {
+					prefix = "(?i)"
+				}
+				got := gotRe.String()
+				assert.Equal(t, prefix+test.want, got, test.in)
+			} else {
+				require.Error(t, err, test.in)
+				assert.Contains(t, err.Error(), test.error, test.in)
+				assert.Nil(t, gotRe)
+			}
+		}
+	}
+}
+
+func TestGlobStringToRegexpWithoutAnchors(t *testing.T) {
+	for _, test := range []struct {
+		in    string
+		want  string
+		error string
+	}{
+		{``, ``, ``},
+		{`potato`, `potato`, ``},
+		{`potato,sausage`, `potato,sausage`, ``},
+		{`/potato`, `/potato`, ``},
+		{`potato?sausage`, `potato.sausage`, ``},
+		{`potat[oa]`, `potat[oa]`, ``},
+		{`potat[a-z]or`, `potat[a-z]or`, ``},
+		{`potat[[:alpha:]]or`, `potat[[:alpha:]]or`, ``},
+		{`'.' '+' '(' ')' '|' '^' '$'`, `'\.' '\+' '\(' '\)' '\|' '\^' '\$'`, ``},
+		{`*.jpg`, `.*\.jpg`, ``},
+		{`a{b,c,d}e`, `a(b|c|d)e`, ``},
+		{`potato**`, ``, `too many stars`},
+		{`potato**sausage`, ``, `too many stars`},
+		{`*.p[lm]`, `.*\.p[lm]`, ``},
+		{`[\[\]]`, `[\[\]]`, ``},
+		{`***potato`, ``, `too many stars`},
+		{`***`, ``, `too many stars`},
+		{`ab]c`, ``, `mismatched ']'`},
+		{`ab[c`, ``, `mismatched '[' and ']'`},
+		{`ab{x{cd`, ``, `can't nest`},
+		{`ab{}}cd`, ``, `mismatched '{' and '}'`},
+		{`ab}c`, ``, `mismatched '{' and '}'`},
+		{`ab{c`, ``, `mismatched '{' and '}'`},
+		{`*.{jpg,png,gif}`, `.*\.(jpg|png|gif)`, ``},
+		{`[a--b]`, ``, `bad glob pattern`},
+		{`a\*b`, `a\*b`, ``},
+		{`a\\b`, `a\\b`, ``},
+		{`a{{.*}}b`, `a(.*)b`, ``},
+		{`a{{.*}`, ``, `mismatched '{{' and '}}'`},
+		{`{{regexp}}`, `(regexp)`, ``},
+		{`\{{{regexp}}`, `\{(regexp)`, ``},
+		{`/{{regexp}}`, `/(regexp)`, ``},
+		{`/{{\d{8}}}`, `/(\d{8})`, ``},
+		{`/{{\}}}`, `/(\})`, ``},
+		{`{{(?i)regexp}}`, `((?i)regexp)`, ``},
+	} {
+		for _, ignoreCase := range []bool{false, true} {
+			gotRe, err := globToRegexp(test.in, false, false, ignoreCase)
+			if test.error == "" {
+				require.NoError(t, err, test.in)
+				prefix := ""
+				if ignoreCase {
+					prefix = "(?i)"
+				}
+				got := gotRe.String()
+				assert.Equal(t, prefix+test.want, got, test.in)
+			} else {
+				require.Error(t, err, test.in)
+				assert.Contains(t, err.Error(), test.error, test.in)
+				assert.Nil(t, gotRe)
+			}
+		}
+	}
+}
+
+func TestGlobPathToRegexp(t *testing.T) {
 	for _, test := range []struct {
 		in    string
 		want  string
@@ -28,20 +150,20 @@ func TestGlobToRegexp(t *testing.T) {
 		{`potato**sausage`, `(^|/)potato.*sausage$`, ``},
 		{`*.p[lm]`, `(^|/)[^/]*\.p[lm]$`, ``},
 		{`[\[\]]`, `(^|/)[\[\]]$`, ``},
-		{`***potato`, `(^|/)`, `too many stars`},
-		{`***`, `(^|/)`, `too many stars`},
-		{`ab]c`, `(^|/)`, `mismatched ']'`},
-		{`ab[c`, `(^|/)`, `mismatched '[' and ']'`},
-		{`ab{x{cd`, `(^|/)`, `can't nest`},
-		{`ab{}}cd`, `(^|/)`, `mismatched '{' and '}'`},
-		{`ab}c`, `(^|/)`, `mismatched '{' and '}'`},
-		{`ab{c`, `(^|/)`, `mismatched '{' and '}'`},
+		{`***potato`, ``, `too many stars`},
+		{`***`, ``, `too many stars`},
+		{`ab]c`, ``, `mismatched ']'`},
+		{`ab[c`, ``, `mismatched '[' and ']'`},
+		{`ab{x{cd`, ``, `can't nest`},
+		{`ab{}}cd`, ``, `mismatched '{' and '}'`},
+		{`ab}c`, ``, `mismatched '{' and '}'`},
+		{`ab{c`, ``, `mismatched '{' and '}'`},
 		{`*.{jpg,png,gif}`, `(^|/)[^/]*\.(jpg|png|gif)$`, ``},
-		{`[a--b]`, `(^|/)`, `bad glob pattern`},
+		{`[a--b]`, ``, `bad glob pattern`},
 		{`a\*b`, `(^|/)a\*b$`, ``},
 		{`a\\b`, `(^|/)a\\b$`, ``},
 		{`a{{.*}}b`, `(^|/)a(.*)b$`, ``},
-		{`a{{.*}`, `(^|/)a(.*)b$`, `mismatched '{{' and '}}'`},
+		{`a{{.*}`, ``, `mismatched '{{' and '}}'`},
 		{`{{regexp}}`, `(^|/)(regexp)$`, ``},
 		{`\{{{regexp}}`, `(^|/)\{(regexp)$`, ``},
 		{`/{{regexp}}`, `^(regexp)$`, ``},
@@ -50,7 +172,7 @@ func TestGlobToRegexp(t *testing.T) {
 		{`{{(?i)regexp}}`, `(^|/)((?i)regexp)$`, ``},
 	} {
 		for _, ignoreCase := range []bool{false, true} {
-			gotRe, err := GlobToRegexp(test.in, ignoreCase)
+			gotRe, err := GlobPathToRegexp(test.in, ignoreCase)
 			if test.error == "" {
 				require.NoError(t, err, test.in)
 				prefix := ""
@@ -111,7 +233,7 @@ func TestGlobToDirGlobs(t *testing.T) {
 		{"/sausage3**", []string{`/sausage3**/`, "/"}},
 		{"/a/*.jpg", []string{`/a/`, "/"}},
 	} {
-		_, err := GlobToRegexp(test.in, false)
+		_, err := GlobPathToRegexp(test.in, false)
 		assert.NoError(t, err)
 		got := globToDirGlobs(test.in)
 		assert.Equal(t, test.want, got, test.in)
diff --git a/fstest/fstests/fstests.go b/fstest/fstests/fstests.go
index 00e5372f5..be43a015a 100644
--- a/fstest/fstests/fstests.go
+++ b/fstest/fstests/fstests.go
@@ -83,6 +83,7 @@ type SetUploadCutoffer interface {
 type SetCopyCutoffer interface {
 	// Change the configured CopyCutoff.
 	// Will only be called while no transfer is in progress.
+	// Return fs.ErrorNotImplemented if you can't implement this
 	SetCopyCutoff(fs.SizeSuffix) (fs.SizeSuffix, error)
 }
 
@@ -428,7 +429,7 @@ func Run(t *testing.T, opt *Opt) {
 
 	// Set extra config if supplied
 	for _, item := range opt.ExtraConfig {
-		config.FileSet(item.Name, item.Key, item.Value)
+		config.FileSetValue(item.Name, item.Key, item.Value)
 	}
 	if *fstest.RemoteName != "" {
 		remoteName = *fstest.RemoteName
@@ -2345,10 +2346,6 @@ func Run(t *testing.T, opt *Opt) {
 				t.Skip("skipping as ChunkedUpload.Skip is set")
 			}
 
-			if strings.HasPrefix(f.Name(), "serves3") || strings.HasPrefix(f.Name(), "TestS3Rclone") {
-				t.Skip("FIXME skip test - see #7454")
-			}
-
 			do, _ := f.(SetCopyCutoffer)
 			if do == nil {
 				t.Skipf("%T does not implement SetCopyCutoff", f)
@@ -2362,6 +2359,13 @@ func Run(t *testing.T, opt *Opt) {
 				minChunkSize = opt.ChunkedUpload.CeilChunkSize(minChunkSize)
 			}
 
+			// Test setting the copy cutoff before we get going
+			_, err := do.SetCopyCutoff(minChunkSize)
+			if errors.Is(err, fs.ErrorNotImplemented) {
+				t.Skipf("%T does not support SetCopyCutoff: %v", f, err)
+			}
+			require.NoError(t, err)
+
 			chunkSizes := fs.SizeSuffixList{
 				minChunkSize,
 				minChunkSize + 1,
diff --git a/fstest/test_all/config.yaml b/fstest/test_all/config.yaml
index 10279dd66..37a561b81 100644
--- a/fstest/test_all/config.yaml
+++ b/fstest/test_all/config.yaml
@@ -215,6 +215,23 @@ backends:
      - TestIntegration/FsMkdir/FsEncoding/leading_HT
      - TestIntegration/FsMkdir/FsEncoding/leading_VT
      - TestIntegration/FsMkdir/FsPutFiles/FsPutStream/0
+ - backend:  "s3"
+   remote:   "TestS3GCS:"
+   fastlist: true
+   ignore:
+     - TestIntegration/FsMkdir/FsEncoding/control_chars
+     - TestIntegration/FsMkdir/FsEncoding/leading_CR
+     - TestIntegration/FsMkdir/FsEncoding/leading_LF
+     - TestIntegration/FsMkdir/FsEncoding/trailing_CR
+     - TestIntegration/FsMkdir/FsEncoding/trailing_LF
+     - TestIntegration/FsMkdir/FsPutFiles/PublicLink
+     - TestIntegration/FsMkdir/FsPutFiles/SetTier
+     - TestIntegration/FsMkdir/FsPutFiles/Internal/Metadata/GzipEncoding
+     - TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterDelete/List
+     - TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterDelete/NewObject
+     - TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterTwo/List
+     - TestIntegration/FsMkdir/FsPutFiles/Internal/Versions/VersionAt/AfterTwo/NewObject
+     - TestBisyncRemoteRemote/extended_filenames
  # Disabled due to excessive rate limiting at DO which cause the tests never to pass
  # This hits the rate limit as documented here: https://www.digitalocean.com/docs/spaces/#limits
  # 2 COPYs per 5 minutes on any individual object in a Space