Repository: rclone/rclone
Pull request #3607 (2019-10-09 15:37:29)

chunker: reservations for future, security improvements, complete sha1/md5 overlay
#### What is the purpose of this change?

Goal:
Reserve possibility for future chunker/rclone releases to add more control information (which can be used to implement resumable uploads etc) in a way that future types of hidden chunks don't play jack-in-the-box in file listings of unaware past releases.

Subgoals:
* keep metadata format estableshed earlier
* postpone creation of internal structures for future, only prepare chunk name and metadata validators
* rectify validation so that future control structures can't be accidentally corrupted by past releases
* polish terminology in docs and comments (in fact 70% of the diff).

Security fixes:
* reject or report attempts to corrupt chunks (in fact remaining 20% of the diff)
* reject huge numbers in chunk name, prevent memory overflow
* specially handle user files containing valid chunker metadata  (currently treated as corrupted composites upon upload)

New Features:
* implement consistent hashing option

#### Was the change discussed in an issue or in the forum before?

No, but it's good :)

#### Checklist

- [x] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-pull-request).
- [x] I have added tests for all changes in this PR if appropriate.
- [x] I have added documentation for the changes if appropriate.
- [x] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages).
- [x] I'm done, this Pull Request is ready for review :-)


ivandeex: Of Testing:
I'm not adding remotes with consistent hashing into "config.yaml". They are rather superfluous (rclone config is copy/pasted from similar hashing remotes with substitution "hash_type = sha1all"):
```
- backend:  "chunker"
  remote:   "TestChunkerChunk50bSha1AllMailru:"
  fastlist: true
  maxfile:  3k
  ignore:
    - TestSyncWithTrackRenames
```
Tracking test detects new counters because hash-based sync jumps in. Expected.

Integration tests (`test_all -backends=chunker`) pass, except for 3 issues I'm being investigating:
* test `TestIntegration/FsMkdir/FsPutFiles/Internal/MetadataInput` fails on remote `TestChunkerChunk50bYandex` with non-standar error  message `"can't move file - destination already exists"` (translated from russian). I will change Fs.baseMove() to delete destination and repeat move if first move fails with whatever error
* Tests on remote `TestChunkerChunk3bLocal` have become slow, under investigation
* Encoding tests fail on `TestChunkerChunk3bLoca`, with lots of debug messages `"chunk without meta object"`, under investigation
ncw: I fixed the build :-(

Do you want me to merge this now or wait until you've investigated the test failures?
ivandeex: > Do you want me to merge this now or wait until you've investigated the test failures?
Do I have to rebase to pick the fixed build?
I'd rather take one-two days to investigate.


ncw: Yes rebase needed. I think you've done that already though 

Ok will await your updates.
ivandeex: Integration tests pass now. Please review.

Patch:
diff --git a/backend/chunker/chunker.go b/backend/chunker/chunker.go
index 6d5e25bed..ca7305201 100644
--- a/backend/chunker/chunker.go
+++ b/backend/chunker/chunker.go
@@ -29,30 +29,93 @@ import (
 	"github.com/rclone/rclone/fs/operations"
 )
 
+//
+// Chunker's composite files have one or more chunks
+// and optional metadata object. If it's present,
+// meta object is named after the original file.
+//
+// The following types of chunks are supported:
+// data and control, active and temporary.
+// Chunk type is identified by matching chunk file name
+// based on the chunk name format configured by user.
+//
+// Both data and control chunks can be either temporary or
+// active (non-temporary).
+// An operation creates temporary chunks while it runs.
+// By completion it removes temporary and leaves active
+// (aka normal aka permanent) chunks.
+//
+// Temporary (aka hidden) chunks have a special hardcoded suffix
+// in addition to the configured name pattern. The suffix comes last
+// to prevent name collisions with non-temporary chunks.
+// Temporary suffix includes so called transaction number usually
+// abbreviated as `xactNo` below, a generic non-negative integer
+// used by parallel operations to share a composite object.
+//
+// Chunker can tell data chunks from control chunks by the characters
+// located in the "hash placeholder" position of configured format.
+// Data chunks have decimal digits there.
+// Control chunks have a short lowercase literal prepended by underscore
+// in that position.
+//
+// Metadata format v1 does not define any control chunk types,
+// they are currently ignored aka reserved.
+// In future they can be used to implement resumable uploads etc.
+//
 const (
-	// optimizeFirstChunk enables the following Put optimization:
-	// if a single chunk is expected, put the first chunk using base target
-	// name instead of temporary name, thus avoiding extra rename operation.
-	// WARNING: this optimization is not transaction safe!
-	optimizeFirstChunk = false
-
-	// Normally metadata is a small (100-200 bytes) piece of JSON.
-	// Valid metadata size should not exceed this limit.
-	maxMetaDataSize = 199
+	ctrlTypeRegStr  = `[a-z]{3,9}`
+	tempChunkFormat = `%s..tmp_%010d`
+	tempChunkRegStr = `\.\.tmp_([0-9]{10,19})`
+)
 
-	metaDataVersion = 1
+var (
+	ctrlTypeRegexp = regexp.MustCompile(`^` + ctrlTypeRegStr + `$`)
 )
 
-// Formatting of temporary chunk names. Temporary suffix *follows* chunk
-// suffix to prevent name collisions between chunks and ordinary files.
+// Normally metadata is a small piece of JSON (about 100-300 bytes).
+// The size of valid metadata size must never exceed this limit.
+// Current maximum provides a reasonable room for future extensions.
+//
+// Please refrain from increasing it, this can cause old rclone versions
+// to fail, or worse, treat meta object as a normal file (see NewObject).
+// If more room is needed please bump metadata version forcing previous
+// releases to ask for upgrade, and offload extra info to a control chunk.
+//
+// And still chunker's primary function is to chunk large files
+// rather than serve as a generic metadata container.
+const maxMetadataSize = 255
+
+// Current/highest supported metadata format.
+const metadataVersion = 1
+
+// optimizeFirstChunk enables the following optimization in the Put:
+// If a single chunk is expected, put the first chunk using the
+// base target name instead of a temporary name, thus avoiding
+// extra rename operation.
+// Warning: this optimization is not transaction safe.
+const optimizeFirstChunk = false
+
+// revealHidden is a stub until chunker lands the `reveal hidden` option.
+const revealHidden = false
+
+// Prevent memory overflow due to specially crafted chunk name
+const maxSafeChunkNumber = 10000000
+
+// standard chunker errors
 var (
-	tempChunkFormat = `%s..tmp_%010d`
-	tempChunkRegexp = regexp.MustCompile(`^(.+)\.\.tmp_([0-9]{10,19})$`)
+	ErrChunkOverflow = errors.New("chunk number overflow")
+)
+
+// variants of baseMove's parameter delMode
+const (
+	delNever  = 0 // don't delete, just move
+	delAlways = 1 // delete destination before moving
+	delFailed = 2 // move, then delete and try again if failed
 )
 
 // Note: metadata logic is tightly coupled with chunker code in many
-// places of the code, eg. in checks whether a file can have meta object
-// or is eligible for chunking.
+// places, eg. in checks whether a file should have meta object or is
+// eligible for chunking.
 // If more metadata formats (or versions of a format) are added in future,
 // it may be advisable to factor it into a "metadata strategy" interface
 // similar to chunkingReader or linearReader below.
@@ -102,44 +165,47 @@ Metadata is a small JSON file named after the composite file.`,
 			}, {
 				Value: "simplejson",
 				Help: `Simple JSON supports hash sums and chunk validation.
-It has the following fields: size, nchunks, md5, sha1.`,
+It has the following fields: ver, size, nchunks, md5, sha1.`,
 			}},
 		}, {
 			Name:     "hash_type",
 			Advanced: false,
 			Default:  "md5",
-			Help:     `Choose how chunker handles hash sums.`,
+			Help:     `Choose how chunker handles hash sums. All modes but "none" require metadata.`,
 			Examples: []fs.OptionExample{{
 				Value: "none",
-				Help: `Chunker can pass any hash supported by wrapped remote
-for a single-chunk file but returns nothing otherwise.`,
+				Help:  `Pass any hash supported by wrapped remote for non-chunked files, return nothing otherwise`,
 			}, {
 				Value: "md5",
-				Help:  `MD5 for multi-chunk files. Requires "simplejson".`,
+				Help:  `MD5 for composite files`,
 			}, {
 				Value: "sha1",
-				Help:  `SHA1 for multi-chunk files. Requires "simplejson".`,
+				Help:  `SHA1 for composite files`,
+			}, {
+				Value: "md5all",
+				Help:  `MD5 for all files`,
+			}, {
+				Value: "sha1all",
+				Help:  `SHA1 for all files`,
 			}, {
 				Value: "md5quick",
-				Help: `Copying a file to chunker will request MD5 from the source
-falling back to SHA1 if unsupported. Requires "simplejson".`,
+				Help:  `Copying a file to chunker will request MD5 from the source falling back to SHA1 if unsupported`,
 			}, {
 				Value: "sha1quick",
-				Help:  `Similar to "md5quick" but prefers SHA1 over MD5. Requires "simplejson".`,
+				Help:  `Similar to "md5quick" but prefers SHA1 over MD5`,
 			}},
 		}, {
-			Name:     "fail_on_bad_chunks",
+			Name:     "fail_hard",
 			Advanced: true,
 			Default:  false,
-			Help: `The list command might encounter files with missinng or invalid chunks.
-This boolean flag tells what rclone should do in such cases.`,
+			Help:     `Choose how chunker should handle files with missing or invalid chunks.`,
 			Examples: []fs.OptionExample{
 				{
 					Value: "true",
-					Help:  "Fail with error.",
+					Help:  "Report errors and abort current command.",
 				}, {
 					Value: "false",
-					Help:  "Silently ignore invalid object.",
+					Help:  "Warn user, skip incomplete file and proceed.",
 				},
 			},
 		}},
@@ -183,39 +249,10 @@ func NewFs(name, rpath string, m configmap.Mapper) (fs.Fs, error) {
 		root: rpath,
 		opt:  *opt,
 	}
+	f.dirSort = true // processEntries requires that meta Objects prerun data chunks atm.
 
-	switch opt.MetaFormat {
-	case "none":
-		f.useMeta = false
-	case "simplejson":
-		f.useMeta = true
-	default:
-		return nil, fmt.Errorf("unsupported meta format '%s'", opt.MetaFormat)
-	}
-
-	requireMetaHash := true
-	switch opt.HashType {
-	case "none":
-		requireMetaHash = false
-	case "md5":
-		f.useMD5 = true
-	case "sha1":
-		f.useSHA1 = true
-	case "md5quick":
-		f.useMD5 = true
-		f.quickHash = true
-	case "sha1quick":
-		f.useSHA1 = true
-		f.quickHash = true
-	default:
-		return nil, fmt.Errorf("unsupported hash type '%s'", opt.HashType)
-	}
-	if requireMetaHash && opt.MetaFormat != "simplejson" {
-		return nil, fmt.Errorf("hash type '%s' requires meta format 'simplejson'", opt.HashType)
-	}
-
-	if err := f.parseNameFormat(opt.NameFormat); err != nil {
-		return nil, fmt.Errorf("invalid name format '%s': %v", opt.NameFormat, err)
+	if err := f.configure(opt.NameFormat, opt.MetaFormat, opt.HashType); err != nil {
+		return nil, err
 	}
 
 	// Handle the tricky case detected by FsMkdir/FsPutFiles/FsIsFile
@@ -224,7 +261,7 @@ func NewFs(name, rpath string, m configmap.Mapper) (fs.Fs, error) {
 	// detects a composite file because it finds the first chunk!
 	// (yet can't satisfy fstest.CheckListing, will ignore)
 	if err == nil && !f.useMeta && strings.Contains(rpath, "/") {
-		firstChunkPath := f.makeChunkName(remotePath, 0, -1)
+		firstChunkPath := f.makeChunkName(remotePath, 0, "", -1)
 		_, testErr := baseInfo.NewFs(baseName, firstChunkPath, baseConfig)
 		if testErr == fs.ErrorIsFile {
 			err = testErr
@@ -250,53 +287,105 @@ func NewFs(name, rpath string, m configmap.Mapper) (fs.Fs, error) {
 
 // Options defines the configuration for this backend
 type Options struct {
-	Remote          string        `config:"remote"`
-	ChunkSize       fs.SizeSuffix `config:"chunk_size"`
-	NameFormat      string        `config:"name_format"`
-	StartFrom       int           `config:"start_from"`
-	MetaFormat      string        `config:"meta_format"`
-	HashType        string        `config:"hash_type"`
-	FailOnBadChunks bool          `config:"fail_on_bad_chunks"`
+	Remote     string        `config:"remote"`
+	ChunkSize  fs.SizeSuffix `config:"chunk_size"`
+	NameFormat string        `config:"name_format"`
+	StartFrom  int           `config:"start_from"`
+	MetaFormat string        `config:"meta_format"`
+	HashType   string        `config:"hash_type"`
+	FailHard   bool          `config:"fail_hard"`
 }
 
 // Fs represents a wrapped fs.Fs
 type Fs struct {
-	base       fs.Fs
-	wrapper    fs.Fs
-	name       string
-	root       string
-	useMeta    bool
-	useMD5     bool // mutually exclusive with useSHA1
-	useSHA1    bool // mutually exclusive with useMD5
-	quickHash  bool
-	nameFormat string
-	nameRegexp *regexp.Regexp
-	opt        Options
-	features   *fs.Features // optional features
-}
-
-// Name of the remote (as passed into NewFs)
-func (f *Fs) Name() string {
-	return f.name
+	name         string
+	root         string
+	base         fs.Fs          // remote wrapped by chunker overlay
+	wrapper      fs.Fs          // wrapper is used by SetWrapper
+	useMeta      bool           // false if metadata format is 'none'
+	useMD5       bool           // mutually exclusive with useSHA1
+	useSHA1      bool           // mutually exclusive with useMD5
+	hashFallback bool           // allows fallback from MD5 to SHA1 and vice versa
+	hashAll      bool           // hash all files, mutually exclusive with hashFallback
+	dataNameFmt  string         // name format of data chunks
+	ctrlNameFmt  string         // name format of control chunks
+	nameRegexp   *regexp.Regexp // regular expression to match chunk names
+	opt          Options        // copy of Options
+	features     *fs.Features   // optional features
+	dirSort      bool           // reserved for future, ignored
+}
+
+// configure must be called only from NewFs or by unit tests
+func (f *Fs) configure(nameFormat, metaFormat, hashType string) error {
+	if err := f.setChunkNameFormat(nameFormat); err != nil {
+		return errors.Wrapf(err, "invalid name format '%s'", nameFormat)
+	}
+	if err := f.setMetaFormat(metaFormat); err != nil {
+		return err
+	}
+	if err := f.setHashType(hashType); err != nil {
+		return err
+	}
+	return nil
 }
 
-// Root of the remote (as passed into NewFs)
-func (f *Fs) Root() string {
-	return f.root
+func (f *Fs) setMetaFormat(metaFormat string) error {
+	switch metaFormat {
+	case "none":
+		f.useMeta = false
+	case "simplejson":
+		f.useMeta = true
+	default:
+		return fmt.Errorf("unsupported meta format '%s'", metaFormat)
+	}
+	return nil
 }
 
-// Features returns the optional features of this Fs
-func (f *Fs) Features() *fs.Features {
-	return f.features
-}
+// setHashType
+// must be called *after* setMetaFormat.
+//
+// In the "All" mode chunker will force metadata on all files
+// if the wrapped remote can't provide given hashsum.
+func (f *Fs) setHashType(hashType string) error {
+	f.useMD5 = false
+	f.useSHA1 = false
+	f.hashFallback = false
+	f.hashAll = false
+	requireMetaHash := true
 
-// String returns a description of the FS
-func (f *Fs) String() string {
-	return fmt.Sprintf("Chunked '%s:%s'", f.name, f.root)
+	switch hashType {
+	case "none":
+		requireMetaHash = false
+	case "md5":
+		f.useMD5 = true
+	case "sha1":
+		f.useSHA1 = true
+	case "md5quick":
+		f.useMD5 = true
+		f.hashFallback = true
+	case "sha1quick":
+		f.useSHA1 = true
+		f.hashFallback = true
+	case "md5all":
+		f.useMD5 = true
+		f.hashAll = !f.base.Hashes().Contains(hash.MD5)
+	case "sha1all":
+		f.useSHA1 = true
+		f.hashAll = !f.base.Hashes().Contains(hash.SHA1)
+	default:
+		return fmt.Errorf("unsupported hash type '%s'", hashType)
+	}
+	if requireMetaHash && !f.useMeta {
+		return fmt.Errorf("hash type '%s' requires compatible meta format", hashType)
+	}
+	return nil
 }
 
-// parseNameFormat converts pattern-based name format into Printf format and Regexp
-func (f *Fs) parseNameFormat(pattern string) error {
+// setChunkNameFormat converts pattern based chunk name format
+// into Printf format and Regular expressions for data and
+// control chunks.
+func (f *Fs) setChunkNameFormat(pattern string) error {
+	// validate pattern
 	if strings.Count(pattern, "*") != 1 {
 		return errors.New("pattern must have exactly one asterisk (*)")
 	}
@@ -310,86 +399,140 @@ func (f *Fs) parseNameFormat(pattern string) error {
 	if ok, _ := regexp.MatchString("^[^#]*[#]+[^#]*$", pattern); !ok {
 		return errors.New("hashes (#) in pattern must be consecutive")
 	}
+	if dir, _ := path.Split(pattern); dir != "" {
+		return errors.New("directory separator prohibited")
+	}
+	if pattern[0] != '*' {
+		return errors.New("pattern must start with asterisk") // to be lifted later
+	}
 
+	// craft a unified regular expression for all types of chunks
 	reHashes := regexp.MustCompile("[#]+")
-	strRegex := regexp.QuoteMeta(pattern)
-	reDigits := "([0-9]+)"
+	reDigits := "[0-9]+"
 	if numDigits > 1 {
-		reDigits = fmt.Sprintf("([0-9]{%d,})", numDigits)
+		reDigits = fmt.Sprintf("[0-9]{%d,}", numDigits)
 	}
-	strRegex = reHashes.ReplaceAllLiteralString(strRegex, reDigits)
-	strRegex = strings.Replace(strRegex, "\\*", "(.+)", -1)
-	f.nameRegexp = regexp.MustCompile("^" + strRegex + "$")
+	reDataOrCtrl := fmt.Sprintf("(?:(%s)|_(%s))", reDigits, ctrlTypeRegStr)
 
-	strFmt := strings.Replace(pattern, "%", "%%", -1) // escape percent signs for name format
+	// this must be non-greedy or else it can eat up temporary suffix
+	const mainNameRegStr = "(.+?)"
+
+	strRegex := regexp.QuoteMeta(pattern)
+	strRegex = reHashes.ReplaceAllLiteralString(strRegex, reDataOrCtrl)
+	strRegex = strings.Replace(strRegex, "\\*", mainNameRegStr, -1)
+	strRegex = fmt.Sprintf("^%s(?:%s)?$", strRegex, tempChunkRegStr)
+	f.nameRegexp = regexp.MustCompile(strRegex)
+
+	// craft printf formats for active data/control chunks
 	fmtDigits := "%d"
 	if numDigits > 1 {
 		fmtDigits = fmt.Sprintf("%%0%dd", numDigits)
 	}
-	strFmt = reHashes.ReplaceAllLiteralString(strFmt, fmtDigits)
-	f.nameFormat = strings.Replace(strFmt, "*", "%s", -1)
+	strFmt := strings.Replace(pattern, "%", "%%", -1)
+	strFmt = strings.Replace(strFmt, "*", "%s", 1)
+	f.dataNameFmt = reHashes.ReplaceAllLiteralString(strFmt, fmtDigits)
+	f.ctrlNameFmt = reHashes.ReplaceAllLiteralString(strFmt, "_%s")
 	return nil
 }
 
-// makeChunkName produces a chunk name for a given file name.
-// chunkNo must be a zero based index.
-// A negative tempNo (eg. -1) indicates normal chunk, else temporary.
-func (f *Fs) makeChunkName(mainName string, chunkNo int, tempNo int64) string {
-	name := mainName
-	name = fmt.Sprintf(f.nameFormat, name, chunkNo+f.opt.StartFrom)
-	if tempNo < 0 {
-		return name
+// makeChunkName produces chunk name (or path) for given file.
+//
+// mainPath can be name, relative or absolute path of main file.
+//
+// chunkNo must be a zero based index of data chunk.
+// Negative chunkNo eg. -1 indicates a control chunk.
+// ctrlType is type of control chunk (must be valid).
+// ctrlType must be "" for data chunks.
+//
+// xactNo is a transaction number.
+// Negative xactNo eg. -1 indicates an active chunk,
+// otherwise produce temporary chunk name.
+//
+func (f *Fs) makeChunkName(mainPath string, chunkNo int, ctrlType string, xactNo int64) string {
+	dir, mainName := path.Split(mainPath)
+	var name string
+	switch {
+	case chunkNo >= 0 && ctrlType == "":
+		name = fmt.Sprintf(f.dataNameFmt, mainName, chunkNo+f.opt.StartFrom)
+	case chunkNo < 0 && ctrlTypeRegexp.MatchString(ctrlType):
+		name = fmt.Sprintf(f.ctrlNameFmt, mainName, ctrlType)
+	default:
+		panic("makeChunkName: invalid argument") // must not produce something we can't consume
+	}
+	if xactNo >= 0 {
+		name = fmt.Sprintf(tempChunkFormat, name, xactNo)
 	}
-	return fmt.Sprintf(tempChunkFormat, name, tempNo)
+	return dir + name
 }
 
-// parseChunkName validates and parses a given chunk name.
-// Returned mainName is "" if it's not a chunk name.
-// Returned chunkNo is zero based.
-// Returned tempNo is -1 for a normal chunk
-// or non-negative integer for a temporary chunk.
-func (f *Fs) parseChunkName(name string) (mainName string, chunkNo int, tempNo int64) {
+// parseChunkName checks whether given file path belongs to
+// a chunk and extracts chunk name parts.
+//
+// filePath can be name, relative or absolute path of a file.
+//
+// Returned mainPath is a non-empty string if valid chunk name
+// is detected or "" if it's not a chunk.
+// Other returned values depend on detected chunk type:
+// data or control, active or temporary:
+//
+// data chunk - the returned chunkNo is non-negative and ctrlType is ""
+// control chunk - the chunkNo is -1 and ctrlType is non-empty string
+// active chunk - the returned xactNo is -1
+// temporary chunk - the xactNo is non-negative integer
+func (f *Fs) parseChunkName(filePath string) (mainPath string, chunkNo int, ctrlType string, xactNo int64) {
+	dir, name := path.Split(filePath)
+	match := f.nameRegexp.FindStringSubmatch(name)
+	if match == nil || match[1] == "" {
+		return "", -1, "", -1
+	}
 	var err error
-	chunkMatch := f.nameRegexp.FindStringSubmatchIndex(name)
-	chunkName := name
-	tempNo = -1
-	if chunkMatch == nil {
-		tempMatch := tempChunkRegexp.FindStringSubmatchIndex(name)
-		if tempMatch == nil {
-			return "", -1, -1
-		}
-		chunkName = name[tempMatch[2]:tempMatch[3]]
-		tempNo, err = strconv.ParseInt(name[tempMatch[4]:tempMatch[5]], 10, 64)
-		if err != nil {
-			return "", -1, -1
+
+	chunkNo = -1
+	if match[2] != "" {
+		if chunkNo, err = strconv.Atoi(match[2]); err != nil {
+			chunkNo = -1
 		}
-		chunkMatch = f.nameRegexp.FindStringSubmatchIndex(chunkName)
-		if chunkMatch == nil {
-			return "", -1, -1
+		if chunkNo -= f.opt.StartFrom; chunkNo < 0 {
+			fs.Infof(f, "invalid data chunk number in file %q", name)
+			return "", -1, "", -1
 		}
 	}
-	mainName = chunkName[chunkMatch[2]:chunkMatch[3]]
-	chunkNo, err = strconv.Atoi(chunkName[chunkMatch[4]:chunkMatch[5]])
-	if err != nil {
-		return "", -1, -1
+
+	xactNo = -1
+	if match[4] != "" {
+		if xactNo, err = strconv.ParseInt(match[4], 10, 64); err != nil || xactNo < 0 {
+			fs.Infof(f, "invalid transaction number in file %q", name)
+			return "", -1, "", -1
+		}
 	}
-	chunkNo -= f.opt.StartFrom
-	if chunkNo < 0 {
-		fs.Infof(f, "invalid chunk number in name %q", name)
-		return "", -1, -1
+
+	mainPath = dir + match[1]
+	ctrlType = match[3]
+	return
+}
+
+// forbidChunk prints error message or raises error if file is chunk.
+// First argument sets log prefix, use `false` to suppress message.
+func (f *Fs) forbidChunk(o interface{}, filePath string) error {
+	if mainPath, _, _, _ := f.parseChunkName(filePath); mainPath != "" {
+		if f.opt.FailHard {
+			return fmt.Errorf("chunk overlap with %q", mainPath)
+		}
+		if boolVal, isBool := o.(bool); !isBool || boolVal {
+			fs.Errorf(o, "chunk overlap with %q", mainPath)
+		}
 	}
-	return mainName, chunkNo, tempNo
+	return nil
 }
 
-// List the objects and directories in dir into entries.  The
-// entries can be returned in any order but should be for a
-// complete directory.
+// List the objects and directories in dir into entries.
+// The entries can be returned in any order but should be
+// for a complete directory.
 //
 // dir should be "" to list the root, and should not have
 // trailing slashes.
 //
-// This should return ErrDirNotFound if the directory isn't
-// found.
+// This should return ErrDirNotFound if the directory isn't found.
 //
 // Commands normally cleanup all temporary chunks in case of a failure.
 // However, if rclone dies unexpectedly, it can leave behind a bunch of
@@ -397,7 +540,7 @@ func (f *Fs) parseChunkName(name string) (mainName string, chunkNo int, tempNo i
 // silently skip all temporary chunks in the directory. It's okay if
 // they belong to an unfinished command running in parallel.
 //
-// However, there is no way to discover dead temporary chunks a.t.m.
+// However, there is no way to discover dead temporary chunks atm.
 // As a workaround users can use `purge` to forcibly remove the whole
 // directory together with dead chunks.
 // In future a flag named like `--chunker-list-hidden` may be added to
@@ -408,7 +551,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
 	if err != nil {
 		return nil, err
 	}
-	return f.chunkEntries(ctx, entries, f.opt.FailOnBadChunks)
+	return f.processEntries(ctx, entries, dir)
 }
 
 // ListR lists the objects and directories of the Fs starting
@@ -426,11 +569,11 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
 // immediately.
 //
 // Don't implement this unless you have a more efficient way
-// of listing recursively that doing a directory traversal.
+// of listing recursively than doing a directory traversal.
 func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (err error) {
 	do := f.base.Features().ListR
 	return do(ctx, dir, func(entries fs.DirEntries) error {
-		newEntries, err := f.chunkEntries(ctx, entries, f.opt.FailOnBadChunks)
+		newEntries, err := f.processEntries(ctx, entries, dir)
 		if err != nil {
 			return err
 		}
@@ -438,13 +581,17 @@ func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (
 	})
 }
 
-// chunkEntries is called by List(R). It merges chunk entries from
-// wrapped remote into composite directory entries.
-func (f *Fs) chunkEntries(ctx context.Context, origEntries fs.DirEntries, hardErrors bool) (chunkedEntries fs.DirEntries, err error) {
-	// sort entries, so that meta objects (if any) appear before their chunks
-	sortedEntries := make(fs.DirEntries, len(origEntries))
-	copy(sortedEntries, origEntries)
-	sort.Sort(sortedEntries)
+// processEntries assembles chunk entries into composite entries
+func (f *Fs) processEntries(ctx context.Context, origEntries fs.DirEntries, dirPath string) (newEntries fs.DirEntries, err error) {
+	var sortedEntries fs.DirEntries
+	if f.dirSort {
+		// sort entries so that meta objects go before their chunks
+		sortedEntries = make(fs.DirEntries, len(origEntries))
+		copy(sortedEntries, origEntries)
+		sort.Sort(sortedEntries)
+	} else {
+		sortedEntries = origEntries
+	}
 
 	byRemote := make(map[string]*Object)
 	badEntry := make(map[string]bool)
@@ -455,9 +602,17 @@ func (f *Fs) chunkEntries(ctx context.Context, origEntries fs.DirEntries, hardEr
 		switch entry := dirOrObject.(type) {
 		case fs.Object:
 			remote := entry.Remote()
-			if mainRemote, chunkNo, tempNo := f.parseChunkName(remote); mainRemote != "" {
-				if tempNo != -1 {
-					fs.Debugf(f, "skip temporary chunk %q", remote)
+			if mainRemote, chunkNo, ctrlType, xactNo := f.parseChunkName(remote); mainRemote != "" {
+				if xactNo != -1 {
+					if revealHidden {
+						fs.Infof(f, "ignore temporary chunk %q", remote)
+					}
+					break
+				}
+				if ctrlType != "" {
+					if revealHidden {
+						fs.Infof(f, "ignore control chunk %q", remote)
+					}
 					break
 				}
 				mainObject := byRemote[mainRemote]
@@ -466,7 +621,7 @@ func (f *Fs) chunkEntries(ctx context.Context, origEntries fs.DirEntries, hardEr
 					break
 				}
 				if mainObject == nil {
-					// useMeta is false - create chunked object without meta data
+					// useMeta is false - create chunked object without metadata
 					mainObject = f.newObject(mainRemote, nil, nil)
 					byRemote[mainRemote] = mainObject
 					if !badEntry[mainRemote] {
@@ -474,7 +629,7 @@ func (f *Fs) chunkEntries(ctx context.Context, origEntries fs.DirEntries, hardEr
 					}
 				}
 				if err := mainObject.addChunk(entry, chunkNo); err != nil {
-					if hardErrors {
+					if f.opt.FailHard {
 						return nil, err
 					}
 					badEntry[mainRemote] = true
@@ -490,8 +645,8 @@ func (f *Fs) chunkEntries(ctx context.Context, origEntries fs.DirEntries, hardEr
 			wrapDir.SetRemote(entry.Remote())
 			tempEntries = append(tempEntries, wrapDir)
 		default:
-			if hardErrors {
-				return nil, errors.Errorf("Unknown object type %T", entry)
+			if f.opt.FailHard {
+				return nil, fmt.Errorf("Unknown object type %T", entry)
 			}
 			fs.Debugf(f, "unknown object type %T", entry)
 		}
@@ -501,7 +656,7 @@ func (f *Fs) chunkEntries(ctx context.Context, origEntries fs.DirEntries, hardEr
 		if object, ok := entry.(*Object); ok {
 			remote := object.Remote()
 			if isSubdir[remote] {
-				if hardErrors {
+				if f.opt.FailHard {
 					return nil, fmt.Errorf("%q is both meta object and directory", remote)
 				}
 				badEntry[remote] = true // fall thru
@@ -511,28 +666,35 @@ func (f *Fs) chunkEntries(ctx context.Context, origEntries fs.DirEntries, hardEr
 				continue
 			}
 			if err := object.validate(); err != nil {
-				if hardErrors {
+				if f.opt.FailHard {
 					return nil, err
 				}
 				fs.Debugf(f, "invalid chunks in object %q", remote)
 				continue
 			}
 		}
-		chunkedEntries = append(chunkedEntries, entry)
+		newEntries = append(newEntries, entry)
 	}
 
-	return chunkedEntries, nil
+	if f.dirSort {
+		sort.Sort(newEntries)
+	}
+	return newEntries, nil
 }
 
 // NewObject finds the Object at remote.
 //
 // Please note that every NewObject invocation will scan the whole directory.
-// Using here something like fs.DirCache might improve performance (and make
-// logic more complex though).
+// Using here something like fs.DirCache might improve performance
+// (but will make logic more complex, though).
+//
+// Note that chunker prefers analyzing file names rather than reading
+// the content of meta object assuming that directory scans are fast
+// but opening even a small file can be slow on some backends.
 //
 func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
-	if mainRemote, _, _ := f.parseChunkName(remote); mainRemote != "" {
-		return nil, fmt.Errorf("%q should be meta object, not a chunk", remote)
+	if err := f.forbidChunk(false, remote); err != nil {
+		return nil, errors.Wrap(err, "can't access")
 	}
 
 	var (
@@ -548,17 +710,25 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
 		}
 		remote = baseObj.Remote()
 
-		// meta object cannot be large - assume single chunk
+		// Chunker's meta object cannot be large and maxMetadataSize acts
+		// as a hard limit. Anything larger than that is treated as a
+		// non-chunked file without even checking its contents, so it's
+		// paramount to prevent metadata from exceeding the maximum size.
 		o = f.newObject("", baseObj, nil)
-		if o.size > maxMetaDataSize {
+		if o.size > maxMetadataSize {
 			return o, nil
 		}
 	} else {
-		// will read single wrapped object later
+		// Metadata is disabled, hence this is either a multi-chunk
+		// composite file without meta object or a non-chunked file.
+		// Create an empty wrapper here, scan directory to determine
+		// which case it is and postpone reading if it's the latter one.
 		o = f.newObject(remote, nil, nil)
 	}
 
-	// the object is small, probably it contains meta data
+	// If the object is small, it's probably a meta object.
+	// However, composite file must have data chunks besides it.
+	// Scan directory for possible data chunks now and decide later on.
 	dir := path.Dir(strings.TrimRight(remote, "/"))
 	if dir == "." {
 		dir = ""
@@ -570,7 +740,7 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
 	case fs.ErrorDirNotFound:
 		entries = nil
 	default:
-		return nil, errors.Wrap(err, "can't detect chunked object")
+		return nil, errors.Wrap(err, "can't detect composite file")
 	}
 
 	for _, dirOrObject := range entries {
@@ -582,9 +752,9 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
 		if !strings.Contains(entryRemote, remote) {
 			continue // bypass regexp to save cpu
 		}
-		mainRemote, chunkNo, tempNo := f.parseChunkName(entryRemote)
-		if mainRemote == "" || mainRemote != remote || tempNo != -1 {
-			continue // skip non-matching or temporary chunks
+		mainRemote, chunkNo, ctrlType, xactNo := f.parseChunkName(entryRemote)
+		if mainRemote == "" || mainRemote != remote || ctrlType != "" || xactNo != -1 {
+			continue // skip non-conforming, temporary and control chunks
 		}
 		//fs.Debugf(f, "%q belongs to %q as chunk %d", entryRemote, mainRemote, chunkNo)
 		if err := o.addChunk(entry, chunkNo); err != nil {
@@ -593,10 +763,17 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
 	}
 
 	if o.main == nil && (o.chunks == nil || len(o.chunks) == 0) {
+		// Scanning hasn't found data chunks with conforming names.
 		if f.useMeta {
+			// Metadata is required but absent and there are no chunks.
 			return nil, fs.ErrorObjectNotFound
 		}
-		// lazily read single wrapped object
+
+		// Data chunks are not found and metadata is disabled.
+		// Thus, we are in the "latter case" from above.
+		// Let's try the postponed reading of a non-chunked file and add it
+		// as a single chunk to the empty composite wrapper created above
+		// with nil metadata.
 		baseObj, err = f.base.NewObject(ctx, remote)
 		if err == nil {
 			err = o.addChunk(baseObj, 0)
@@ -606,39 +783,41 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
 		}
 	}
 
-	// calculate total file size
+	// This is either a composite object with metadata or a non-chunked
+	// file without metadata. Validate it and update the total data size.
+	// As an optimization, skip metadata reading here - we will call
+	// readMetadata lazily when needed.
 	if err := o.validate(); err != nil {
 		return nil, err
 	}
-	// note: will call readMetaData lazily when needed
 	return o, nil
 }
 
-func (o *Object) readMetaData(ctx context.Context) error {
+func (o *Object) readMetadata(ctx context.Context) error {
 	if o.isFull {
 		return nil
 	}
-	if !o.isChunked() || !o.f.useMeta {
+	if !o.isComposite() || !o.f.useMeta {
 		o.isFull = true
 		return nil
 	}
 
-	// validate meta data
+	// validate metadata
 	metaObject := o.main
 	reader, err := metaObject.Open(ctx)
 	if err != nil {
 		return err
 	}
-	metaData, err := ioutil.ReadAll(reader)
+	metadata, err := ioutil.ReadAll(reader)
+	_ = reader.Close() // ensure file handle is freed on windows
 	if err != nil {
 		return err
 	}
 
 	switch o.f.opt.MetaFormat {
 	case "simplejson":
-		metaInfo, err := unmarshalSimpleJSON(ctx, metaObject, metaData)
+		metaInfo, err := unmarshalSimpleJSON(ctx, metaObject, metadata, true)
 		if err != nil {
-			// TODO: in a rare case we might mistake a small file for metadata
 			return errors.Wrap(err, "invalid metadata")
 		}
 		if o.size != metaInfo.Size() || len(o.chunks) != metaInfo.nChunks {
@@ -664,15 +843,22 @@ func (f *Fs) put(ctx context.Context, in io.Reader, src fs.ObjectInfo, remote st
 		}
 	}()
 
-	baseRemote := remote
-	tempNo := time.Now().Unix()
-	if tempNo < 0 {
-		tempNo = -tempNo // unlikely but must be positive
+	// Use system timer as a trivial source of transaction numbers,
+	// don't try hard to safeguard against chunk collisions between
+	// parallel transactions.
+	xactNo := time.Now().Unix()
+	if xactNo < 0 {
+		xactNo = -xactNo // unlikely but transaction number must be positive
 	}
+	baseRemote := remote
 
 	// Transfer chunks data
-	for chunkNo := 0; !c.done; chunkNo++ {
-		tempRemote := f.makeChunkName(baseRemote, chunkNo, tempNo)
+	for c.chunkNo = 0; !c.done; c.chunkNo++ {
+		if c.chunkNo > maxSafeChunkNumber {
+			return nil, ErrChunkOverflow
+		}
+
+		tempRemote := f.makeChunkName(baseRemote, c.chunkNo, "", xactNo)
 		size := c.sizeLeft
 		if size > c.chunkSize {
 			size = c.chunkSize
@@ -681,7 +867,7 @@ func (f *Fs) put(ctx context.Context, in io.Reader, src fs.ObjectInfo, remote st
 
 		// If a single chunk is expected, avoid the extra rename operation
 		chunkRemote := tempRemote
-		if c.expectSingle && chunkNo == 0 && optimizeFirstChunk {
+		if c.expectSingle && c.chunkNo == 0 && optimizeFirstChunk {
 			chunkRemote = baseRemote
 		}
 		info := f.wrapInfo(src, chunkRemote, size)
@@ -693,11 +879,13 @@ func (f *Fs) put(ctx context.Context, in io.Reader, src fs.ObjectInfo, remote st
 		}
 
 		if size > 0 && c.readCount == savedReadCount && c.expectSingle {
-			// basePut returned success but did not call chunking Read.
-			// This is possible if wrapped remote has performed put by hash
-			// (chunker bridges Hash from source for *single-chunk* files).
-			// Account for actually read bytes here as a workaround.
-			c.accountBytes(size)
+			// basePut returned success but didn't call chunkingReader's Read.
+			// This is possible if wrapped remote has performed the put by hash
+			// because chunker bridges Hash from source for non-chunked files.
+			// Hence, force Read here to update accounting and hashsums.
+			if err := c.dummyRead(wrapIn, size); err != nil {
+				return nil, err
+			}
 		}
 		if c.sizeLeft == 0 && !c.done {
 			// The file has been apparently put by hash, force completion.
@@ -707,9 +895,9 @@ func (f *Fs) put(ctx context.Context, in io.Reader, src fs.ObjectInfo, remote st
 		// Expected a single chunk but more to come, so name it as usual.
 		if !c.done && chunkRemote != tempRemote {
 			fs.Infof(chunk, "Expected single chunk, got more")
-			chunkMoved, errMove := f.baseMove(ctx, chunk, tempRemote, false)
+			chunkMoved, errMove := f.baseMove(ctx, chunk, tempRemote, delFailed)
 			if errMove != nil {
-				_ = chunk.Remove(ctx) // ignore error
+				silentlyRemove(ctx, chunk)
 				return nil, errMove
 			}
 			chunk = chunkMoved
@@ -717,9 +905,9 @@ func (f *Fs) put(ctx context.Context, in io.Reader, src fs.ObjectInfo, remote st
 
 		// Wrapped remote may or may not have seen EOF from chunking reader,
 		// eg. the box multi-uploader reads exactly the chunk size specified
-		// and skips the "EOF" read - switch to next limit here.
+		// and skips the "EOF" read. Hence, switch to next limit here.
 		if !(c.chunkLimit == 0 || c.chunkLimit == c.chunkSize || c.sizeTotal == -1 || c.done) {
-			_ = chunk.Remove(ctx) // ignore error
+			silentlyRemove(ctx, chunk)
 			return nil, fmt.Errorf("Destination ignored %d data bytes", c.chunkLimit)
 		}
 		c.chunkLimit = c.chunkSize
@@ -732,17 +920,26 @@ func (f *Fs) put(ctx context.Context, in io.Reader, src fs.ObjectInfo, remote st
 		return nil, fmt.Errorf("Incorrect upload size %d != %d", c.readCount, c.sizeTotal)
 	}
 
-	// Finalize the single-chunk object
-	if len(c.chunks) == 1 {
-		// If old object was chunked, remove its chunks
+	// Check for input that looks like valid metadata
+	needMeta := len(c.chunks) > 1
+	if c.readCount <= maxMetadataSize && len(c.chunks) == 1 {
+		_, err := unmarshalSimpleJSON(ctx, c.chunks[0], c.smallHead, false)
+		needMeta = err == nil
+	}
+
+	// Finalize small object as non-chunked.
+	// This can be bypassed, and single chunk with metadata will be
+	// created if forced by consistent hashing or due to unsafe input.
+	if !needMeta && !f.hashAll && f.useMeta {
+		// If previous object was chunked, remove its chunks
 		f.removeOldChunks(ctx, baseRemote)
 
-		// Rename single chunk in place
+		// Rename single data chunk in place
 		chunk := c.chunks[0]
 		if chunk.Remote() != baseRemote {
-			chunkMoved, errMove := f.baseMove(ctx, chunk, baseRemote, true)
+			chunkMoved, errMove := f.baseMove(ctx, chunk, baseRemote, delAlways)
 			if errMove != nil {
-				_ = chunk.Remove(ctx) // ignore error
+				silentlyRemove(ctx, chunk)
 				return nil, errMove
 			}
 			chunk = chunkMoved
@@ -751,7 +948,7 @@ func (f *Fs) put(ctx context.Context, in io.Reader, src fs.ObjectInfo, remote st
 		return f.newObject("", chunk, nil), nil
 	}
 
-	// Validate total size of chunks
+	// Validate total size of data chunks
 	var sizeTotal int64
 	for _, chunk := range c.chunks {
 		sizeTotal += chunk.Size()
@@ -760,13 +957,13 @@ func (f *Fs) put(ctx context.Context, in io.Reader, src fs.ObjectInfo, remote st
 		return nil, fmt.Errorf("Incorrect chunks size %d != %d", sizeTotal, c.readCount)
 	}
 
-	// If old object was chunked, remove its chunks
+	// If previous object was chunked, remove its chunks
 	f.removeOldChunks(ctx, baseRemote)
 
-	// Rename chunks from temporary to final names
+	// Rename data chunks from temporary to final names
 	for chunkNo, chunk := range c.chunks {
-		chunkRemote := f.makeChunkName(baseRemote, chunkNo, -1)
-		chunkMoved, errMove := f.baseMove(ctx, chunk, chunkRemote, false)
+		chunkRemote := f.makeChunkName(baseRemote, chunkNo, "", -1)
+		chunkMoved, errMove := f.baseMove(ctx, chunk, chunkRemote, delFailed)
 		if errMove != nil {
 			return nil, errMove
 		}
@@ -777,7 +974,7 @@ func (f *Fs) put(ctx context.Context, in io.Reader, src fs.ObjectInfo, remote st
 		// Remove stale metadata, if any
 		oldMeta, errOldMeta := f.base.NewObject(ctx, baseRemote)
 		if errOldMeta == nil {
-			_ = oldMeta.Remove(ctx) // ignore error
+			silentlyRemove(ctx, oldMeta)
 		}
 
 		o := f.newObject(baseRemote, nil, c.chunks)
@@ -786,15 +983,15 @@ func (f *Fs) put(ctx context.Context, in io.Reader, src fs.ObjectInfo, remote st
 	}
 
 	// Update meta object
-	var metaData []byte
+	var metadata []byte
 	switch f.opt.MetaFormat {
 	case "simplejson":
 		c.updateHashes()
-		metaData, err = marshalSimpleJSON(ctx, sizeTotal, len(c.chunks), c.md5, c.sha1)
+		metadata, err = marshalSimpleJSON(ctx, sizeTotal, len(c.chunks), c.md5, c.sha1)
 	}
 	if err == nil {
-		metaInfo := f.wrapInfo(src, baseRemote, int64(len(metaData)))
-		metaObject, err = basePut(ctx, bytes.NewReader(metaData), metaInfo)
+		metaInfo := f.wrapInfo(src, baseRemote, int64(len(metadata)))
+		metaObject, err = basePut(ctx, bytes.NewReader(metadata), metaInfo)
 	}
 	if err != nil {
 		return nil, err
@@ -814,10 +1011,12 @@ type chunkingReader struct {
 	readCount    int64
 	chunkSize    int64
 	chunkLimit   int64
+	chunkNo      int
 	err          error
 	done         bool
 	chunks       []fs.Object
 	expectSingle bool
+	smallHead    []byte
 	fs           *Fs
 	hasher       gohash.Hash
 	md5          string
@@ -827,7 +1026,6 @@ type chunkingReader struct {
 func (f *Fs) newChunkingReader(src fs.ObjectInfo) *chunkingReader {
 	c := &chunkingReader{
 		fs:        f,
-		readCount: 0,
 		chunkSize: int64(f.opt.ChunkSize),
 		sizeTotal: src.Size(),
 	}
@@ -843,7 +1041,7 @@ func (c *chunkingReader) wrapStream(ctx context.Context, in io.Reader, src fs.Ob
 	switch {
 	case c.fs.useMD5:
 		if c.md5, _ = src.Hash(ctx, hash.MD5); c.md5 == "" {
-			if c.fs.quickHash {
+			if c.fs.hashFallback {
 				c.sha1, _ = src.Hash(ctx, hash.SHA1)
 			} else {
 				c.hasher = md5.New()
@@ -851,7 +1049,7 @@ func (c *chunkingReader) wrapStream(ctx context.Context, in io.Reader, src fs.Ob
 		}
 	case c.fs.useSHA1:
 		if c.sha1, _ = src.Hash(ctx, hash.SHA1); c.sha1 == "" {
-			if c.fs.quickHash {
+			if c.fs.hashFallback {
 				c.md5, _ = src.Hash(ctx, hash.MD5)
 			} else {
 				c.hasher = sha1.New()
@@ -898,6 +1096,9 @@ func (c *chunkingReader) Read(buf []byte) (bytesRead int, err error) {
 		return
 	}
 	c.accountBytes(int64(bytesRead))
+	if c.chunkNo == 0 && c.expectSingle && bytesRead > 0 && c.readCount <= maxMetadataSize {
+		c.smallHead = append(c.smallHead, buf[:bytesRead]...)
+	}
 	if bytesRead == 0 && c.sizeLeft == 0 {
 		err = io.EOF // Force EOF when no data left.
 	}
@@ -915,7 +1116,28 @@ func (c *chunkingReader) accountBytes(bytesRead int64) {
 	}
 }
 
-// rollback removes uploaded temporary chunk
+// dummyRead updates accounting, hashsums etc by simulating reads
+func (c *chunkingReader) dummyRead(in io.Reader, size int64) error {
+	if c.hasher == nil && c.readCount+size > maxMetadataSize {
+		c.accountBytes(size)
+		return nil
+	}
+	const bufLen = 1048576 // 1MB
+	buf := make([]byte, bufLen)
+	for size > 0 {
+		n := size
+		if n > bufLen {
+			n = bufLen
+		}
+		if _, err := io.ReadFull(in, buf[0:n]); err != nil {
+			return err
+		}
+		size -= n
+	}
+	return nil
+}
+
+// rollback removes uploaded temporary chunks
 func (c *chunkingReader) rollback(ctx context.Context, metaObject fs.Object) {
 	if metaObject != nil {
 		c.chunks = append(c.chunks, metaObject)
@@ -939,24 +1161,34 @@ func (f *Fs) removeOldChunks(ctx context.Context, remote string) {
 	}
 }
 
-// Put in to the remote path with the modTime given of the given size
+// Put into the remote path with the given modTime and size.
 //
 // May create the object even if it returns an error - if so
 // will return the object and the error, otherwise will return
 // nil and the error
 func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
+	if err := f.forbidChunk(src, src.Remote()); err != nil {
+		return nil, errors.Wrap(err, "refusing to put")
+	}
 	return f.put(ctx, in, src, src.Remote(), options, f.base.Put)
 }
 
 // PutStream uploads to the remote path with the modTime given of indeterminate size
 func (f *Fs) PutStream(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
+	if err := f.forbidChunk(src, src.Remote()); err != nil {
+		return nil, errors.Wrap(err, "refusing to upload")
+	}
 	return f.put(ctx, in, src, src.Remote(), options, f.base.Features().PutStream)
 }
 
 // Update in to the object with the modTime given of the given size
 func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
-	if err := o.readMetaData(ctx); err != nil {
-		return err
+	if err := o.f.forbidChunk(o, o.Remote()); err != nil {
+		return errors.Wrap(err, "update refused")
+	}
+	if err := o.readMetadata(ctx); err != nil {
+		// refuse to update a file of unsupported format
+		return errors.Wrap(err, "refusing to update")
 	}
 	basePut := o.f.base.Put
 	if src.Size() < 0 {
@@ -976,13 +1208,12 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
 //
 // This will create a duplicate if we upload a new file without
 // checking to see if there is one already - use Put() for that.
-// TODO: really split stream here
 func (f *Fs) PutUnchecked(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
 	do := f.base.Features().PutUnchecked
 	if do == nil {
 		return nil, errors.New("can't PutUnchecked")
 	}
-	// TODO: handle options and chunking!
+	// TODO: handle range/limit options and really chunk stream here!
 	o, err := do(ctx, in, f.wrapInfo(src, "", -1))
 	if err != nil {
 		return nil, err
@@ -997,13 +1228,13 @@ func (f *Fs) Precision() time.Duration {
 
 // Hashes returns the supported hash sets.
 // Chunker advertises a hash type if and only if it can be calculated
-// for files of any size, multi-chunked or small.
+// for files of any size, non-chunked or composite.
 func (f *Fs) Hashes() hash.Set {
-	// composites && all of them && small files supported by wrapped remote
-	if f.useMD5 && !f.quickHash && f.base.Hashes().Contains(hash.MD5) {
+	// composites AND no fallback AND (chunker OR wrapped Fs will hash all non-chunked's)
+	if f.useMD5 && !f.hashFallback && (f.hashAll || f.base.Hashes().Contains(hash.MD5)) {
 		return hash.NewHashSet(hash.MD5)
 	}
-	if f.useSHA1 && !f.quickHash && f.base.Hashes().Contains(hash.SHA1) {
+	if f.useSHA1 && !f.hashFallback && (f.hashAll || f.base.Hashes().Contains(hash.SHA1)) {
 		return hash.NewHashSet(hash.SHA1)
 	}
 	return hash.NewHashSet() // can't provide strong guarantees
@@ -1013,6 +1244,9 @@ func (f *Fs) Hashes() hash.Set {
 //
 // Shouldn't return an error if it already exists
 func (f *Fs) Mkdir(ctx context.Context, dir string) error {
+	if err := f.forbidChunk(dir, dir); err != nil {
+		return errors.Wrap(err, "can't mkdir")
+	}
 	return f.base.Mkdir(ctx, dir)
 }
 
@@ -1031,8 +1265,8 @@ func (f *Fs) Rmdir(ctx context.Context, dir string) error {
 // Return an error if it doesn't exist.
 //
 // This command will chain to `purge` from wrapped remote.
-// As a result it removes not only chunker files with their
-// active chunks but also all hidden chunks in the directory.
+// As a result it removes not only composite chunker files with their
+// active chunks but also all hidden temporary chunks in the directory.
 //
 func (f *Fs) Purge(ctx context.Context) error {
 	do := f.base.Features().Purge
@@ -1044,14 +1278,14 @@ func (f *Fs) Purge(ctx context.Context) error {
 
 // Remove an object (chunks and metadata, if any)
 //
-// Remove deletes only active chunks of the object.
+// Remove deletes only active chunks of the composite object.
 // It does not try to look for temporary chunks because they could belong
 // to another command modifying this composite file in parallel.
 //
 // Commands normally cleanup all temporary chunks in case of a failure.
 // However, if rclone dies unexpectedly, it can leave hidden temporary
 // chunks, which cannot be discovered using the `list` command.
-// Remove does not try to search for such chunks or delete them.
+// Remove does not try to search for such chunks or to delete them.
 // Sometimes this can lead to strange results eg. when `list` shows that
 // directory is empty but `rmdir` refuses to remove it because on the
 // level of wrapped remote it's actually *not* empty.
@@ -1061,22 +1295,56 @@ func (f *Fs) Purge(ctx context.Context) error {
 // Remove to search directory for hidden chunks and remove them too
 // (at the risk of breaking parallel commands).
 //
+// Remove is the only operation allowed on the composite files with
+// invalid or future metadata format.
+// We don't let user copy/move/update unsupported composite files.
+// Let's at least let her get rid of them, just complain loudly.
+//
+// This can litter directory with orphan chunks of unsupported types,
+// but as long as we remove meta object, even future releases will
+// treat the composite file as removed and refuse to act upon it.
+//
+// Disclaimer: corruption can still happen if unsupported file is removed
+// and then recreated with the same name.
+// Unsupported control chunks will get re-picked by a more recent
+// rclone version with unexpected results. This can be helped by
+// the `delete hidden` flag above or at least the user has been warned.
+//
 func (o *Object) Remove(ctx context.Context) (err error) {
+	if err := o.f.forbidChunk(o, o.Remote()); err != nil {
+		// operations.Move can still call Remove if chunker's Move refuses
+		// to corrupt file in hard mode. Hence, refuse to Remove, too.
+		return errors.Wrap(err, "refuse to corrupt")
+	}
+	if err := o.readMetadata(ctx); err != nil {
+		// Proceed but warn user that unexpected things can happen.
+		fs.Errorf(o, "Removing a file with unsupported metadata: %v", err)
+	}
+
+	// Remove non-chunked file or meta object of a composite file.
 	if o.main != nil {
 		err = o.main.Remove(ctx)
 	}
+
+	// Remove only active data chunks, ignore any temporary chunks that
+	// might probably be created in parallel by other transactions.
 	for _, chunk := range o.chunks {
 		chunkErr := chunk.Remove(ctx)
 		if err == nil {
 			err = chunkErr
 		}
 	}
+
+	// There are no known control chunks to remove atm.
 	return err
 }
 
 // copyOrMove implements copy or move
 func (f *Fs) copyOrMove(ctx context.Context, o *Object, remote string, do copyMoveFn, md5, sha1, opName string) (fs.Object, error) {
-	if !o.isChunked() {
+	if err := f.forbidChunk(o, remote); err != nil {
+		return nil, errors.Wrapf(err, "can't %s", opName)
+	}
+	if !o.isComposite() {
 		fs.Debugf(o, "%s non-chunked object...", opName)
 		oResult, err := do(ctx, o.mainChunk(), remote) // chain operation to a single wrapped chunk
 		if err != nil {
@@ -1084,17 +1352,23 @@ func (f *Fs) copyOrMove(ctx context.Context, o *Object, remote string, do copyMo
 		}
 		return f.newObject("", oResult, nil), nil
 	}
+	if err := o.readMetadata(ctx); err != nil {
+		// Refuse to copy/move composite files with invalid or future
+		// metadata format which might involve unsupported chunk types.
+		return nil, errors.Wrapf(err, "can't %s this file", opName)
+	}
 
-	fs.Debugf(o, "%s %d chunks...", opName, len(o.chunks))
+	fs.Debugf(o, "%s %d data chunks...", opName, len(o.chunks))
 	mainRemote := o.remote
 	var newChunks []fs.Object
 	var err error
 
-	// Copy or move chunks
+	// Copy/move active data chunks.
+	// Ignore possible temporary chunks being created by parallel operations.
 	for _, chunk := range o.chunks {
 		chunkRemote := chunk.Remote()
 		if !strings.HasPrefix(chunkRemote, mainRemote) {
-			err = fmt.Errorf("invalid chunk %q", chunkRemote)
+			err = fmt.Errorf("invalid chunk name %q", chunkRemote)
 			break
 		}
 		chunkSuffix := chunkRemote[len(mainRemote):]
@@ -1105,14 +1379,15 @@ func (f *Fs) copyOrMove(ctx context.Context, o *Object, remote string, do copyMo
 		newChunks = append(newChunks, chunkResult)
 	}
 
-	// Copy or move old metadata
+	// Copy or move old metadata.
+	// There are no known control chunks to move/copy atm.
 	var metaObject fs.Object
 	if err == nil && o.main != nil {
 		metaObject, err = do(ctx, o.main, remote)
 	}
 	if err != nil {
 		for _, chunk := range newChunks {
-			_ = chunk.Remove(ctx) // ignore error
+			silentlyRemove(ctx, chunk)
 		}
 		return nil, err
 	}
@@ -1121,18 +1396,18 @@ func (f *Fs) copyOrMove(ctx context.Context, o *Object, remote string, do copyMo
 	newObj := f.newObject(remote, metaObject, newChunks)
 	err = newObj.validate()
 	if err != nil {
-		_ = newObj.Remove(ctx) // ignore error
+		silentlyRemove(ctx, newObj)
 		return nil, err
 	}
 
 	// Update metadata
-	var metaData []byte
+	var metadata []byte
 	switch f.opt.MetaFormat {
 	case "simplejson":
-		metaData, err = marshalSimpleJSON(ctx, newObj.size, len(newChunks), md5, sha1)
+		metadata, err = marshalSimpleJSON(ctx, newObj.size, len(newChunks), md5, sha1)
 		if err == nil {
-			metaInfo := f.wrapInfo(metaObject, "", int64(len(metaData)))
-			err = newObj.main.Update(ctx, bytes.NewReader(metaData), metaInfo)
+			metaInfo := f.wrapInfo(metaObject, "", int64(len(metadata)))
+			err = newObj.main.Update(ctx, bytes.NewReader(metadata), metaInfo)
 		}
 	case "none":
 		if newObj.main != nil {
@@ -1140,9 +1415,9 @@ func (f *Fs) copyOrMove(ctx context.Context, o *Object, remote string, do copyMo
 		}
 	}
 
-	// Return wrapping object
+	// Return the composite object
 	if err != nil {
-		_ = newObj.Remove(ctx) // ignore error
+		silentlyRemove(ctx, newObj)
 		return nil, err
 	}
 	return newObj, nil
@@ -1172,8 +1447,9 @@ func (f *Fs) okForServerSide(ctx context.Context, src fs.Object, opName string)
 		return
 	}
 
-	if f.opt.MetaFormat != "simplejson" || !obj.isChunked() {
-		ok = true // hash is not required for meta data
+	requireMetaHash := obj.isComposite() && f.opt.MetaFormat == "simplejson"
+	if !requireMetaHash && !f.hashAll {
+		ok = true // hash is not required for metadata
 		return
 	}
 
@@ -1181,14 +1457,14 @@ func (f *Fs) okForServerSide(ctx context.Context, src fs.Object, opName string)
 	case f.useMD5:
 		md5, _ = obj.Hash(ctx, hash.MD5)
 		ok = md5 != ""
-		if !ok && f.quickHash {
+		if !ok && f.hashFallback {
 			sha1, _ = obj.Hash(ctx, hash.SHA1)
 			ok = sha1 != ""
 		}
 	case f.useSHA1:
 		sha1, _ = obj.Hash(ctx, hash.SHA1)
 		ok = sha1 != ""
-		if !ok && f.quickHash {
+		if !ok && f.hashFallback {
 			md5, _ = obj.Hash(ctx, hash.MD5)
 			ok = md5 != ""
 		}
@@ -1233,7 +1509,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
 // If it isn't possible then return fs.ErrorCantMove
 func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
 	baseMove := func(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
-		return f.baseMove(ctx, src, remote, false)
+		return f.baseMove(ctx, src, remote, delNever)
 	}
 	obj, md5, sha1, ok := f.okForServerSide(ctx, src, "move")
 	if !ok {
@@ -1243,14 +1519,25 @@ func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object,
 }
 
 // baseMove chains to the wrapped Move or simulates it by Copy+Delete
-func (f *Fs) baseMove(ctx context.Context, src fs.Object, remote string, deleteDest bool) (fs.Object, error) {
-	var dest fs.Object
-	if deleteDest {
-		var err error
+func (f *Fs) baseMove(ctx context.Context, src fs.Object, remote string, delMode int) (fs.Object, error) {
+	var (
+		dest fs.Object
+		err  error
+	)
+	switch delMode {
+	case delAlways:
 		dest, err = f.base.NewObject(ctx, remote)
-		if err != nil {
-			dest = nil
+	case delFailed:
+		dest, err = operations.Move(ctx, f.base, nil, remote, src)
+		if err == nil {
+			return dest, err
 		}
+		dest, err = f.base.NewObject(ctx, remote)
+	case delNever:
+		// fall thru, the default
+	}
+	if err != nil {
+		dest = nil
 	}
 	return operations.Move(ctx, f.base, dest, remote, src)
 }
@@ -1315,6 +1602,9 @@ func (f *Fs) SetWrapper(wrapper fs.Fs) {
 // ChangeNotify calls the passed function with a path
 // that has had changes. If the implementation
 // uses polling, it should adhere to the given interval.
+//
+// Replace data chunk names by the name of composite file.
+// Ignore temporary and control chunks.
 func (f *Fs) ChangeNotify(ctx context.Context, notifyFunc func(string, fs.EntryType), pollIntervalChan <-chan time.Duration) {
 	do := f.base.Features().ChangeNotify
 	if do == nil {
@@ -1323,7 +1613,8 @@ func (f *Fs) ChangeNotify(ctx context.Context, notifyFunc func(string, fs.EntryT
 	wrappedNotifyFunc := func(path string, entryType fs.EntryType) {
 		//fs.Debugf(f, "ChangeNotify: path %q entryType %d", path, entryType)
 		if entryType == fs.EntryObject {
-			if mainPath, _, tempNo := f.parseChunkName(path); mainPath != "" && tempNo == -1 {
+			mainPath, _, _, xactNo := f.parseChunkName(path)
+			if mainPath != "" && xactNo == -1 {
 				path = mainPath
 			}
 		}
@@ -1332,13 +1623,13 @@ func (f *Fs) ChangeNotify(ctx context.Context, notifyFunc func(string, fs.EntryT
 	do(ctx, wrappedNotifyFunc, pollIntervalChan)
 }
 
-// Object wraps one or more remote data chunks
+// Object represents a composite file wrapping one or more data chunks
 type Object struct {
 	remote string
-	main   fs.Object // optional meta object if file is chunked, or single data chunk
-	chunks []fs.Object
-	size   int64 // cached total size of chunks in a chunked file
-	isFull bool
+	main   fs.Object   // meta object if file is composite, or wrapped non-chunked file, nil if meta format is 'none'
+	chunks []fs.Object // active data chunks if file is composite, or wrapped file as a single chunk if meta format is 'none'
+	size   int64       // cached total size of chunks in a composite file or -1 for non-chunked files
+	isFull bool        // true if metadata has been read
 	md5    string
 	sha1   string
 	f      *Fs
@@ -1352,6 +1643,9 @@ func (o *Object) addChunk(chunk fs.Object, chunkNo int) error {
 		o.chunks = append(o.chunks, chunk)
 		return nil
 	}
+	if chunkNo > maxSafeChunkNumber {
+		return ErrChunkOverflow
+	}
 	if chunkNo > len(o.chunks) {
 		newChunks := make([]fs.Object, (chunkNo + 1), (chunkNo+1)*2)
 		copy(newChunks, o.chunks)
@@ -1363,13 +1657,13 @@ func (o *Object) addChunk(chunk fs.Object, chunkNo int) error {
 
 // validate verifies the object internals and updates total size
 func (o *Object) validate() error {
-	if !o.isChunked() {
+	if !o.isComposite() {
 		_ = o.mainChunk() // verify that single wrapped chunk exists
 		return nil
 	}
 
-	metaObject := o.main // this file is chunked - o.main refers to optional meta object
-	if metaObject != nil && metaObject.Size() > maxMetaDataSize {
+	metaObject := o.main // this file is composite - o.main refers to meta object (or nil if meta format is 'none')
+	if metaObject != nil && metaObject.Size() > maxMetadataSize {
 		// metadata of a chunked file must be a tiny piece of json
 		o.size = -1
 		return fmt.Errorf("%q metadata is too large", o.remote)
@@ -1383,7 +1677,7 @@ func (o *Object) validate() error {
 		}
 		totalSize += chunk.Size()
 	}
-	o.size = totalSize // cache up total size
+	o.size = totalSize // cache up the total data size
 	return nil
 }
 
@@ -1405,20 +1699,21 @@ func (f *Fs) newObject(remote string, main fs.Object, chunks []fs.Object) *Objec
 }
 
 // mainChunk returns:
-// - a single wrapped chunk for non-chunked files
+// - a wrapped object for non-chunked files
 // - meta object for chunked files with metadata
 // - first chunk for chunked files without metadata
+// Never returns nil.
 func (o *Object) mainChunk() fs.Object {
 	if o.main != nil {
-		return o.main // meta object or single wrapped chunk
+		return o.main // meta object or non-chunked wrapped file
 	}
 	if o.chunks != nil {
-		return o.chunks[0] // first chunk for chunked files
+		return o.chunks[0] // first chunk of a chunked composite file
 	}
-	panic("invalid chunked object") // unlikely
+	panic("invalid chunked object") // very unlikely
 }
 
-func (o *Object) isChunked() bool {
+func (o *Object) isComposite() bool {
 	return o.chunks != nil
 }
 
@@ -1442,10 +1737,10 @@ func (o *Object) Remote() string {
 
 // Size returns the size of the file
 func (o *Object) Size() int64 {
-	if o.isChunked() {
-		return o.size // total size of chunks in a chunked file
+	if o.isComposite() {
+		return o.size // total size of data chunks in a composite file
 	}
-	return o.mainChunk().Size() // size of a single wrapped chunk
+	return o.mainChunk().Size() // size of wrapped non-chunked file
 }
 
 // Storable returns whether object is storable
@@ -1460,8 +1755,8 @@ func (o *Object) ModTime(ctx context.Context) time.Time {
 
 // SetModTime sets the modification time of the file
 func (o *Object) SetModTime(ctx context.Context, mtime time.Time) error {
-	if err := o.readMetaData(ctx); err != nil {
-		return err
+	if err := o.readMetadata(ctx); err != nil {
+		return err // refuse to act on unsupported format
 	}
 	return o.mainChunk().SetModTime(ctx, mtime)
 }
@@ -1469,32 +1764,32 @@ func (o *Object) SetModTime(ctx context.Context, mtime time.Time) error {
 // Hash returns the selected checksum of the file.
 // If no checksum is available it returns "".
 //
-// Hash prefers wrapped hashsum for a non-chunked file, then tries to
-// read it from metadata. This in theory handles an unusual case when
-// a small file is modified on the lower level by wrapped remote
-// but chunker is not yet aware of changes.
+// Hash won't fail with `unsupported` error but return empty
+// hash string if a particular hashsum type is not supported
 //
-// Currently metadata (if not configured as 'none') is kept only for
-// multi-chunk files, but for small files chunker obtains hashsums from
-// wrapped remote. If a particular hashsum type is not supported,
-// chunker won't fail with `unsupported` error but return empty hash.
+// Hash takes hashsum from metadata if available or requests it
+// from wrapped remote for non-chunked files.
+// Metadata (if meta format is not 'none') is by default kept
+// only for composite files. In the "All" hashing mode chunker
+// will force metadata on all files if particular hashsum type
+// is not supported by wrapped remote.
 //
-// In future metadata logic can be extended: if a normal (non-quick)
-// hash type is configured, chunker will check whether wrapped remote
-// supports it (see Fs.Hashes as an example). If not, it will add metadata
-// to small files as well, thus providing hashsums for all files.
+// Note that Hash prefers the wrapped hashsum for non-chunked
+// file, then tries to read it from metadata. This in theory
+// handles the unusual case when a small file has been tampered
+// on the level of wrapped remote but chunker is unaware of that.
 //
 func (o *Object) Hash(ctx context.Context, hashType hash.Type) (string, error) {
-	if !o.isChunked() {
-		// First, chain to the single wrapped chunk, if possible.
+	if !o.isComposite() {
+		// First, chain to the wrapped non-chunked file if possible.
 		if value, err := o.mainChunk().Hash(ctx, hashType); err == nil && value != "" {
 			return value, nil
 		}
 	}
-	if err := o.readMetaData(ctx); err != nil {
-		return "", err
+	if err := o.readMetadata(ctx); err != nil {
+		return "", err // valid metadata is required to get hash, abort
 	}
-	// Try saved hash if the file is chunked or the wrapped remote fails.
+	// Try hash from metadata if the file is composite or if wrapped remote fails.
 	switch hashType {
 	case hash.MD5:
 		if o.md5 == "" {
@@ -1518,11 +1813,12 @@ func (o *Object) UnWrap() fs.Object {
 
 // Open opens the file for read.  Call Close() on the returned io.ReadCloser
 func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (rc io.ReadCloser, err error) {
-	if !o.isChunked() {
-		return o.mainChunk().Open(ctx, options...) // chain to a single wrapped chunk
+	if !o.isComposite() {
+		return o.mainChunk().Open(ctx, options...) // chain to wrapped non-chunked file
 	}
-	if err := o.readMetaData(ctx); err != nil {
-		return nil, err
+	if err := o.readMetadata(ctx); err != nil {
+		// refuse to open unsupported format
+		return nil, errors.Wrap(err, "can't open")
 	}
 
 	var openOptions []fs.OpenOption
@@ -1535,7 +1831,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (rc io.Read
 		case *fs.RangeOption:
 			offset, limit = opt.Decode(o.size)
 		default:
-			// pass on Options to wrapped open if appropriate
+			// pass Options on to the wrapped open, if appropriate
 			openOptions = append(openOptions, option)
 		}
 	}
@@ -1659,8 +1955,8 @@ func (r *linearReader) Close() (err error) {
 type ObjectInfo struct {
 	src     fs.ObjectInfo
 	fs      *Fs
-	nChunks int
-	size    int64  // overrides source size by the total size of chunks
+	nChunks int    // number of data chunks
+	size    int64  // overrides source size by the total size of data chunks
 	remote  string // overrides remote name
 	md5     string // overrides MD5 checksum
 	sha1    string // overrides SHA1 checksum
@@ -1714,8 +2010,9 @@ func (oi *ObjectInfo) ModTime(ctx context.Context) time.Time {
 	return oi.src.ModTime(ctx)
 }
 
-// Hash returns the selected checksum of the file
-// If no checksum is available it returns ""
+// Hash returns the selected checksum of the wrapped file
+// It returns "" if no checksum is available or if this
+// info doesn't wrap the complete file.
 func (oi *ObjectInfo) Hash(ctx context.Context, hashType hash.Type) (string, error) {
 	var errUnsupported error
 	switch hashType {
@@ -1731,7 +2028,7 @@ func (oi *ObjectInfo) Hash(ctx context.Context, hashType hash.Type) (string, err
 		errUnsupported = hash.ErrUnsupported
 	}
 	if oi.Size() != oi.src.Size() {
-		// fail if this info wraps a file part
+		// fail if this info wraps only a part of the file
 		return "", errUnsupported
 	}
 	// chain to full source if possible
@@ -1752,76 +2049,134 @@ func (o *Object) ID() string {
 
 // Meta format `simplejson`
 type metaSimpleJSON struct {
-	Version int    `json:"ver"`
-	Size    int64  `json:"size"`
-	NChunks int    `json:"nchunks"`
-	MD5     string `json:"md5"`
-	SHA1    string `json:"sha1"`
-}
-
-func marshalSimpleJSON(ctx context.Context, size int64, nChunks int, md5, sha1 string) (data []byte, err error) {
-	metaData := &metaSimpleJSON{
-		Version: metaDataVersion,
-		Size:    size,
-		NChunks: nChunks,
-		MD5:     md5,
-		SHA1:    sha1,
-	}
-	return json.Marshal(&metaData)
+	// required core fields
+	Version  *int   `json:"ver"`
+	Size     *int64 `json:"size"`    // total size of data chunks
+	ChunkNum *int   `json:"nchunks"` // number of data chunks
+	// optional extra fields
+	MD5  string `json:"md5,omitempty"`
+	SHA1 string `json:"sha1,omitempty"`
 }
 
-// Note: only metadata format version 1 is supported a.t.m.
+// marshalSimpleJSON
 //
-// Current implementation creates metadata only for files larger than
-// configured chunk size. This approach has drawback: availability of
-// configured hashsum type for small files depends on the wrapped remote.
-// Future versions of chunker may change approach as described in comment
-// to the Hash method. They can transparently migrate older metadata.
+// Current implementation creates metadata in three cases:
+// - for files larger than chunk size
+// - if file contents can be mistaken as meta object
+// - if consistent hashing is on but wrapped remote can't provide given hash
+//
+func marshalSimpleJSON(ctx context.Context, size int64, nChunks int, md5, sha1 string) ([]byte, error) {
+	version := metadataVersion
+	metadata := metaSimpleJSON{
+		// required core fields
+		Version:  &version,
+		Size:     &size,
+		ChunkNum: &nChunks,
+		// optional extra fields
+		MD5:  md5,
+		SHA1: sha1,
+	}
+	data, err := json.Marshal(&metadata)
+	if err == nil && data != nil && len(data) >= maxMetadataSize {
+		// be a nitpicker, never produce something you can't consume
+		return nil, errors.New("metadata can't be this big, please report to rclone developers")
+	}
+	return data, err
+}
+
+// unmarshalSimpleJSON
+//
+// Only metadata format version 1 is supported atm.
+// Future releases will transparently migrate older metadata objects.
 // New format will have a higher version number and cannot be correctly
-// hanled by current implementation.
+// handled by current implementation.
 // The version check below will then explicitly ask user to upgrade rclone.
 //
-func unmarshalSimpleJSON(ctx context.Context, metaObject fs.Object, data []byte) (info *ObjectInfo, err error) {
-	var metaData *metaSimpleJSON
-	err = json.Unmarshal(data, &metaData)
+func unmarshalSimpleJSON(ctx context.Context, metaObject fs.Object, data []byte, strictChecks bool) (info *ObjectInfo, err error) {
+	// Be strict about JSON format
+	// to reduce possibility that a random small file resembles metadata.
+	if data != nil && len(data) > maxMetadataSize {
+		return nil, errors.New("too big")
+	}
+	if data == nil || len(data) < 2 || data[0] != '{' || data[len(data)-1] != '}' {
+		return nil, errors.New("invalid json")
+	}
+	var metadata metaSimpleJSON
+	err = json.Unmarshal(data, &metadata)
 	if err != nil {
 		return nil, err
 	}
-
+	// Basic fields are strictly required
+	// to reduce possibility that a random small file resembles metadata.
+	if metadata.Version == nil || metadata.Size == nil || metadata.ChunkNum == nil {
+		return nil, errors.New("missing required field")
+	}
 	// Perform strict checks, avoid corruption of future metadata formats.
-	if metaData.Size < 0 {
+	if *metadata.Version < 1 {
+		return nil, errors.New("wrong version")
+	}
+	if *metadata.Size < 0 {
 		return nil, errors.New("negative file size")
 	}
-	if metaData.NChunks <= 0 {
-		return nil, errors.New("wrong number of chunks")
+	if *metadata.ChunkNum < 0 {
+		return nil, errors.New("negative number of chunks")
 	}
-	if metaData.MD5 != "" {
-		_, err = hex.DecodeString(metaData.MD5)
-		if len(metaData.MD5) != 32 || err != nil {
+	if *metadata.ChunkNum > maxSafeChunkNumber {
+		return nil, ErrChunkOverflow
+	}
+	if metadata.MD5 != "" {
+		_, err = hex.DecodeString(metadata.MD5)
+		if len(metadata.MD5) != 32 || err != nil {
 			return nil, errors.New("wrong md5 hash")
 		}
 	}
-	if metaData.SHA1 != "" {
-		_, err = hex.DecodeString(metaData.SHA1)
-		if len(metaData.SHA1) != 40 || err != nil {
+	if metadata.SHA1 != "" {
+		_, err = hex.DecodeString(metadata.SHA1)
+		if len(metadata.SHA1) != 40 || err != nil {
 			return nil, errors.New("wrong sha1 hash")
 		}
 	}
-	if metaData.Version <= 0 {
-		return nil, errors.New("wrong version number")
+	// ChunkNum is allowed to be 0 in future versions
+	if *metadata.ChunkNum < 1 && *metadata.Version <= metadataVersion {
+		return nil, errors.New("wrong number of chunks")
 	}
-	if metaData.Version != metaDataVersion {
-		return nil, errors.Errorf("version %d is not supported, please upgrade rclone", metaData.Version)
+	// Non-strict mode also accepts future metadata versions
+	if *metadata.Version > metadataVersion && strictChecks {
+		return nil, fmt.Errorf("version %d is not supported, please upgrade rclone", metadata.Version)
 	}
 
 	var nilFs *Fs // nil object triggers appropriate type method
-	info = nilFs.wrapInfo(metaObject, "", metaData.Size)
-	info.md5 = metaData.MD5
-	info.sha1 = metaData.SHA1
-	info.nChunks = metaData.NChunks
+	info = nilFs.wrapInfo(metaObject, "", *metadata.Size)
+	info.nChunks = *metadata.ChunkNum
+	info.md5 = metadata.MD5
+	info.sha1 = metadata.SHA1
 	return info, nil
 }
 
+func silentlyRemove(ctx context.Context, o fs.Object) {
+	_ = o.Remove(ctx) // ignore error
+}
+
+// Name of the remote (as passed into NewFs)
+func (f *Fs) Name() string {
+	return f.name
+}
+
+// Root of the remote (as passed into NewFs)
+func (f *Fs) Root() string {
+	return f.root
+}
+
+// Features returns the optional features of this Fs
+func (f *Fs) Features() *fs.Features {
+	return f.features
+}
+
+// String returns a description of the FS
+func (f *Fs) String() string {
+	return fmt.Sprintf("Chunked '%s:%s'", f.name, f.root)
+}
+
 // Check the interfaces are satisfied
 var (
 	_ fs.Fs              = (*Fs)(nil)
diff --git a/backend/dropbox/dropbox.go b/backend/dropbox/dropbox.go
index 1ddefce12..8d683c5aa 100644
--- a/backend/dropbox/dropbox.go
+++ b/backend/dropbox/dropbox.go
@@ -388,8 +388,7 @@ func (f *Fs) getMetadata(objPath string) (entry files.IsMetadata, notFound bool,
 	if err != nil {
 		switch e := err.(type) {
 		case files.GetMetadataAPIError:
-			switch e.EndpointError.Path.Tag {
-			case files.LookupErrorNotFound:
+			if e.EndpointError != nil && e.EndpointError.Path != nil && e.EndpointError.Path.Tag == files.LookupErrorNotFound {
 				notFound = true
 				err = nil
 			}
@@ -489,8 +488,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
 			if err != nil {
 				switch e := err.(type) {
 				case files.ListFolderAPIError:
-					switch e.EndpointError.Path.Tag {
-					case files.LookupErrorNotFound:
+					if e.EndpointError != nil && e.EndpointError.Path != nil && e.EndpointError.Path.Tag == files.LookupErrorNotFound {
 						err = fs.ErrorDirNotFound
 					}
 				}
@@ -1012,7 +1010,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
 	switch e := err.(type) {
 	case files.DownloadAPIError:
 		// Don't attempt to retry copyright violation errors
-		if e.EndpointError.Path.Tag == files.LookupErrorRestrictedContent {
+		if e.EndpointError != nil && e.EndpointError.Path != nil && e.EndpointError.Path.Tag == files.LookupErrorRestrictedContent {
 			return nil, fserrors.NoRetryError(err)
 		}
 	}
diff --git a/backend/onedrive/onedrive.go b/backend/onedrive/onedrive.go
index 7ff407de4..f9b8e6d4c 100644
--- a/backend/onedrive/onedrive.go
+++ b/backend/onedrive/onedrive.go
@@ -225,9 +225,9 @@ func init() {
 			Help: "Microsoft App Client Secret\nLeave blank normally.",
 		}, {
 			Name: "chunk_size",
-			Help: `Chunk size to upload files with - must be multiple of 320k.
+			Help: `Chunk size to upload files with - must be multiple of 320k (327,680 bytes).
 
-Above this size files will be chunked - must be multiple of 320k. Note
+Above this size files will be chunked - must be multiple of 320k (327,680 bytes). Note
 that the chunks will be buffered into memory.`,
 			Default:  defaultChunkSize,
 			Advanced: true,
diff --git a/docs/content/authors.md b/docs/content/authors.md
index 0a380d43a..f30f041ed 100644
--- a/docs/content/authors.md
+++ b/docs/content/authors.md
@@ -299,3 +299,5 @@ Contributors
   * Tyler <TylerNakamura@users.noreply.github.com>
   * Brett Dutro <brett.dutro@gmail.com>
   * Vighnesh SK <booterror99@gmail.com>
+  * Arijit Biswas <dibbyo456@gmail.com>
+  * Michele Caci <michele.caci@gmail.com>
diff --git a/docs/content/chunker.md b/docs/content/chunker.md
index f35fc8cf9..ae0eb73d5 100644
--- a/docs/content/chunker.md
+++ b/docs/content/chunker.md
@@ -22,7 +22,7 @@ won't. This means that if you are using a bucket based remote (eg S3, B2, swift)
 then you should probably put the bucket in the remote `s3:bucket`.
 
 Now configure `chunker` using `rclone config`. We will call this one `overlay`
-to separate it from the `remote`.
+to separate it from the `remote` itself.
 
 ```
 No remotes found - make a new one
@@ -46,20 +46,22 @@ remote> remote:path
 Files larger than chunk size will be split in chunks.
 Enter a size with suffix k,M,G,T. Press Enter for the default ("2G").
 chunk_size> 100M
-Choose how chunker handles hash sums.
+Choose how chunker handles hash sums. All modes but "none" require metadata.
 Enter a string value. Press Enter for the default ("md5").
 Choose a number from below, or type in your own value
-   / Chunker can pass any hash supported by wrapped remote
- 1 | for a single-chunk file but returns nothing otherwise.
+ 1 / Pass any hash supported by wrapped remote for non-chunked files, return nothing otherwise
    \ "none"
- 2 / MD5 for multi-chunk files. Requires "simplejson".
+ 2 / MD5 for composite files
    \ "md5"
- 3 / SHA1 for multi-chunk files. Requires "simplejson".
+ 3 / SHA1 for composite files
    \ "sha1"
-   / Copying a file to chunker will request MD5 from the source
- 4 | falling back to SHA1 if unsupported. Requires "simplejson".
+ 4 / MD5 for all files
+   \ "md5all"
+ 5 / SHA1 for all files
+   \ "sha1all"
+ 6 / Copying a file to chunker will request MD5 from the source falling back to SHA1 if unsupported
    \ "md5quick"
- 5 / Similar to "md5quick" but prefers SHA1 over MD5. Requires "simplejson".
+ 7 / Similar to "md5quick" but prefers SHA1 over MD5
    \ "sha1quick"
 hash_type> md5
 Edit advanced config? (y/n)
@@ -95,28 +97,49 @@ When rclone starts a file upload, chunker checks the file size. If it
 doesn't exceed the configured chunk size, chunker will just pass the file
 to the wrapped remote. If a file is large, chunker will transparently cut
 data in pieces with temporary names and stream them one by one, on the fly.
-Each chunk will contain the specified number of data byts, except for the
+Each data chunk will contain the specified number of bytes, except for the
 last one which may have less data. If file size is unknown in advance
 (this is called a streaming upload), chunker will internally create
 a temporary copy, record its size and repeat the above process.
+
 When upload completes, temporary chunk files are finally renamed.
-This scheme guarantees that operations look from outside as atomic.
+This scheme guarantees that operations can be run in parallel and look
+from outside as atomic.
 A similar method with hidden temporary chunks is used for other operations
 (copy/move/rename etc). If an operation fails, hidden chunks are normally
-destroyed, and the destination composite file stays intact.
+destroyed, and the target composite file stays intact.
+
+When a composite file download is requested, chunker transparently
+assembles it by concatenating data chunks in order. As the split is trivial
+one could even manually concatenate data chunks together to obtain the
+original content.
+
+When the `list` rclone command scans a directory on wrapped remote,
+the potential chunk files are accounted for, grouped and assembled into
+composite directory entries. Any temporary chunks are hidden.
+
+List and other commands can sometimes come across composite files with
+missing or invalid chunks, eg. shadowed by like-named directory or
+another file. This usually means that wrapped file system has been directly
+tampered with or damaged. If chunker detects a missing chunk it will
+by default print warning, skip the whole incomplete group of chunks but
+proceed with current command.
+You can set the `--chunker-fail-hard` flag to have commands abort with
+error message in such cases.
+
 
 #### Chunk names
 
-By default chunk names are `BIG_FILE_NAME.rclone-chunk.001`,
-`BIG_FILE_NAME.rclone-chunk.002` etc, because the default chunk name
-format is `*.rclone-chunk.###`. You can configure another name format
-using the `--chunker-name-format` option. The format uses asterisk
+The default chunk name format is `*.rclone-chunk.###`, hence by default
+chunk names are `BIG_FILE_NAME.rclone-chunk.001`,
+`BIG_FILE_NAME.rclone-chunk.002` etc. You can configure a different name
+format using the `--chunker-name-format` option. The format uses asterisk
 `*` as a placeholder for the base file name and one or more consecutive
 hash characters `#` as a placeholder for sequential chunk number.
 There must be one and only one asterisk. The number of consecutive hash
 characters defines the minimum length of a string representing a chunk number.
 If decimal chunk number has less digits than the number of hashes, it is
-left-padded by zeros. If the number stringis longer, it is left intact.
+left-padded by zeros. If the decimal string is longer, it is left intact.
 By default numbering starts from 1 but there is another option that allows
 user to start from 0, eg. for compatibility with legacy software.
 
@@ -125,24 +148,18 @@ For example, if name format is `big_*-##.part` and original file name is
 `big_data.txt-00.part`, the 99th chunk will be `big_data.txt-98.part`
 and the 302nd chunk will become `big_data.txt-301.part`.
 
-When the `list` rclone command scans a directory on wrapped remote, the
-potential chunk files are accounted for and merged into composite directory
-entries only if their names match the configured format. All other files
-are ignored, including temporary chunks.
-The list command might encounter composite files with missing or invalid
-chunks. If chunker detects a missing chunk it will by default silently
-ignore the whole group. You can use the `--chunker-fail-on-bad-chunks`
-command line flag to make `list` fail with an error message.
+Note that `list` assembles composite directory entries only when chunk names
+match the configured format and treats non-conforming file names as normal
+non-chunked files.
 
 
 ### Metadata
 
-By default when a file is large enough, chunker will create a metadata
-object besides data chunks. The object is named after the original file.
+Besides data chunks chunker will by default create metadata object for
+a composite file. The object is named after the original file.
 Chunker allows user to disable metadata completely (the `none` format).
-Please note that currently metadata is not created for files smaller
-than configured chunk size. This may change in future as new formats
-are developed.
+Note that metadata is normally not created for files smaller than the
+configured chunk size. This may change in future rclone releases.
 
 #### Simple JSON metadata format
 
@@ -151,13 +168,13 @@ for composite files. Meta objects carry the following fields:
 
 - `ver`     - version of format, currently `1`
 - `size`    - total size of composite file
-- `nchunks` - number of chunks in the file
+- `nchunks` - number of data chunks in file
 - `md5`     - MD5 hashsum of composite file (if present)
 - `sha1`    - SHA1 hashsum (if present)
 
 There is no field for composite file name as it's simply equal to the name
 of meta object on the wrapped remote. Please refer to respective sections
-for detils on hashsums and handling of modified time.
+for details on hashsums and modified time handling.
 
 #### No metadata
 
@@ -165,36 +182,42 @@ You can disable meta objects by setting the meta format option to `none`.
 In this mode chunker will scan directory for all files that follow
 configured chunk name format, group them by detecting chunks with the same
 base name and show group names as virtual composite files.
-When a download is requested, chunker will transparently assemble compound
-files by merging chunks in order. This method is more prone to missing chunk
-errors (especially missing last chunk) than metadata-enabled formats.
+This method is more prone to missing chunk errors (especially missing
+last chunk) than format with metadata enabled.
 
 
 ### Hashsums
 
 Chunker supports hashsums only when a compatible metadata is present.
-Thus, if you choose metadata format of `none`, chunker will return
-`UNSUPPORTED` as hashsum.
+Hence, if you choose metadata format of `none`, chunker will report hashsum
+as `UNSUPPORTED`.
 
-Please note that metadata is stored only for composite files. If a file
-is small (smaller than configured chunk size), chunker will transparently
+Please note that by default metadata is stored only for composite files.
+If a file is smaller than configured chunk size, chunker will transparently
 redirect hash requests to wrapped remote, so support depends on that.
 You will see the empty string as a hashsum of requested type for small
 files if the wrapped remote doesn't support it.
 
 Many storage backends support MD5 and SHA1 hash types, so does chunker.
-Currently you can choose one or another but not both.
+With chunker you can choose one or another but not both.
 MD5 is set by default as the most supported type.
 Since chunker keeps hashes for composite files and falls back to the
-wrapped remote hash for small ones, we advise you to choose the same
-hash type as wrapped remote so that your file listings look coherent.
+wrapped remote hash for non-chunked ones, we advise you to choose the same
+hash type as supported by wrapped remote so that your file listings
+look coherent.
+
+If your storage backend does not support MD5 or SHA1 but you need consistent
+file hashing, configure chunker with `md5all` or `sha1all`. These two modes
+guarantee given hash for all files. If wrapped remote doesn't support it,
+chunker will then add metadata to all files, even small. However, this can
+double the amount of small files in storage and incur additional service charges.
 
-Normally, when a file is copied to a chunker controlled remote, chunker
+Normally, when a file is copied to chunker controlled remote, chunker
 will ask the file source for compatible file hash and revert to on-the-fly
 calculation if none is found. This involves some CPU overhead but provides
 a guarantee that given hashsum is available. Also, chunker will reject
 a server-side copy or move operation if source and destination hashsum
-types are different, resulting in the extra network bandwidth, too.
+types are different resulting in the extra network bandwidth, too.
 In some rare cases this may be undesired, so chunker provides two optional
 choices: `sha1quick` and `md5quick`. If the source does not support primary
 hash type and the quick mode is enabled, chunker will try to fall back to
@@ -209,10 +232,10 @@ between source and target are not found.
 Chunker stores modification times using the wrapped remote so support
 depends on that. For a small non-chunked file the chunker overlay simply
 manipulates modification time of the wrapped remote file.
-If file is large and metadata is present, then chunker will get and set
+For a composite file with metadata chunker will get and set
 modification time of the metadata object on the wrapped remote.
 If file is chunked but metadata format is `none` then chunker will
-use modification time of the first chunk.
+use modification time of the first data chunk.
 
 
 ### Migrations
@@ -222,11 +245,11 @@ chunk naming scheme is to:
 
 - Collect all your chunked files under a directory and have your
   chunker remote point to it.
-- Create another directory (possibly on the same cloud storage)
+- Create another directory (most probably on the same cloud storage)
   and configure a new remote with desired metadata format,
   hash type, chunk naming etc.
 - Now run `rclone sync oldchunks: newchunks:` and all your data
-  will be transparently converted at transfer.
+  will be transparently converted in transfer.
   This may take some time, yet chunker will try server-side
   copy if possible.
 - After checking data integrity you may remove configuration section
@@ -235,11 +258,11 @@ chunk naming scheme is to:
 If rclone gets killed during a long operation on a big composite file,
 hidden temporary chunks may stay in the directory. They will not be
 shown by the `list` command but will eat up your account quota.
-Please note that the `deletefile` rclone command deletes only active
+Please note that the `deletefile` command deletes only active
 chunks of a file. As a workaround, you can use remote of the wrapped
 file system to see them.
 An easy way to get rid of hidden garbage is to copy littered directory
-somewhere using the chunker remote and purge original directory.
+somewhere using the chunker remote and purge the original directory.
 The `copy` command will copy only active chunks while the `purge` will
 remove everything including garbage.
 
@@ -260,7 +283,7 @@ Beware that in result of this some files which have been treated as chunks
 before the change can pop up in directory listings as normal files
 and vice versa. The same warning holds for the chunk size.
 If you desperately need to change critical chunking setings, you should
-run data migration as described in a dedicated section.
+run data migration as described above.
 
 If wrapped remote is case insensitive, the chunker overlay will inherit
 that property (so you can't have a file called "Hello.doc" and "hello.doc"
@@ -294,7 +317,7 @@ Files larger than chunk size will be split in chunks.
 
 #### --chunker-hash-type
 
-Choose how chunker handles hash sums.
+Choose how chunker handles hash sums. All modes but "none" require metadata.
 
 - Config:      hash_type
 - Env Var:     RCLONE_CHUNKER_HASH_TYPE
@@ -302,21 +325,23 @@ Choose how chunker handles hash sums.
 - Default:     "md5"
 - Examples:
     - "none"
-        - Chunker can pass any hash supported by wrapped remote
-        - for a single-chunk file but returns nothing otherwise.
+        - Pass any hash supported by wrapped remote for non-chunked files, return nothing otherwise
     - "md5"
-        - MD5 for multi-chunk files. Requires "simplejson".
+        - MD5 for composite files
     - "sha1"
-        - SHA1 for multi-chunk files. Requires "simplejson".
+        - SHA1 for composite files
+    - "md5all"
+        - MD5 for all files
+    - "sha1all"
+        - SHA1 for all files
     - "md5quick"
-        - Copying a file to chunker will request MD5 from the source
-        - falling back to SHA1 if unsupported. Requires "simplejson".
+        - Copying a file to chunker will request MD5 from the source falling back to SHA1 if unsupported
     - "sha1quick"
-        - Similar to "md5quick" but prefers SHA1 over MD5. Requires "simplejson".
+        - Similar to "md5quick" but prefers SHA1 over MD5
 
 ### Advanced Options
 
-Here are the advanced options specific to chunker (Transparently chunk/split large files).
+Here are the advanced options specific to chunker.
 
 #### --chunker-name-format
 
@@ -356,21 +381,20 @@ Metadata is a small JSON file named after the composite file.
         - Do not use metadata files at all. Requires hash type "none".
     - "simplejson"
         - Simple JSON supports hash sums and chunk validation.
-        - It has the following fields: size, nchunks, md5, sha1.
+        - It has the following fields: ver, size, nchunks, md5, sha1.
 
-#### --chunker-fail-on-bad-chunks
+#### --chunker-fail-hard
 
-The list command might encounter files with missinng or invalid chunks.
-This boolean flag tells what rclone should do in such cases.
+Choose how chunker should handle files with missing or invalid chunks.
 
-- Config:      fail_on_bad_chunks
-- Env Var:     RCLONE_CHUNKER_FAIL_ON_BAD_CHUNKS
+- Config:      fail_hard
+- Env Var:     RCLONE_CHUNKER_FAIL_HARD
 - Type:        bool
 - Default:     false
 - Examples:
     - "true"
-        - Fail with error.
+        - Report errors and abort current command.
     - "false"
-        - Silently ignore invalid object.
+        - Warn user, skip incomplete file and proceed.
 
 <!--- autogenerated options stop -->
diff --git a/docs/content/docs.md b/docs/content/docs.md
index bfd6178fa..4c912e1c6 100644
--- a/docs/content/docs.md
+++ b/docs/content/docs.md
@@ -1098,9 +1098,18 @@ The default is to run 4 file transfers in parallel.
 This forces rclone to skip any files which exist on the destination
 and have a modified time that is newer than the source file.
 
+This can be useful when transferring to a remote which doesn't support
+mod times directly (or when using `--use-server-mod-time` to avoid extra
+API calls) as it is more accurate than a `--size-only` check and faster
+than using `--checksum`.
+
 If an existing destination file has a modification time equal (within
 the computed modify window precision) to the source file's, it will be
-updated if the sizes are different.
+updated if the sizes are different.  If `--checksum` is set then
+rclone will update the destination if the checksums differ too.
+
+If an existing destination file is older than the source file then
+it will be updated if the size or checksum differs from the source file.
 
 On remotes which don't support mod time directly (or when using
 `--use-server-mod-time`) the time checked will be the uploaded time.
@@ -1108,11 +1117,6 @@ This means that if uploading to one of these remotes, rclone will skip
 any files which exist on the destination and have an uploaded time that
 is newer than the modification time of the source file.
 
-This can be useful when transferring to a remote which doesn't support
-mod times directly (or when using `--use-server-mod-time` to avoid extra
-API calls) as it is more accurate than a `--size-only` check and faster
-than using `--checksum`.
-
 ### --use-mmap ###
 
 If this flag is set then rclone will use anonymous memory allocated by
diff --git a/docs/content/onedrive.md b/docs/content/onedrive.md
index 55c72cafd..26acdf5a2 100644
--- a/docs/content/onedrive.md
+++ b/docs/content/onedrive.md
@@ -124,19 +124,16 @@ rclone uses a pair of Client ID and Key shared by all rclone users when performi
 If you are having problems with them (E.g., seeing a lot of throttling), you can get your own
 Client ID and Key by following the steps below:
 
-1. Open https://apps.dev.microsoft.com/#/appList, then click `Add an app` (Choose `Converged applications` if applicable)
-2. Enter a name for your app, and click continue. Copy and keep the `Application Id` under the app name for later use.
-3. Under section `Application Secrets`, click `Generate New Password`. Copy and keep that password for later use.
-4. Under section `Platforms`, click `Add platform`, then `Web`. Enter `http://localhost:53682/` in
-`Redirect URLs`.
-5. Under section `Microsoft Graph Permissions`, `Add` these `delegated permissions`:
-`Files.Read`, `Files.ReadWrite`, `Files.Read.All`, `Files.ReadWrite.All`, `offline_access`, `User.Read`.
-6. Scroll to the bottom and click `Save`.
+1. Open https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade, then click `New registration`.
+2. Enter a name for your app, choose your account type, select `Web` in `Redirect URI` Enter `http://localhost:53682/` and click Register. Copy and keep the `Application (client) ID` under the app name for later use.
+3. Under `manage` select `Certificates & secrets`, click `New client secret`. Copy and keep that secret for later use.
+4. Under `manage` select `API permissions`, click `Add a permission` and select `Microsoft Graph` then select `delegated permissions`.
+5. Search and select the follwing permssions: `Files.Read`, `Files.ReadWrite`, `Files.Read.All`, `Files.ReadWrite.All`, `offline_access`, `User.Read`. Once selected click `Add permissions` at the bottom.
 
 Now the application is complete. Run `rclone config` to create or edit a OneDrive remote.
 Supply the app ID and password as Client ID and Secret, respectively. rclone will walk you through the remaining steps.
 
-### Modified time and hashes ###
+### Modification time and hashes ###
 
 OneDrive allows modification times to be set on objects accurate to 1
 second.  These will be used to detect whether objects need syncing or
@@ -148,7 +145,7 @@ Sharepoint Server support
 
 For all types of OneDrive you can use the `--checksum` flag.
 
-#### Restricted filename characters
+### Restricted filename characters ###
 
 In addition to the [default restricted characters set](/overview/#restricted-characters)
 the following characters are also replaced:
@@ -223,9 +220,9 @@ Here are the advanced options specific to onedrive (Microsoft OneDrive).
 
 #### --onedrive-chunk-size
 
-Chunk size to upload files with - must be multiple of 320k.
+Chunk size to upload files with - must be multiple of 320k (327,680 bytes).
 
-Above this size files will be chunked - must be multiple of 320k. Note
+Above this size files will be chunked - must be multiple of 320k (327,680 bytes). Note
 that the chunks will be buffered into memory.
 
 - Config:      chunk_size
@@ -270,6 +267,8 @@ listing, set this option.
 
 ### Limitations ###
 
+#### Naming ####
+
 Note that OneDrive is case insensitive so you can't have a
 file called "Hello.doc" and one called "hello.doc".
 
@@ -279,16 +278,22 @@ platforms they are common.  Rclone will map these names to and from an
 identical looking unicode equivalent.  For example if a file has a `?`
 in it will be mapped to `？` instead.
 
+#### File sizes ####
+
 The largest allowed file sizes are 15GB for OneDrive for Business and 35GB for OneDrive Personal (Updated 4 Jan 2019).
 
+#### Path length ####
+
 The entire path, including the file name, must contain fewer than 400 characters for OneDrive, OneDrive for Business and SharePoint Online. If you are encrypting file and folder names with rclone, you may want to pay attention to this limitation because the encrypted names are typically longer than the original ones.
 
+#### Number of files ####
+
 OneDrive seems to be OK with at least 50,000 files in a folder, but at
 100,000 rclone will get errors listing the directory like `couldn’t
 list files: UnknownError:`.  See
 [#2707](https://github.com/rclone/rclone/issues/2707) for more info.
 
-An official document about the limitations for different types of OneDrive can be found [here](https://support.office.com/en-us/article/invalid-file-names-and-file-types-in-onedrive-onedrive-for-business-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa). 
+An official document about the limitations for different types of OneDrive can be found [here](https://support.office.com/en-us/article/invalid-file-names-and-file-types-in-onedrive-onedrive-for-business-and-sharepoint-64883a5d-228e-48f5-b3d2-eb39e07630fa).
 
 ### Versioning issue ###
 
@@ -327,6 +332,36 @@ Note: This will disable the creation of new file versions, but will not remove a
 
 ### Troubleshooting ###
 
+#### Unexpected file size/hash differences on Sharepoint ####
+
+It is a
+[known](https://github.com/OneDrive/onedrive-api-docs/issues/935#issuecomment-441741631)
+issue that Sharepoint (not OneDrive or OneDrive for Business) silently modifies
+uploaded files, mainly Office files (.docx, .xlsx, etc.), causing file size and
+hash checks to fail. To use rclone with such affected files on Sharepoint, you
+may disable these checks with the following command line arguments:
+
+```
+--ignore-checksum --ignore-size
+```
+
+#### Replacing/deleting existing files on Sharepoint gets "item not found" ####
+
+It is a [known](https://github.com/OneDrive/onedrive-api-docs/issues/1068) issue
+that Sharepoint (not OneDrive or OneDrive for Business) may return "item not
+found" errors when users try to replace or delete uploaded files; this seems to
+mainly affect Office files (.docx, .xlsx, etc.). As a workaround, you may use
+the `--backup-dir <BACKUP_DIR>` command line argument so rclone moves the
+files to be replaced/deleted into a given backup directory (instead of directly
+replacing/deleting them). For example, to instruct rclone to move the files into
+the directory `rclone-backup-dir` on backend `mysharepoint`, you may use:
+
+```
+--backup-dir mysharepoint:rclone-backup-dir
+```
+
+#### access\_denied (AADSTS65005) ####
+
 ```
 Error: access_denied
 Code: AADSTS65005
@@ -337,6 +372,7 @@ This means that rclone can't use the OneDrive for Business API with your account
 
 However, there are other ways to interact with your OneDrive account. Have a look at the webdav backend: https://rclone.org/webdav/#sharepoint
 
+#### invalid\_grant (AADSTS50076) ####
 
 ```
 Error: invalid_grant
diff --git a/docs/content/webdav.md b/docs/content/webdav.md
index 70a2f1574..0ffa99d7e 100644
--- a/docs/content/webdav.md
+++ b/docs/content/webdav.md
@@ -246,7 +246,7 @@ pass = encryptedpassword
 #### Required Flags for SharePoint ####
 As SharePoint does some special things with uploaded documents, you won't be able to use the documents size or the documents hash to compare if a file has been changed since the upload / which file is newer.
 
-For all Rclone calls copying files from/to SharePoint or OneDrive (like copy, sync, etc.), you should append these flags to ensure Rclone uses the "Last Modified" datetime property to compare your documents:
+For Rclone calls copying files (especially Office files such as .docx, .xlsx, etc.) from/to SharePoint (like copy, sync, etc.), you should append these flags to ensure Rclone uses the "Last Modified" datetime property to compare your documents:
 
 ```
 --ignore-size --ignore-checksum --update
diff --git a/fs/filter/filter.go b/fs/filter/filter.go
index b693c2129..460fad6a6 100644
--- a/fs/filter/filter.go
+++ b/fs/filter/filter.go
@@ -191,7 +191,12 @@ func NewFilter(opt *Opt) (f *Filter, err error) {
 			return nil, err
 		}
 	}
+
+	inActive := f.InActive()
 	for _, rule := range f.Opt.FilesFrom {
+		if !inActive {
+			return nil, fmt.Errorf("The usage of --files-from overrides all other filters, it should be used alone")
+		}
 		f.initAddFile() // init to show --files-from set even if no files within
 		err := forEachLine(rule, func(line string) error {
 			return f.AddFile(line)
diff --git a/fs/operations/operations.go b/fs/operations/operations.go
index a5c9158cd..6e56e7c7e 100644
--- a/fs/operations/operations.go
+++ b/fs/operations/operations.go
@@ -114,7 +114,7 @@ func checkHashes(ctx context.Context, src fs.ObjectInfo, dst fs.Object, ht hash.
 // Otherwise the file is considered to be not equal including if there
 // were errors reading info.
 func Equal(ctx context.Context, src fs.ObjectInfo, dst fs.Object) bool {
-	return equal(ctx, src, dst, fs.Config.SizeOnly, fs.Config.CheckSum, !fs.Config.NoUpdateModTime)
+	return equal(ctx, src, dst, defaultEqualOpt())
 }
 
 // sizeDiffers compare the size of src and dst taking into account the
@@ -128,12 +128,30 @@ func sizeDiffers(src, dst fs.ObjectInfo) bool {
 
 var checksumWarning sync.Once
 
-func equal(ctx context.Context, src fs.ObjectInfo, dst fs.Object, sizeOnly, checkSum, UpdateModTime bool) bool {
+// options for equal function()
+type equalOpt struct {
+	sizeOnly          bool // if set only check size
+	checkSum          bool // if set check checksum+size instead of modtime+size
+	updateModTime     bool // if set update the modtime if hashes identical and checking with modtime+size
+	forceModTimeMatch bool // if set assume modtimes match
+}
+
+// default set of options for equal()
+func defaultEqualOpt() equalOpt {
+	return equalOpt{
+		sizeOnly:          fs.Config.SizeOnly,
+		checkSum:          fs.Config.CheckSum,
+		updateModTime:     !fs.Config.NoUpdateModTime,
+		forceModTimeMatch: false,
+	}
+}
+
+func equal(ctx context.Context, src fs.ObjectInfo, dst fs.Object, opt equalOpt) bool {
 	if sizeDiffers(src, dst) {
 		fs.Debugf(src, "Sizes differ (src %d vs dst %d)", src.Size(), dst.Size())
 		return false
 	}
-	if sizeOnly {
+	if opt.sizeOnly {
 		fs.Debugf(src, "Sizes identical")
 		return true
 	}
@@ -141,7 +159,7 @@ func equal(ctx context.Context, src fs.ObjectInfo, dst fs.Object, sizeOnly, chec
 	// Assert: Size is equal or being ignored
 
 	// If checking checksum and not modtime
-	if checkSum {
+	if opt.checkSum {
 		// Check the hash
 		same, ht, _ := CheckHashes(ctx, src, dst)
 		if !same {
@@ -159,21 +177,23 @@ func equal(ctx context.Context, src fs.ObjectInfo, dst fs.Object, sizeOnly, chec
 		return true
 	}
 
-	// Sizes the same so check the mtime
-	modifyWindow := fs.GetModifyWindow(src.Fs(), dst.Fs())
-	if modifyWindow == fs.ModTimeNotSupported {
-		fs.Debugf(src, "Sizes identical")
-		return true
-	}
 	srcModTime := src.ModTime(ctx)
-	dstModTime := dst.ModTime(ctx)
-	dt := dstModTime.Sub(srcModTime)
-	if dt < modifyWindow && dt > -modifyWindow {
-		fs.Debugf(src, "Size and modification time the same (differ by %s, within tolerance %s)", dt, modifyWindow)
-		return true
-	}
+	if !opt.forceModTimeMatch {
+		// Sizes the same so check the mtime
+		modifyWindow := fs.GetModifyWindow(src.Fs(), dst.Fs())
+		if modifyWindow == fs.ModTimeNotSupported {
+			fs.Debugf(src, "Sizes identical")
+			return true
+		}
+		dstModTime := dst.ModTime(ctx)
+		dt := dstModTime.Sub(srcModTime)
+		if dt < modifyWindow && dt > -modifyWindow {
+			fs.Debugf(src, "Size and modification time the same (differ by %s, within tolerance %s)", dt, modifyWindow)
+			return true
+		}
 
-	fs.Debugf(src, "Modification times differ by %s: %v, %v", dt, srcModTime, dstModTime)
+		fs.Debugf(src, "Modification times differ by %s: %v, %v", dt, srcModTime, dstModTime)
+	}
 
 	// Check if the hashes are the same
 	same, ht, _ := CheckHashes(ctx, src, dst)
@@ -187,7 +207,7 @@ func equal(ctx context.Context, src fs.ObjectInfo, dst fs.Object, sizeOnly, chec
 	}
 
 	// mod time differs but hash is the same to reset mod time if required
-	if UpdateModTime {
+	if opt.updateModTime {
 		if fs.Config.DryRun {
 			fs.Logf(src, "Not updating modification time as --dry-run")
 		} else {
@@ -1444,7 +1464,9 @@ func copyDest(ctx context.Context, fdst fs.Fs, dst, src fs.Object, CopyDest, bac
 	default:
 		return false, err
 	}
-	if equal(ctx, src, CopyDestFile, fs.Config.SizeOnly, fs.Config.CheckSum, false) {
+	opt := defaultEqualOpt()
+	opt.updateModTime = false
+	if equal(ctx, src, CopyDestFile, opt) {
 		if dst == nil || !Equal(ctx, src, dst) {
 			if dst != nil && backupDir != nil {
 				err = MoveBackupDir(ctx, backupDir, dst)
@@ -1520,13 +1542,22 @@ func NeedTransfer(ctx context.Context, dst, src fs.Object) bool {
 			fs.Debugf(src, "Destination is newer than source, skipping")
 			return false
 		case dt <= -modifyWindow:
-			fs.Debugf(src, "Destination is older than source, transferring")
+			// force --checksum on for the check and do update modtimes by default
+			opt := defaultEqualOpt()
+			opt.forceModTimeMatch = true
+			if equal(ctx, src, dst, opt) {
+				fs.Debugf(src, "Unchanged skipping")
+				return false
+			}
 		default:
-			if !sizeDiffers(src, dst) {
-				fs.Debugf(src, "Destination mod time is within %v of source and sizes identical, skipping", modifyWindow)
+			// Do a size only compare unless --checksum is set
+			opt := defaultEqualOpt()
+			opt.sizeOnly = !fs.Config.CheckSum
+			if equal(ctx, src, dst, opt) {
+				fs.Debugf(src, "Destination mod time is within %v of source and files identical, skipping", modifyWindow)
 				return false
 			}
-			fs.Debugf(src, "Destination mod time is within %v of source but sizes differ, transferring", modifyWindow)
+			fs.Debugf(src, "Destination mod time is within %v of source but files differ, transferring", modifyWindow)
 		}
 	} else {
 		// Check to see if changed or not


Test patch:
diff --git a/backend/chunker/chunker_internal_test.go b/backend/chunker/chunker_internal_test.go
index ea9c107bf..372fa7bc6 100644
--- a/backend/chunker/chunker_internal_test.go
+++ b/backend/chunker/chunker_internal_test.go
@@ -1,15 +1,24 @@
 package chunker
 
 import (
+	"bytes"
 	"context"
 	"flag"
 	"fmt"
+	"io/ioutil"
+	"path"
+	"regexp"
+	"strings"
 	"testing"
 
 	"github.com/rclone/rclone/fs"
+	"github.com/rclone/rclone/fs/hash"
+	"github.com/rclone/rclone/fs/operations"
 	"github.com/rclone/rclone/fstest"
 	"github.com/rclone/rclone/fstest/fstests"
+	"github.com/rclone/rclone/lib/random"
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 )
 
 // Command line flags
@@ -18,7 +27,7 @@ var (
 )
 
 // test that chunking does not break large uploads
-func (f *Fs) InternalTestPutLarge(t *testing.T, kilobytes int) {
+func testPutLarge(t *testing.T, f *Fs, kilobytes int) {
 	t.Run(fmt.Sprintf("PutLarge%dk", kilobytes), func(t *testing.T) {
 		fstests.TestPutLarge(context.Background(), t, f, &fstest.Item{
 			ModTime: fstest.Time("2001-02-03T04:05:06.499999999Z"),
@@ -28,119 +37,568 @@ func (f *Fs) InternalTestPutLarge(t *testing.T, kilobytes int) {
 	})
 }
 
-func (f *Fs) InternalTestChunkNameFormat(t *testing.T) {
-	savedNameFormat := f.opt.NameFormat
-	savedStartFrom := f.opt.StartFrom
+// test chunk name parser
+func testChunkNameFormat(t *testing.T, f *Fs) {
+	saveOpt := f.opt
 	defer func() {
-		// restore original settings
-		_ = f.parseNameFormat(savedNameFormat)
-		f.opt.StartFrom = savedStartFrom
+		// restore original settings (f is pointer, f.opt is struct)
+		f.opt = saveOpt
+		_ = f.setChunkNameFormat(f.opt.NameFormat)
 	}()
-	var err error
 
-	err = f.parseNameFormat("*.rclone_chunk.###")
-	assert.NoError(t, err)
-	assert.Equal(t, `%s.rclone_chunk.%03d`, f.nameFormat)
-	assert.Equal(t, `^(.+)\.rclone_chunk\.([0-9]{3,})$`, f.nameRegexp.String())
+	assertFormat := func(pattern, wantDataFormat, wantCtrlFormat, wantNameRegexp string) {
+		err := f.setChunkNameFormat(pattern)
+		assert.NoError(t, err)
+		assert.Equal(t, wantDataFormat, f.dataNameFmt)
+		assert.Equal(t, wantCtrlFormat, f.ctrlNameFmt)
+		assert.Equal(t, wantNameRegexp, f.nameRegexp.String())
+	}
 
-	err = f.parseNameFormat("*.rclone_chunk.#")
-	assert.NoError(t, err)
-	assert.Equal(t, `%s.rclone_chunk.%d`, f.nameFormat)
-	assert.Equal(t, `^(.+)\.rclone_chunk\.([0-9]+)$`, f.nameRegexp.String())
+	assertFormatValid := func(pattern string) {
+		err := f.setChunkNameFormat(pattern)
+		assert.NoError(t, err)
+	}
 
-	err = f.parseNameFormat("*_chunk_#####")
-	assert.NoError(t, err)
-	assert.Equal(t, `%s_chunk_%05d`, f.nameFormat)
-	assert.Equal(t, `^(.+)_chunk_([0-9]{5,})$`, f.nameRegexp.String())
+	assertFormatInvalid := func(pattern string) {
+		err := f.setChunkNameFormat(pattern)
+		assert.Error(t, err)
+	}
 
-	err = f.parseNameFormat("*-chunk-#")
-	assert.NoError(t, err)
-	assert.Equal(t, `%s-chunk-%d`, f.nameFormat)
-	assert.Equal(t, `^(.+)-chunk-([0-9]+)$`, f.nameRegexp.String())
+	assertMakeName := func(wantChunkName, mainName string, chunkNo int, ctrlType string, xactNo int64) {
+		gotChunkName := f.makeChunkName(mainName, chunkNo, ctrlType, xactNo)
+		assert.Equal(t, wantChunkName, gotChunkName)
+	}
 
-	err = f.parseNameFormat("_*-chunk-##,")
-	assert.NoError(t, err)
-	assert.Equal(t, `_%s-chunk-%02d,`, f.nameFormat)
-	assert.Equal(t, `^_(.+)-chunk-([0-9]{2,}),$`, f.nameRegexp.String())
+	assertMakeNamePanics := func(mainName string, chunkNo int, ctrlType string, xactNo int64) {
+		assert.Panics(t, func() {
+			_ = f.makeChunkName(mainName, chunkNo, ctrlType, xactNo)
+		}, "makeChunkName(%q,%d,%q,%d) should panic", mainName, chunkNo, ctrlType, xactNo)
+	}
 
-	err = f.parseNameFormat(`*-chunk-#-%^$()[]{}.+-!?:\/`)
-	assert.NoError(t, err)
-	assert.Equal(t, `%s-chunk-%d-%%^$()[]{}.+-!?:\/`, f.nameFormat)
-	assert.Equal(t, `^(.+)-chunk-([0-9]+)-%\^\$\(\)\[\]\{\}\.\+-!\?:\\/$`, f.nameRegexp.String())
+	assertParseName := func(fileName, wantMainName string, wantChunkNo int, wantCtrlType string, wantXactNo int64) {
+		gotMainName, gotChunkNo, gotCtrlType, gotXactNo := f.parseChunkName(fileName)
+		assert.Equal(t, wantMainName, gotMainName)
+		assert.Equal(t, wantChunkNo, gotChunkNo)
+		assert.Equal(t, wantCtrlType, gotCtrlType)
+		assert.Equal(t, wantXactNo, gotXactNo)
+	}
 
-	err = f.parseNameFormat("chunk-#")
-	assert.Error(t, err)
+	const newFormatSupported = false // support for patterns not starting with base name (*)
 
-	err = f.parseNameFormat("*-chunk")
-	assert.Error(t, err)
+	// valid formats
+	assertFormat(`*.rclone_chunk.###`, `%s.rclone_chunk.%03d`, `%s.rclone_chunk._%s`, `^(.+?)\.rclone_chunk\.(?:([0-9]{3,})|_([a-z]{3,9}))(?:\.\.tmp_([0-9]{10,19}))?$`)
+	assertFormat(`*.rclone_chunk.#`, `%s.rclone_chunk.%d`, `%s.rclone_chunk._%s`, `^(.+?)\.rclone_chunk\.(?:([0-9]+)|_([a-z]{3,9}))(?:\.\.tmp_([0-9]{10,19}))?$`)
+	assertFormat(`*_chunk_#####`, `%s_chunk_%05d`, `%s_chunk__%s`, `^(.+?)_chunk_(?:([0-9]{5,})|_([a-z]{3,9}))(?:\.\.tmp_([0-9]{10,19}))?$`)
+	assertFormat(`*-chunk-#`, `%s-chunk-%d`, `%s-chunk-_%s`, `^(.+?)-chunk-(?:([0-9]+)|_([a-z]{3,9}))(?:\.\.tmp_([0-9]{10,19}))?$`)
+	assertFormat(`*-chunk-#-%^$()[]{}.+-!?:\`, `%s-chunk-%d-%%^$()[]{}.+-!?:\`, `%s-chunk-_%s-%%^$()[]{}.+-!?:\`, `^(.+?)-chunk-(?:([0-9]+)|_([a-z]{3,9}))-%\^\$\(\)\[\]\{\}\.\+-!\?:\\(?:\.\.tmp_([0-9]{10,19}))?$`)
+	if newFormatSupported {
+		assertFormat(`_*-chunk-##,`, `_%s-chunk-%02d,`, `_%s-chunk-_%s,`, `^_(.+?)-chunk-(?:([0-9]{2,})|_([a-z]{3,9})),(?:\.\.tmp_([0-9]{10,19}))?$`)
+	}
 
-	err = f.parseNameFormat("*-*-chunk-#")
-	assert.Error(t, err)
+	// invalid formats
+	assertFormatInvalid(`chunk-#`)
+	assertFormatInvalid(`*-chunk`)
+	assertFormatInvalid(`*-*-chunk-#`)
+	assertFormatInvalid(`*-chunk-#-#`)
+	assertFormatInvalid(`#-chunk-*`)
+	assertFormatInvalid(`*/#`)
+
+	assertFormatValid(`*#`)
+	assertFormatInvalid(`**#`)
+	assertFormatInvalid(`#*`)
+	assertFormatInvalid(``)
+	assertFormatInvalid(`-`)
+
+	// quick tests
+	if newFormatSupported {
+		assertFormat(`part_*_#`, `part_%s_%d`, `part_%s__%s`, `^part_(.+?)_(?:([0-9]+)|_([a-z]{3,9}))(?:\.\.tmp_([0-9]{10,19}))?$`)
+		f.opt.StartFrom = 1
+
+		assertMakeName(`part_fish_1`, "fish", 0, "", -1)
+		assertParseName(`part_fish_43`, "fish", 42, "", -1)
+		assertMakeName(`part_fish_3..tmp_0000000004`, "fish", 2, "", 4)
+		assertParseName(`part_fish_4..tmp_0000000005`, "fish", 3, "", 5)
+		assertMakeName(`part_fish__locks`, "fish", -2, "locks", -3)
+		assertParseName(`part_fish__locks`, "fish", -1, "locks", -1)
+		assertMakeName(`part_fish__blockinfo..tmp_1234567890123456789`, "fish", -3, "blockinfo", 1234567890123456789)
+		assertParseName(`part_fish__blockinfo..tmp_1234567890123456789`, "fish", -1, "blockinfo", 1234567890123456789)
+	}
+
+	// prepare format for long tests
+	assertFormat(`*.chunk.###`, `%s.chunk.%03d`, `%s.chunk._%s`, `^(.+?)\.chunk\.(?:([0-9]{3,})|_([a-z]{3,9}))(?:\.\.tmp_([0-9]{10,19}))?$`)
+	f.opt.StartFrom = 2
+
+	// valid data chunks
+	assertMakeName(`fish.chunk.003`, "fish", 1, "", -1)
+	assertMakeName(`fish.chunk.011..tmp_0000054321`, "fish", 9, "", 54321)
+	assertMakeName(`fish.chunk.011..tmp_1234567890`, "fish", 9, "", 1234567890)
+	assertMakeName(`fish.chunk.1916..tmp_123456789012345`, "fish", 1914, "", 123456789012345)
+
+	assertParseName(`fish.chunk.003`, "fish", 1, "", -1)
+	assertParseName(`fish.chunk.004..tmp_0000000021`, "fish", 2, "", 21)
+	assertParseName(`fish.chunk.021`, "fish", 19, "", -1)
+	assertParseName(`fish.chunk.323..tmp_1234567890123456789`, "fish", 321, "", 1234567890123456789)
+
+	// parsing invalid data chunk names
+	assertParseName(`fish.chunk.3`, "", -1, "", -1)
+	assertParseName(`fish.chunk.001`, "", -1, "", -1)
+	assertParseName(`fish.chunk.21`, "", -1, "", -1)
+	assertParseName(`fish.chunk.-21`, "", -1, "", -1)
+
+	assertParseName(`fish.chunk.004.tmp_0000000021`, "", -1, "", -1)
+	assertParseName(`fish.chunk.003..tmp_123456789`, "", -1, "", -1)
+	assertParseName(`fish.chunk.003..tmp_012345678901234567890123456789`, "", -1, "", -1)
+	assertParseName(`fish.chunk.003..tmp_-1`, "", -1, "", -1)
+
+	// valid control chunks
+	assertMakeName(`fish.chunk._info`, "fish", -1, "info", -1)
+	assertMakeName(`fish.chunk._locks`, "fish", -2, "locks", -1)
+	assertMakeName(`fish.chunk._blockinfo`, "fish", -3, "blockinfo", -1)
+
+	assertParseName(`fish.chunk._info`, "fish", -1, "info", -1)
+	assertParseName(`fish.chunk._locks`, "fish", -1, "locks", -1)
+	assertParseName(`fish.chunk._blockinfo`, "fish", -1, "blockinfo", -1)
+
+	// valid temporary control chunks
+	assertMakeName(`fish.chunk._info..tmp_0000000021`, "fish", -1, "info", 21)
+	assertMakeName(`fish.chunk._locks..tmp_0000054321`, "fish", -2, "locks", 54321)
+	assertMakeName(`fish.chunk._uploads..tmp_0000000000`, "fish", -3, "uploads", 0)
+	assertMakeName(`fish.chunk._blockinfo..tmp_1234567890123456789`, "fish", -4, "blockinfo", 1234567890123456789)
+
+	assertParseName(`fish.chunk._info..tmp_0000000021`, "fish", -1, "info", 21)
+	assertParseName(`fish.chunk._locks..tmp_0000054321`, "fish", -1, "locks", 54321)
+	assertParseName(`fish.chunk._uploads..tmp_0000000000`, "fish", -1, "uploads", 0)
+	assertParseName(`fish.chunk._blockinfo..tmp_1234567890123456789`, "fish", -1, "blockinfo", 1234567890123456789)
+
+	// parsing invalid control chunk names
+	assertParseName(`fish.chunk.info`, "", -1, "", -1)
+	assertParseName(`fish.chunk.locks`, "", -1, "", -1)
+	assertParseName(`fish.chunk.uploads`, "", -1, "", -1)
+	assertParseName(`fish.chunk.blockinfo`, "", -1, "", -1)
+
+	assertParseName(`fish.chunk._os`, "", -1, "", -1)
+	assertParseName(`fish.chunk._futuredata`, "", -1, "", -1)
+	assertParseName(`fish.chunk._me_ta`, "", -1, "", -1)
+	assertParseName(`fish.chunk._in-fo`, "", -1, "", -1)
+	assertParseName(`fish.chunk._.bin`, "", -1, "", -1)
+
+	assertParseName(`fish.chunk._locks..tmp_123456789`, "", -1, "", -1)
+	assertParseName(`fish.chunk._meta..tmp_-1`, "", -1, "", -1)
+	assertParseName(`fish.chunk._blockinfo..tmp_012345678901234567890123456789`, "", -1, "", -1)
+
+	// short control chunk names: 3 letters ok, 1-2 letters not allowed
+	assertMakeName(`fish.chunk._ext`, "fish", -1, "ext", -1)
+	assertMakeName(`fish.chunk._ext..tmp_0000000021`, "fish", -1, "ext", 21)
+	assertParseName(`fish.chunk._int`, "fish", -1, "int", -1)
+	assertParseName(`fish.chunk._int..tmp_0000000021`, "fish", -1, "int", 21)
+	assertMakeNamePanics("fish", -1, "in", -1)
+	assertMakeNamePanics("fish", -1, "up", 4)
+	assertMakeNamePanics("fish", -1, "x", -1)
+	assertMakeNamePanics("fish", -1, "c", 4)
+
+	// base file name can sometimes look like a valid chunk name
+	assertParseName(`fish.chunk.003.chunk.004`, "fish.chunk.003", 2, "", -1)
+	assertParseName(`fish.chunk.003.chunk.005..tmp_0000000021`, "fish.chunk.003", 3, "", 21)
+	assertParseName(`fish.chunk.003.chunk._info`, "fish.chunk.003", -1, "info", -1)
+	assertParseName(`fish.chunk.003.chunk._blockinfo..tmp_1234567890123456789`, "fish.chunk.003", -1, "blockinfo", 1234567890123456789)
+	assertParseName(`fish.chunk.003.chunk._Meta`, "", -1, "", -1)
+	assertParseName(`fish.chunk.003.chunk._x..tmp_0000054321`, "", -1, "", -1)
+
+	assertParseName(`fish.chunk.004..tmp_0000000021.chunk.004`, "fish.chunk.004..tmp_0000000021", 2, "", -1)
+	assertParseName(`fish.chunk.004..tmp_0000000021.chunk.005..tmp_0000000021`, "fish.chunk.004..tmp_0000000021", 3, "", 21)
+	assertParseName(`fish.chunk.004..tmp_0000000021.chunk._info`, "fish.chunk.004..tmp_0000000021", -1, "info", -1)
+	assertParseName(`fish.chunk.004..tmp_0000000021.chunk._blockinfo..tmp_1234567890123456789`, "fish.chunk.004..tmp_0000000021", -1, "blockinfo", 1234567890123456789)
+	assertParseName(`fish.chunk.004..tmp_0000000021.chunk._Meta`, "", -1, "", -1)
+	assertParseName(`fish.chunk.004..tmp_0000000021.chunk._x..tmp_0000054321`, "", -1, "", -1)
+
+	assertParseName(`fish.chunk._info.chunk.004`, "fish.chunk._info", 2, "", -1)
+	assertParseName(`fish.chunk._info.chunk.005..tmp_0000000021`, "fish.chunk._info", 3, "", 21)
+	assertParseName(`fish.chunk._info.chunk._info`, "fish.chunk._info", -1, "info", -1)
+	assertParseName(`fish.chunk._info.chunk._blockinfo..tmp_1234567890123456789`, "fish.chunk._info", -1, "blockinfo", 1234567890123456789)
+	assertParseName(`fish.chunk._info.chunk._info.chunk._Meta`, "", -1, "", -1)
+	assertParseName(`fish.chunk._info.chunk._info.chunk._x..tmp_0000054321`, "", -1, "", -1)
+
+	assertParseName(`fish.chunk._blockinfo..tmp_1234567890123456789.chunk.004`, "fish.chunk._blockinfo..tmp_1234567890123456789", 2, "", -1)
+	assertParseName(`fish.chunk._blockinfo..tmp_1234567890123456789.chunk.005..tmp_0000000021`, "fish.chunk._blockinfo..tmp_1234567890123456789", 3, "", 21)
+	assertParseName(`fish.chunk._blockinfo..tmp_1234567890123456789.chunk._info`, "fish.chunk._blockinfo..tmp_1234567890123456789", -1, "info", -1)
+	assertParseName(`fish.chunk._blockinfo..tmp_1234567890123456789.chunk._blockinfo..tmp_1234567890123456789`, "fish.chunk._blockinfo..tmp_1234567890123456789", -1, "blockinfo", 1234567890123456789)
+	assertParseName(`fish.chunk._blockinfo..tmp_1234567890123456789.chunk._info.chunk._Meta`, "", -1, "", -1)
+	assertParseName(`fish.chunk._blockinfo..tmp_1234567890123456789.chunk._info.chunk._x..tmp_0000054321`, "", -1, "", -1)
+
+	// attempts to make invalid chunk names
+	assertMakeNamePanics("fish", -1, "", -1)           // neither data nor control
+	assertMakeNamePanics("fish", 0, "info", -1)        // both data and control
+	assertMakeNamePanics("fish", -1, "futuredata", -1) // control type too long
+	assertMakeNamePanics("fish", -1, "123", -1)        // digits not allowed
+	assertMakeNamePanics("fish", -1, "Meta", -1)       // only lower case letters allowed
+	assertMakeNamePanics("fish", -1, "in-fo", -1)      // punctuation not allowed
+	assertMakeNamePanics("fish", -1, "_info", -1)
+	assertMakeNamePanics("fish", -1, "info_", -1)
+	assertMakeNamePanics("fish", -2, ".bind", -3)
+	assertMakeNamePanics("fish", -2, "bind.", -3)
+
+	assertMakeNamePanics("fish", -1, "", 1)            // neither data nor control
+	assertMakeNamePanics("fish", 0, "info", 12)        // both data and control
+	assertMakeNamePanics("fish", -1, "futuredata", 45) // control type too long
+	assertMakeNamePanics("fish", -1, "123", 123)       // digits not allowed
+	assertMakeNamePanics("fish", -1, "Meta", 456)      // only lower case letters allowed
+	assertMakeNamePanics("fish", -1, "in-fo", 321)     // punctuation not allowed
+	assertMakeNamePanics("fish", -1, "_info", 15678)
+	assertMakeNamePanics("fish", -1, "info_", 999)
+	assertMakeNamePanics("fish", -2, ".bind", 0)
+	assertMakeNamePanics("fish", -2, "bind.", 0)
+}
+
+func testSmallFileInternals(t *testing.T, f *Fs) {
+	const dir = "small"
+	ctx := context.Background()
+	saveOpt := f.opt
+	defer func() {
+		f.opt.FailHard = false
+		_ = operations.Purge(ctx, f.base, dir)
+		f.opt = saveOpt
+	}()
+	f.opt.FailHard = false
+
+	modTime := fstest.Time("2001-02-03T04:05:06.499999999Z")
+
+	checkSmallFileInternals := func(obj fs.Object) {
+		assert.NotNil(t, obj)
+		o, ok := obj.(*Object)
+		assert.True(t, ok)
+		assert.NotNil(t, o)
+		if o == nil {
+			return
+		}
+		switch {
+		case !f.useMeta:
+			// If meta format is "none", non-chunked file (even empty)
+			// internally is a single chunk without meta object.
+			assert.Nil(t, o.main)
+			assert.True(t, o.isComposite()) // sorry, sometimes a name is misleading
+			assert.Equal(t, 1, len(o.chunks))
+		case f.hashAll:
+			// Consistent hashing forces meta object on small files too
+			assert.NotNil(t, o.main)
+			assert.True(t, o.isComposite())
+			assert.Equal(t, 1, len(o.chunks))
+		default:
+			// normally non-chunked file is kept in the Object's main field
+			assert.NotNil(t, o.main)
+			assert.False(t, o.isComposite())
+			assert.Equal(t, 0, len(o.chunks))
+		}
+	}
+
+	checkContents := func(obj fs.Object, contents string) {
+		assert.NotNil(t, obj)
+		assert.Equal(t, int64(len(contents)), obj.Size())
+
+		r, err := obj.Open(ctx)
+		assert.NoError(t, err)
+		assert.NotNil(t, r)
+		if r == nil {
+			return
+		}
+		data, err := ioutil.ReadAll(r)
+		assert.NoError(t, err)
+		assert.Equal(t, contents, string(data))
+		_ = r.Close()
+	}
+
+	checkHashsum := func(obj fs.Object) {
+		var ht hash.Type
+		switch {
+		case !f.hashAll:
+			return
+		case f.useMD5:
+			ht = hash.MD5
+		case f.useSHA1:
+			ht = hash.SHA1
+		default:
+			return
+		}
+		// even empty files must have hashsum in consistent mode
+		sum, err := obj.Hash(ctx, ht)
+		assert.NoError(t, err)
+		assert.NotEqual(t, sum, "")
+	}
+
+	checkSmallFile := func(name, contents string) {
+		filename := path.Join(dir, name)
+		item := fstest.Item{Path: filename, ModTime: modTime}
+		_, put := fstests.PutTestContents(ctx, t, f, &item, contents, false)
+		assert.NotNil(t, put)
+		checkSmallFileInternals(put)
+		checkContents(put, contents)
+		checkHashsum(put)
+
+		// objects returned by Put and NewObject must have similar structure
+		obj, err := f.NewObject(ctx, filename)
+		assert.NoError(t, err)
+		assert.NotNil(t, obj)
+		checkSmallFileInternals(obj)
+		checkContents(obj, contents)
+		checkHashsum(obj)
+
+		_ = obj.Remove(ctx)
+		_ = put.Remove(ctx) // for good
+	}
+
+	checkSmallFile("emptyfile", "")
+	checkSmallFile("smallfile", "Ok")
+}
+
+func testPreventCorruption(t *testing.T, f *Fs) {
+	if f.opt.ChunkSize > 50 {
+		t.Skip("this test requires small chunks")
+	}
+	const dir = "corrupted"
+	ctx := context.Background()
+	saveOpt := f.opt
+	defer func() {
+		f.opt.FailHard = false
+		_ = operations.Purge(ctx, f.base, dir)
+		f.opt = saveOpt
+	}()
+	f.opt.FailHard = true
+
+	contents := random.String(250)
+	modTime := fstest.Time("2001-02-03T04:05:06.499999999Z")
+	const overlapMessage = "chunk overlap"
 
-	err = f.parseNameFormat("*-chunk-#-#")
+	assertOverlapError := func(err error) {
+		assert.Error(t, err)
+		if err != nil {
+			assert.Contains(t, err.Error(), overlapMessage)
+		}
+	}
+
+	newFile := func(name string) fs.Object {
+		item := fstest.Item{Path: path.Join(dir, name), ModTime: modTime}
+		_, obj := fstests.PutTestContents(ctx, t, f, &item, contents, true)
+		require.NotNil(t, obj)
+		return obj
+	}
+	billyObj := newFile("billy")
+
+	billyChunkName := func(chunkNo int) string {
+		return f.makeChunkName(billyObj.Remote(), chunkNo, "", -1)
+	}
+
+	err := f.Mkdir(ctx, billyChunkName(1))
+	assertOverlapError(err)
+
+	_, err = f.Move(ctx, newFile("silly1"), billyChunkName(2))
 	assert.Error(t, err)
+	assert.True(t, err == fs.ErrorCantMove || (err != nil && strings.Contains(err.Error(), overlapMessage)))
 
-	err = f.parseNameFormat("#-chunk-*")
+	_, err = f.Copy(ctx, newFile("silly2"), billyChunkName(3))
 	assert.Error(t, err)
+	assert.True(t, err == fs.ErrorCantCopy || (err != nil && strings.Contains(err.Error(), overlapMessage)))
+
+	// accessing chunks in strict mode is prohibited
+	f.opt.FailHard = true
+	billyChunk4Name := billyChunkName(4)
+	billyChunk4, err := f.NewObject(ctx, billyChunk4Name)
+	assertOverlapError(err)
 
-	err = f.parseNameFormat("*#")
+	f.opt.FailHard = false
+	billyChunk4, err = f.NewObject(ctx, billyChunk4Name)
 	assert.NoError(t, err)
+	require.NotNil(t, billyChunk4)
 
-	err = f.parseNameFormat("**#")
-	assert.Error(t, err)
-	err = f.parseNameFormat("#*")
+	f.opt.FailHard = true
+	_, err = f.Put(ctx, bytes.NewBufferString(contents), billyChunk4)
+	assertOverlapError(err)
+
+	// you can freely read chunks (if you have an object)
+	r, err := billyChunk4.Open(ctx)
+	assert.NoError(t, err)
+	var chunkContents []byte
+	assert.NotPanics(t, func() {
+		chunkContents, err = ioutil.ReadAll(r)
+		_ = r.Close()
+	})
+	assert.NoError(t, err)
+	assert.NotEqual(t, contents, string(chunkContents))
+
+	// but you can't change them
+	err = billyChunk4.Update(ctx, bytes.NewBufferString(contents), newFile("silly3"))
+	assertOverlapError(err)
+
+	// Remove isn't special, you can't corrupt files even if you have an object
+	err = billyChunk4.Remove(ctx)
+	assertOverlapError(err)
+
+	// recreate billy in case it was anyhow corrupted
+	willyObj := newFile("willy")
+	willyChunkName := f.makeChunkName(willyObj.Remote(), 1, "", -1)
+	f.opt.FailHard = false
+	willyChunk, err := f.NewObject(ctx, willyChunkName)
+	f.opt.FailHard = true
+	assert.NoError(t, err)
+	require.NotNil(t, willyChunk)
+
+	_, err = operations.Copy(ctx, f, willyChunk, willyChunkName, newFile("silly4"))
+	assertOverlapError(err)
+
+	// operations.Move will return error when chunker's Move refused
+	// to corrupt target file, but reverts to copy/delete method
+	// still trying to delete target chunk. Chunker must come to rescue.
+	_, err = operations.Move(ctx, f, willyChunk, willyChunkName, newFile("silly5"))
+	assertOverlapError(err)
+	r, err = willyChunk.Open(ctx)
+	assert.NoError(t, err)
+	assert.NotPanics(t, func() {
+		_, err = ioutil.ReadAll(r)
+		_ = r.Close()
+	})
+	assert.NoError(t, err)
+}
+
+func testChunkNumberOverflow(t *testing.T, f *Fs) {
+	if f.opt.ChunkSize > 50 {
+		t.Skip("this test requires small chunks")
+	}
+	const dir = "wreaked"
+	const wreakNumber = 10200300
+	ctx := context.Background()
+	saveOpt := f.opt
+	defer func() {
+		f.opt.FailHard = false
+		_ = operations.Purge(ctx, f.base, dir)
+		f.opt = saveOpt
+	}()
+
+	modTime := fstest.Time("2001-02-03T04:05:06.499999999Z")
+	contents := random.String(100)
+
+	newFile := func(f fs.Fs, name string) (fs.Object, string) {
+		filename := path.Join(dir, name)
+		item := fstest.Item{Path: filename, ModTime: modTime}
+		_, obj := fstests.PutTestContents(ctx, t, f, &item, contents, true)
+		require.NotNil(t, obj)
+		return obj, filename
+	}
+
+	f.opt.FailHard = false
+	file, fileName := newFile(f, "wreaker")
+	wreak, _ := newFile(f.base, f.makeChunkName("wreaker", wreakNumber, "", -1))
+
+	f.opt.FailHard = false
+	fstest.CheckListingWithRoot(t, f, dir, nil, nil, f.Precision())
+	_, err := f.NewObject(ctx, fileName)
 	assert.Error(t, err)
-	err = f.parseNameFormat("")
+
+	f.opt.FailHard = true
+	_, err = f.List(ctx, dir)
 	assert.Error(t, err)
-	err = f.parseNameFormat("-")
+	_, err = f.NewObject(ctx, fileName)
 	assert.Error(t, err)
 
-	f.opt.StartFrom = 2
-	err = f.parseNameFormat("*.chunk.###")
-	assert.NoError(t, err)
-	assert.Equal(t, `%s.chunk.%03d`, f.nameFormat)
-	assert.Equal(t, `^(.+)\.chunk\.([0-9]{3,})$`, f.nameRegexp.String())
-
-	assert.Equal(t, "fish.chunk.003", f.makeChunkName("fish", 1, -1))
-	assert.Equal(t, "fish.chunk.011..tmp_0000054321", f.makeChunkName("fish", 9, 54321))
-	assert.Equal(t, "fish.chunk.011..tmp_1234567890", f.makeChunkName("fish", 9, 1234567890))
-	assert.Equal(t, "fish.chunk.1916..tmp_123456789012345", f.makeChunkName("fish", 1914, 123456789012345))
-
-	name, chunkNo, tempNo := f.parseChunkName("fish.chunk.003")
-	assert.True(t, name == "fish" && chunkNo == 1 && tempNo == -1)
-	name, chunkNo, tempNo = f.parseChunkName("fish.chunk.004..tmp_0000000021")
-	assert.True(t, name == "fish" && chunkNo == 2 && tempNo == 21)
-	name, chunkNo, tempNo = f.parseChunkName("fish.chunk.021")
-	assert.True(t, name == "fish" && chunkNo == 19 && tempNo == -1)
-	name, chunkNo, tempNo = f.parseChunkName("fish.chunk.323..tmp_1234567890123456789")
-	assert.True(t, name == "fish" && chunkNo == 321 && tempNo == 1234567890123456789)
-	name, chunkNo, tempNo = f.parseChunkName("fish.chunk.3")
-	assert.True(t, name == "" && chunkNo == -1 && tempNo == -1)
-	name, chunkNo, tempNo = f.parseChunkName("fish.chunk.001")
-	assert.True(t, name == "" && chunkNo == -1 && tempNo == -1)
-	name, chunkNo, tempNo = f.parseChunkName("fish.chunk.21")
-	assert.True(t, name == "" && chunkNo == -1 && tempNo == -1)
-	name, chunkNo, tempNo = f.parseChunkName("fish.chunk.-21")
-	assert.True(t, name == "" && chunkNo == -1 && tempNo == -1)
-	name, chunkNo, tempNo = f.parseChunkName("fish.chunk.004.tmp_0000000021")
-	assert.True(t, name == "" && chunkNo == -1 && tempNo == -1)
-	name, chunkNo, tempNo = f.parseChunkName("fish.chunk.003..tmp_123456789")
-	assert.True(t, name == "" && chunkNo == -1 && tempNo == -1)
-	name, chunkNo, tempNo = f.parseChunkName("fish.chunk.003..tmp_012345678901234567890123456789")
-	assert.True(t, name == "" && chunkNo == -1 && tempNo == -1)
-	name, chunkNo, tempNo = f.parseChunkName("fish.chunk.003..tmp_-1")
-	assert.True(t, name == "" && chunkNo == -1 && tempNo == -1)
+	f.opt.FailHard = false
+	_ = wreak.Remove(ctx)
+	_ = file.Remove(ctx)
+}
+
+func testMetadataInput(t *testing.T, f *Fs) {
+	const minChunkForTest = 50
+	if f.opt.ChunkSize < minChunkForTest {
+		t.Skip("this test requires chunks that fit metadata")
+	}
+
+	const dir = "usermeta"
+	ctx := context.Background()
+	saveOpt := f.opt
+	defer func() {
+		f.opt.FailHard = false
+		_ = operations.Purge(ctx, f.base, dir)
+		f.opt = saveOpt
+	}()
+	f.opt.FailHard = false
+
+	modTime := fstest.Time("2001-02-03T04:05:06.499999999Z")
+
+	putFile := func(f fs.Fs, name, contents, message string, check bool) fs.Object {
+		item := fstest.Item{Path: name, ModTime: modTime}
+		_, obj := fstests.PutTestContents(ctx, t, f, &item, contents, check)
+		assert.NotNil(t, obj, message)
+		return obj
+	}
+
+	runSubtest := func(contents, name string) {
+		description := fmt.Sprintf("file with %s metadata", name)
+		filename := path.Join(dir, name)
+		require.True(t, len(contents) > 2 && len(contents) < minChunkForTest, description+" test data is correct")
+
+		part := putFile(f.base, f.makeChunkName(filename, 0, "", -1), "oops", "", true)
+		_ = putFile(f, filename, contents, "upload "+description, false)
+
+		obj, err := f.NewObject(ctx, filename)
+		assert.NoError(t, err, "access "+description)
+		assert.NotNil(t, obj)
+		assert.Equal(t, int64(len(contents)), obj.Size(), "size "+description)
+
+		o, ok := obj.(*Object)
+		assert.NotNil(t, ok)
+		if o != nil {
+			assert.True(t, o.isComposite() && len(o.chunks) == 1, description+" is forced composite")
+			o = nil
+		}
+
+		defer func() {
+			_ = obj.Remove(ctx)
+			_ = part.Remove(ctx)
+		}()
+
+		r, err := obj.Open(ctx)
+		assert.NoError(t, err, "open "+description)
+		assert.NotNil(t, r, "open stream of "+description)
+		if err == nil && r != nil {
+			data, err := ioutil.ReadAll(r)
+			assert.NoError(t, err, "read all of "+description)
+			assert.Equal(t, contents, string(data), description+" contents is ok")
+			_ = r.Close()
+		}
+	}
+
+	metaData, err := marshalSimpleJSON(ctx, 3, 1, "", "")
+	require.NoError(t, err)
+	todaysMeta := string(metaData)
+	runSubtest(todaysMeta, "today")
+
+	pastMeta := regexp.MustCompile(`"ver":[0-9]+`).ReplaceAllLiteralString(todaysMeta, `"ver":1`)
+	pastMeta = regexp.MustCompile(`"size":[0-9]+`).ReplaceAllLiteralString(pastMeta, `"size":0`)
+	runSubtest(pastMeta, "past")
+
+	futureMeta := regexp.MustCompile(`"ver":[0-9]+`).ReplaceAllLiteralString(todaysMeta, `"ver":999`)
+	futureMeta = regexp.MustCompile(`"nchunks":[0-9]+`).ReplaceAllLiteralString(futureMeta, `"nchunks":0,"x":"y"`)
+	runSubtest(futureMeta, "future")
 }
 
+// InternalTest dispatches all internal tests
 func (f *Fs) InternalTest(t *testing.T) {
 	t.Run("PutLarge", func(t *testing.T) {
 		if *UploadKilobytes <= 0 {
 			t.Skip("-upload-kilobytes is not set")
 		}
-		f.InternalTestPutLarge(t, *UploadKilobytes)
+		testPutLarge(t, f, *UploadKilobytes)
 	})
 	t.Run("ChunkNameFormat", func(t *testing.T) {
-		f.InternalTestChunkNameFormat(t)
+		testChunkNameFormat(t, f)
+	})
+	t.Run("SmallFileInternals", func(t *testing.T) {
+		testSmallFileInternals(t, f)
+	})
+	t.Run("PreventCorruption", func(t *testing.T) {
+		testPreventCorruption(t, f)
+	})
+	t.Run("ChunkNumberOverflow", func(t *testing.T) {
+		testChunkNumberOverflow(t, f)
+	})
+	t.Run("MetadataInput", func(t *testing.T) {
+		testMetadataInput(t, f)
 	})
 }
 
diff --git a/fs/filter/filter_test.go b/fs/filter/filter_test.go
index 36c425cba..773b3eb04 100644
--- a/fs/filter/filter_test.go
+++ b/fs/filter/filter_test.go
@@ -42,7 +42,58 @@ func testFile(t *testing.T, contents string) string {
 	return s
 }
 
-func TestNewFilterFull(t *testing.T) {
+func TestNewFilterForbiddenMixOfFilesFromAndFilterRule(t *testing.T) {
+	Opt := DefaultOpt
+
+	// Set up the input
+	Opt.FilterRule = []string{"- filter1", "- filter1b"}
+	Opt.FilesFrom = []string{testFile(t, "#comment\nfiles1\nfiles2\n")}
+
+	rm := func(p string) {
+		err := os.Remove(p)
+		if err != nil {
+			t.Logf("error removing %q: %v", p, err)
+		}
+	}
+	// Reset the input
+	defer func() {
+		rm(Opt.FilesFrom[0])
+	}()
+
+	_, err := NewFilter(&Opt)
+	require.Error(t, err)
+	require.Contains(t, err.Error(), "The usage of --files-from overrides all other filters")
+}
+
+func TestNewFilterWithFilesFromAlone(t *testing.T) {
+	Opt := DefaultOpt
+
+	// Set up the input
+	Opt.FilesFrom = []string{testFile(t, "#comment\nfiles1\nfiles2\n")}
+
+	rm := func(p string) {
+		err := os.Remove(p)
+		if err != nil {
+			t.Logf("error removing %q: %v", p, err)
+		}
+	}
+	// Reset the input
+	defer func() {
+		rm(Opt.FilesFrom[0])
+	}()
+
+	f, err := NewFilter(&Opt)
+	require.NoError(t, err)
+	assert.Len(t, f.files, 2)
+	for _, name := range []string{"files1", "files2"} {
+		_, ok := f.files[name]
+		if !ok {
+			t.Errorf("Didn't find file %q in f.files", name)
+		}
+	}
+}
+
+func TestNewFilterFullExceptFilesFromOpt(t *testing.T) {
 	Opt := DefaultOpt
 
 	mins := fs.SizeSuffix(100 * 1024)
@@ -56,7 +107,6 @@ func TestNewFilterFull(t *testing.T) {
 	Opt.ExcludeFrom = []string{testFile(t, "#comment\nexclude2\nexclude3\n")}
 	Opt.IncludeRule = []string{"include1"}
 	Opt.IncludeFrom = []string{testFile(t, "#comment\ninclude2\ninclude3\n")}
-	Opt.FilesFrom = []string{testFile(t, "#comment\nfiles1\nfiles2\n")}
 	Opt.MinSize = mins
 	Opt.MaxSize = maxs
 
@@ -71,7 +121,6 @@ func TestNewFilterFull(t *testing.T) {
 		rm(Opt.FilterFrom[0])
 		rm(Opt.ExcludeFrom[0])
 		rm(Opt.IncludeFrom[0])
-		rm(Opt.FilesFrom[0])
 	}()
 
 	f, err := NewFilter(&Opt)
@@ -96,13 +145,6 @@ func TestNewFilterFull(t *testing.T) {
 + ^.*$
 - ^.*$`
 	assert.Equal(t, want, got)
-	assert.Len(t, f.files, 2)
-	for _, name := range []string{"files1", "files2"} {
-		_, ok := f.files[name]
-		if !ok {
-			t.Errorf("Didn't find file %q in f.files", name)
-		}
-	}
 	assert.False(t, f.InActive())
 }
 
diff --git a/fs/sync/sync_test.go b/fs/sync/sync_test.go
index da69a48ba..eea3ea282 100644
--- a/fs/sync/sync_test.go
+++ b/fs/sync/sync_test.go
@@ -972,10 +972,22 @@ func TestSyncWithUpdateOlder(t *testing.T) {
 		fs.Config.ModifyWindow = oldModifyWindow
 	}()
 
-	accounting.GlobalStats().ResetCounters()
 	err := Sync(context.Background(), r.Fremote, r.Flocal, false)
 	require.NoError(t, err)
 	fstest.CheckItems(t, r.Fremote, oneO, twoF, threeO, fourF, fiveF)
+
+	if r.Fremote.Hashes().Count() == 0 {
+		t.Logf("Skip test with --checksum as no hashes supported")
+		return
+	}
+
+	// now enable checksum
+	fs.Config.CheckSum = true
+	defer func() { fs.Config.CheckSum = false }()
+
+	err = Sync(context.Background(), r.Fremote, r.Flocal, false)
+	require.NoError(t, err)
+	fstest.CheckItems(t, r.Fremote, oneO, twoF, threeF, fourF, fiveF)
 }
 
 // Test with TrackRenames set
diff --git a/fstest/fstests/fstests.go b/fstest/fstests/fstests.go
index 547f68895..4f12391b0 100644
--- a/fstest/fstests/fstests.go
+++ b/fstest/fstests/fstests.go
@@ -151,16 +151,19 @@ func retry(t *testing.T, what string, f func() error) {
 	require.NoError(t, err, what)
 }
 
-// testPut puts file to the remote
+// testPut puts file with random contents to the remote
 func testPut(ctx context.Context, t *testing.T, f fs.Fs, file *fstest.Item) (string, fs.Object) {
+	return PutTestContents(ctx, t, f, file, random.String(100), true)
+}
+
+// PutTestContents puts file with given contents to the remote and checks it but unlike TestPutLarge doesn't remove
+func PutTestContents(ctx context.Context, t *testing.T, f fs.Fs, file *fstest.Item, contents string, check bool) (string, fs.Object) {
 	var (
 		err        error
 		obj        fs.Object
 		uploadHash *hash.MultiHasher
-		contents   string
 	)
 	retry(t, "Put", func() error {
-		contents = random.String(100)
 		buf := bytes.NewBufferString(contents)
 		uploadHash = hash.NewMultiHasher()
 		in := io.TeeReader(buf, uploadHash)
@@ -171,10 +174,12 @@ func testPut(ctx context.Context, t *testing.T, f fs.Fs, file *fstest.Item) (str
 		return err
 	})
 	file.Hashes = uploadHash.Sums()
-	file.Check(t, obj, f.Precision())
-	// Re-read the object and check again
-	obj = findObject(ctx, t, f, file.Path)
-	file.Check(t, obj, f.Precision())
+	if check {
+		file.Check(t, obj, f.Precision())
+		// Re-read the object and check again
+		obj = findObject(ctx, t, f, file.Path)
+		file.Check(t, obj, f.Precision())
+	}
 	return contents, obj
 }
 
@@ -1310,7 +1315,7 @@ func Run(t *testing.T, opt *Opt) {
 				require.NotNil(t, fileRemote)
 				assert.Equal(t, fs.ErrorIsFile, err)
 
-				if strings.HasPrefix(remoteName, "TestChunkerChunk") && strings.Contains(remoteName, "Nometa") {
+				if strings.HasPrefix(remoteName, "TestChunker") && strings.Contains(remoteName, "Nometa") {
 					// TODO fix chunker and remove this bypass
 					t.Logf("Skip listing check -- chunker can't yet handle this tricky case")
 					return