neulab/tldr huggingface dataset:
Cache (DEPRECATED) The cache remote wraps another existing remote and stores file structure and its data for long running tasks like rclone mount.
Status The cache backend code is working but it currently doesn't have a maintainer so there are outstanding bugs (https://github.com/rclone/rclone/issues?q=is%3Aopen+is%3Aissue+label%3Abug+label%3A%22Remote%3A+Cache%22) which aren't getting fixed.
The cache backend is due to be phased out in favour of the VFS caching layer eventually which is more tightly integrated into rclone.
Until this happens we recommend only using the cache backend if you find you can't work without it.  There are many docs online describing the use of the cache backend to minimize API hits and by-and-large these are out of date and the cache backend isn't needed in those scenarios any more.
Configuration To get started you just need to have an existing remote which can be configured with cache.
Here is an example of how to make a remote called test-cache.  First run:
No remotes found - make a new one n) New remote r) Rename remote c) Copy remote s) Set configuration password q) Quit config n/r/c/s/q> n name> test-cache Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Cache a remote \ "cache" [snip] Storage> cache Remote to cache. Normally should contain a ':' and a path, e.g. "myremote:path/to/dir", "myremote:bucket" or maybe "myremote:" (not recommended). remote> local:/test Optional: The URL of the Plex server plex_url> http://127.0.0.1:32400 Optional: The username of the Plex user plex_username> dummyusername Optional: The password of the Plex user y) Yes type in my own password g) Generate random password n) No leave this optional password blank y/g/n> y Enter the password: password: Confirm the password: password: The size of a chunk. Lower value good for slow connections but can affect seamless reading. Default: 5M Choose a number from below, or type in your own value 1 / 1 MiB \ "1M" 2 / 5 MiB \ "5M" 3 / 10 MiB \ "10M" chunk_size> 2 How much time should object info (file size, file hashes, etc.) be stored in cache. Use a very high value if you don't plan on changing the source FS from outside the cache. Accepted units are: "s", "m", "h". Default: 5m Choose a number from below, or type in your own value 1 / 1 hour \ "1h" 2 / 24 hours \ "24h" 3 / 24 hours \ "48h" info_age> 2 The maximum size of stored chunks. When the storage grows beyond this size, the oldest chunks will be deleted. Default: 10G Choose a number from below, or type in your own value 1 / 500 MiB \ "500M" 2 / 1 GiB \ "1G" 3 / 10 GiB \ "10G" chunk_total_size> 3 Remote config -------------------- [test-cache] remote = local:/test plex_url = http://127.0.0.1:32400 plex_username = dummyusername plex_password = *** ENCRYPTED *** chunk_size = 5M info_age = 48h chunk_total_size = 10G
You can then use it like this,
List directories in top level of your drive
rclone lsd test-cache:
List all the files in your drive
rclone ls test-cache:
To start a cached mount
rclone mount --allow-other test-cache: /var/tmp/test-cache
Write Features Offline uploading In an effort to make writing through cache more reliable, the backend now supports this feature which can be activated by specifying a cache-tmp-upload-path.
A files goes through these states when using this feature:
1. An upload is started (usually by copying a file on the cache remote)
2. When the copy to the temporary location is complete the file is part of the cached remote and looks and behaves like any other file (reading included)
3. After cache-tmp-wait-time passes and the file is next in line, rclone move is used to move the file to the cloud provider
4. Reading the file still works during the upload but most modifications on it will be prohibited
5. Once the move is complete the file is unlocked for modifications as it becomes as any other regular file
6. If the file is being read through cache when it's actually deleted from the temporary path then cache will simply swap the source to the cloud provider without interrupting the reading (small blip can happen though)
Files are uploaded in sequence and only one file is uploaded at a time. Uploads will be stored in a queue and be processed based on the order they were added.  The queue and the temporary storage is persistent across restarts but can be cleared on startup with the --cache-db-purge flag.
Write Support Writes are supported through cache.  One caveat is that a mounted cache remote does not add any retry or fallback mechanism to the upload operation.  This will depend on the implementation of the wrapped remote. Consider using Offline uploading for reliable writes.
One special case is covered with cache-writes which will cache the file data at the same time as the upload when it is enabled making it available from the cache store immediately once the upload is finished.
Read Features Multiple connections To counter the high latency between a local PC where rclone is running and cloud providers, the cache remote can split multiple requests to the cloud provider for smaller file chunks and combines them together locally where they can be available almost immediately before the reader usually needs them.
This is similar to buffering when media files are played online.  Rclone will stay around the current marker but always try its best to stay ahead and prepare the data before.
Plex Integration There is a direct integration with Plex which allows cache to detect during reading if the file is in playback or not.  This helps cache to adapt how it queries the cloud provider depending on what is needed for.
Scans will have a minimum amount of workers (1) while in a confirmed playback cache will deploy the configured number of workers.
This integration opens the doorway to additional performance improvements which will be explored in the near future.
Note: If Plex options are not configured, cache will function with its configured options without adapting any of its settings.
How to enable?  Run rclone config and add all the Plex options (endpoint, username and password) in your remote and it will be automatically enabled.
Affected settings: - cache-workers: Configured value during confirmed playback or 1 all the other times
Certificate Validation When the Plex server is configured to only accept secure connections, it is possible to use .plex.direct URLs to ensure certificate validation succeeds.  These URLs are used by Plex internally to connect to the Plex server securely.
The format for these URLs is the following:
https://ip-with-dots-replaced.server-hash.plex.direct:32400/
The ip-with-dots-replaced part can be any IPv4 address, where the dots have been replaced with dashes, e.g.  127.0.0.1 becomes 127-0-0-1.
To get the server-hash part, the easiest way is to visit
https://plex.tv/api/resources?includeHttps=1&X-Plex-Token=your-plex-token
This page will list all the available Plex servers for your account with at least one .plex.direct link for each.  Copy one URL and replace the IP address with the desired address.  This can be used as the plex_url value.
Known issues Mount and --dir-cache-time --dir-cache-time controls the first layer of directory caching which works at the mount layer.  Being an independent caching mechanism from the cache backend, it will manage its own entries based on the configured time.
To avoid getting in a scenario where dir cache has obsolete data and cache would have the correct one, try to set --dir-cache-time to a lower time than --cache-info-age.  Default values are already configured in this way.
Windows support - Experimental There are a couple of issues with Windows mount functionality that still require some investigations.  It should be considered as experimental thus far as fixes come in for this OS.
Most of the issues seem to be related to the difference between filesystems on Linux flavors and Windows as cache is heavily dependent on them.
Any reports or feedback on how cache behaves on this OS is greatly appreciated.
• https://github.com/rclone/rclone/issues/1935
• https://github.com/rclone/rclone/issues/1907
• https://github.com/rclone/rclone/issues/1834
Risk of throttling Future iterations of the cache backend will make use of the pooling functionality of the cloud provider to synchronize and at the same time make writing through it more tolerant to failures.
There are a couple of enhancements in track to add these but in the meantime there is a valid concern that the expiring cache listings can lead to cloud provider throttles or bans due to repeated queries on it for very large mounts.
Some recommendations: - don't use a very small interval for entry information (--cache-info-age) - while writes aren't yet optimised, you can still write through cache which gives you the advantage of adding the file in the cache at the same time if configured to do so.
Future enhancements:
• https://github.com/rclone/rclone/issues/1937
• https://github.com/rclone/rclone/issues/1936
cache and crypt One common scenario is to keep your data encrypted in the cloud provider using the crypt remote.  crypt uses a similar technique to wrap around an existing remote and handles this translation in a seamless way.
There is an issue with wrapping the remotes in this order: cloud remote -> crypt -> cache
During testing, I experienced a lot of bans with the remotes in this order.  I suspect it might be related to how crypt opens files on the cloud provider which makes it think we're downloading the full file instead of small chunks.  Organizing the remotes in this order yields better results: cloud remote -> cache -> crypt
absolute remote paths cache can not differentiate between relative and absolute paths for the wrapped remote.  Any path given in the remote config setting and on the command line will be passed to the wrapped remote as is, but for storing the chunks on disk the path will be made relative by removing any leading / character.
This behavior is irrelevant for most backend types, but there are backends where a leading / changes the effective directory, e.g.  in the sftp backend paths starting with a / are relative to the root of the SSH server and paths without are relative to the user home directory.  As a result sftp:bin and sftp:/bin will share the same cache folder, even if they represent a different directory on the SSH server.
Cache and Remote Control (--rc) Cache supports the new --rc mode in rclone and can be remote controlled through the following end points: By default, the listener is disabled if you do not add the flag.
rc cache/expire Purge a remote from the cache backend.  Supports either a directory or a file.  It supports both encrypted and unencrypted file names if cache is wrapped by crypt.
Params: - remote = path to remote (required) - withData = true/false to delete cached data (chunks) as well (optional, false by default)
Standard options Here are the standard options specific to cache (Cache a remote).
--cache-remote Remote to cache.
Normally should contain a ':' and a path, e.g.  "myremote:path/to/dir", "myremote:bucket" or maybe "myremote:" (not recommended).
• Env Var: RCLONE_CACHE_REMOTE
--cache-plex-url The URL of the Plex server.
• Config: plex_url
• Env Var: RCLONE_CACHE_PLEX_URL
--cache-plex-username The username of the Plex user.
• Config: plex_username
• Env Var: RCLONE_CACHE_PLEX_USERNAME
--cache-plex-password The password of the Plex user.
• Config: plex_password
• Env Var: RCLONE_CACHE_PLEX_PASSWORD
--cache-chunk-size The size of a chunk (partial file data).
Use lower numbers for slower connections.  If the chunk size is changed, any downloaded chunks will be invalid and cache-chunk-path will need to be cleared or unexpected EOF errors will occur.
• Env Var: RCLONE_CACHE_CHUNK_SIZE
• "1M"
• 1 MiB
• "5M"
• 5 MiB
• "10M"
• 10 MiB
--cache-info-age How long to cache file structure information (directory listings, file size, times, etc.).  If all write operations are done through the cache then you can safely make this value very large as the cache store will also be updated in real time.
• Config: info_age
• Env Var: RCLONE_CACHE_INFO_AGE
• Default: 6h0m0s
• "1h"
• 1 hour
• "24h"
• 24 hours
• "48h"
• 48 hours
--cache-chunk-total-size The total size that the chunks can take up on the local disk.
If the cache exceeds this value then it will start to delete the oldest chunks until it goes under this value.
• Config: chunk_total_size
• Env Var: RCLONE_CACHE_CHUNK_TOTAL_SIZE
• Default: 10Gi
• "500M"
• 500 MiB
• "1G"
• 1 GiB
• "10G"
• 10 GiB
Advanced options Here are the advanced options specific to cache (Cache a remote).
--cache-plex-token The plex token for authentication - auto set normally.
• Config: plex_token
• Env Var: RCLONE_CACHE_PLEX_TOKEN
--cache-plex-insecure Skip all certificate verification when connecting to the Plex server.
• Config: plex_insecure
• Env Var: RCLONE_CACHE_PLEX_INSECURE
--cache-db-path Directory to store file structure metadata DB.
The remote name is used as the DB file name.
• Config: db_path
• Env Var: RCLONE_CACHE_DB_PATH
• Default: "$HOME/.cache/rclone/cache-backend"
--cache-chunk-path Directory to cache chunk files.
Path to where partial file data (chunks) are stored locally.  The remote name is appended to the final path.
This config follows the "--cache-db-path".  If you specify a custom location for "--cache-db-path" and don't specify one for "--cache-chunk-path" then "--cache-chunk-path" will use the same path as "--cache-db-path".
• Config: chunk_path
• Env Var: RCLONE_CACHE_CHUNK_PATH
--cache-db-purge Clear all the cached data for this remote on start.
• Config: db_purge
• Env Var: RCLONE_CACHE_DB_PURGE
--cache-chunk-clean-interval How often should the cache perform cleanups of the chunk storage.
The default value should be ok for most people.  If you find that the cache goes over "cache-chunk-total-size" too often then try to lower this value to force it to perform cleanups more often.
• Config: chunk_clean_interval
• Env Var: RCLONE_CACHE_CHUNK_CLEAN_INTERVAL
--cache-read-retries How many times to retry a read from a cache storage.
Since reading from a cache stream is independent from downloading file data, readers can get to a point where there's no more data in the cache. Most of the times this can indicate a connectivity issue if cache isn't able to provide file data anymore.
For really slow connections, increase this to a point where the stream is able to provide data but your experience will be very stuttering.
• Config: read_retries
• Env Var: RCLONE_CACHE_READ_RETRIES
• Default: 10
--cache-workers How many workers should run in parallel to download chunks.
Higher values will mean more parallel processing (better CPU needed) and more concurrent requests on the cloud provider.  This impacts several aspects like the cloud provider API limits, more stress on the hardware that rclone runs on but it also means that streams will be more fluid and data will be available much more faster to readers.
Note: If the optional Plex integration is enabled then this setting will adapt to the type of reading performed and the value specified here will be used as a maximum number of workers to use.
• Config: workers
• Env Var: RCLONE_CACHE_WORKERS
--cache-chunk-no-memory Disable the in-memory cache for storing chunks during streaming.
By default, cache will keep file data during streaming in RAM as well to provide it to readers as fast as possible.
This transient data is evicted as soon as it is read and the number of chunks stored doesn't exceed the number of workers.  However, depending on other settings like "cache-chunk-size" and "cache-workers" this footprint can increase if there are parallel streams too (multiple files being read at the same time).
If the hardware permits it, use this feature to provide an overall better performance during streaming but it can also be disabled if RAM is not available on the local machine.
• Config: chunk_no_memory
• Env Var: RCLONE_CACHE_CHUNK_NO_MEMORY
--cache-rps Limits the number of requests per second to the source FS (-1 to disable).
This setting places a hard limit on the number of requests per second that cache will be doing to the cloud provider remote and try to respect that value by setting waits between reads.
If you find that you're getting banned or limited on the cloud provider through cache and know that a smaller number of requests per second will allow you to work with it then you can use this setting for that.
A good balance of all the other settings should make this setting useless but it is available to set for more special cases.
NOTE: This will limit the number of requests during streams but other API calls to the cloud provider like directory listings will still pass.
• Config: rps
• Env Var: RCLONE_CACHE_RPS
• Default: -1
--cache-writes Cache file data on writes through the FS.
If you need to read files immediately after you upload them through cache you can enable this flag to have their data stored in the cache store at the same time during upload.
• Config: writes
• Env Var: RCLONE_CACHE_WRITES
--cache-tmp-upload-path Directory to keep temporary files until they are uploaded.
This is the path where cache will use as a temporary storage for new files that need to be uploaded to the cloud provider.
Specifying a value will enable this feature.  Without it, it is completely disabled and files will be uploaded directly to the cloud provider
• Config: tmp_upload_path
• Env Var: RCLONE_CACHE_TMP_UPLOAD_PATH
--cache-tmp-wait-time How long should files be stored in local cache before being uploaded.
This is the duration that a file must wait in the temporary location cache-tmp-upload-path before it is selected for upload.
Note that only one file is uploaded at a time and it can take longer to start the upload if a queue formed for this purpose.
• Config: tmp_wait_time
• Env Var: RCLONE_CACHE_TMP_WAIT_TIME
• Default: 15s
--cache-db-wait-time How long to wait for the DB to be available - 0 is unlimited.
Only one process can have the DB open at any one time, so rclone waits for this duration for the DB to become available before it gives an error.
If you set it to 0 then it will wait forever.
• Config: db_wait_time
• Env Var: RCLONE_CACHE_DB_WAIT_TIME
• Default: 1s
Backend commands Here are the commands specific to the cache backend.
stats Print stats on the cache backend in JSON format.
rclone backend stats remote: [options] [<arguments>+]
Chunker (BETA) The chunker overlay transparently splits large files into smaller chunks during upload to wrapped remote and transparently assembles them back when the file is downloaded.  This allows to effectively overcome size limits imposed by storage providers.
Configuration To use it, first set up the underlying remote following the configuration instructions for that remote.  You can also use a local pathname instead of a remote.
First check your chosen remote is working - we'll call it remote:path here.  Note that anything inside remote:path will be chunked and anything outside won't.  This means that if you are using a bucket based remote (e.g.  S3, B2, swift) then you should probably put the bucket in the remote s3:bucket.
Now configure chunker using rclone config.  We will call this one overlay to separate it from the remote itself.
No remotes found - make a new one n) New remote s) Set configuration password q) Quit config n/s/q> n name> overlay Type of storage to configure. Choose a number from below, or type in your own value [snip] XX / Transparently chunk/split large files \ "chunker" [snip] Storage> chunker Remote to chunk/unchunk. Normally should contain a ':' and a path, e.g. "myremote:path/to/dir", "myremote:bucket" or maybe "myremote:" (not recommended). Enter a string value. Press Enter for the default (""). remote> remote:path Files larger than chunk size will be split in chunks. Enter a size with suffix K,M,G,T. Press Enter for the default ("2G"). chunk_size> 100M Choose how chunker handles hash sums. All modes but "none" require metadata. Enter a string value. Press Enter for the default ("md5"). Choose a number from below, or type in your own value 1 / Pass any hash supported by wrapped remote for non-chunked files, return nothing otherwise \ "none" 2 / MD5 for composite files \ "md5" 3 / SHA1 for composite files \ "sha1" 4 / MD5 for all files \ "md5all" 5 / SHA1 for all files \ "sha1all" 6 / Copying a file to chunker will request MD5 from the source falling back to SHA1 if unsupported \ "md5quick" 7 / Similar to "md5quick" but prefers SHA1 over MD5 \ "sha1quick" hash_type> md5 Edit advanced config? (y/n) y) Yes n) No y/n> n Remote config -------------------- [overlay] type = chunker remote = remote:bucket chunk_size = 100M hash_type = md5 -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
Specifying the remote In normal use, make sure the remote has a : in.  If you specify the remote without a : then rclone will use a local directory of that name. So if you use a remote of /path/to/secret/files then rclone will chunk stuff in that directory.  If you use a remote of name then rclone will put files in a directory called name in the current directory.
Chunking When rclone starts a file upload, chunker checks the file size.  If it doesn't exceed the configured chunk size, chunker will just pass the file to the wrapped remote.  If a file is large, chunker will transparently cut data in pieces with temporary names and stream them one by one, on the fly.  Each data chunk will contain the specified number of bytes, except for the last one which may have less data.  If file size is unknown in advance (this is called a streaming upload), chunker will internally create a temporary copy, record its size and repeat the above process.
When upload completes, temporary chunk files are finally renamed.  This scheme guarantees that operations can be run in parallel and look from outside as atomic.  A similar method with hidden temporary chunks is used for other operations (copy/move/rename, etc.).  If an operation fails, hidden chunks are normally destroyed, and the target composite file stays intact.
When a composite file download is requested, chunker transparently assembles it by concatenating data chunks in order.  As the split is trivial one could even manually concatenate data chunks together to obtain the original content.
When the list rclone command scans a directory on wrapped remote, the potential chunk files are accounted for, grouped and assembled into composite directory entries.  Any temporary chunks are hidden.
List and other commands can sometimes come across composite files with missing or invalid chunks, e.g.  shadowed by like-named directory or another file.  This usually means that wrapped file system has been directly tampered with or damaged.  If chunker detects a missing chunk it will by default print warning, skip the whole incomplete group of chunks but proceed with current command.  You can set the --chunker-fail-hard flag to have commands abort with error message in such cases.
Chunk names The default chunk name format is *.rclone_chunk.###, hence by default chunk names are BIG_FILE_NAME.rclone_chunk.001, BIG_FILE_NAME.rclone_chunk.002 etc.  You can configure another name format using the name_format configuration file option.  The format uses asterisk * as a placeholder for the base file name and one or more consecutive hash characters # as a placeholder for sequential chunk number.  There must be one and only one asterisk.  The number of consecutive hash characters defines the minimum length of a string representing a chunk number.  If decimal chunk number has less digits than the number of hashes, it is left-padded by zeros.  If the decimal string is longer, it is left intact.  By default numbering starts from 1 but there is another option that allows user to start from 0, e.g.  for compatibility with legacy software.
For example, if name format is big_*-##.part and original file name is data.txt and numbering starts from 0, then the first chunk will be named big_data.txt-00.part, the 99th chunk will be big_data.txt-98.part and the 302nd chunk will become big_data.txt-301.part.
Note that list assembles composite directory entries only when chunk names match the configured format and treats non-conforming file names as normal non-chunked files.
When using norename transactions, chunk names will additionally have a unique file version suffix.  For example, BIG_FILE_NAME.rclone_chunk.001_bp562k.
Metadata Besides data chunks chunker will by default create metadata object for a composite file.  The object is named after the original file.  Chunker allows user to disable metadata completely (the none format).  Note that metadata is normally not created for files smaller than the configured chunk size.  This may change in future rclone releases.
Simple JSON metadata format This is the default format.  It supports hash sums and chunk validation for composite files.  Meta objects carry the following fields:
• ver - version of format, currently 1
• size - total size of composite file
• nchunks - number of data chunks in file
• md5 - MD5 hashsum of composite file (if present)
• sha1 - SHA1 hashsum (if present)
• txn - identifies current version of the file
There is no field for composite file name as it's simply equal to the name of meta object on the wrapped remote.  Please refer to respective sections for details on hashsums and modified time handling.
No metadata You can disable meta objects by setting the meta format option to none. In this mode chunker will scan directory for all files that follow configured chunk name format, group them by detecting chunks with the same base name and show group names as virtual composite files.  This method is more prone to missing chunk errors (especially missing last chunk) than format with metadata enabled.
Hashsums Chunker supports hashsums only when a compatible metadata is present. Hence, if you choose metadata format of none, chunker will report hashsum as UNSUPPORTED.
Please note that by default metadata is stored only for composite files. If a file is smaller than configured chunk size, chunker will transparently redirect hash requests to wrapped remote, so support depends on that.  You will see the empty string as a hashsum of requested type for small files if the wrapped remote doesn't support it.
Many storage backends support MD5 and SHA1 hash types, so does chunker. With chunker you can choose one or another but not both.  MD5 is set by default as the most supported type.  Since chunker keeps hashes for composite files and falls back to the wrapped remote hash for non-chunked ones, we advise you to choose the same hash type as supported by wrapped remote so that your file listings look coherent.
If your storage backend does not support MD5 or SHA1 but you need consistent file hashing, configure chunker with md5all or sha1all.  These two modes guarantee given hash for all files.  If wrapped remote doesn't support it, chunker will then add metadata to all files, even small. However, this can double the amount of small files in storage and incur additional service charges.  You can even use chunker to force md5/sha1 support in any other remote at expense of sidecar meta objects by setting e.g.  chunk_type=sha1all to force hashsums and chunk_size=1P to effectively disable chunking.
Normally, when a file is copied to chunker controlled remote, chunker will ask the file source for compatible file hash and revert to on-the-fly calculation if none is found.  This involves some CPU overhead but provides a guarantee that given hashsum is available.  Also, chunker will reject a server-side copy or move operation if source and destination hashsum types are different resulting in the extra network bandwidth, too.  In some rare cases this may be undesired, so chunker provides two optional choices: sha1quick and md5quick.  If the source does not support primary hash type and the quick mode is enabled, chunker will try to fall back to the secondary type.  This will save CPU and bandwidth but can result in empty hashsums at destination.  Beware of consequences: the sync command will revert (sometimes silently) to time/size comparison if compatible hashsums between source and target are not found.
Modified time Chunker stores modification times using the wrapped remote so support depends on that.  For a small non-chunked file the chunker overlay simply manipulates modification time of the wrapped remote file.  For a composite file with metadata chunker will get and set modification time of the metadata object on the wrapped remote.  If file is chunked but metadata format is none then chunker will use modification time of the first data chunk.
Migrations The idiomatic way to migrate to a different chunk size, hash type, transaction style or chunk naming scheme is to:
• Collect all your chunked files under a directory and have your chunker remote point to it.
• Create another directory (most probably on the same cloud storage) and configure a new remote with desired metadata format, hash type, chunk naming etc.
• Now run rclone sync -i oldchunks: newchunks: and all your data will be transparently converted in transfer.  This may take some time, yet chunker will try server-side copy if possible.
• After checking data integrity you may remove configuration section of the old remote.
If rclone gets killed during a long operation on a big composite file, hidden temporary chunks may stay in the directory.  They will not be shown by the list command but will eat up your account quota.  Please note that the deletefile command deletes only active chunks of a file. As a workaround, you can use remote of the wrapped file system to see them.  An easy way to get rid of hidden garbage is to copy littered directory somewhere using the chunker remote and purge the original directory.  The copy command will copy only active chunks while the purge will remove everything including garbage.
Caveats and Limitations Chunker requires wrapped remote to support server-side move (or copy + delete) operations, otherwise it will explicitly refuse to start.  This is because it internally renames temporary chunk files to their final names when an operation completes successfully.
Chunker encodes chunk number in file name, so with default name_format setting it adds 17 characters.  Also chunker adds 7 characters of temporary suffix during operations.  Many file systems limit base file name without path by 255 characters.  Using rclone's crypt remote as a base file system limits file name by 143 characters.  Thus, maximum name length is 231 for most files and 119 for chunker-over-crypt.  A user in need can change name format to e.g.  *.rcc## and save 10 characters (provided at most 99 chunks per file).
Note that a move implemented using the copy-and-delete method may incur double charging with some cloud storage providers.
Chunker will not automatically rename existing chunks when you run rclone config on a live remote and change the chunk name format.  Beware that in result of this some files which have been treated as chunks before the change can pop up in directory listings as normal files and vice versa. The same warning holds for the chunk size.  If you desperately need to change critical chunking settings, you should run data migration as described above.
If wrapped remote is case insensitive, the chunker overlay will inherit that property (so you can't have a file called "Hello.doc" and "hello.doc" in the same directory).
Chunker included in rclone releases up to v1.54 can sometimes fail to detect metadata produced by recent versions of rclone.  We recommend users to keep rclone up-to-date to avoid data corruption.
Changing transactions is dangerous and requires explicit migration.
Standard options Here are the standard options specific to chunker (Transparently chunk/split large files).
--chunker-remote Remote to chunk/unchunk.
• Env Var: RCLONE_CHUNKER_REMOTE
--chunker-chunk-size Files larger than chunk size will be split in chunks.
• Env Var: RCLONE_CHUNKER_CHUNK_SIZE
• Default: 2Gi
--chunker-hash-type Choose how chunker handles hash sums.
All modes but "none" require metadata.
• Config: hash_type
• Env Var: RCLONE_CHUNKER_HASH_TYPE
• Default: "md5"
• "none"
• Pass any hash supported by wrapped remote for non-chunked files.
• Return nothing otherwise.
• "md5"
• MD5 for composite files.
• "sha1"
• SHA1 for composite files.
• "md5all"
• MD5 for all files.
• "sha1all"
• SHA1 for all files.
• "md5quick"
• Copying a file to chunker will request MD5 from the source.
• Falling back to SHA1 if unsupported.
• "sha1quick"
• Similar to "md5quick" but prefers SHA1 over MD5.
Advanced options Here are the advanced options specific to chunker (Transparently chunk/split large files).
--chunker-name-format String format of chunk file names.
The two placeholders are: base file name (*) and chunk number (#...). There must be one and only one asterisk and one or more consecutive hash characters.  If chunk number has less digits than the number of hashes, it is left-padded by zeros.  If there are more digits in the number, they are left as is.  Possible chunk files are ignored if their name does not match given format.
• Config: name_format
• Env Var: RCLONE_CHUNKER_NAME_FORMAT
• Default: "*.rclone_chunk.###"
--chunker-start-from Minimum valid chunk number.  Usually 0 or 1.
By default chunk numbers start from 1.
• Config: start_from
• Env Var: RCLONE_CHUNKER_START_FROM
• Default: 1
--chunker-meta-format Format of the metadata object or "none".
By default "simplejson".  Metadata is a small JSON file named after the composite file.
• Config: meta_format
• Env Var: RCLONE_CHUNKER_META_FORMAT
• Default: "simplejson"
• Do not use metadata files at all.
• Requires hash type "none".
• "simplejson"
• Simple JSON supports hash sums and chunk validation.
• It has the following fields: ver, size, nchunks, md5, sha1.
--chunker-fail-hard Choose how chunker should handle files with missing or invalid chunks.
• Config: fail_hard
• Env Var: RCLONE_CHUNKER_FAIL_HARD
• Report errors and abort current command.
• Warn user, skip incomplete file and proceed.
--chunker-transactions Choose how chunker should handle temporary files during transactions.
• Config: transactions
• Env Var: RCLONE_CHUNKER_TRANSACTIONS
• Default: "rename"
• "rename"
• Rename temporary files after a successful transaction.
• "norename"
• Leave temporary file names and write transaction ID to metadata file.
• Metadata is required for no rename transactions (meta format cannot be "none").
• If you are using norename transactions you should be careful not to downgrade Rclone
• as older versions of Rclone don't support this transaction style and will misinterpret
• files manipulated by norename transactions.
• This method is EXPERIMENTAL, don't use on production systems.
• "auto"
• Rename or norename will be used depending on capabilities of the backend.
• If meta format is set to "none", rename transactions will always be used.
Citrix ShareFile Citrix ShareFile (https://sharefile.com) is a secure file sharing and transfer service aimed as business.
Configuration The initial setup for Citrix ShareFile involves getting a token from Citrix ShareFile which you can in your browser.  rclone config walks you through it.
No remotes found - make a new one n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value XX / Citrix Sharefile \ "sharefile" Storage> sharefile ** See help for sharefile backend at: https://rclone.org/sharefile/ **
ID of the root folder
Leave blank to access "Personal Folders".  You can use one of the standard values here or any folder ID (long hex number ID). Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value 1 / Access the Personal Folders. (Default) \ "" 2 / Access the Favorites folder. \ "favorites" 3 / Access all the shared folders. \ "allshared" 4 / Access all the individual connectors. \ "connectors" 5 / Access the home, favorites, and shared folders as well as the connectors. \ "top" root_folder_id> Edit advanced config? (y/n) y) Yes n) No y/n> n Remote config Use auto config? * Say Y if not sure * Say N if you are working on a remote or headless machine y) Yes n) No y/n> y If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth?state=XXX Log in and authorize rclone for access Waiting for code... Got code -------------------- [remote] type = sharefile endpoint = https://XXX.sharefile.com token = {"access_token":"XXX","token_type":"bearer","refresh_token":"XXX","expiry":"2019-09-30T19:41:45.878561877+01:00"} -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
Note that rclone runs a webserver on your local machine to collect the token as returned from Citrix ShareFile.  This only runs from the moment it opens your browser to the moment you get back the verification code. This is on http://127.0.0.1:53682/ and this it may require you to unblock it temporarily if you are running a host firewall.
List directories in top level of your ShareFile
List all the files in your ShareFile
To copy a local directory to an ShareFile directory called backup
Modified time and hashes ShareFile allows modification times to be set on objects accurate to 1 second.  These will be used to detect whether objects need syncing or not.
ShareFile supports MD5 type hashes, so you can use the --checksum flag.
Transfers For files above 128 MiB rclone will use a chunked transfer.  Rclone will upload up to --transfers chunks at the same time (shared among all the multipart uploads).  Chunks are buffered in memory and are normally 64 MiB so increasing --transfers will increase memory use.
Character   Value   Replacement ──────────────────────────────── \           0x5C         ＼ *           0x2A         ＊ <           0x3C         ＜ >           0x3E         ＞ ?           0x3F         ？ :           0x3A         ： |           0x7C         ｜ "           0x22         ＂
Character   Value   Replacement ──────────────────────────────── SP          0x20         ␠ .           0x2E         ．
Standard options Here are the standard options specific to sharefile (Citrix Sharefile).
--sharefile-root-folder-id ID of the root folder.
Leave blank to access "Personal Folders".  You can use one of the standard values here or any folder ID (long hex number ID).
• Env Var: RCLONE_SHAREFILE_ROOT_FOLDER_ID
• Access the Personal Folders (default).
• "favorites"
• Access the Favorites folder.
• "allshared"
• Access all the shared folders.
• "connectors"
• Access all the individual connectors.
• "top"
• Access the home, favorites, and shared folders as well as the connectors.
Advanced options Here are the advanced options specific to sharefile (Citrix Sharefile).
--sharefile-upload-cutoff Cutoff for switching to multipart upload.
• Env Var: RCLONE_SHAREFILE_UPLOAD_CUTOFF
• Default: 128Mi
--sharefile-chunk-size Upload chunk size.
Must a power of 2 >= 256k.
Making this larger will improve performance, but note that each chunk is buffered in memory one per transfer.
Reducing this will reduce memory usage but decrease performance.
• Env Var: RCLONE_SHAREFILE_CHUNK_SIZE
• Default: 64Mi
--sharefile-endpoint Endpoint for API calls.
This is usually auto discovered as part of the oauth process, but can be set manually to something like: https://XXX.sharefile.com
• Env Var: RCLONE_SHAREFILE_ENDPOINT
--sharefile-encoding This sets the encoding for the backend.
• Env Var: RCLONE_SHAREFILE_ENCODING
• Default: Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,LeftSpace,LeftPeriod,RightSpace,RightPeriod,InvalidUtf8,Dot
Limitations Note that ShareFile is case insensitive so you can't have a file called "Hello.doc" and one called "hello.doc".
ShareFile only supports filenames up to 256 characters in length.
rclone about is not supported by the Citrix ShareFile backend.  Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.
Crypt Rclone crypt remotes encrypt and decrypt other remotes.
A remote of type crypt does not access a storage system (https://rclone.org/overview/) directly, but instead wraps another remote, which in turn accesses the storage system.  This is similar to how alias (https://rclone.org/alias/), union (https://rclone.org/union/), chunker (https://rclone.org/chunker/) and a few others work.  It makes the usage very flexible, as you can add a layer, in this case an encryption layer, on top of any other backend, even in multiple layers. Rclone's functionality can be used as with any other remote, for example you can mount (https://rclone.org/commands/rclone_mount/) a crypt remote.
Accessing a storage system through a crypt remote realizes client-side encryption, which makes it safe to keep your data in a location you do not trust will not get compromised.  When working against the crypt remote, rclone will automatically encrypt (before uploading) and decrypt (after downloading) on your local system as needed on the fly, leaving the data encrypted at rest in the wrapped remote.  If you access the storage system using an application other than rclone, or access the wrapped remote directly using rclone, there will not be any encryption/decryption: Downloading existing content will just give you the encrypted (scrambled) format, and anything you upload will not become encrypted.
The encryption is a secret-key encryption (also called symmetric key encryption) algorithm, where a password (or pass phrase) is used to generate real encryption key.  The password can be supplied by user, or you may chose to let rclone generate one.  It will be stored in the configuration file, in a lightly obscured form.  If you are in an environment where you are not able to keep your configuration secured, you should add configuration encryption (https://rclone.org/docs/#configuration-encryption) as protection.  As long as you have this configuration file, you will be able to decrypt your data.  Without the configuration file, as long as you remember the password (or keep it in a safe place), you can re-create the configuration and gain access to the existing data.  You may also configure a corresponding remote in a different installation to access the same data.  See below for guidance to changing password.
Encryption uses cryptographic salt (https://en.wikipedia.org/wiki/Salt_(cryptography)), to permute the encryption key so that the same string may be encrypted in different ways.  When configuring the crypt remote it is optional to enter a salt, or to let rclone generate a unique salt.  If omitted, rclone uses a built-in unique string.  Normally in cryptography, the salt is stored together with the encrypted content, and do not have to be memorized by the user.  This is not the case in rclone, because rclone does not store any additional information on the remotes.  Use of custom salt is effectively a second password that must be memorized.
File content encryption is performed using NaCl SecretBox (https://godoc.org/golang.org/x/crypto/nacl/secretbox), based on XSalsa20 cipher and Poly1305 for integrity.  Names (file- and directory names) are also encrypted by default, but this has some implications and is therefore possible to turned off.
Configuration Here is an example of how to make a remote called secret.
To use crypt, first set up the underlying remote.  Follow the rclone config instructions for the specific backend.
Before configuring the crypt remote, check the underlying remote is working.  In this example the underlying remote is called remote.  We will configure a path path within this remote to contain the encrypted content.  Anything inside remote:path will be encrypted and anything outside will not.
Configure crypt using rclone config.  In this example the crypt remote is called secret, to differentiate it from the underlying remote.
When you are done you can use the crypt remote named secret just as you would with any other remote, e.g.  rclone copy D:\docs secret:\docs, and rclone will encrypt and decrypt as needed on the fly.  If you access the wrapped remote remote:path directly you will bypass the encryption, and anything you read will be in encrypted form, and anything you write will be unencrypted.  To avoid issues it is best to configure a dedicated path for encrypted content, and access it exclusively through a crypt remote.
No remotes found - make a new one n) New remote s) Set configuration password q) Quit config n/s/q> n name> secret Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [snip] XX / Encrypt/Decrypt a remote \ "crypt" [snip] Storage> crypt ** See help for crypt backend at: https://rclone.org/crypt/ **
Remote to encrypt/decrypt. Normally should contain a ':' and a path, eg "myremote:path/to/dir", "myremote:bucket" or maybe "myremote:" (not recommended). Enter a string value. Press Enter for the default (""). remote> remote:path How to encrypt the filenames. Enter a string value. Press Enter for the default ("standard"). Choose a number from below, or type in your own value. / Encrypt the filenames. 1 | See the docs for the details. \ "standard" 2 / Very simple filename obfuscation. \ "obfuscate" / Don't encrypt the file names. 3 | Adds a ".bin" extension only. \ "off" filename_encryption> Option to either encrypt directory names or leave them intact.
NB If filename_encryption is "off" then this option will do nothing. Enter a boolean value (true or false). Press Enter for the default ("true"). Choose a number from below, or type in your own value 1 / Encrypt directory names. \ "true" 2 / Don't encrypt directory names, leave them intact. \ "false" directory_name_encryption> Password or pass phrase for encryption. y) Yes type in my own password g) Generate random password y/g> y Enter the password: password: Confirm the password: password: Password or pass phrase for salt. Optional but recommended. Should be different to the previous password. y) Yes type in my own password g) Generate random password n) No leave this optional password blank (default) y/g/n> g Password strength in bits. 64 is just about memorable 128 is secure 1024 is the maximum Bits> 128 Your password is: JAsJvRcgR-_veXNfy_sGmQ Use this password? Please note that an obscured version of this password (and not the password itself) will be stored under your configuration file, so keep this generated password in a safe place. y) Yes (default) n) No y/n> Edit advanced config? (y/n) y) Yes n) No (default) y/n> Remote config -------------------- [secret] type = crypt remote = remote:path password = *** ENCRYPTED *** password2 = *** ENCRYPTED *** -------------------- y) Yes this is OK (default) e) Edit this remote d) Delete this remote y/e/d>
Important The crypt password stored in rclone.conf is lightly obscured. That only protects it from cursory inspection.  It is not secure unless configuration encryption (https://rclone.org/docs/#configuration- encryption) of rclone.conf is specified.
A long passphrase is recommended, or rclone config can generate a random one.
The obscured password is created using AES-CTR with a static key.  The salt is stored verbatim at the beginning of the obscured password.  This static key is shared between all versions of rclone.
If you reconfigure rclone with the same passwords/passphrases elsewhere it will be compatible, but the obscured version will be different due to the different salt.
Rclone does not encrypt
• file length - this can be calculated within 16 bytes
• modification time - used for syncing
Specifying the remote When configuring the remote to encrypt/decrypt, you may specify any string that rclone accepts as a source/destination of other commands.
The primary use case is to specify the path into an already configured remote (e.g.  remote:path/to/dir or remote:bucket), such that data in a remote untrusted location can be stored encrypted.
You may also specify a local filesystem path, such as /path/to/dir on Linux, C:\path\to\dir on Windows.  By creating a crypt remote pointing to such a local filesystem path, you can use rclone as a utility for pure local file encryption, for example to keep encrypted files on a removable USB drive.
Note: A string which do not contain a : will by rclone be treated as a relative path in the local filesystem.  For example, if you enter the name remote without the trailing :, it will be treated as a subdirectory of the current directory with name "remote".