neulab/tldr huggingface dataset:
Name               Hash       ModTime   Case          Duplicate   MIME Type Insensitive   Files ──────────────────────────────────────────────────────────────────────────── 1Fichier         Whirlpool      No          No           Yes          R Amazon              MD5         No          Yes          No           R Drive Amazon S3           MD5         Yes         No           No          R/W (or S3 compatible) Backblaze          SHA1         Yes         No           No          R/W B2 Box                SHA1         Yes         Yes          No           - Citrix              MD5         Yes         Yes          No           - ShareFile Dropbox          DBHASH ¹       Yes         Yes          No           - Enterprise           -          Yes         Yes          No          R/W File Fabric FTP                  -          No          No           No           - Google              MD5         Yes         No           No          R/W Cloud Storage Google              MD5         Yes         No           Yes         R/W Drive Google               -          No          No           Yes          R Photos HDFS                 -          Yes         No           No           - HTTP                 -          No          No           No           R Hubic               MD5         Yes         No           No          R/W Jottacloud          MD5         Yes         Yes          No           R Koofr               MD5         No          Yes          No           - Mail.ru          Mailru ⁶       Yes         Yes          No           - Cloud Mega                 -          No          No           Yes          - Memory              MD5         Yes         No           No           - Microsoft           MD5         Yes         No           No          R/W Azure Blob Storage Microsoft         SHA1 ⁵        Yes         Yes          No           R OneDrive OpenDrive           MD5         Yes         Yes       Partial ⁸       -
OpenStack           MD5         Yes         No           No          R/W Swift pCloud          MD5, SHA1 ⁷     Yes         No           No           W premiumize.me        -          No          Yes          No           R put.io            CRC-32        Yes         No           Yes          R QingStor            MD5         No          No           No          R/W Seafile              -          No          No           No           - SFTP            MD5, SHA1 ²     Yes       Depends        No           - Sia                  -          No          No           No           - SugarSync            -          No          No           No           - Tardigrade           -          Yes         No           No           - Uptobox              -          No          No           Yes          - WebDAV          MD5, SHA1 ³    Yes ⁴      Depends        No           - Yandex Disk         MD5         Yes         No           No           R Zoho                 -          No          No           No           - WorkDrive The local           All         Yes       Depends        No           - filesystem
Notes ¹ Dropbox supports its own custom hash (https://www.dropbox.com/developers/reference/content-hash).  This is an SHA256 sum of all the 4 MiB block SHA256s.
² SFTP supports checksums if the same login has shell access and md5sum or sha1sum as well as echo are in the remote's PATH.
³ WebDAV supports hashes when used with Owncloud and Nextcloud only.
⁴ WebDAV supports modtimes when used with Owncloud and Nextcloud only.
⁵ Microsoft OneDrive Personal supports SHA1 hashes, whereas OneDrive for business and SharePoint server support Microsoft's own QuickXorHash (https://docs.microsoft.com/en-us/onedrive/developer/code- snippets/quickxorhash).
⁶ Mail.ru uses its own modified SHA1 hash
⁷ pCloud only supports SHA1 (not MD5) in its EU region
⁸ Opendrive does not support creation of duplicate files using their web client interface or other stock clients, but the underlying storage platform has been determined to allow duplicate files, and it is possible to create them with rclone.  It may be that this is a mistake or an unsupported feature.
Hash The cloud storage system supports various hash types of the objects.  The hashes are used when transferring data as an integrity check and can be specifically used with the --checksum flag in syncs and in the check command.
To use the verify checksums when transferring between cloud storage systems they must support a common hash type.
ModTime The cloud storage system supports setting modification times on objects. If it does then this enables a using the modification times as part of the sync.  If not then only the size will be checked by default, though the MD5SUM can be checked with the --checksum flag.
All cloud storage systems support some kind of date on the object and these will be set when transferring from the cloud storage system.
Case Insensitive If a cloud storage systems is case sensitive then it is possible to have two files which differ only in case, e.g.  file.txt and FILE.txt.  If a cloud storage system is case insensitive then that isn't possible.
This can cause problems when syncing between a case insensitive system and a case sensitive system.  The symptom of this is that no matter how many times you run the sync it never completes fully.
The local filesystem and SFTP may or may not be case sensitive depending on OS.
• Windows - usually case insensitive, though case is preserved
• OSX - usually case insensitive, though it is possible to format case sensitive
• Linux - usually case sensitive, but there are case insensitive file systems (e.g.  FAT formatted USB keys)
Most of the time this doesn't cause any problems as people tend to avoid files whose name differs only by case even on case sensitive systems.
Duplicate files If a cloud storage system allows duplicate files then it can have two objects with the same name.
This confuses rclone greatly when syncing - use the rclone dedupe command to rename or remove duplicates.
Restricted filenames Some cloud storage systems might have restrictions on the characters that are usable in file or directory names.  When rclone detects such a name during a file upload, it will transparently replace the restricted characters with similar looking Unicode characters.  To handle the different sets of restricted characters for different backends, rclone uses something it calls encoding.
This process is designed to avoid ambiguous file names as much as possible and allow to move files between many cloud storage systems transparently.
The name shown by rclone to the user or during log output will only contain a minimal set of replaced characters to ensure correct formatting and not necessarily the actual name used on the cloud storage.
This transformation is reversed when downloading a file or parsing rclone arguments.  For example, when uploading a file named my file?.txt to Onedrive, it will be displayed as my file?.txt on the console, but stored as my file？.txt to Onedrive (the ?  gets replaced by the similar looking ？ character, the so-called "fullwidth question mark").  The reverse transformation allows to read a file unusual/name.txt from Google Drive, by passing the name unusual／name.txt on the command line (the / needs to be replaced by the similar looking ／ character).
Caveats The filename encoding system works well in most cases, at least where file names are written in English or similar languages.  You might not even notice it: It just works.  In some cases it may lead to issues, though.  E.g.  when file names are written in Chinese, or Japanese, where it is always the Unicode fullwidth variants of the punctuation marks that are used.
On Windows, the characters :, * and ? are examples of restricted characters.  If these are used in filenames on a remote that supports it, Rclone will transparently convert them to their fullwidth Unicode variants ＊, ？ and ： when downloading to Windows, and back again when uploading.  This way files with names that are not allowed on Windows can still be stored.
However, if you have files on your Windows system originally with these same Unicode characters in their names, they will be included in the same conversion process.  E.g.  if you create a file in your Windows filesystem with name Test：1.jpg, where ： is the Unicode fullwidth colon symbol, and use rclone to upload it to Google Drive, which supports regular : (halfwidth question mark), rclone will replace the fullwidth : with the halfwidth : and store the file as Test:1.jpg in Google Drive. Since both Windows and Google Drive allows the name Test：1.jpg, it would probably be better if rclone just kept the name as is in this case.
With the opposite situation; if you have a file named Test:1.jpg, in your Google Drive, e.g.  uploaded from a Linux system where : is valid in file names.  Then later use rclone to copy this file to your Windows computer you will notice that on your local disk it gets renamed to Test：1.jpg. The original filename is not legal on Windows, due to the :, and rclone therefore renames it to make the copy possible.  That is all good. However, this can also lead to an issue: If you already had a different file named Test：1.jpg on Windows, and then use rclone to copy either way. Rclone will then treat the file originally named Test:1.jpg on Google Drive and the file originally named Test：1.jpg on Windows as the same file, and replace the contents from one with the other.
Its virtually impossible to handle all cases like these correctly in all situations, but by customizing the encoding option, changing the set of characters that rclone should convert, you should be able to create a configuration that works well for your specific situation.  See also the example (https://rclone.org/overview/#encoding-example-windows) below.
(Windows was used as an example of a file system with many restricted characters, and Google drive a storage system with few.)
Default restricted characters The table below shows the characters that are replaced by default.
When a replacement character is found in a filename, this character will be escaped with the ‛ character to avoid ambiguous file names.  (e.g.  a file named ␀.txt would shown as ‛␀.txt)
Each cloud storage backend can use a different set of characters, which will be specified in the documentation for each backend.
Character   Value   Replacement ──────────────────────────────── NUL         0x00         ␀ SOH         0x01         ␁ STX         0x02         ␂ ETX         0x03         ␃ EOT         0x04         ␄ ENQ         0x05         ␅ ACK         0x06         ␆ BEL         0x07         ␇ BS          0x08         ␈ HT          0x09         ␉
LF          0x0A         ␊ VT          0x0B         ␋ FF          0x0C         ␌ CR          0x0D         ␍ SO          0x0E         ␎ SI          0x0F         ␏ DLE         0x10         ␐ DC1         0x11         ␑ DC2         0x12         ␒ DC3         0x13         ␓ DC4         0x14         ␔ NAK         0x15         ␕ SYN         0x16         ␖ ETB         0x17         ␗ CAN         0x18         ␘ EM          0x19         ␙ SUB         0x1A         ␚ ESC         0x1B         ␛ FS          0x1C         ␜ GS          0x1D         ␝ RS          0x1E         ␞ US          0x1F         ␟ /           0x2F         ／ DEL         0x7F         ␡
The default encoding will also encode these file names as they are problematic with many cloud storage systems.
File name   Replacement ──────────────────────── .                ． ..              ．．
Invalid UTF-8 bytes Some backends only support a sequence of well formed UTF-8 bytes as file or directory names.
In this case all invalid UTF-8 bytes will be replaced with a quoted representation of the byte value to allow uploading a file to such a backend.  For example, the invalid byte 0xFE will be encoded as ‛FE.
A common source of invalid UTF-8 bytes are local filesystems, that store names in a different encoding than UTF-8 or UTF-16, like latin1.  See the local filenames (https://rclone.org/local/#filenames) section for details.
Encoding option Most backends have an encoding option, specified as a flag --backend-encoding where backend is the name of the backend, or as a config parameter encoding (you'll need to select the Advanced config in rclone config to see it).
This will have default value which encodes and decodes characters in such a way as to preserve the maximum number of characters (see above).
However this can be incorrect in some scenarios, for example if you have a Windows file system with Unicode fullwidth characters ＊, ？ or ：, that you want to remain as those characters on the remote rather than being translated to regular (halfwidth) *, ? and :.
The --backend-encoding flags allow you to change that.  You can disable the encoding completely with --backend-encoding None or set encoding = None in the config file.
Encoding takes a comma separated list of encodings.  You can see the list of all possible values by passing an invalid value to this flag, e.g. --local-encoding "help".  The command rclone help flags encoding will show you the defaults for the backends.
Encoding        Characters ─────────────────────────────────────────── Asterisk        * BackQuote       ` BackSlash       \ Colon           : CrLf            CR 0x0D, LF 0x0A Ctl             All control characters 0x00-0x1F Del             DEL 0x7F Dollar          $ Dot             . or .. as entire string DoubleQuote     " Hash            # InvalidUtf8     An invalid UTF-8 character (e.g.  latin1) LeftCrLfHtVt    CR 0x0D, LF 0x0A,HT 0x09, VT 0x0B on the left of a string LeftPeriod      . on the left of a string LeftSpace       SPACE on the left of a string LeftTilde       ~ on the left of a string LtGt            <, >
None            No characters are encoded Percent         % Pipe            | Question        ? RightCrLfHtVt   CR 0x0D, LF 0x0A, HT 0x09, VT 0x0B on the right of a string RightPeriod     . on the right of a string RightSpace      SPACE on the right of a string SingleQuote     ' Slash           / SquareBracket   [, ]
Encoding example: FTP To take a specific example, the FTP backend's default encoding is
--ftp-encoding "Slash,Del,Ctl,RightSpace,Dot"
However, let's say the FTP server is running on Windows and can't have any of the invalid Windows characters in file names.  You are backing up Linux servers to this FTP server which do have those characters in file names.  So you would add the Windows set which are
Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot
to the existing ones, giving:
Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot,Del,RightSpace
This can be specified using the --ftp-encoding flag or using an encoding parameter in the config file.
Encoding example: Windows As a nother example, take a Windows system where there is a file with name Test：1.jpg, where ： is the Unicode fullwidth colon symbol.  When using rclone to copy this to a remote which supports :, the regular (halfwidth) colon (such as Google Drive), you will notice that the file gets renamed to Test:1.jpg.
To avoid this you can change the set of characters rclone should convert for the local filesystem, using command-line argument --local-encoding. Rclone's default behavior on Windows corresponds to
--local-encoding "Slash,LtGt,DoubleQuote,Colon,Question,Asterisk,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"
If you want to use fullwidth characters ：, ＊ and ？ in your filenames without rclone changing them when uploading to a remote, then set the same as the default value but without Colon,Question,Asterisk:
--local-encoding "Slash,LtGt,DoubleQuote,Pipe,BackSlash,Ctl,RightSpace,RightPeriod,InvalidUtf8,Dot"
Alternatively, you can disable the conversion of any characters with --local-encoding None.
Instead of using command-line argument --local-encoding, you may also set it as environment variable (https://rclone.org/docs/#environment- variables) RCLONE_LOCAL_ENCODING, or configure (https://rclone.org/docs/#configure) a remote of type local in your config, and set the encoding option there.
The risk by doing this is that if you have a filename with the regular (halfwidth) :, * and ? in your cloud storage, and you try to download it to your Windows filesystem, this will fail.  These characters are not valid in filenames on Windows, and you have told rclone not to work around this by converting them to valid fullwidth variants.
MIME Type MIME types (also known as media types) classify types of documents using a simple text classification, e.g.  text/html or application/pdf.
Some cloud storage systems support reading (R) the MIME type of objects and some support writing (W) the MIME type of objects.
The MIME type can be important if you are serving files directly to HTTP from the storage system.
If you are copying from a remote which supports reading (R) to a remote which supports writing (W) then rclone will preserve the MIME types. Otherwise they will be guessed from the extension, or the remote itself may assign the MIME type.
Optional Features All rclone remotes support a base command set.  Other features depend upon backend specific capabilities.
Name            Purge   Copy   Move   DirMove   CleanUp   ListR   StreamUpload   LinkSharing   About   EmptyDir ──────────────────────────────────────────────────────────────────────────────────────────────────────────────── 1Fichier         No     Yes    Yes      No        No       No          No            Yes        No       Yes Amazon           Yes     No    Yes      Yes       No       No          No            No         No       Yes Drive Amazon           No     Yes     No      No        Yes      Yes        Yes            Yes        No        No S3
Purge This deletes a directory quicker than just deleting all the files in the directory.
† Note Swift, Hubic, and Tardigrade implement this in order to delete directory markers but they don't actually have a quicker way of deleting files other than deleting them individually.
‡ StreamUpload is not supported with Nextcloud
Copy Used when copying an object to and from the same remote.  This known as a server-side copy so you can copy a file without downloading it and uploading it again.  It is used if you use rclone copy or rclone move if the remote doesn't support Move directly.
If the server doesn't support Copy directly then for copy operations the file is downloaded then re-uploaded.
Move Used when moving/renaming an object on the same remote.  This is known as a server-side move of a file.  This is used in rclone move if the server doesn't support DirMove.
If the server isn't capable of Move then rclone simulates it with Copy then delete.  If the server doesn't support Copy then rclone will download the file and re-upload it.
DirMove This is used to implement rclone move to move a directory if possible. If it isn't then it will use Move on each file (which falls back to Copy then download and upload - see Move section).
CleanUp This is used for emptying the trash for a remote by rclone cleanup.
If the server can't do CleanUp then rclone cleanup will return an error.
‡‡ Note that while Box implements this it has to delete every file individually so it will be slower than emptying the trash via the WebUI
ListR The remote supports a recursive list to list all the contents beneath a directory quickly.  This enables the --fast-list flag to work.  See the rclone docs (https://rclone.org/docs/#fast-list) for more details.
StreamUpload Some remotes allow files to be uploaded without knowing the file size in advance.  This allows certain operations to work without spooling the file to local disk first, e.g.  rclone rcat.
LinkSharing Sets the necessary permissions on a file or folder and prints a link that allows others to access them, even if they don't have an account on the particular cloud provider.
About Rclone about prints quota information for a remote.  Typical output includes bytes used, free, quota and in trash.
If a remote lacks about capability rclone about remote:returns an error.
Backends without about capability cannot determine free space for an rclone mount, or use policy mfs (most free space) as a member of an rclone union remote.
See rclone about command (https://rclone.org/commands/rclone_about/)
EmptyDir The remote supports empty directories.  See Limitations (https://rclone.org/bugs/#limitations) for details.  Most Object/Bucket based remotes do not support this.
Global Flags This describes the global flags available to every rclone command split into two groups, non backend and backend flags.
Non Backend Flags These flags are available for every command.
Docker Volume Plugin Introduction Docker 1.9 has added support for creating named volumes (https://docs.docker.com/storage/volumes/) via command-line interface (https://docs.docker.com/engine/reference/commandline/volume_create/) and mounting them in containers as a way to share data between them.  Since Docker 1.10 you can create named volumes with Docker Compose (https://docs.docker.com/compose/) by descriptions in docker-compose.yml (https://docs.docker.com/compose/compose-file/compose-file-v2/#volume- configuration-reference) files for use by container groups on a single host.  As of Docker 1.12 volumes are supported by Docker Swarm (https://docs.docker.com/engine/swarm/key-concepts/) included with Docker Engine and created from descriptions in swarm compose v3 (https://docs.docker.com/compose/compose-file/compose-file-v3/#volume- configuration-reference) files for use with swarm stacks across multiple cluster nodes.
Docker Volume Plugins (https://docs.docker.com/engine/extend/plugins_volume/) augment the default local volume driver included in Docker with stateful volumes shared across containers and hosts.  Unlike local volumes, your data will not be deleted when such volume is removed.  Plugins can run managed by the docker daemon, as a native system service (under systemd, sysv or upstart) or as a standalone executable.  Rclone can run as docker volume plugin in all these modes.  It interacts with the local docker daemon via plugin API (https://docs.docker.com/engine/extend/plugin_api/) and handles mounting of remote file systems into docker containers so it must run on the same host as the docker daemon or on every Swarm node.
Getting started In the first example we will use the SFTP (https://rclone.org/sftp/) rclone volume with Docker engine on a standalone Ubuntu machine.
Start from installing Docker (https://docs.docker.com/engine/install/) on the host.
The FUSE driver is a prerequisite for rclone mounting and should be installed on host:
sudo apt-get -y install fuse
Create two directories required by rclone docker plugin:
sudo mkdir -p /var/lib/docker-plugins/rclone/config sudo mkdir -p /var/lib/docker-plugins/rclone/cache
Install the managed rclone docker plugin for your architecture (here amd64):
docker plugin install rclone/docker-volume-rclone:amd64 args="-v" --alias rclone --grant-all-permissions docker plugin list
Create your SFTP volume (https://rclone.org/sftp/#standard-options):
docker volume create firstvolume -d rclone -o type=sftp -o sftp-host=_hostname_ -o sftp-user=_username_ -o sftp-pass=_password_ -o allow-other=true
Note that since all options are static, you don't even have to run rclone config or create the rclone.conf file (but the config directory should still be present).  In the simplest case you can use localhost as hostname and your SSH credentials as username and password.  You can also change the remote path to your home directory on the host, for example -o path=/home/username.
Time to create a test container and mount the volume into it:
docker run --rm -it -v firstvolume:/mnt --workdir /mnt ubuntu:latest bash
If all goes well, you will enter the new container and change right to the mounted SFTP remote.  You can type ls to list the mounted directory or otherwise play with it.  Type exit when you are done.  The container will stop but the volume will stay, ready to be reused.  When it's not needed anymore, remove it:
docker volume list docker volume remove firstvolume
Now let us try something more elaborate: Google Drive (https://rclone.org/drive/) volume on multi-node Docker Swarm.
You should start from installing Docker and FUSE, creating plugin directories and installing rclone plugin on every swarm node.  Then setup the Swarm (https://docs.docker.com/engine/swarm/swarm-mode/).
Google Drive volumes need an access token which can be setup via web browser and will be periodically renewed by rclone.  The managed plugin cannot run a browser so we will use a technique similar to the rclone setup on a headless box (https://rclone.org/remote_setup/).
Run rclone config (https://rclone.org/commands/rclone_config_create/) on another machine equipped with web browser and graphical user interface. Create the Google Drive remote (https://rclone.org/drive/#standard- options).  When done, transfer the resulting rclone.conf to the Swarm cluster and save as /var/lib/docker-plugins/rclone/config/rclone.conf on every node.  By default this location is accessible only to the root user so you will need appropriate privileges.  The resulting config will look like this:
[gdrive] type = drive scope = drive drive_id = 1234567... root_folder_id = 0Abcd... token = {"access_token":...}
Now create the file named example.yml with a swarm stack description like this:
version: '3' services: heimdall: image: linuxserver/heimdall:latest ports: [8080:80] volumes: [configdata:/config] volumes: configdata: driver: rclone driver_opts: remote: 'gdrive:heimdall' allow_other: 'true' vfs_cache_mode: full poll_interval: 0
and run the stack:
docker stack deploy example -c ./example.yml
After a few seconds docker will spread the parsed stack description over cluster, create the example_heimdall service on port 8080, run service containers on one or more cluster nodes and request the example_configdata volume from rclone plugins on the node hosts.  You can use the following commands to confirm results:
docker service ls docker service ps example_heimdall docker volume ls
Point your browser to http://cluster.host.address:8080 and play with the service.  Stop it with docker stack remove example when you are done. Note that the example_configdata volume(s) created on demand at the cluster nodes will not be automatically removed together with the stack but stay for future reuse.  You can remove them manually by invoking the docker volume remove example_configdata command on every node.
Creating Volumes via CLI Volumes can be created with docker volume create (https://docs.docker.com/engine/reference/commandline/volume_create/). Here are a few examples:
docker volume create vol1 -d rclone -o remote=storj: -o vfs-cache-mode=full docker volume create vol2 -d rclone -o remote=:tardigrade,access_grant=xxx:heimdall docker volume create vol3 -d rclone -o type=tardigrade -o path=heimdall -o tardigrade-access-grant=xxx -o poll-interval=0
Note the -d rclone flag that tells docker to request volume from the rclone driver.  This works even if you installed managed driver by its full name rclone/docker-volume-rclone because you provided the --alias rclone option.
Volumes can be inspected as follows:
docker volume list docker volume inspect vol1
Volume Configuration Rclone flags and volume options are set via the -o flag to the docker volume create command.  They include backend-specific parameters as well as mount and VFS options.  Also there are a few special -o options: remote, fs, type, path, mount-type and persist.
remote determines an existing remote name from the config file, with trailing colon and optionally with a remote path.  See the full syntax in the rclone documentation (https://rclone.org/docs/#syntax-of-remote- paths).  This option can be aliased as fs to prevent confusion with the remote parameter of such backends as crypt or alias.
The remote=:backend:dir/subdir syntax can be used to create on-the-fly (config-less) remotes (https://rclone.org/docs/#backend-path-to-dir), while the type and path options provide a simpler alternative for this. Using two split options
-o type=backend -o path=dir/subdir
is equivalent to the combined syntax
-o remote=:backend:dir/subdir
but is arguably easier to parameterize in scripts.  The path part is optional.
Mount and VFS options (https://rclone.org/commands/rclone_serve_docker/#options) as well as backend parameters (https://rclone.org/flags/#backend-flags) are named like their twin command-line flags without the -- CLI prefix.  Optionally you can use underscores instead of dashes in option names.  For example, --vfs-cache-mode full becomes -o vfs-cache-mode=full or -o vfs_cache_mode=full.  Boolean CLI flags without value will gain the true value, e.g.  --allow-other becomes -o allow-other=true or -o allow_other=true.
Please note that you can provide parameters only for the backend immediately referenced by the backend type of mounted remote.  If this is a wrapping backend like alias, chunker or crypt, you cannot provide options for the referred to remote or backend.  This limitation is imposed by the rclone connection string parser.  The only workaround is to feed plugin with rclone.conf or configure plugin arguments (see below).
Special Volume Options mount-type determines the mount method and in general can be one of: mount, cmount, or mount2.  This can be aliased as mount_type.  It should be noted that the managed rclone docker plugin currently does not support the cmount method and mount2 is rarely needed.  This option defaults to the first found method, which is usually mount so you generally won't need it.
persist is a reserved boolean (true/false) option.  In future it will allow to persist on-the-fly remotes in the plugin rclone.conf file.
Connection Strings The remote value can be extended with connection strings (https://rclone.org/docs/#connection-strings) as an alternative way to supply backend parameters.  This is equivalent to the -o backend options with one syntactic difference.  Inside connection string the backend prefix must be dropped from parameter names but in the -o param=value array it must be present.  For instance, compare the following option array
-o remote=:sftp:/home -o sftp-host=localhost
with equivalent connection string:
-o remote=:sftp,host=localhost:/home
This difference exists because flag options -o key=val include not only backend parameters but also mount/VFS flags and possibly other settings. Also it allows to discriminate the remote option from the crypt-remote (or similarly named backend parameters) and arguably simplifies scripting due to clearer value substitution.
Using with Swarm or Compose Both Docker Swarm and Docker Compose use YAML (http://yaml.org/spec/1.2/spec.html)-formatted text files to describe groups (stacks) of containers, their properties, networks and volumes. Compose uses the compose v2 (https://docs.docker.com/compose/compose- file/compose-file-v2/#volume-configuration-reference) format, Swarm uses the compose v3 (https://docs.docker.com/compose/compose-file/compose- file-v3/#volume-configuration-reference) format.  They are mostly similar, differences are explained in the docker documentation (https://docs.docker.com/compose/compose-file/compose- versioning/#upgrading).
Volumes are described by the children of the top-level volumes: node. Each of them should be named after its volume and have at least two elements, the self-explanatory driver: rclone value and the driver_opts: structure playing the same role as -o key=val CLI flags:
volumes: volume_name_1: driver: rclone driver_opts: remote: 'gdrive:' allow_other: 'true' vfs_cache_mode: full token: '{"type": "borrower", "expires": "2021-12-31"}' poll_interval: 0
Notice a few important details: - YAML prefers _ in option names instead of -.  - YAML treats single and double quotes interchangeably.  Simple strings and integers can be left unquoted.  - Boolean values must be quoted like 'true' or "false" because these two words are reserved by YAML.  - The filesystem string is keyed with remote (or with fs). Normally you can omit quotes here, but if the string ends with colon, you must quote it like remote: "storage_box:".  - YAML is picky about surrounding braces in values as this is in fact another syntax for key/value mappings (http://yaml.org/spec/1.2/spec.html#id2790832).  For example, JSON access tokens usually contain double quotes and surrounding braces, so you must put them in single quotes.
Installing as Managed Plugin Docker daemon can install plugins from an image registry and run them managed.  We maintain the docker-volume-rclone (https://hub.docker.com/p/rclone/docker-volume-rclone/) plugin image on Docker Hub (https://hub.docker.com).
Rclone volume plugin requires Docker Engine >= 19.03.15
The plugin requires presence of two directories on the host before it can be installed.  Note that plugin will not create them automatically.  By default they must exist on host at the following locations (though you can tweak the paths): - /var/lib/docker-plugins/rclone/config is reserved for the rclone.conf config file and must exist even if it's empty and the config file is not present.  - /var/lib/docker-plugins/rclone/cache holds the plugin state file as well as optional VFS caches.
You can install managed plugin (https://docs.docker.com/engine/reference/commandline/plugin_install/) with default settings as follows:
docker plugin install rclone/docker-volume-rclone:amd64 --grant-all-permissions --alias rclone
The :amd64 part of the image specification after colon is called a tag. Usually you will want to install the latest plugin for your architecture. In this case the tag will just name it, like amd64 above.  The following plugin architectures are currently available: - amd64 - arm64 - arm-v7
Sometimes you might want a concrete plugin version, not the latest one. Then you should use image tag in the form :ARCHITECTURE-VERSION.  For example, to install plugin version v1.56.2 on architecture arm64 you will use tag arm64-1.56.2 (note the removed v) so the full image specification becomes rclone/docker-volume-rclone:arm64-1.56.2.
We also provide the latest plugin tag, but since docker does not support multi-architecture plugins as of the time of this writing, this tag is currently an alias for amd64.  By convention the latest tag is the default one and can be omitted, thus both rclone/docker-volume-rclone:latest and just rclone/docker-volume-rclone will refer to the latest plugin release for the amd64 platform.
Also the amd64 part can be omitted from the versioned rclone plugin tags. For example, rclone image reference rclone/docker-volume-rclone:amd64-1.56.2 can be abbreviated as rclone/docker-volume-rclone:1.56.2 for convenience.  However, for non-intel architectures you still have to use the full tag as amd64 or latest will fail to start.
Managed plugin is in fact a special container running in a namespace separate from normal docker containers.  Inside it runs the rclone serve docker command.  The config and cache directories are bind-mounted into the container at start.  The docker daemon connects to a unix socket created by the command inside the container.  The command creates on-demand remote mounts right inside, then docker machinery propagates them through kernel mount namespaces and bind-mounts into requesting user containers.
You can tweak a few plugin settings after installation when it's disabled (not in use), for instance:
docker plugin disable rclone docker plugin set rclone RCLONE_VERBOSE=2 config=/etc/rclone args="--vfs-cache-mode=writes --allow-other" docker plugin enable rclone docker plugin inspect rclone
Note that if docker refuses to disable the plugin, you should find and remove all active volumes connected with it as well as containers and swarm services that use them.  This is rather tedious so please carefully plan in advance.
You can tweak the following settings: args, config, cache, HTTP_PROXY, HTTPS_PROXY, NO_PROXY and RCLONE_VERBOSE.  It's your task to keep plugin settings in sync across swarm cluster nodes.
args sets command-line arguments for the rclone serve docker command (none by default).  Arguments should be separated by space so you will normally want to put them in quotes on the docker plugin set (https://docs.docker.com/engine/reference/commandline/plugin_set/) command line.  Both serve docker flags (https://rclone.org/commands/rclone_serve_docker/#options) and generic rclone flags (https://rclone.org/flags/) are supported, including backend parameters that will be used as defaults for volume creation.  Note that plugin will fail (due to this docker bug (https://github.com/moby/moby/blob/v20.10.7/plugin/v2/plugin.go#L195)) if the args value is empty.  Use e.g.  args="-v" as a workaround.
config=/host/dir sets alternative host location for the config directory. Plugin will look for rclone.conf here.  It's not an error if the config file is not present but the directory must exist.  Please note that plugin can periodically rewrite the config file, for example when it renews storage access tokens.  Keep this in mind and try to avoid races between the plugin and other instances of rclone on the host that might try to change the config simultaneously resulting in corrupted rclone.conf.  You can also put stuff like private key files for SFTP remotes in this directory.  Just note that it's bind-mounted inside the plugin container at the predefined path /data/config.  For example, if your key file is named sftp-box1.key on the host, the corresponding volume config option should read -o sftp-key-file=/data/config/sftp-box1.key.
cache=/host/dir sets alternative host location for the cache directory. The plugin will keep VFS caches here.  Also it will create and maintain the docker-plugin.state file in this directory.  When the plugin is restarted or reinstalled, it will look in this file to recreate any volumes that existed previously.  However, they will not be re-mounted into consuming containers after restart.  Usually this is not a problem as the docker daemon normally will restart affected user containers after failures, daemon restarts or host reboots.
RCLONE_VERBOSE sets plugin verbosity from 0 (errors only, by default) to 2 (debugging).  Verbosity can be also tweaked via args="-v [-v] ...". Since arguments are more generic, you will rarely need this setting.  The plugin output by default feeds the docker daemon log on local host.  Log entries are reflected as errors in the docker log but retain their actual level assigned by rclone in the encapsulated message string.
HTTP_PROXY, HTTPS_PROXY, NO_PROXY customize the plugin proxy settings.
You can set custom plugin options right when you install it, in one go:
docker plugin remove rclone docker plugin install rclone/docker-volume-rclone:amd64 \ --alias rclone --grant-all-permissions \ args="-v --allow-other" config=/etc/rclone docker plugin inspect rclone
Healthchecks The docker plugin volume protocol doesn't provide a way for plugins to inform the docker daemon that a volume is (un-)available.  As a workaround you can setup a healthcheck to verify that the mount is responding, for example:
services: my_service: image: my_image healthcheck: test: ls /path/to/rclone/mount || exit 1 interval: 1m timeout: 15s retries: 3 start_period: 15s
Running Plugin under Systemd In most cases you should prefer managed mode.  Moreover, MacOS and Windows do not support native Docker plugins.  Please use managed mode on these systems.  Proceed further only if you are on Linux.
First, install rclone (https://rclone.org/install/).  You can just run it (type rclone serve docker and hit enter) for the test.
Install FUSE:
Download two systemd configuration files: docker-volume-rclone.service (https://raw.githubusercontent.com/rclone/rclone/master/contrib/docker- plugin/systemd/docker-volume-rclone.service) and docker-volume-rclone.socket (https://raw.githubusercontent.com/rclone/rclone/master/contrib/docker- plugin/systemd/docker-volume-rclone.socket).
Put them to the /etc/systemd/system/ directory:
cp docker-volume-plugin.service /etc/systemd/system/ cp docker-volume-plugin.socket  /etc/systemd/system/
Please note that all commands in this section must be run as root but we omit sudo prefix for brevity.  Now create directories required by the service:
mkdir -p /var/lib/docker-volumes/rclone mkdir -p /var/lib/docker-plugins/rclone/config mkdir -p /var/lib/docker-plugins/rclone/cache
Run the docker plugin service in the socket activated mode:
systemctl daemon-reload systemctl start docker-volume-rclone.service systemctl enable docker-volume-rclone.socket systemctl start docker-volume-rclone.socket systemctl restart docker
Or run the service directly: - run systemctl daemon-reload to let systemd pick up new config - run systemctl enable docker-volume-rclone.service to make the new service start automatically when you power on your machine. - run systemctl start docker-volume-rclone.service to start the service now.  - run systemctl restart docker to restart docker daemon and let it detect the new plugin socket.  Note that this step is not needed in managed mode where docker knows about plugin state changes.
The two methods are equivalent from the user perspective, but I personally prefer socket activation.
Troubleshooting You can see managed plugin settings (https://docs.docker.com/engine/extend/#debugging-plugins) with
docker plugin list docker plugin inspect rclone
Note that docker (including latest 20.10.7) will not show actual values of args, just the defaults.
Use journalctl --unit docker to see managed plugin output as part of the docker daemon log.  Note that docker reflects plugin lines as errors but their actual level can be seen from encapsulated message string.
You will usually install the latest version of managed plugin for your platform.  Use the following commands to print the actual installed version:
PLUGID=$(docker plugin list --no-trunc | awk '/rclone/{print$1}') sudo runc --root /run/docker/runtime-runc/plugins.moby exec $PLUGID rclone version
You can even use runc to run shell inside the plugin container:
sudo runc --root /run/docker/runtime-runc/plugins.moby exec --tty $PLUGID bash
Also you can use curl to check the plugin socket connectivity:
docker plugin list --no-trunc PLUGID=123abc... sudo curl -H Content-Type:application/json -XPOST -d {} --unix-socket /run/docker/plugins/$PLUGID/rclone.sock http://localhost/Plugin.Activate
though this is rarely needed.
Caveats Finally I'd like to mention a caveat with updating volume settings. Docker CLI does not have a dedicated command like docker volume update. It may be tempting to invoke docker volume create with updated options on existing volume, but there is a gotcha.  The command will do nothing, it won't even return an error.  I hope that docker maintainers will fix this some day.  In the meantime be aware that you must remove your volume before recreating it with new settings:
docker volume remove my_vol docker volume create my_vol -d rclone -o opt1=new_val1 ...
and verify that settings did update:
docker volume list docker volume inspect my_vol
If docker refuses to remove the volume, you should find containers or swarm services that use it and stop them first.
1Fichier This is a backend for the 1fichier (https://1fichier.com) cloud storage service.  Note that a Premium subscription is required to use the API.
Paths are specified as remote:path
Paths may be as deep as required, e.g.  remote:directory/subdirectory.
Configuration The initial setup for 1Fichier involves getting the API key from the website which you need to do in your browser.
Here is an example of how to make a remote called remote.  First run:
This will guide you through an interactive setup process:
No remotes found - make a new one n) New remote s) Set configuration password q) Quit config n/s/q> n name> remote Type of storage to configure. Enter a string value. Press Enter for the default (""). Choose a number from below, or type in your own value [snip] XX / 1Fichier \ "fichier" [snip] Storage> fichier ** See help for fichier backend at: https://rclone.org/fichier/ **
Your API Key, get it from https://1fichier.com/console/params.pl Enter a string value. Press Enter for the default (""). api_key> example_key
Edit advanced config? (y/n) y) Yes n) No y/n> Remote config -------------------- [remote] type = fichier api_key = example_key -------------------- y) Yes this is OK e) Edit this remote d) Delete this remote y/e/d> y
Once configured you can then use rclone like this,
List directories in top level of your 1Fichier account
rclone lsd remote:
List all the files in your 1Fichier account
rclone ls remote:
To copy a local directory to a 1Fichier directory called backup
rclone copy /home/source remote:backup
Modified time and hashes 1Fichier does not support modification times.  It supports the Whirlpool hash algorithm.
Duplicated files 1Fichier can have two files with exactly the same name and path (unlike a normal file system).
Duplicated files cause problems with the syncing and you will see messages in the log about duplicates.
Restricted filename characters In addition to the default restricted characters set (https://rclone.org/overview/#restricted-characters) the following characters are also replaced:
Character   Value   Replacement ──────────────────────────────── \           0x5C         ＼ <           0x3C         ＜ >           0x3E         ＞ "           0x22         ＂ $           0x24         ＄ `           0x60         ｀ '           0x27         ＇
File names can also not start or end with the following characters. These only get replaced if they are the first or last character in the name:
Character   Value   Replacement ──────────────────────────────── SP          0x20         ␠
Invalid UTF-8 bytes will also be replaced (https://rclone.org/overview/#invalid-utf8), as they can't be used in JSON strings.
Standard options Here are the standard options specific to fichier (1Fichier).
--fichier-api-key Your API Key, get it from https://1fichier.com/console/params.pl.
• Config: api_key
• Env Var: RCLONE_FICHIER_API_KEY
• Type: string
• Default: ""
Advanced options Here are the advanced options specific to fichier (1Fichier).
--fichier-shared-folder If you want to download a shared folder, add this parameter.
• Config: shared_folder
• Env Var: RCLONE_FICHIER_SHARED_FOLDER
--fichier-file-password If you want to download a shared file that is password protected, add this parameter.
NB Input to this must be obscured - see rclone obscure (https://rclone.org/commands/rclone_obscure/).
• Config: file_password
• Env Var: RCLONE_FICHIER_FILE_PASSWORD
--fichier-folder-password If you want to list the files in a shared folder that is password protected, add this parameter.
• Config: folder_password
• Env Var: RCLONE_FICHIER_FOLDER_PASSWORD
--fichier-encoding This sets the encoding for the backend.
See the encoding section in the overview (https://rclone.org/overview/#encoding) for more info.
• Config: encoding
• Env Var: RCLONE_FICHIER_ENCODING
• Type: MultiEncoder
• Default: Slash,LtGt,DoubleQuote,SingleQuote,BackQuote,Dollar,BackSlash,Del,Ctl,LeftSpace,RightSpace,InvalidUtf8,Dot
Limitations rclone about is not supported by the 1Fichier backend.  Backends without this capability cannot determine free space for an rclone mount or use policy mfs (most free space) as a member of an rclone union remote.
See List of backends that do not support rclone about (https://rclone.org/overview/#optional-features) See rclone about (https://rclone.org/commands/rclone_about/)
Alias The alias remote provides a new name for another remote.
Paths may be as deep as required or a local path, e.g. remote:directory/subdirectory or /directory/subdirectory.
During the initial setup with rclone config you will specify the target remote.  The target remote can either be a local path or another remote.
Subfolders can be used in target remote.  Assume an alias remote named backup with the target mydrive:private/backup.  Invoking rclone mkdir backup:desktop is exactly the same as invoking rclone mkdir mydrive:private/backup/desktop.
There will be no special handling of paths containing ..  segments. Invoking rclone mkdir backup:../desktop is exactly the same as invoking rclone mkdir mydrive:private/backup/../desktop.  The empty path is not allowed as a remote.  To alias the current directory use . instead.
Configuration Here is an example of how to make an alias called remote for local folder.  First run: