Repository: rclone/rclone
Pull request #7717 (2024-04-02 11:54:02)

backend: Add Apple iCloud Drive backend
#### What is the purpose of this change?

Add support for a new backend, iCloud Drive

#### Was the change discussed in an issue or in the forum before?

#1778 

#### Checklist

- [x] I have read the [contribution guidelines](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#submitting-a-new-feature-or-bug-fix).
- [x] I have added tests for all changes in this PR if appropriate.
- [x] I have added documentation for the changes if appropriate.
- [x] All commit messages are in [house style](https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#commit-messages).
- [x] I'm done, this Pull Request is ready for review :-)

#### Notes

This feature branch is in a working state, feedback on finished implementations is appreciated. Ill keep it in draft untill everything is implemented and ill update the TODO accordingly

This PR is only for iCloud Drive. Ill continue work on Photos once this PR is merged.

#### Todo

Rclone Interfaces

##### Fs
- [x] List
- [x] Mkdir
- [x] Rmdir
- [x]  Put

##### Object
- [x] Open
- [x] Update
- [x] Remove

##### Mover
- [x] Move

##### DirMove
- [x] DirMove

##### Copier
- [x] Copy



lostb1t: fyi: so iCloud doesn't have an replace/overwrite api call when uploading files. At least not something I can find reverse engineering

So if a file exists I move it to iCloud trash. And upload a new version of the file. 
After successful upload the file is removed from trash or restored from trash if upload failed
ncw: @lostb1t 
> not sure what client_id you are referring to?

This one at the top of the source!

```go
clientId      = "e9f98057fb916de2bbd755ef280d7257146a76e5118f27ab2e9a3d065c20c17e"
```

> and no I haven't run any integration test yet. Will do when implemented all interfaces.

Running the integration tests will guide you through what you need to do. You can start running them as soon as you've done `List`.

> fyi: so iCloud doesn't have functionality for updating/replacing files. At least not something I can find reverse engineering
> 
> So if a file exists I move it to iCloud trash. And upload a new version of the file. After successful upload the file is removed from trash or restored from trash if upload failed.

Seems OK!

> how one handle folder invalidation? drive uses etags for files and folders. If a file or folder within a folder changes then the parent folders etag changes. This means I have to clear almost the whole tree from child to parent.
> Not sure how to handle that in general, what if a users change it locally but file/folder has changed remotely in between. What error should I raise on that? (if etags are not matching)

Do you have to use the etags? They are optional for most cloud storage systems and rclone doesn't use them. This means that the last write wins which is what we want. If you can't ignore etags then refresh the etag and try again.
 
> cachedir doesn't seem to refresh after invalidation, see:
> 
> https://github.com/rclone/rclone/blob/37816e9fbafdb8eee1483a7eab34aa2ff9b1f191/backend/icloud/icloud.go#L424
> 
> When calling flushdir at the end of rmdir I see the call for FindLeaf but the cachedir entries aren't updated.

Not sure!


lostb1t: > @lostb1t
> 
> > not sure what client_id you are referring to?
> 
> This one at the top of the source!
> 
> ```go
> clientId      = "e9f98057fb916de2bbd755ef280d7257146a76e5118f27ab2e9a3d065c20c17e"
> ```
> 
> > and no I haven't run any integration test yet. Will do when implemented all interfaces.
> 
> Running the integration tests will guide you through what you need to do. You can start running them as soon as you've done `List`.
> 
> > fyi: so iCloud doesn't have functionality for updating/replacing files. At least not something I can find reverse engineering
> > So if a file exists I move it to iCloud trash. And upload a new version of the file. After successful upload the file is removed from trash or restored from trash if upload failed.
> 
> Seems OK!
> 
> > how one handle folder invalidation? drive uses etags for files and folders. If a file or folder within a folder changes then the parent folders etag changes. This means I have to clear almost the whole tree from child to parent.
> > Not sure how to handle that in general, what if a users change it locally but file/folder has changed remotely in between. What error should I raise on that? (if etags are not matching)
> 
> Do you have to use the etags? They are optional for most cloud storage systems and rclone doesn't use them. This means that the last write wins which is what we want. If you can't ignore etags then refresh the etag and try again.
> 
> > cachedir doesn't seem to refresh after invalidation, see:
> > https://github.com/rclone/rclone/blob/37816e9fbafdb8eee1483a7eab34aa2ff9b1f191/backend/icloud/icloud.go#L424
> > 
> > When calling flushdir at the end of rmdir I see the call for FindLeaf but the cachedir entries aren't updated.
> 
> Not sure!

etags are needed for file operations unfortunately, but I can  just refresh it when it got stale.

the clientid is a static that icloud.com uses. So I reuse it to mimic the browser.
lostb1t: @ncw how do I handle characters that are not allowed? for example an dotting the path name results in an externs

I could encode it that it for rclone, but it will show undecided characters on anything but rclone. Could I raise an error that also satisfies the unit tests? I think just not allowing a dot should be preferred


lostb1t: @ncw small question. iCloud Drive does not allow uploading empty files so check the size and raise ErrorCantUploadEmptyFiles when its 0

But when using macOS finder to copy a file, I get an partial file with zero length (while the actually file isn't empty)

Not sure howto handle this.
ncw: @lostb1t 
> small question. iCloud Drive does not allow uploading empty files so check the size and raise ErrorCantUploadEmptyFiles when its 0

That sounds correct.

> But when using macOS finder to copy a file, I get an partial file with zero length (while the actually file isn't empty)

Are you using `rclone mount` in the finder?

So you are seeing a zero length file in the finder? I'm not sure what you mean by a partial file?

Can you reproduce this without the `rclone mount`? So using `rclone copy`? If so please paste your command line and the log with -vv.
 
lostb1t: Its slow but all integration test are passing :tada:

Ill add documentation soon.
ncw: > Its slow but finished all interfaces and all integration test are passing 🎉

Well done!

You are running the full suite from `test_all`?

> Only documentation left before its ready for review.

:-)

epetousis: @lostb1t not sure if it's too early for bug reports, but currently it looks like app folders aren't being represented as folders, but rather as files.
lostb1t: > @lostb1t not sure if it's too early for bug reports, but currently it looks like app folders aren't being represented as folders, but rather as files.

bug reports are welcome. Will fix with the following pass, tnx
lostb1t: @epetousis this has been fixed
lostb1t: crap i broke icloud. Too many trash items and now the endpoint is broken lol 

the icloud web api is a mess
epetousis: Fix works great. Much appreciated!
cyayon: Hi,
I setup successfully and authentication seems ok, but I do not see any files or directories on my iCloud account.
I tried ls, lsl, lsd... no file.

thanks.
lostb1t: > Hi,
> 
> I setup successfully and authentication seems ok, but I do not see any files or directories on my iCloud account.
> 
> I tried ls, lsl, lsd... no file.
> 
> 
> 
> thanks.

Could you post a log from the command with the -vv flag (e.g. output from rclone -vv ls 
cyayon: Hi,

here is the verbose.
```
./rclone-icloud -vvv --config rclone-icloud.conf lsd icloud:/
2024/04/20 11:04:04 DEBUG : rclone: Version "v1.67.0-DEV" starting with parameters ["./rclone-icloud" "-vvv" "--config" "rclone-icloud.conf" "lsd" "icloud:/"]
2024/04/20 11:04:04 DEBUG : Creating backend with remote "icloud:/"
2024/04/20 11:04:04 DEBUG : Using config file from "/home/pkg/pkgbuild/rclone/rclone-icloud/rclone/rclone-icloud.conf"
2024/04/20 11:04:04 DEBUG : icloud: Valid session, no need to reauth
2024/04/20 11:04:04 DEBUG : fs cache: renaming cache item "icloud:/" to be canonical "icloud:"
2024/04/20 11:04:05 DEBUG : 8 go routines active
```
lostb1t: do you maybe have advanced data protection enabled?
cyayon: Yes I have.
Is there an issue with that ?
lostb1t: yes its currently unsupported
cyayon: Ok, I understand.
Do you plan to support it or is it too complicated to implement ?
Thanks
lostb1t: i hope to add support for it eventually.Its a different flow, need to dive in a bit to see what it entails
cyayon: Thanks !--Christophe YayonOn 20 Apr 2024, at 12:56, lostb1t ***@***.***> wrote:﻿
i hope to add support for it eventually.Its a different flow, need to dive in a bit to see what it entails

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
cyayon: I just disabled the advanced protection and it do not change everything. No file or directory found ...
Same verbose infos.
lostb1t: > I just disabled the advanced protection and it do not change everything. No file or directory found ...
> Same verbose infos.

delete the remote and add it again (reauthenticate)
cyayon: Great ! it works.
Thanks
cescofry: Works for me  on `ArchLinux 6.8.7-1`.

I had an issue with 2fa, when prompted none of my devices would show the code on screen. However I managed to get it working by going on my iPhone to `settings->AppleID->Sigh-in & Security->Two-Factor Authentication->Get Verification Code`.

I also struggled when looking for iCloud in the list of rclone remotes as it has not be inserted in the right alphabetical spot.

Great work @lostb1t, cannot wait to see this landed on main. If you need some help to finish this work or just for some testing let me know. 

cyayon: Hi,

Everything seems to work as expected for a few days.
But today, while trying to synchronise just uploaded files to my iCloud Drive, I got an error on rclone sync command : 
```
20/4/04/25 15:58:27 DEBUG : pacer: low level retry 3/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/25 15:58:27 DEBUG : pacer: low level retry 3/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/25 15:58:29 DEBUG : pacer: low level retry 3/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/25 15:58:31 DEBUG : pacer: low level retry 3/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/25 15:58:33 DEBUG : pacer: low level retry 4/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/25 15:58:35 DEBUG : pacer: low level retry 4/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/25 15:58:37 DEBUG : pacer: low level retry 4/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/25 15:58:39 DEBUG : pacer: low level retry 4/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
```

Tried to re-launch multiple times, it is not better.
any idea ?
thanks.
lostb1t: @cyayon does listing etc your drive works? or do all file operations return a 503?

cyayon: lsf seems to work, only sync when trying to download failed 
lostb1t: you might be overloading the server, try lowering the amount of transfers
cyayon: same issue with --multi-thread-streams 1 option. 
lostb1t: also try https://rclone.org/docs/#transfers-n
lostb1t: and https://rclone.org/docs/#checkers-n
cyayon: same with --transfers=1 --checkers=1 
lostb1t: the clientid is a static. Its the same for everybody. Login to icloud.com and you will see the same id
lostb1t: maybe removing and readding ylur remote might work (new credentials)
cyayon: I wil try again tomorrow.
But for information, I logged to iCloud with browser (safari on macOS) and my clientId in headers from https://idmsa.apple.com/appleauth/auth/authorize/signin is : d39ba9916b725105xxxxxxxxxxxxxxxxxxxxxxxxxxxx8d9d1a815d

it is different from yours.
cyayon: I don't know if this could help about clientId : https://backstage.forgerock.com/docs/sdks/latest/sdks/use-cases/social-login/creatingappleclient.html


cescofry: Since this morning I am seeing a very similar error.
I would exclude rate limiting since I am not using my main icloud account, therefore I have only a handful of files.

```
2024/04/24 20:21:57 NOTICE: Time may be set wrong - time from "p109-docws.icloud.com" is -15h11m26.497311883s different from this computer
2024/04/24 20:22:04 ERROR : IO error: HTTP error 503 (503 Service Unavailable) returned body: ""
2024/04/24 20:22:24 ERROR : IO error: HTTP error 503 (503 Service Unavailable) returned body: ""
2024/04/24 20:22:44 ERROR : IO error: HTTP error 503 (503 Service Unavailable) returned body: ""
2024/04/24 20:23:05 ERROR : IO error: HTTP error 503 (503 Service Unavailable) returned body: ""

```
cyayon: ClientId ban by apple ?
cyayon: Hi,
Same issue persist this morning (from France).
All others apps which use iCloud seems to work correctly.
Did you try to download a file (sync or copy) on your side ?
lostb1t: does downloading the file from icloud.com work?
cyayon: Yes of course 
lostb1t: well its the same api rclone uses so if that works then might be on my side. Might be they are blockong yeah, figure out on what, might be the user. agent.

could you try with something like

`--user-agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0`
cyayon: Hi,
Same issue, always 503 error from iCloud and as soon as there is something to download.
cyayon: Hi,

I tried to fully re-create config and re-authenticate on iCloud, also with my "own" clientId (found in headers on iCloud.com).
No change, always get 503 error.

```
2024/04/27 17:44:22 DEBUG : Library_bear.tgz.7z: Need to transfer - File not found at Destination
2024/04/27 17:44:22 DEBUG : Library_goodlinks.tgz.7z: Need to transfer - File not found at Destination
2024/04/27 17:44:22 DEBUG : Library_notes.tgz.7z: Need to transfer - File not found at Destination
2024/04/27 17:44:22 DEBUG : Local file system at /Users/cya/tmp/icloud: Waiting for checks to finish
2024/04/27 17:44:22 DEBUG : Local file system at /Users/cya/tmp/icloud: Waiting for transfers to finish
2024/04/27 17:44:22 DEBUG : pacer: low level retry 1/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/27 17:44:22 DEBUG : pacer: Rate limited, increasing sleep to 20ms
2024/04/27 17:44:22 DEBUG : pacer: low level retry 1/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/27 17:44:22 DEBUG : pacer: Rate limited, increasing sleep to 40ms
2024/04/27 17:44:22 DEBUG : pacer: low level retry 1/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/27 17:44:22 DEBUG : pacer: Rate limited, increasing sleep to 80ms
2024/04/27 17:44:23 DEBUG : pacer: low level retry 2/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/27 17:44:23 DEBUG : pacer: Rate limited, increasing sleep to 160ms
2024/04/27 17:44:23 DEBUG : pacer: low level retry 2/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/27 17:44:23 DEBUG : pacer: Rate limited, increasing sleep to 320ms
2024/04/27 17:44:23 DEBUG : pacer: low level retry 2/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/27 17:44:23 DEBUG : pacer: Rate limited, increasing sleep to 640ms
2024/04/27 17:44:23 DEBUG : pacer: low level retry 3/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
2024/04/27 17:44:23 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
^C2024/04/27 17:44:23 INFO  : Signal received: interrupt
```
pedrohgmacedo: > Hi,
> 
> 
> 
> I tried to fully re-create config and re-authenticate on iCloud, also with my "own" clientId (found in headers on iCloud.com).
> 
> No change, always get 503 error.
> 
> 
> 
> ```
> 
> 2024/04/27 17:44:22 DEBUG : Library_bear.tgz.7z: Need to transfer - File not found at Destination
> 
> 2024/04/27 17:44:22 DEBUG : Library_goodlinks.tgz.7z: Need to transfer - File not found at Destination
> 
> 2024/04/27 17:44:22 DEBUG : Library_notes.tgz.7z: Need to transfer - File not found at Destination
> 
> 2024/04/27 17:44:22 DEBUG : Local file system at /Users/cya/tmp/icloud: Waiting for checks to finish
> 
> 2024/04/27 17:44:22 DEBUG : Local file system at /Users/cya/tmp/icloud: Waiting for transfers to finish
> 
> 2024/04/27 17:44:22 DEBUG : pacer: low level retry 1/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
> 
> 2024/04/27 17:44:22 DEBUG : pacer: Rate limited, increasing sleep to 20ms
> 
> 2024/04/27 17:44:22 DEBUG : pacer: low level retry 1/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
> 
> 2024/04/27 17:44:22 DEBUG : pacer: Rate limited, increasing sleep to 40ms
> 
> 2024/04/27 17:44:22 DEBUG : pacer: low level retry 1/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
> 
> 2024/04/27 17:44:22 DEBUG : pacer: Rate limited, increasing sleep to 80ms
> 
> 2024/04/27 17:44:23 DEBUG : pacer: low level retry 2/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
> 
> 2024/04/27 17:44:23 DEBUG : pacer: Rate limited, increasing sleep to 160ms
> 
> 2024/04/27 17:44:23 DEBUG : pacer: low level retry 2/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
> 
> 2024/04/27 17:44:23 DEBUG : pacer: Rate limited, increasing sleep to 320ms
> 
> 2024/04/27 17:44:23 DEBUG : pacer: low level retry 2/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
> 
> 2024/04/27 17:44:23 DEBUG : pacer: Rate limited, increasing sleep to 640ms
> 
> 2024/04/27 17:44:23 DEBUG : pacer: low level retry 3/10 (error HTTP error 503 (503 Service Unavailable) returned body: "")
> 
> 2024/04/27 17:44:23 DEBUG : pacer: Rate limited, increasing sleep to 1.28s
> 
> ^C2024/04/27 17:44:23 INFO  : Signal received: interrupt
> 
> ```

I'm getting the same errors here. 
lostb1t: Alright this should be fixed. At least in my testing, let me know if it still occures.
cyayon: Hi,

With the last patch today, it is better. Some download work as expected.
But with another directory download I have this error : 
```
024/05/01 18:11:20 DEBUG : Added delayed dir = "Perso/nbux/recovery", newDst=Perso/nbux/recovery
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x1ecb8e7]

goroutine 93 [running]:
github.com/rclone/rclone/backend/icloud/api.(*DriveService).GetDownloadURLByDriveID(0xc000f16d80, {0x3aa73d0, 0xc000f26820}, {0xc0007f9f50?, 0x3a?})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/backend/icloud/api/drive.go:228 +0x2e7
github.com/rclone/rclone/backend/icloud.(*Object).Open.func1()
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/backend/icloud/icloud.go:1038 +0xe5
github.com/rclone/rclone/fs.pacerInvoker(0x1, 0xa, 0x50?)
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/pacer.go:88 +0x32
github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc000f220c0, 0xc000814050, 0xa)
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/lib/pacer/pacer.go:197 +0x7c
github.com/rclone/rclone/lib/pacer.(*Pacer).Call(0xc000f220c0, 0xc000814050)
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/lib/pacer/pacer.go:216 +0x92
github.com/rclone/rclone/backend/icloud.(*Object).Open(0xc000ed6320, {0x3aa73d0, 0xc000f26820}, {0xc0009d0010, 0x1, 0x1})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/backend/icloud/icloud.go:1032 +0x329
github.com/rclone/rclone/fs/operations.(*ReOpen).open(0xc000254d20)
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/reopen.go:145 +0xc5
github.com/rclone/rclone/fs/operations.NewReOpen({0x3aa73d0, 0xc000f26820}, {0x3abd140, 0xc000ed6320}, 0xa, {0xc0009d0010, 0x1, 0x1})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/reopen.go:102 +0x50b
github.com/rclone/rclone/fs/operations.Open({0x3aa73d0, 0xc000f26820}, {0x3abd140, 0xc000ed6320}, {0xc0009d0010, 0x1, 0x1})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/reopen.go:124 +0x6b
github.com/rclone/rclone/fs/operations.(*copy).manualCopy(0xc000612510, {0x3aa73d0, 0xc000f26820})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/copy.go:253 +0x4e5
github.com/rclone/rclone/fs/operations.(*copy).copy(0xc000612510, {0x3aa73d0, 0xc000f26820})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/copy.go:302 +0x16c
github.com/rclone/rclone/fs/operations.Copy({0x3aa73d0, 0xc000f26820}, {0x3abd610, 0xc000f483c0}, {0x0, 0x0}, {0xc000b80180, 0x13}, {0x3abd140, 0xc000ed6320})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/copy.go:404 +0x485
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0xc00017ca08, {0x3aa73d0, 0xc000f26820}, 0xc000f26780, {0x3abd610, 0xc000f483c0}, 0x32, 0x0?)
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/sync/sync.go:498 +0x205
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startTransfers in goroutine 1
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/sync/sync.go:528 +0x6b
```

cyayon: after the last commit 98ebc42, I got the same error : 

```
2024/05/01 19:19:50 DEBUG : Perso/perso/family/Maman.md: Need to transfer - File not found at Destination
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x1ecbb12]

goroutine 102 [running]:
github.com/rclone/rclone/backend/icloud/api.(*DriveService).DownloadFile(0xc000ec0d80, {0x3aa73d0, 0xc000ecc5f0}, {0x0, 0x0}, {0xc000970870, 0x1, 0x1})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/backend/icloud/api/drive.go:253 +0x192
github.com/rclone/rclone/backend/icloud.(*Object).Open.func1()
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/backend/icloud/icloud.go:1040 +0x139
github.com/rclone/rclone/fs.pacerInvoker(0x1, 0xa, 0x50?)
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/pacer.go:88 +0x32
github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc000ec8180, 0xc00110c690, 0xa)
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/lib/pacer/pacer.go:197 +0x7c
github.com/rclone/rclone/lib/pacer.(*Pacer).Call(0xc000ec8180, 0xc00110c690)
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/lib/pacer/pacer.go:216 +0x92
github.com/rclone/rclone/backend/icloud.(*Object).Open(0xc000f481e0, {0x3aa73d0, 0xc000ecc5f0}, {0xc000970870, 0x1, 0x1})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/backend/icloud/icloud.go:1032 +0x329
github.com/rclone/rclone/fs/operations.(*ReOpen).open(0xc000ed80e0)
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/reopen.go:145 +0xc5
github.com/rclone/rclone/fs/operations.NewReOpen({0x3aa73d0, 0xc000ecc5f0}, {0x3abd140, 0xc000f481e0}, 0xa, {0xc000970870, 0x1, 0x1})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/reopen.go:102 +0x50b
github.com/rclone/rclone/fs/operations.Open({0x3aa73d0, 0xc000ecc5f0}, {0x3abd140, 0xc000f481e0}, {0xc000970870, 0x1, 0x1})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/reopen.go:124 +0x6b
github.com/rclone/rclone/fs/operations.(*copy).manualCopy(0xc0009a25a0, {0x3aa73d0, 0xc000ecc5f0})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/copy.go:253 +0x4e5
github.com/rclone/rclone/fs/operations.(*copy).copy(0xc0009a25a0, {0x3aa73d0, 0xc000ecc5f0})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/copy.go:302 +0x16c
github.com/rclone/rclone/fs/operations.Copy({0x3aa73d0, 0xc000ecc5f0}, {0x3abd610, 0xc000eec1e0}, {0x0, 0x0}, {0xc0009055c0, 0x13}, {0x3abd140, 0xc000f481e0})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/copy.go:404 +0x485
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0xc0004dca08, {0x3aa73d0, 0xc000ecc5f0}, 0xc000ecc550, {0x3abd610, 0xc000eec1e0}, 0x19, 0x0?)
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/sync/sync.go:498 +0x205
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startTransfers in goroutine 1
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/sync/sync.go:528 +0x6b
```

lostb1t: @cyayon thanks for pointing me in the right direction, this was a bug with application files and has been fixed.
cyayon: Great, perfect !

Just a suggestion, I think it would be a good thing if we you could specify a different clientId in config (rclone config question). My clientId in web headers is different from yours (d39ba9916b725105xxxxxxxxxxxxxxxxd9d1a815d).
 
lostb1t: yeah ill check it out and replace it when i have the change
cyayon: Great !

Another thing, I found the download very slow (about 1MB/s) for a large 3GB file.

When using Finder, the download speed is much higher (about 20MB/s).

Is there some reason ?

Thank you very much.

lostb1t: > Great !
> 
> Another thing, I found the download very slow (about 1MB/s) for a large 3GB file.
> 
> When using Finder, the download speed is much higher (about 20MB/s).
> 
> Is there some reason ?
> 
> Thank you very much.
> 

if you say finder so you mean apples native drive integration?
cyayon: Yes on macOS (and files.app on iOS).
lostb1t: different apis. Apple probaply using a private api we dont have access to. This uses the web api.

So if you want to compare use icloud.com, those speeds should
match with the rclone implementation 
cyayon: 
I understand. 
It is not really important as I use Rclone only for backup all my cloud storage accounts.
Thanks !
hurryman2212: With the current HEAD commit, I did `rclone config` with successful 2FA, and `rclone mount iCloud: ~/iCloud`/`rclone ls iCloud:`/`rclone lsd iCloud:` but the remote always shows nothing (0 items, Free space: 1.0PiB).

When I try to copy a non-zero-bytes file into it, when mounted, it says `Cannot find PCS auth info from request`:
```
./Sources/rclone/rclone mount --verbose iCloud: ~/iCloud
2024/05/09 20:21:54 INFO  : : poll-interval is not supported by this remote
2024/05/09 20:21:54 NOTICE: : --vfs-cache-mode writes or full is recommended for this remote as it can't stream
2024/05/09 20:22:00 NOTICE: Local file system at /tmp/rclone-spool480232071: --checksum is in use but the source and destination have no hashes in common; falling back to --size-only
2024/05/09 20:22:01 ERROR : 1.pdf: Failed to copy: GET https://p126-docws.icloud.com:443/ws/com.apple.CloudDocs/list/lookup_by_id?document_id=root&unified_format=false failed, status 400, err: HTTP error 400 (400 Bad Request) returned body: "{\n  \"uuid\" : \"9cdfa44c-a396-46a9-b175-fe5e4d414870\",\n  \"messageForDeveloper\" : \"Cannot find PCS auth info from request\",\n  \"error_code\" : \"InvalidRequestException\",\n  \"serverErrorCode\" : \"BAD_REQUEST\",\n  \"reason\" : \"Cannot find PCS auth info from request\"\n}"
2024/05/09 20:22:01 INFO  : 1.pdf: Deleted
2024/05/09 20:22:01 INFO  : Local file system at /tmp/rclone-spool480232071: Removing directory
2024/05/09 20:22:01 ERROR : 1.pdf: WriteFileHandle.New Rcat failed: GET https://p126-docws.icloud.com:443/ws/com.apple.CloudDocs/list/lookup_by_id?document_id=root&unified_format=false failed, status 400, err: HTTP error 400 (400 Bad Request) returned body: "{\n  \"uuid\" : \"9cdfa44c-a396-46a9-b175-fe5e4d414870\",\n  \"messageForDeveloper\" : \"Cannot find PCS auth info from request\",\n  \"error_code\" : \"InvalidRequestException\",\n  \"serverErrorCode\" : \"BAD_REQUEST\",\n  \"reason\" : \"Cannot find PCS auth info from request\"\n}"
2024/05/09 20:22:01 ERROR : 1.pdf: WriteFileHandle.Flush error: GET https://p126-docws.icloud.com:443/ws/com.apple.CloudDocs/list/lookup_by_id?document_id=root&unified_format=false failed, status 400, err: HTTP error 400 (400 Bad Request) returned body: "{\n  \"uuid\" : \"9cdfa44c-a396-46a9-b175-fe5e4d414870\",\n  \"messageForDeveloper\" : \"Cannot find PCS auth info from request\",\n  \"error_code\" : \"InvalidRequestException\",\n  \"serverErrorCode\" : \"BAD_REQUEST\",\n  \"reason\" : \"Cannot find PCS auth info from request\"\n}"
2024/05/09 20:22:01 ERROR : IO error: GET https://p126-docws.icloud.com:443/ws/com.apple.CloudDocs/list/lookup_by_id?document_id=root&unified_format=false failed, status 400, err: HTTP error 400 (400 Bad Request) returned body: "{\n  \"uuid\" : \"9cdfa44c-a396-46a9-b175-fe5e4d414870\",\n  \"messageForDeveloper\" : \"Cannot find PCS auth info from request\",\n  \"error_code\" : \"InvalidRequestException\",\n  \"serverErrorCode\" : \"BAD_REQUEST\",\n  \"reason\" : \"Cannot find PCS auth info from request\"\n}"
2024/05/09 20:22:01 ERROR : IO error: can't open file - writer failed
2024/05/09 20:22:01 ERROR : IO error: can't open file - writer failed
```

ENV: Xubuntu 24.04 (6.8.0-31-generic)
lostb1t: @hurryman2212 > do you maybe have advanced data protection enabled? thats currently not supported and need to be disabled.
hurryman2212: > @hurryman2212 > do you maybe have advanced data protection enabled? thats currently not supported and need to be disabled.

It has been almost a year, I think, so I have completely forgotten about it.. Thank you!
cyayon: Hi,
Today, I had to refresh 2FA authentication manually (rclone config, edit, re-enter 2FA code).

Is it normal ? I thank that it will be done without human interaction.

Thanks.
lostb1t: @cyayon it should selfrenew. Could you send me the log when this occurs again?
cyayon: > @cyayon it should selfrenew. Could you send me the log when this occurs again?

Hi,

Here is a log  : 
```
2024/05/24 19:40:14 DEBUG : rclone: Version "v1.67.0-DEV" starting with parameters ["/opt/rclone/rclone-icloud" "sync" "--config" "/etc/rclone/rclone.conf" "-x" "--local-no-check-updated" "--progress" "-vv" "--suffix=.BackupMGR_3274985_2024-05-24-19:40" " "icloud_xxx:/Documents" "[/mnt/datas/cloud/users/lya/iCloud/xxxx](mailto:%2Fmnt%2Fdatas%2Fcloud%2Fusers%2Flya%2Ficloud%2Fxxxx)/Documents"]
2024/05/24 19:40:14 DEBUG : Creating backend with remote "icloud_xxxx:/Documents"
2024/05/24 19:40:14 DEBUG : Using config file from "/etc/rclone/rclone.conf"
2024/05/24 19:40:15 DEBUG : icloud: Authenticating as [xxxx](mailto:xxx)
2024/05/24 19:40:16 DEBUG : Saving config "cookies" in section "icloud_xxxx" of the config file
2024/05/24 19:40:16 DEBUG : Keeping previous permissions for config file: -r--r-----
2024/05/24 19:40:16 DEBUG : fs cache: renaming cache item "icloud_xxxx:/Documents" to be canonical "icloud_xxxx:Documents"
2024/05/24 19:40:16 DEBUG : Creating backend with remote "[/mnt/datas/cloud/users/lya/iCloud/xxxx](mailto:%2Fmnt%2Fdatas%2Fcloud%2Fusers%2Flya%2Ficloud%2Fxxxx)/Documents"
2024/05/24 19:40:16 DEBUG : local: detected overridden config - adding "{RZq88}" suffix to name
2024/05/24 19:40:16 DEBUG : fs cache: renaming cache item "[/mnt/datas/cloud/users/lya/iCloud/xxxx](mailto:%2Fmnt%2Fdatas%2Fcloud%2Fusers%2Flya%2Ficloud%2Fxxx)/Documents" to be canonical "local{RZq88}:[/mnt/datas/cloud/users/lya/iCloud/xxxx](mailto:%2Fmnt%2Fdatas%2Fcloud%2Fusers%2Flya%2Ficloud%2Fxxx/Documents"

2024/05/24 19:40:16 ERROR : Documents: error reading source root directory: directory not found
Transferred:   	          0 B / 0 B, -, 0 B/s, ETA -
Elapsed time:         1.9s
```
cyayon: Hi, 

Same issue today.

```
/opt/rclone/rclone-icloud ls --config /etc/rclone/rclone.conf -vvvv 'icloud_xxx:/'
2024/06/06 14:13:20 DEBUG : rclone: Version "v1.67.0-DEV" starting with parameters ["/opt/rclone/rclone-icloud" "ls" "--config" "/etc/rclone/rclone.conf" "-vvvv" "icloud_xxxx:/"]
2024/06/06 14:13:20 DEBUG : Creating backend with remote "icloud_sxxx:/"
2024/06/06 14:13:20 DEBUG : Using config file from "/etc/rclone/rclone.conf"
2024/06/06 14:13:20 DEBUG : icloud: Authenticating as xxxx
2024/06/06 14:13:21 DEBUG : Saving config "cookies" in section "icloud_xxx" of the config file
2024/06/06 14:13:21 DEBUG : Keeping previous permissions for config file: -r--r-----
2024/06/06 14:13:21 DEBUG : fs cache: renaming cache item "icloud_xxx:/" to be canonical "icloud_xxx:"
2024/06/06 14:13:22 DEBUG : 10 go routines active
```

I have to refresh token and 2FA again...
fcrespo82: Hi @lostb1t I was trying to test it and got this error

> 2024/06/19 11:58:54 Fatal error: POST https://setup.icloud.com/setup/ws/1/accountLogin failed, status 400, err: HTTP error 400 (400 Bad Request) returned body: "{\"success\":false,\"error\":\"Invalid Session Token\"}"

How can I debug it? And how can I help in the project overall?
lostb1t: > Hi @lostb1t I was trying to test it and got this error
> 
> 
> 
> > 2024/06/19 11:58:54 Fatal error: POST https://setup.icloud.com/setup/ws/1/accountLogin failed, status 400, err: HTTP error 400 (400 Bad Request) returned body: "{\"success\":false,\"error\":\"Invalid Session Token\"}"
> 
> 
> 
> How can I debug it? And how can I help in the project overall?

did you run the config succesfully? (2fa)
fcrespo82: > > Hi @lostb1t I was trying to test it and got this error
> > > 2024/06/19 11:58:54 Fatal error: POST https://setup.icloud.com/setup/ws/1/accountLogin failed, status 400, err: HTTP error 400 (400 Bad Request) returned body: "{"success":false,"error":"Invalid Session Token"}"
> > 
> > 
> > How can I debug it? And how can I help in the project overall?
> 
> did you run the config succesfully? (2fa)

This happened when trying to create the remote. Below is the attempt to create the remote.

```
❯ ./rclone config -vvv
2024/06/19 16:21:48 DEBUG : rclone: Version "v1.67.0-beta.7874.d2259ab8d.feature/icloud" starting with parameters ["./rclone" "config" "-vvv"]
2024/06/19 16:21:48 DEBUG : Using config file from "/home/fernando/.config/rclone/rclone.conf"
Current remotes:

Name                 Type
====                 ====
OneDrive             onedrive

e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n

Enter name for new remote.
name> iCloudDrive

Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
...
55 / iCloud Drive
   \ (iclouddrive)
...
Storage> 55

2024/06/19 16:22:03 DEBUG : iCloudDrive: config in: state="*all", result=""
2024/06/19 16:22:03 DEBUG : iCloudDrive: config out: out=&{State:*all-set,0,false Option: OAuth:<nil> Error: Result:}, err=<nil>
2024/06/19 16:22:03 DEBUG : iCloudDrive: config: reading config parameter "apple_id"
Option apple_id.
Apple ID.
Enter a value.
apple_id> fernando82@gmail.com

2024/06/19 16:22:13 DEBUG : iCloudDrive: config in: state="*all-set,0,false", result="fernando82@gmail.com"
2024/06/19 16:22:13 DEBUG : Saving config "apple_id" in section "iCloudDrive" of the config file
2024/06/19 16:22:13 DEBUG : iCloudDrive: config out: out=&{State:*all,1,false Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/06/19 16:22:13 DEBUG : iCloudDrive: config in: state="*all,1,false", result=""
2024/06/19 16:22:13 DEBUG : iCloudDrive: config out: out=&{State:*all-set,1,false Option: OAuth:<nil> Error: Result:}, err=<nil>
2024/06/19 16:22:13 DEBUG : iCloudDrive: config: reading config parameter "password"
Option password.
Password.
Choose an alternative below. Press Enter for the default (n).
y) Yes, type in my own password
g) Generate random password
n) No, leave this optional password blank (default)
y/g/n> 

2024/06/19 16:22:18 DEBUG : iCloudDrive: config in: state="*all-set,1,false", result=""
2024/06/19 16:22:18 DEBUG : iCloudDrive: config out: out=&{State:*all,2,false Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/06/19 16:22:18 DEBUG : iCloudDrive: config in: state="*all,2,false", result=""
2024/06/19 16:22:18 DEBUG : iCloudDrive: config out: out=&{State:*all,3,false Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/06/19 16:22:18 DEBUG : iCloudDrive: config in: state="*all,3,false", result=""
2024/06/19 16:22:18 DEBUG : iCloudDrive: config out: out=&{State:*all,4,false Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/06/19 16:22:18 DEBUG : iCloudDrive: config in: state="*all,4,false", result=""
2024/06/19 16:22:18 DEBUG : iCloudDrive: config out: out=&{State:*all,5,false Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/06/19 16:22:18 DEBUG : iCloudDrive: config in: state="*all,5,false", result=""
2024/06/19 16:22:18 DEBUG : iCloudDrive: config out: out=&{State:*all,6,false Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/06/19 16:22:18 DEBUG : iCloudDrive: config in: state="*all,6,false", result=""
2024/06/19 16:22:18 DEBUG : iCloudDrive: config out: out=&{State:*all-advanced Option:false OAuth:<nil> Error: Result:}, err=<nil>
2024/06/19 16:22:18 DEBUG : iCloudDrive: config: reading config parameter "config_fs_advanced"
Edit advanced config?
y) Yes
n) No (default)
y/n> 

2024/06/19 16:22:39 DEBUG : iCloudDrive: config in: state="*all-advanced", result="false"
2024/06/19 16:22:39 DEBUG : iCloudDrive: config out: out=&{State:*postconfig Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/06/19 16:22:39 DEBUG : iCloudDrive: config in: state="*postconfig", result=""
2024/06/19 16:22:39 DEBUG : iCloudDrive: config in: state="", result=""
2024/06/19 16:22:39 DEBUG : iCloudDrive: config out: out=&{State:password Option: OAuth:<nil> Error: Result:}, err=<nil>
2024/06/19 16:22:39 DEBUG : iCloudDrive: config out: out=&{State:password Option: OAuth:<nil> Error: Result:}, err=<nil>
2024/06/19 16:22:39 DEBUG : iCloudDrive: config: reading config parameter "config_password"
Option config_password.
Two-factor authentication: please enter your password (it won't be saved in the configuration)
Choose an alternative below. Press Enter for the default (n).
y) Yes, type in my own password
g) Generate random password
n) No, leave this optional password blank (default)
y/g/n> y
Enter the password:
password:
Confirm the password:
password:

2024/06/19 16:23:54 DEBUG : iCloudDrive: config in: state="password", result="[REDACTED]"
2024/06/19 16:23:54 DEBUG : Saving config "password" in section "iCloudDrive" of the config file
2024/06/19 16:23:54 DEBUG : iCloudDrive: config out: out=&{State:authenticate Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/06/19 16:23:54 DEBUG : iCloudDrive: config in: state="authenticate", result=""
2024/06/19 16:23:54 DEBUG : icloud: Authenticating as fernando82@gmail.com
2024/06/19 16:23:55 DEBUG : iCloudDrive: config out: out=<nil>, err=POST https://setup.icloud.com/setup/ws/1/accountLogin failed, status 400, err: HTTP error 400 (400 Bad Request) returned body: "{\"success\":false,\"error\":\"Invalid Session Token\"}"
Error: POST https://setup.icloud.com/setup/ws/1/accountLogin failed, status 400, err: HTTP error 400 (400 Bad Request) returned body: "{\"success\":false,\"error\":\"Invalid Session Token\"}"
Usage:
  rclone config [flags]
  rclone config [command]

Available Commands:
  create      Create a new remote with name, type and options.
  delete      Delete an existing remote.
  disconnect  Disconnects user from remote
  dump        Dump the config file as JSON.
  edit        Enter an interactive configuration session.
  file        Show path of configuration file in use.
  password    Update password in an existing remote.
  paths       Show paths used for configuration, cache, temp etc.
  providers   List in JSON format all the providers and options.
  reconnect   Re-authenticates user with remote.
  redacted    Print redacted (decrypted) config file, or the redacted config for a single remote.
  show        Print (decrypted) config file, or the config for a single remote.
  touch       Ensure configuration file exists.
  update      Update options in an existing remote.
  userinfo    Prints info about logged in user of remote.

Flags:
  -h, --help   help for config

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

2024/06/19 16:23:55 Fatal error: POST https://setup.icloud.com/setup/ws/1/accountLogin failed, status 400, err: HTTP error 400 (400 Bad Request) returned body: "{\"success\":false,\"error\":\"Invalid Session Token\"}"
```
lostb1t: do you have 2fa enabled for your icloud account?
lostb1t: > Hi, 
> 
> 
> 
> Same issue today.
> 
> 
> 
> ```
> 
> /opt/rclone/rclone-icloud ls --config /etc/rclone/rclone.conf -vvvv 'icloud_xxx:/'
> 
> 2024/06/06 14:13:20 DEBUG : rclone: Version "v1.67.0-DEV" starting with parameters ["/opt/rclone/rclone-icloud" "ls" "--config" "/etc/rclone/rclone.conf" "-vvvv" "icloud_xxxx:/"]
> 
> 2024/06/06 14:13:20 DEBUG : Creating backend with remote "icloud_sxxx:/"
> 
> 2024/06/06 14:13:20 DEBUG : Using config file from "/etc/rclone/rclone.conf"
> 
> 2024/06/06 14:13:20 DEBUG : icloud: Authenticating as xxxx
> 
> 2024/06/06 14:13:21 DEBUG : Saving config "cookies" in section "icloud_xxx" of the config file
> 
> 2024/06/06 14:13:21 DEBUG : Keeping previous permissions for config file: -r--r-----
> 
> 2024/06/06 14:13:21 DEBUG : fs cache: renaming cache item "icloud_xxx:/" to be canonical "icloud_xxx:"
> 
> 2024/06/06 14:13:22 DEBUG : 10 go routines active
> 
> ```
> 
> 
> 
> I have to refresh token and 2FA again...

Seems something not going right when reauthenticating. Not sure why on first glance. Cookies are
refreshed but seems something else is off. Will have to find some time to dig in.
fcrespo82: > Make sure 2fa is enabled on your account. I made the password required just now, could you pull the changes and try again?

Yes, I had 2fa setup in my account.

And yes, the change made it work, it was a couple of things that made it not work.

1. My account was, for some reason, blocked; and
2. I wasn't putting my password on the first field.

I successfully mounted the drive and now will use it an make some tests. If anything goes wrong, I'll report it back.

Thank you very much for this rClone backend!

I am recently migrating to Linux after the hole Microsoft Recall fiasco, and I rely some of my daily workflow on iCloud, this was the missing piece for the migration.

PS: Do you have GitHub sponsor, ko-fi, ...?
lostb1t: k this sucks but it seems the trust token is only valid for 30 days. After that it has to be renewed with an MFA code.... not ... ideal...
cyayon: Hi,

It seems that the other GitHub project icloud_photos_downloader have also the same issue.

I am using this while waiting for native rclone support for iCloud photo (backup).  And each 30 days (I think) I have to renew with MFA (for me, my wife and son...)

I will ask ask directly on this other GitHub project if there is a solution. I will post here the issue link.
cyayon: Someone already ask for the expiration of the cookie : https://github.com/icloud-photos-downloader/icloud_photos_downloader/issues/803#issuecomment-2043416590

cyayon: An idea here (not great) : https://github.com/boredazfcuk/docker-icloudpd/issues/524
Another (not great too) : https://github.com/icloud-photos-downloader/icloud_photos_downloader/issues/803#issuecomment-2180784435
byte-bandit: I spent a few minutes debugging the "iCloud Drive" app for Windows, and it looks like they're using gRPC & protobuf in those. Unfortunately my reverse engineering skills are pretty limited, but maybe it's worth pursuing this in the future to improve the performance and speed here.
lostb1t: > I spent a few minutes debugging the "iCloud Drive" app for Windows, and it looks like they're using gRPC & protobuf in those. Unfortunately my reverse engineering skills are pretty limited, but maybe it's worth pursuing this in the future to improve the performance and speed here.

Apple uses SSL pinning so its unfortunately impossible to reverse engineer it with a MITM.

The only way this was possible
is because its and web app which can be reverse engineerd as the https connections is terlinated by the browser
byte-bandit: > > I spent a few minutes debugging the "iCloud Drive" app for Windows, and it looks like they're using gRPC & protobuf in those. Unfortunately my reverse engineering skills are pretty limited, but maybe it's worth pursuing this in the future to improve the performance and speed here.
> 
> Apple uses SSL pinning so its unfortunately impossible to reverse engineer it with a MITM.
> 
> The only way this was possible is because of the icloud web app which can be reverse engineered as the https connection is terminated by the browser and not a private app

I think by using something like IDA to inspect actual control flow it might be possible to reverse engineer the protocol from the binary, but it's very tedious and definitely above my experience with this. Just leaving it for inspiration. 
fcrespo82: Hi, it's me again. I'm trying this integration, and it is in a good working condition with minor hiccups.

I don't know if i did something wrong, but now I'm getting errors when trying to sync.

Here is the log [rclone-iCloudDrive.log](https://github.com/user-attachments/files/16041625/rclone-iCloudDrive.log)

One specific folder is not able to send files, but all my iCloud is not seeing changes that I make on other locations.

What can I do to help debug this?
lostb1t: @fcrespo82 can you post your mount command
fcrespo82: > @fcrespo82 can you post your mount command

I am using a service to mount. Here its contents:

```
# User service for Rclone mounting
#
# Place in ~/.config/systemd/user/
# File must include the '@' (ex rclone@.service)
# As your normal user, run 
#   systemctl --user daemon-reload
# You can now start/enable each remote by using rclone@<remote>
#   systemctl --user enable --now rclone@dropbox

[Unit]
Description=rclone: Remote FUSE filesystem for cloud storage config %i
Documentation=man:rclone(1)
After=network-online.target
Wants=network-online.target

[Service]
Type=notify
ExecStartPre=-/usr/bin/mkdir -p %h/mnt/%i
ExecStart= \
  /usr/bin/rclone mount \
    --config=%h/.config/rclone/rclone.conf \
    --vfs-cache-mode writes \
    --vfs-cache-max-size 100M \
    --log-level INFO \
    --log-file /tmp/rclone-%i.log \
    --umask 022 \
    --allow-other \
    %i: %h/mnt/%i
ExecStop=/bin/fusermount -u %h/mnt/%i

[Install]
WantedBy=default.target
```

Thanks.
lostb1t: @fcrespo82 could you try to disable vfs-cache-mode (off) and see if that resolves the issue?
fcrespo82: > @fcrespo82 could you try to disable vfs-cache-mode (off) and see if that resolves the issue?

Tried with vfs-cache-mode off, but the error messages persist.
vtolstov: hi, what needs to be fixed to get this merged ?
lostb1t: Some integration tests are failing ('test_all -backend iclouddrive -timeout 99999s') I think if those are fixed.

Tests that are failing are mostly about parallel functionality, where sometimes things go out of sync.
stv0g: @lostb1t Do you accept funding for this work?

Or is there some other way how I could support you? Writing more tests or docs, testing, reveng?

I am personally quite interested in the feature it would enable me to ingest/backup my iCloud with git-annex.
hurryman2212: Has the current implementation of yours reached the theoretically possible maximum performance, with the hard constraints from using the web version of iCloud? Or, is there something that can be improved in the later implementation? With the current implementation, it is just bearable minimum in terms of performance, at least on Linux.
lostb1t: > Has the current implementation of yours reached the theoretically possible maximum performance, with the hard constraints from using the web version of iCloud? Or, is there something that can be improved in the later implementation? With the current implementation, it is just bearable minimum, or worse, in terms of performance, at least on Linux.

depends, it will always be slow as the web api is just slow.

What exactly do experience as extreme slow? listings? downloads?
hurryman2212: > > Has the current implementation of yours reached the theoretically possible maximum performance, with the hard constraints from using the web version of iCloud? Or, is there something that can be improved in the later implementation? With the current implementation, it is just bearable minimum, or worse, in terms of performance, at least on Linux.
> 
> depends, it will always be slow as the web api is just slow.
> 
> What exactly do experience as extreme slow? listings? downloads?

Listing and moving existing file/directory within that cloud (even with ` --vfs-cache-mode full`). Uploading file/directory is fast due to caching (but the actual uploading is delayed and it is indeed slow to be synced to the cloud). Downloading file/directory is initially slow for each file iteration, but for each file, the speed is comparable to the web interface; Moving existing file/directory within the cloud shows similar behavior as Downloading, without caching in effect.

For listing:

I've noticed while testing with it, that the performance is significantly worse with the GUI file browser (`thunar`)  than with CLI. With CLI, basic listing with `ls` is actually quite bearable and the delay before printing list is (thankfully) non-linear by number of items to list at that depth. Multi-depth listing (i.e. `find`) is slow for each step.

With `thunar`, it is SO SLOW and basically makes that instance freeze until the listing is done. I think it is related to thumbnail, but the result is same with file types without thumbnail.
lostb1t: im gonna guess a gui downloads previews which slows things down. Not sure if theres a setting in rclone for parallel downloads
cyayon: Hi,
Do you plan to add icloud photos support ?
It would be great to be able to use rclone to backup photos...
thanks 
lostb1t: > Hi,
> 
> Do you plan to add icloud photos support ?
> 
> It would be great to be able to use rclone to backup photos...
> 
> thanks 

not anytime soon. Good news is though that the auth layer can be used for photos service aswell and the photos api is mich simpler and modern then the drive api.

It's just not something on my personal list but if anyone wants to fund it i might think about it.
cyayon:  
> It's just not something on my personal list but if anyone wants to fund it i might think about it.

Why not, how much could it be ?

Scripter17: I've already committed $100 USD to this project and am willing to raise it to $200 for icloud photos
cescofry: @Scripter17 the aim of the current milestone is to add basic functionality for iCloud syncing to rclone. Considering how hard it has been for @lostb1t to get this far (the iCloud API is definitely not an easy one to work with), it would probably be wise to not expand the scope of the current work to Photos. This does not mean Photos should never been implemented in the future and in fact you should probably create an issue so work can be organized there.
stv0g: @Scripter17 @cescofry I've created a new issue / feature request for the Photos support: https://github.com/rclone/rclone/issues/7982

Lets continue the discussion about the Photos suppport there.
lostb1t: @ncw All tests are passing. This is ready for review.
lostb1t: > Thanks for addressing my comments to quickly. I've gave it another look and found some more dead code and inconsistencies in the commenting style.

uch ill adjust my linting to catch these automatic
cyayon: Hi,For information, my id is different.I already posted about this in the feature related GitHub thread.I think it would be better to be able to define in config (with a default pre-defined).--Christophe YayonOn 8 Aug 2024, at 10:15, Steffen Vogel ***@***.***> wrote:﻿
@stv0g commented on this pull request.



In backend/icloud/api/session.go:
> +		return fmt.Errorf("validateSession failed: %w", err)
+	}
+
+	return nil
+}
+
+// GetAuthHeaders returns the authentication headers for the session.
+//
+// It takes an `overwrite` map[string]string parameter which allows
+// overwriting the default headers. It returns a map[string]string.
+func (s *Session) GetAuthHeaders(overwrite map[string]string) map[string]string {
+	headers := map[string]string{
+		// "Accept": "*.*",
+		"Accept":                           "application/json",
+		"Content-Type":                     "application/json",
+		"X-Apple-OAuth-Client-Id":          "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",

They are extracted from Apple's iCloud web app. And, as we try to emulate the web-app, we need to use those IDs. As far as I am aware Apple also does not allow to create new OAuth2 IDs..
But I think we should document this better, maybe give some instructions how they can be updated in case Apple changes the IDs in their web app.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
cyayon: Ok, but i do not understand why my id is different. Perhaps it could be related to the location ?--Christophe YayonOn 8 Aug 2024, at 11:24, lostb1t ***@***.***> wrote:﻿
@lostb1t commented on this pull request.



In backend/icloud/api/session.go:
> +		return fmt.Errorf("validateSession failed: %w", err)
+	}
+
+	return nil
+}
+
+// GetAuthHeaders returns the authentication headers for the session.
+//
+// It takes an `overwrite` map[string]string parameter which allows
+// overwriting the default headers. It returns a map[string]string.
+func (s *Session) GetAuthHeaders(overwrite map[string]string) map[string]string {
+	headers := map[string]string{
+		// "Accept": "*.*",
+		"Accept":                           "application/json",
+		"Content-Type":                     "application/json",
+		"X-Apple-OAuth-Client-Id":          "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",

added some info comments.
I don't really see a need to add it as an setting right now. As changing it isnt needed.
If apple decides to change that behavior its a small patch to add

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
lostb1t: maybe its browser specific. But it doesnt matter, functionality is not impacted by it.
cyayon: Thanks.--Christophe YayonOn 8 Aug 2024, at 12:22, lostb1t ***@***.***> wrote:﻿
maybe its browser specific. But it doesnt matter, functionality is not impacted by it.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
lvjonok: > > Hi,
> > Same issue today.
> > ```
> > 
> > /opt/rclone/rclone-icloud ls --config /etc/rclone/rclone.conf -vvvv 'icloud_xxx:/'
> > 
> > 2024/06/06 14:13:20 DEBUG : rclone: Version "v1.67.0-DEV" starting with parameters ["/opt/rclone/rclone-icloud" "ls" "--config" "/etc/rclone/rclone.conf" "-vvvv" "icloud_xxxx:/"]
> > 
> > 2024/06/06 14:13:20 DEBUG : Creating backend with remote "icloud_sxxx:/"
> > 
> > 2024/06/06 14:13:20 DEBUG : Using config file from "/etc/rclone/rclone.conf"
> > 
> > 2024/06/06 14:13:20 DEBUG : icloud: Authenticating as xxxx
> > 
> > 2024/06/06 14:13:21 DEBUG : Saving config "cookies" in section "icloud_xxx" of the config file
> > 
> > 2024/06/06 14:13:21 DEBUG : Keeping previous permissions for config file: -r--r-----
> > 
> > 2024/06/06 14:13:21 DEBUG : fs cache: renaming cache item "icloud_xxx:/" to be canonical "icloud_xxx:"
> > 
> > 2024/06/06 14:13:22 DEBUG : 10 go routines active
> > ```
> > 
> > 
> >     
> >       
> >     
> > 
> >       
> >     
> > 
> >     
> >   
> > I have to refresh token and 2FA again...
> 
> Seems something not going right when reauthenticating. Not sure why on first glance. Cookies are refreshed but seems something else is off. Will have to find some time to dig in.

Hello! I am experiencing the same kind of problems. From what I have read in this PR comments:
- 2FA is enabled
- Advanced Data Protection is disabled

```
2024/09/06 12:22:22 DEBUG : icloud: config in: state="*all-advanced", result="false"
2024/09/06 12:22:22 DEBUG : icloud: config out: out=&{State:*postconfig Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/09/06 12:22:22 DEBUG : icloud: config in: state="*postconfig", result=""
2024/09/06 12:22:22 DEBUG : icloud: config in: state="", result=""
2024/09/06 12:22:22 DEBUG : icloud: Authenticating as *********
2024/09/06 12:22:24 DEBUG : icloud: config out: out=<nil>, err=HTTP error 400 (400 Bad Request) returned body: "{\"success\":false,\"error\":\"Invalid Session Token\"}"
2024/09/06 12:22:24 DEBUG : icloud: config out: out=<nil>, err=HTTP error 400 (400 Bad Request) returned body: "{\"success\":false,\"error\":\"Invalid Session Token\"}"
Error: HTTP error 400 (400 Bad Request) returned body: "{\"success\":false,\"error\":\"Invalid Session Token\"}"
```

I cannot even successfully authenticate. Would appreciate some help with this.
lostb1t: @lvjonok did you rerun rclone config?
lvjonok: > @lvjonok did you rerun rclone config?

Yes, I did. It looks like icloud was added to configuration, but cannot be used.

For instance, when I try to run `rclone lsd icloud:/` the response is:
```
2024/09/06 16:26:08 Failed to create file system for "icloud:": didn't find backend called "iclouddrive"
```
lostb1t: iclouddrive most definitely exists. Are you using the correct fork/branch?
lvjonok: > iclouddrive most definitely exists. Are you using the correct fork/branch?

I am on commit fcb68cc82c9826e6583ab5d54bd859019ec2215a.

My initial thought was that rclone I installed previously and the one after `go build` conflict with each other, so I removed the first one. Anyway I run into the issue.

My steps:
1. Start setting up icloud as a new remote
2. Enter the password two times
3. Crash
lvjonok: > > iclouddrive most definitely exists. Are you using the correct fork/branch?
> 
> I am on commit [fcb68cc](https://github.com/rclone/rclone/commit/fcb68cc82c9826e6583ab5d54bd859019ec2215a).
> 
> My initial thought was that rclone I installed previously and the one after `go build` conflict with each other, so I removed the first one. Anyway I run into the issue.
> 
> My steps:
> 
> 1. Start setting up icloud as a new remote
> 2. Enter the password two times
> 3. Crash

It was just the wrong password :) Sorry to bother, great contribution!
ncw: @lostb1t do you want to try to get this backend in to v1.69
hurryman2212: I am using the latest commit and now I am no longer able to do `rclone mount` or even `rclone ls`. For `rclone mount`, it mounts but no file or directory is listed.

When using `rclone mount -v`:
```
2024/09/11 10:31:51 INFO  : : poll-interval is not supported by this remote
2024/09/11 10:31:51 NOTICE: : --vfs-cache-mode writes or full is recommended for this remote as it can't stream
2024/09/11 10:32:41 NOTICE: : --checksum is in use but the source and destination have no hashes in common; falling back to --size-only
^C2024/09/11 10:33:26 INFO  : Signal received: interrupt
2024/09/11 10:33:26 ERROR : /home/ANON/iCloud: Failed to unmount: exit status 1: fusermount3: failed to unmount /home/ANON/iCloud: Device or resource busy
2024/09/11 10:33:26 INFO  : Exiting...
```

The mount point shows no file or directory, and has capacity of 1TB.

The message `2024/09/11 10:32:41 NOTICE: : --checksum is in use but the source and destination have no hashes in common; falling back to --size-only` appears when the command `touch hello.txt` finishes. The created file shows normally on other iCloud devices. The file is no longer visible when I remount.
lostb1t: > @lostb1t do you want to try to get this backend in to v1.69

sure, waiting on a review from one of the maintainers tho
lostb1t: @hurryman2212 cqn you run it with -vvv and post the whole log. As i dont see any icloud info. It should say something about icloud valid session when starting up
ncw: @lostb1t Firstly, thank you for a great piece of work - this will be a very popular backend I think.

> > do you want to try to get this backend in to v1.69
> 
> sure, waiting on a review from one of the maintainers tho

That would be me! I've been keeping an eye on this thread waiting for it to settle down and waiting for the right time in the release cycle. I think the timing looks good for merging now :-)

## Things to do

- Get the CI running clean
- Rebase on master
- Make sure the `test_all` integration tests are passing
- I'll do a thorough review once those 3 things are good

### Other OS build

For the `other os` build the easiest way will be to exclude these OSes

-  plan9
-  solaris

See the sftp backend for an example of how to do this

https://github.com/rclone/rclone/blob/1bc9b94cf203ae25da293df5d51f7d9ba51c21ac/backend/sftp/sftp.go#L1

Note you will need an unsupported file also

https://github.com/rclone/rclone/blob/1bc9b94cf203ae25da293df5d51f7d9ba51c21ac/backend/sftp/sftp_unsupported.go#L1-L2

### Lint

If you rebase the PR on `master` most of the lint errors will go away. There may be some remaining - I didn't check

## Questions

- Do the `test_all` integration tests pass?
- How likely is this backend to keep working - I've noticed a lot of back and forward in this thread changing stuff? Is this Apple changing stuff or just stuff which isn't quite right yet?
- How likely is it to upset Apple in some way?
- Have all the learnings from this PR gone into the docs? (eg notes on 2FA etc)
- Would you like to become an rclone maintainer to help look after this backend once it is merged?

Thank you :-)

lostb1t: @cyayon seems Apple changed an api endpoint i was using to get change date of folders. Will investigate 
cyayon: > @cyayon seems Apple changed an api endpoint i was using to get change date of folders. Will investigate

thanks !
lostb1t: @ncw ill get back to you on almost the points, but what do you mean with the CI running "clean"?
ncw: @lostb1t 

> ill get back on all your points, but what do you mean with the CI running "clean"?

Just make this pass :-)

![Screenshot_20240912-110120](https://github.com/user-attachments/assets/07bf1b8c-db2c-42dc-8672-d4e05ac18db6)


lostb1t: icloud has some issues. Cant for instance duplicate files from the browser. So have to wait until they fix this.
cescofry: > @cyayon seems Apple changed an api endpoint i was using to get change date of folders. Will investigate

same here.
lostb1t: @cyayon could you try duplicating an file through icloud.com and see if its broken for you aswell? im just baffled apple keeps it broken for so long
cyayon: > @cyayon could you try duplicating an file through icloud.com and see if its broken for you aswell? im just baffled apple keeps it broken for so long

Hi,

Yes it Is broken too.

![Screenshot 2024-09-15 at 10 13 13](https://github.com/user-attachments/assets/77d7f3d8-e998-4ccc-b64b-e5f6218c4479)

cyayon: Hi,
with the very last commits, I got an new error. The previous with error 500 seems to be solved.

```
*               Tickets_2422933948670400550.pdf: checkingpanic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x10 pc=0x211e852]

goroutine 109 [running]:
github.com/rclone/rclone/backend/icloud/api.(*DriveService).DownloadFile(0xc00100b140, {0x3b16e50, 0xc0004d8410}, {0x0, 0x0}, {0xc0005c4860, 0x1, 0x1})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/backend/icloud/api/drive.go:247 +0x192
github.com/rclone/rclone/backend/icloud.(*Object).Open.func1()
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/backend/icloud/icloud.go:997 +0x1f7
github.com/rclone/rclone/fs.pacerInvoker(0x1, 0xa, 0x50?)
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/pacer.go:88 +0x32
github.com/rclone/rclone/lib/pacer.(*Pacer).call(0xc000e8a180, 0xc000530780, 0xa)
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/lib/pacer/pacer.go:197 +0x7c
github.com/rclone/rclone/lib/pacer.(*Pacer).Call(0xc000e8a180, 0xc000530780)
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/lib/pacer/pacer.go:216 +0x92
github.com/rclone/rclone/backend/icloud.(*Object).Open(0xc000f30320, {0x3b16e50, 0xc0004d8410}, {0xc0005c4860, 0x1, 0x1})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/backend/icloud/icloud.go:989 +0x1cd
github.com/rclone/rclone/fs/operations.(*ReOpen).open(0xc000f5e000)
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/reopen.go:147 +0x109
github.com/rclone/rclone/fs/operations.NewReOpen({0x3b16e50, 0xc0004d8410}, {0x3b2ddc0, 0xc000f30320}, 0xa, {0xc0005c4830, 0x1, 0x1})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/reopen.go:102 +0x50b
github.com/rclone/rclone/fs/operations.Open({0x3b16e50, 0xc0004d8410}, {0x3b2ddc0, 0xc000f30320}, {0xc0005c4830, 0x1, 0x1})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/reopen.go:124 +0x6b
github.com/rclone/rclone/fs/operations.(*copy).manualCopy(0xc000f3a120, {0x3b16e50, 0xc0004d8410})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/copy.go:261 +0x4dc
github.com/rclone/rclone/fs/operations.(*copy).copy(0xc000f3a120, {0x3b16e50, 0xc0004d8410})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/copy.go:310 +0x176
github.com/rclone/rclone/fs/operations.Copy({0x3b16e50, 0xc0004d8410}, {0x3b2e290, 0xc00054e500}, {0x0, 0x0}, {0xc0005993c0, 0x3e}, {0x3b2ddc0, 0xc000f30320})
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/operations/copy.go:412 +0x493
github.com/rclone/rclone/fs/sync.(*syncCopyMove).pairCopyOrMove(0xc000a26288, {0x3b16e50, 0xc0004d8410}, 0xc0004d8370, {0x3b2e290, 0xc00054e500}, 0x19, 0x0?)
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/sync/sync.go:501 +0x1fe
created by github.com/rclone/rclone/fs/sync.(*syncCopyMove).startTransfers in goroutine 1
        /home/pkg/pkgbuild/rclone/rclone-icloud/rclone/fs/sync/sync.go:531 +0x6b
```
lostb1t: thats because the copy endpoint is broken in icloud web. Hopefulky they fix it soon
cyayon: The 'ls' -like commands work now.
But does this invalid memory error should not be catched properly to avoid segmentation fault ? 
lostb1t: yes, yes it should. Should be fixed.
cyayon: > yes, yes it should. Should be fixed.

Thanks, everything seems to work now.
I am only using copy from iCloud to local drive command and got no more issue.
Thanks !
lostb1t: > @lostb1t Firstly, thank you for a great piece of work - this will be a very popular backend I think.
> 
> 
> 
> > > do you want to try to get this backend in to v1.69
> 
> > 
> 
> > sure, waiting on a review from one of the maintainers tho
> 
> 
> 
> That would be me! I've been keeping an eye on this thread waiting for it to settle down and waiting for the right time in the release cycle. I think the timing looks good for merging now :-)
> 
> 
> 
> ## Things to do
> 
> 
> 
> - Get the CI running clean
> 
> - Rebase on master
> 
> - Make sure the `test_all` integration tests are passing
> 
> - I'll do a thorough review once those 3 things are good
> 
> 
> 
> ### Other OS build
> 
> 
> 
> For the `other os` build the easiest way will be to exclude these OSes
> 
> 
> 
> -  plan9
> 
> -  solaris
> 
> 
> 
> See the sftp backend for an example of how to do this
> 
> 
> 
> https://github.com/rclone/rclone/blob/1bc9b94cf203ae25da293df5d51f7d9ba51c21ac/backend/sftp/sftp.go#L1
> 
> 
> 
> Note you will need an unsupported file also
> 
> 
> 
> https://github.com/rclone/rclone/blob/1bc9b94cf203ae25da293df5d51f7d9ba51c21ac/backend/sftp/sftp_unsupported.go#L1-L2
> 
> 
> 
> ### Lint
> 
> 
> 
> If you rebase the PR on `master` most of the lint errors will go away. There may be some remaining - I didn't check
> 
> 
> 
> ## Questions
> 
> 
> 
> - Do the `test_all` integration tests pass?

once apple fixes icloud.com it should pass. Ill ping you wheh that happens.
> 
> - How likely is this backend to keep working - I've noticed a lot of back and forward in this thread changing stuff? Is this Apple changing stuff or just stuff which isn't quite right yet?

Lots of scripts been using the web api endpoints without big issues for a while now. Sometimes things break like every other products. Alto Apple seems pretty slow with fixing.
> 
> - How likely is it to upset Apple in some way?

Its a public api they don't seem to care. 
> 
> - Have all the learnings from this PR gone into the docs? (eg notes on 2FA etc)

Should be, i dont mention 2FA explicitly as its mandatory and cant be disabled.
> 
> - Would you like to become an rclone maintainer to help look after this backend once it is merged?

Yup im willing to maintain this backend
> 
> 
> 
> Thank you :-)
> 
> 


Kuuhhl: Hi, is there any plans to make it work with Advanced Data Protection enabled?
lostb1t: > Hi, is there any plans to make it work with Advanced Data Protection enabled?

I have currently have no plans to add it. PRs are always welcome ofcourse.
lostb1t: @ncw seems apple is not in a hurry to fix it so i disabled server side copy for now

linting passes and test_all passes so ready for review.
ncw: @lostb1 

> thats because the copy endpoint is broken in icloud web. Hopefulky they fix it soon

Can you make this fail a bit more gracefully than a segfault :-) An error message would be nice!

Will take a look at the code now.



lostb1t: @ncw yeah thats handled properly now
lostb1t: thanks will go through them soon.

I don't have a solution for the 30 day. 2fa cannot be disabled. And can only be send to trusted devices.

mocking the api would be an option but that kinda defeats the purpose of an integration test.
lostb1t: @ncw i think thats everything. Let me know if theres something else
ncw: There is a problem with the `other_os` build

```
2024/09/24 09:50:30 Error compiling plan9/amd64: exit status 1
2024/09/24 09:50:45 ----------------------------
2024/09/24 09:50:45 Failed to run [go build --ldflags -s -X github.com/rclone/rclone/fs.Version=v1.69.0-beta.8333.94a9054bf.merge -trimpath -o rclone-v1.69.0-beta.8333.94a9054bf.merge-solaris-amd64/rclone -tags  ..]: exit status 1
2024/09/24 09:50:45 Command output was:
github.com/rclone/rclone/backend/icloud: build constraints exclude all Go files in /home/runner/work/rclone/rclone/backend/icloud
2024/09/24 09:50:45 ----------------------------
2024/09/24 09:50:45 Error compiling solaris/amd64: exit status 1
2024/09/24 09:50:45 Compiled 15 arches in 13m1.732945759s
2024/09/24 09:50:45 3 compile failures:
  plan9/386
  plan9/amd64
  solaris/amd64
```

I think this will fix it

```diff
--- a/backend/icloud/icloud_unsupported.go
+++ b/backend/icloud/icloud_unsupported.go
@@ -1,7 +1,7 @@
 // Build for icloud for unsupported platforms to stop go complaining
 // about "no buildable Go source files "
 
-//go:build plan9 && solaris
+//go:build plan9 || solaris
 
 // Package icloud implements the iCloud Drive backend
```

Note that the file `icloud_test.go` needs to have the build constraints `//go:build !plan9 && !solaris` otherwise if the tests are run on plan9 or solaris they will fail.

lostb1t: @ncw pushed the changes. Is there a way for me to see the CI for this PR? nothing seem to run under checks
ncw: Looks like it passed https://github.com/rclone/rclone/actions/runs/11032488454/ :-)

I think darthShadow re-ran it so it probably had a hiccough.

Will take another look now.
mattephi: I can not make this backend to work properly. My usecase: mount icloud to use Logseq graphs synced though icloud. Current issue is that app directories (Shortcuts, Logseq, Automator, etc) are just shown as empty 0 bytes files instead of directories. This way they are not properly indexed and do not sync files inside. Is there anything I am doing wrong?
1. add icloud backend
2. mount to some folder / run ls
3. app directories simply indexed as 0 bytes files
4. Trying to directly ls those folders results are on the second screenshot

![image](https://github.com/user-attachments/assets/e04910ba-13d5-4496-8603-e86daba17ed2)
![image](https://github.com/user-attachments/assets/1dd769f1-7f8f-4e01-a279-b712ac040b4f)

@lostb1t 
lostb1t: @mattephi yeah had to change some stuff when apple broke some e endpoints. Pushed a fix for the app directories.

Im not sure whats going on with the directory sizes. Should be working but rclone is not picking it up. Will look into it
lostb1t: @ncw 

Named it to iclouddrive, i can't seem to find any missing docs. Care to point me in the right direction?


Also i cant seem to make the directory size show correctly. I set the size with setsize on the listings: https://github.com/lostb1t/rclone/blob/feature/icloud/backend/iclouddrive/iclouddrive.go#L363

but it doesn't take. Am i missing something?
ncw: > @ncw
> 
> Named it to iclouddrive, i can't seem to find any missing docs. Care to point me in the right direction?

There are two tables in this doc https://github.com/rclone/rclone/blob/master/docs/content/overview.md but you have only edited one of them :-)

> Also i cant seem to make the directory size show correctly. I set the size with setsize on the listings: https://github.com/lostb1t/rclone/blob/feature/icloud/backend/iclouddrive/iclouddrive.go#L363
> 
> but it doesn't take. Am i missing something?

Code looks correct... How are you looking for it? You should see it with `rclone lsd` and `rclone lsjson`


lostb1t: > > @ncw
> 
> > 
> 
> > Named it to iclouddrive, i can't seem to find any missing docs. Care to point me in the right direction?
> 
> 
> 
> There are two tables in this doc https://github.com/rclone/rclone/blob/master/docs/content/overview.md but you have only edited one of them :-)
> 
Scrolling is hard....
> 
> 
> > Also i cant seem to make the directory size show correctly. I set the size with setsize on the listings: https://github.com/lostb1t/rclone/blob/feature/icloud/backend/iclouddrive/iclouddrive.go#L363
> 
> > 
> 
> > but it doesn't take. Am i missing something?
> 
> 
> 
> Code looks correct... How are you looking for it? You should see it with `rclone lsd` and `rclone lsjson`
> 
Using mount it doesn't show. But yes lsd works.
> 
> 
> 

K well that should be all, fixed the doc and squashed commits. 

This bad boy is ready for prime time.

![image](https://github.com/user-attachments/assets/0488de9e-7222-4ecc-8d0a-64063bb26232)

;)

@ncw 

Patch:
diff --git a/README.md b/README.md
index 9cbd74cb5..866986479 100644
--- a/README.md
+++ b/README.md
@@ -66,6 +66,7 @@ Rclone *("rsync for cloud storage")* is a command-line program to sync files and
   * HiDrive [:page_facing_up:](https://rclone.org/hidrive/)
   * HTTP [:page_facing_up:](https://rclone.org/http/)
   * Huawei Cloud Object Storage Service(OBS) [:page_facing_up:](https://rclone.org/s3/#huawei-obs)
+  * iCloud Drive [:page_facing_up:](https://rclone.org/iclouddrive/)
   * ImageKit [:page_facing_up:](https://rclone.org/imagekit/)
   * Internet Archive [:page_facing_up:](https://rclone.org/internetarchive/)
   * Jottacloud [:page_facing_up:](https://rclone.org/jottacloud/)
diff --git a/backend/all/all.go b/backend/all/all.go
index a9191c98d..1abd6770f 100644
--- a/backend/all/all.go
+++ b/backend/all/all.go
@@ -26,6 +26,7 @@ import (
 	_ "github.com/rclone/rclone/backend/hdfs"
 	_ "github.com/rclone/rclone/backend/hidrive"
 	_ "github.com/rclone/rclone/backend/http"
+	_ "github.com/rclone/rclone/backend/iclouddrive"
 	_ "github.com/rclone/rclone/backend/imagekit"
 	_ "github.com/rclone/rclone/backend/internetarchive"
 	_ "github.com/rclone/rclone/backend/jottacloud"
diff --git a/backend/box/box.go b/backend/box/box.go
index 4f38955eb..7183f35e5 100644
--- a/backend/box/box.go
+++ b/backend/box/box.go
@@ -43,6 +43,7 @@ import (
 	"github.com/rclone/rclone/lib/jwtutil"
 	"github.com/rclone/rclone/lib/oauthutil"
 	"github.com/rclone/rclone/lib/pacer"
+	"github.com/rclone/rclone/lib/random"
 	"github.com/rclone/rclone/lib/rest"
 	"github.com/youmark/pkcs8"
 	"golang.org/x/oauth2"
@@ -256,7 +257,6 @@ func getQueryParams(boxConfig *api.ConfigJSON) map[string]string {
 }
 
 func getDecryptedPrivateKey(boxConfig *api.ConfigJSON) (key *rsa.PrivateKey, err error) {
-
 	block, rest := pem.Decode([]byte(boxConfig.BoxAppSettings.AppAuth.PrivateKey))
 	if len(rest) > 0 {
 		return nil, fmt.Errorf("box: extra data included in private key: %w", err)
@@ -619,7 +619,7 @@ func (f *Fs) CreateDir(ctx context.Context, pathID, leaf string) (newID string,
 		return shouldRetry(ctx, resp, err)
 	})
 	if err != nil {
-		//fmt.Printf("...Error %v\n", err)
+		// fmt.Printf("...Error %v\n", err)
 		return "", err
 	}
 	// fmt.Printf("...Id %q\n", *info.Id)
@@ -966,6 +966,26 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
 		return nil, err
 	}
 
+	// check if dest already exists
+	item, err := f.preUploadCheck(ctx, leaf, directoryID, src.Size())
+	if err != nil {
+		return nil, err
+	}
+	if item != nil { // dest already exists, need to copy to temp name and then move
+		tempSuffix := "-rclone-copy-" + random.String(8)
+		fs.Debugf(remote, "dst already exists, copying to temp name %v", remote+tempSuffix)
+		tempObj, err := f.Copy(ctx, src, remote+tempSuffix)
+		if err != nil {
+			return nil, err
+		}
+		fs.Debugf(remote+tempSuffix, "moving to real name %v", remote)
+		err = f.deleteObject(ctx, item.ID)
+		if err != nil {
+			return nil, err
+		}
+		return f.Move(ctx, tempObj, remote)
+	}
+
 	// Copy the object
 	opts := rest.Opts{
 		Method:     "POST",
diff --git a/backend/drive/drive.go b/backend/drive/drive.go
index 3feef0ef2..f39800af0 100644
--- a/backend/drive/drive.go
+++ b/backend/drive/drive.go
@@ -120,6 +120,7 @@ var (
 		"text/html":                 ".html",
 		"text/plain":                ".txt",
 		"text/tab-separated-values": ".tsv",
+		"text/markdown":             ".md",
 	}
 	_mimeTypeToExtensionLinks = map[string]string{
 		"application/x-link-desktop": ".desktop",
diff --git a/backend/iclouddrive/api/client.go b/backend/iclouddrive/api/client.go
new file mode 100644
index 000000000..7cdf74baf
--- /dev/null
+++ b/backend/iclouddrive/api/client.go
@@ -0,0 +1,166 @@
+// Package api provides functionality for interacting with the iCloud API.
+package api
+
+import (
+	"bytes"
+	"context"
+	"encoding/json"
+	"errors"
+	"fmt"
+	"net/http"
+	"strings"
+
+	"github.com/rclone/rclone/fs"
+	"github.com/rclone/rclone/fs/fshttp"
+	"github.com/rclone/rclone/lib/rest"
+)
+
+const (
+	baseEndpoint  = "https://www.icloud.com"
+	homeEndpoint  = "https://www.icloud.com"
+	setupEndpoint = "https://setup.icloud.com/setup/ws/1"
+	authEndpoint  = "https://idmsa.apple.com/appleauth/auth"
+)
+
+type sessionSave func(*Session)
+
+// Client defines the client configuration
+type Client struct {
+	appleID             string
+	password            string
+	srv                 *rest.Client
+	Session             *Session
+	sessionSaveCallback sessionSave
+
+	drive *DriveService
+}
+
+// New creates a new Client instance with the provided Apple ID, password, trust token, cookies, and session save callback.
+//
+// Parameters:
+// - appleID: the Apple ID of the user.
+// - password: the password of the user.
+// - trustToken: the trust token for the session.
+// - clientID: the client id for the session.
+// - cookies: the cookies for the session.
+// - sessionSaveCallback: the callback function to save the session.
+func New(appleID, password, trustToken string, clientID string, cookies []*http.Cookie, sessionSaveCallback sessionSave) (*Client, error) {
+	icloud := &Client{
+		appleID:             appleID,
+		password:            password,
+		srv:                 rest.NewClient(fshttp.NewClient(context.Background())),
+		Session:             NewSession(),
+		sessionSaveCallback: sessionSaveCallback,
+	}
+
+	icloud.Session.TrustToken = trustToken
+	icloud.Session.Cookies = cookies
+	icloud.Session.ClientID = clientID
+	return icloud, nil
+}
+
+// DriveService returns the DriveService instance associated with the Client.
+func (c *Client) DriveService() (*DriveService, error) {
+	var err error
+	if c.drive == nil {
+		c.drive, err = NewDriveService(c)
+		if err != nil {
+			return nil, err
+		}
+	}
+	return c.drive, nil
+}
+
+// Request makes a request and retries it if the session is invalid.
+//
+// This function is the main entry point for making requests to the iCloud
+// API. If the initial request returns a 401 (Unauthorized), it will try to
+// reauthenticate and retry the request.
+func (c *Client) Request(ctx context.Context, opts rest.Opts, request interface{}, response interface{}) (resp *http.Response, err error) {
+	resp, err = c.Session.Request(ctx, opts, request, response)
+	if err != nil && resp != nil {
+		// try to reauth
+		if resp.StatusCode == 401 || resp.StatusCode == 421 {
+			err = c.Authenticate(ctx)
+			if err != nil {
+				return nil, err
+			}
+
+			if c.Session.Requires2FA() {
+				return nil, errors.New("trust token expired, please reauth")
+			}
+			return c.RequestNoReAuth(ctx, opts, request, response)
+		}
+	}
+	return resp, err
+}
+
+// RequestNoReAuth makes a request without re-authenticating.
+//
+// This function is useful when you have a session that is already
+// authenticated, but you need to make a request without triggering
+// a re-authentication.
+func (c *Client) RequestNoReAuth(ctx context.Context, opts rest.Opts, request interface{}, response interface{}) (resp *http.Response, err error) {
+	// Make the request without re-authenticating
+	resp, err = c.Session.Request(ctx, opts, request, response)
+	return resp, err
+}
+
+// Authenticate authenticates the client with the iCloud API.
+func (c *Client) Authenticate(ctx context.Context) error {
+	if c.Session.Cookies != nil {
+		if err := c.Session.ValidateSession(ctx); err == nil {
+			fs.Debugf("icloud", "Valid session, no need to reauth")
+			return nil
+		}
+		c.Session.Cookies = nil
+	}
+
+	fs.Debugf("icloud", "Authenticating as %s\n", c.appleID)
+	err := c.Session.SignIn(ctx, c.appleID, c.password)
+
+	if err == nil {
+		err = c.Session.AuthWithToken(ctx)
+		if err == nil && c.sessionSaveCallback != nil {
+			c.sessionSaveCallback(c.Session)
+		}
+	}
+	return err
+}
+
+// SignIn signs in the client using the provided context and credentials.
+func (c *Client) SignIn(ctx context.Context) error {
+	return c.Session.SignIn(ctx, c.appleID, c.password)
+}
+
+// IntoReader marshals the provided values into a JSON encoded reader
+func IntoReader(values any) (*bytes.Reader, error) {
+	m, err := json.Marshal(values)
+	if err != nil {
+		return nil, err
+	}
+	return bytes.NewReader(m), nil
+}
+
+// RequestError holds info on a result state, icloud can return a 200 but the result is unknown
+type RequestError struct {
+	Status string
+	Text   string
+}
+
+// Error satisfy the error interface.
+func (e *RequestError) Error() string {
+	return fmt.Sprintf("%s: %s", e.Text, e.Status)
+}
+
+func newRequestError(Status string, Text string) *RequestError {
+	return &RequestError{
+		Status: strings.ToLower(Status),
+		Text:   Text,
+	}
+}
+
+// newErr orf makes a new error from sprintf parameters.
+func newRequestErrorf(Status string, Text string, Parameters ...interface{}) *RequestError {
+	return newRequestError(strings.ToLower(Status), fmt.Sprintf(Text, Parameters...))
+}
diff --git a/backend/iclouddrive/api/drive.go b/backend/iclouddrive/api/drive.go
new file mode 100644
index 000000000..5d735ebbd
--- /dev/null
+++ b/backend/iclouddrive/api/drive.go
@@ -0,0 +1,913 @@
+package api
+
+import (
+	"bytes"
+	"context"
+	"io"
+	"mime"
+	"net/http"
+	"net/url"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/google/uuid"
+	"github.com/rclone/rclone/fs"
+	"github.com/rclone/rclone/lib/rest"
+)
+
+const (
+	defaultZone        = "com.apple.CloudDocs"
+	statusOk           = "OK"
+	statusEtagConflict = "ETAG_CONFLICT"
+)
+
+// DriveService represents an iCloud Drive service.
+type DriveService struct {
+	icloud       *Client
+	RootID       string
+	endpoint     string
+	docsEndpoint string
+}
+
+// NewDriveService creates a new DriveService instance.
+func NewDriveService(icloud *Client) (*DriveService, error) {
+	return &DriveService{icloud: icloud, RootID: "FOLDER::com.apple.CloudDocs::root", endpoint: icloud.Session.AccountInfo.Webservices["drivews"].URL, docsEndpoint: icloud.Session.AccountInfo.Webservices["docws"].URL}, nil
+}
+
+// GetItemByDriveID retrieves a DriveItem by its Drive ID.
+func (d *DriveService) GetItemByDriveID(ctx context.Context, id string, includeChildren bool) (*DriveItem, *http.Response, error) {
+	items, resp, err := d.GetItemsByDriveID(ctx, []string{id}, includeChildren)
+	if err != nil {
+		return nil, resp, err
+	}
+	return items[0], resp, err
+}
+
+// GetItemsByDriveID retrieves DriveItems by their Drive IDs.
+func (d *DriveService) GetItemsByDriveID(ctx context.Context, ids []string, includeChildren bool) ([]*DriveItem, *http.Response, error) {
+	var err error
+	_items := []map[string]any{}
+	for _, id := range ids {
+		_items = append(_items, map[string]any{
+			"drivewsid":        id,
+			"partialData":      false,
+			"includeHierarchy": false,
+		})
+	}
+
+	var body *bytes.Reader
+	var path string
+	if !includeChildren {
+		values := []map[string]any{{
+			"items": _items,
+		}}
+		body, err = IntoReader(values)
+		if err != nil {
+			return nil, nil, err
+		}
+		path = "/retrieveItemDetails"
+	} else {
+		values := _items
+		body, err = IntoReader(values)
+		if err != nil {
+			return nil, nil, err
+		}
+		path = "/retrieveItemDetailsInFolders"
+	}
+
+	opts := rest.Opts{
+		Method:       "POST",
+		Path:         path,
+		ExtraHeaders: d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:      d.endpoint,
+		Body:         body,
+	}
+	var items []*DriveItem
+	resp, err := d.icloud.Request(ctx, opts, nil, &items)
+	if err != nil {
+		return nil, resp, err
+	}
+
+	return items, resp, err
+}
+
+// GetDocByPath retrieves a document by its path.
+func (d *DriveService) GetDocByPath(ctx context.Context, path string) (*Document, *http.Response, error) {
+	values := url.Values{}
+	values.Set("unified_format", "false")
+	body, err := IntoReader(path)
+	if err != nil {
+		return nil, nil, err
+	}
+	opts := rest.Opts{
+		Method:       "POST",
+		Path:         "/ws/" + defaultZone + "/list/lookup_by_path",
+		ExtraHeaders: d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:      d.docsEndpoint,
+		Parameters:   values,
+		Body:         body,
+	}
+	var item []*Document
+	resp, err := d.icloud.Request(ctx, opts, nil, &item)
+	if err != nil {
+		return nil, resp, err
+	}
+
+	return item[0], resp, err
+}
+
+// GetItemByPath retrieves a DriveItem by its path.
+func (d *DriveService) GetItemByPath(ctx context.Context, path string) (*DriveItem, *http.Response, error) {
+	values := url.Values{}
+	values.Set("unified_format", "true")
+
+	body, err := IntoReader(path)
+	if err != nil {
+		return nil, nil, err
+	}
+	opts := rest.Opts{
+		Method:       "POST",
+		Path:         "/ws/" + defaultZone + "/list/lookup_by_path",
+		ExtraHeaders: d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:      d.docsEndpoint,
+		Parameters:   values,
+		Body:         body,
+	}
+	var item []*DriveItem
+	resp, err := d.icloud.Request(ctx, opts, nil, &item)
+	if err != nil {
+		return nil, resp, err
+	}
+
+	return item[0], resp, err
+}
+
+// GetDocByItemID retrieves a document by its item ID.
+func (d *DriveService) GetDocByItemID(ctx context.Context, id string) (*Document, *http.Response, error) {
+	values := url.Values{}
+	values.Set("document_id", id)
+	values.Set("unified_format", "false") // important
+	opts := rest.Opts{
+		Method:       "GET",
+		Path:         "/ws/" + defaultZone + "/list/lookup_by_id",
+		ExtraHeaders: d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:      d.docsEndpoint,
+		Parameters:   values,
+	}
+	var item *Document
+	resp, err := d.icloud.Request(ctx, opts, nil, &item)
+	if err != nil {
+		return nil, resp, err
+	}
+
+	return item, resp, err
+}
+
+// GetItemRawByItemID retrieves a DriveItemRaw by its item ID.
+func (d *DriveService) GetItemRawByItemID(ctx context.Context, id string) (*DriveItemRaw, *http.Response, error) {
+	opts := rest.Opts{
+		Method:       "GET",
+		Path:         "/v1/item/" + id,
+		ExtraHeaders: d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:      d.docsEndpoint,
+	}
+	var item *DriveItemRaw
+	resp, err := d.icloud.Request(ctx, opts, nil, &item)
+	if err != nil {
+		return nil, resp, err
+	}
+
+	return item, resp, err
+}
+
+// GetItemsInFolder retrieves a list of DriveItemRaw objects in a folder with the given ID.
+func (d *DriveService) GetItemsInFolder(ctx context.Context, id string, limit int64) ([]*DriveItemRaw, *http.Response, error) {
+	values := url.Values{}
+	values.Set("limit", strconv.FormatInt(limit, 10))
+
+	opts := rest.Opts{
+		Method:       "GET",
+		Path:         "/v1/enumerate/" + id,
+		ExtraHeaders: d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:      d.docsEndpoint,
+		Parameters:   values,
+	}
+
+	items := struct {
+		Items []*DriveItemRaw `json:"drive_item"`
+	}{}
+
+	resp, err := d.icloud.Request(ctx, opts, nil, &items)
+	if err != nil {
+		return nil, resp, err
+	}
+
+	return items.Items, resp, err
+}
+
+// GetDownloadURLByDriveID retrieves the download URL for a file in the DriveService.
+func (d *DriveService) GetDownloadURLByDriveID(ctx context.Context, id string) (string, *http.Response, error) {
+	_, zone, docid := DeconstructDriveID(id)
+	values := url.Values{}
+	values.Set("document_id", docid)
+
+	if zone == "" {
+		zone = defaultZone
+	}
+
+	opts := rest.Opts{
+		Method:       "GET",
+		Path:         "/ws/" + zone + "/download/by_id",
+		Parameters:   values,
+		ExtraHeaders: d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:      d.docsEndpoint,
+	}
+
+	var filer *FileRequest
+	resp, err := d.icloud.Request(ctx, opts, nil, &filer)
+
+	if err != nil {
+		return "", resp, err
+	}
+
+	var url string
+	if filer.DataToken != nil {
+		url = filer.DataToken.URL
+	} else {
+		url = filer.PackageToken.URL
+	}
+
+	return url, resp, err
+}
+
+// DownloadFile downloads a file from the given URL using the provided options.
+func (d *DriveService) DownloadFile(ctx context.Context, url string, opt []fs.OpenOption) (*http.Response, error) {
+	opts := &rest.Opts{
+		Method:       "GET",
+		ExtraHeaders: d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:      url,
+		Options:      opt,
+	}
+
+	resp, err := d.icloud.srv.Call(ctx, opts)
+	if err != nil {
+		// icloud has some weird http codes
+		if resp.StatusCode == 330 {
+			loc, err := resp.Location()
+			if err == nil {
+				return d.DownloadFile(ctx, loc.String(), opt)
+			}
+		}
+
+		return resp, err
+	}
+	return d.icloud.srv.Call(ctx, opts)
+}
+
+// MoveItemToTrashByItemID moves an item to the trash based on the item ID.
+func (d *DriveService) MoveItemToTrashByItemID(ctx context.Context, id, etag string, force bool) (*DriveItem, *http.Response, error) {
+	doc, resp, err := d.GetDocByItemID(ctx, id)
+	if err != nil {
+		return nil, resp, err
+	}
+	return d.MoveItemToTrashByID(ctx, doc.DriveID(), etag, force)
+}
+
+// MoveItemToTrashByID moves an item to the trash based on the item ID.
+func (d *DriveService) MoveItemToTrashByID(ctx context.Context, drivewsid, etag string, force bool) (*DriveItem, *http.Response, error) {
+	values := map[string]any{
+		"items": []map[string]any{{
+			"drivewsid": drivewsid,
+			"etag":      etag,
+			"clientId":  drivewsid,
+		}}}
+
+	body, err := IntoReader(values)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	opts := rest.Opts{
+		Method:       "POST",
+		Path:         "/moveItemsToTrash",
+		ExtraHeaders: d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:      d.endpoint,
+		Body:         body,
+	}
+
+	item := struct {
+		Items []*DriveItem `json:"items"`
+	}{}
+	resp, err := d.icloud.Request(ctx, opts, nil, &item)
+
+	if err != nil {
+		return nil, resp, err
+	}
+
+	if item.Items[0].Status != statusOk {
+		// rerun with latest etag
+		if force && item.Items[0].Status == "ETAG_CONFLICT" {
+			return d.MoveItemToTrashByID(ctx, drivewsid, item.Items[0].Etag, false)
+		}
+
+		err = newRequestError(item.Items[0].Status, "unknown request status")
+	}
+
+	return item.Items[0], resp, err
+}
+
+// CreateNewFolderByItemID creates a new folder by item ID.
+func (d *DriveService) CreateNewFolderByItemID(ctx context.Context, id, name string) (*DriveItem, *http.Response, error) {
+	doc, resp, err := d.GetDocByItemID(ctx, id)
+	if err != nil {
+		return nil, resp, err
+	}
+	return d.CreateNewFolderByDriveID(ctx, doc.DriveID(), name)
+}
+
+// CreateNewFolderByDriveID creates a new folder by its Drive ID.
+func (d *DriveService) CreateNewFolderByDriveID(ctx context.Context, drivewsid, name string) (*DriveItem, *http.Response, error) {
+	values := map[string]any{
+		"destinationDrivewsId": drivewsid,
+		"folders": []map[string]any{{
+			"clientId": "FOLDER::UNKNOWN_ZONE::TempId-" + uuid.New().String(),
+			"name":     name,
+		}},
+	}
+
+	body, err := IntoReader(values)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	opts := rest.Opts{
+		Method:       "POST",
+		Path:         "/createFolders",
+		ExtraHeaders: d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:      d.endpoint,
+		Body:         body,
+	}
+	var fResp *CreateFoldersResponse
+	resp, err := d.icloud.Request(ctx, opts, nil, &fResp)
+	if err != nil {
+		return nil, resp, err
+	}
+	status := fResp.Folders[0].Status
+	if status != statusOk {
+		err = newRequestError(status, "unknown request status")
+	}
+
+	return fResp.Folders[0], resp, err
+}
+
+// RenameItemByItemID renames a DriveItem by its item ID.
+func (d *DriveService) RenameItemByItemID(ctx context.Context, id, etag, name string, force bool) (*DriveItem, *http.Response, error) {
+	doc, resp, err := d.GetDocByItemID(ctx, id)
+	if err != nil {
+		return nil, resp, err
+	}
+	return d.RenameItemByDriveID(ctx, doc.DriveID(), doc.Etag, name, force)
+}
+
+// RenameItemByDriveID renames a DriveItem by its drive ID.
+func (d *DriveService) RenameItemByDriveID(ctx context.Context, id, etag, name string, force bool) (*DriveItem, *http.Response, error) {
+	values := map[string]any{
+		"items": []map[string]any{{
+			"drivewsid": id,
+			"name":      name,
+			"etag":      etag,
+			// "extension": split[1],
+		}},
+	}
+
+	body, err := IntoReader(values)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	opts := rest.Opts{
+		Method:       "POST",
+		Path:         "/renameItems",
+		ExtraHeaders: d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:      d.endpoint,
+		Body:         body,
+	}
+	var items *DriveItem
+	resp, err := d.icloud.Request(ctx, opts, nil, &items)
+
+	if err != nil {
+		return nil, resp, err
+	}
+
+	status := items.Items[0].Status
+	if status != statusOk {
+		// rerun with latest etag
+		if force && status == "ETAG_CONFLICT" {
+			return d.RenameItemByDriveID(ctx, id, items.Items[0].Etag, name, false)
+		}
+		err = newRequestErrorf(status, "unknown inner status for: %s %s", opts.Method, resp.Request.URL)
+	}
+
+	return items.Items[0], resp, err
+}
+
+// MoveItemByItemID moves an item by its item ID to a destination item ID.
+func (d *DriveService) MoveItemByItemID(ctx context.Context, id, etag, dstID string, force bool) (*DriveItem, *http.Response, error) {
+	docSrc, resp, err := d.GetDocByItemID(ctx, id)
+	if err != nil {
+		return nil, resp, err
+	}
+	docDst, resp, err := d.GetDocByItemID(ctx, dstID)
+	if err != nil {
+		return nil, resp, err
+	}
+	return d.MoveItemByDriveID(ctx, docSrc.DriveID(), docSrc.Etag, docDst.DriveID(), force)
+}
+
+// MoveItemByDocID moves an item by its doc ID.
+// func (d *DriveService) MoveItemByDocID(ctx context.Context, srcDocID, srcEtag, dstDocID string, force bool) (*DriveItem, *http.Response, error) {
+// 	return d.MoveItemByDriveID(ctx, srcDocID, srcEtag, docDst.DriveID(), force)
+// }
+
+// MoveItemByDriveID moves an item by its drive ID.
+func (d *DriveService) MoveItemByDriveID(ctx context.Context, id, etag, dstID string, force bool) (*DriveItem, *http.Response, error) {
+	values := map[string]any{
+		"destinationDrivewsId": dstID,
+		"items": []map[string]any{{
+			"drivewsid": id,
+			"etag":      etag,
+			"clientId":  id,
+		}},
+	}
+
+	body, err := IntoReader(values)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	opts := rest.Opts{
+		Method:       "POST",
+		Path:         "/moveItems",
+		ExtraHeaders: d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:      d.endpoint,
+		Body:         body,
+	}
+
+	var items *DriveItem
+	resp, err := d.icloud.Request(ctx, opts, nil, &items)
+
+	if err != nil {
+		return nil, resp, err
+	}
+
+	status := items.Items[0].Status
+	if status != statusOk {
+		// rerun with latest etag
+		if force && status == "ETAG_CONFLICT" {
+			return d.MoveItemByDriveID(ctx, id, items.Items[0].Etag, dstID, false)
+		}
+		err = newRequestErrorf(status, "unknown inner status for: %s %s", opts.Method, resp.Request.URL)
+	}
+
+	return items.Items[0], resp, err
+}
+
+// CopyDocByItemID copies a document by its item ID.
+func (d *DriveService) CopyDocByItemID(ctx context.Context, itemID string) (*DriveItemRaw, *http.Response, error) {
+	// putting name in info doesnt work. extension does work so assume this is a bug in the endpoint
+	values := map[string]any{
+		"info_to_update": map[string]any{},
+	}
+
+	body, err := IntoReader(values)
+	if err != nil {
+		return nil, nil, err
+	}
+	opts := rest.Opts{
+		Method:       "POST",
+		Path:         "/v1/item/copy/" + itemID,
+		ExtraHeaders: d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:      d.docsEndpoint,
+		Body:         body,
+	}
+
+	var info *DriveItemRaw
+	resp, err := d.icloud.Request(ctx, opts, nil, &info)
+	if err != nil {
+		return nil, resp, err
+	}
+	return info, resp, err
+}
+
+// CreateUpload creates an url for an upload.
+func (d *DriveService) CreateUpload(ctx context.Context, size int64, name string) (*UploadResponse, *http.Response, error) {
+	// first we need to request an upload url
+	values := map[string]any{
+		"filename":     name,
+		"type":         "FILE",
+		"size":         strconv.FormatInt(size, 10),
+		"content_type": GetContentTypeForFile(name),
+	}
+	body, err := IntoReader(values)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	opts := rest.Opts{
+		Method:       "POST",
+		Path:         "/ws/" + defaultZone + "/upload/web",
+		ExtraHeaders: d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:      d.docsEndpoint,
+		Body:         body,
+	}
+	var responseInfo []*UploadResponse
+	resp, err := d.icloud.Request(ctx, opts, nil, &responseInfo)
+	if err != nil {
+		return nil, resp, err
+	}
+	return responseInfo[0], resp, err
+}
+
+// Upload uploads a file to the given url
+func (d *DriveService) Upload(ctx context.Context, in io.Reader, size int64, name, uploadURL string) (*SingleFileResponse, *http.Response, error) {
+	// TODO: implement multipart upload
+	opts := rest.Opts{
+		Method:        "POST",
+		ExtraHeaders:  d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:       uploadURL,
+		Body:          in,
+		ContentLength: &size,
+		ContentType:   GetContentTypeForFile(name),
+		// MultipartContentName: "files",
+		MultipartFileName: name,
+	}
+	var singleFileResponse *SingleFileResponse
+	resp, err := d.icloud.Request(ctx, opts, nil, &singleFileResponse)
+	if err != nil {
+		return nil, resp, err
+	}
+	return singleFileResponse, resp, err
+}
+
+// UpdateFile updates a file in the DriveService.
+//
+// ctx: the context.Context object for the request.
+// r: a pointer to the UpdateFileInfo struct containing the information for the file update.
+// Returns a pointer to the DriveItem struct representing the updated file, the http.Response object, and an error if any.
+func (d *DriveService) UpdateFile(ctx context.Context, r *UpdateFileInfo) (*DriveItem, *http.Response, error) {
+	body, err := IntoReader(r)
+	if err != nil {
+		return nil, nil, err
+	}
+	opts := rest.Opts{
+		Method:       "POST",
+		Path:         "/ws/" + defaultZone + "/update/documents",
+		ExtraHeaders: d.icloud.Session.GetHeaders(map[string]string{}),
+		RootURL:      d.docsEndpoint,
+		Body:         body,
+	}
+	var responseInfo *DocumentUpdateResponse
+	resp, err := d.icloud.Request(ctx, opts, nil, &responseInfo)
+	if err != nil {
+		return nil, resp, err
+	}
+
+	doc := responseInfo.Results[0].Document
+	item := DriveItem{
+		Drivewsid:    "FILE::com.apple.CloudDocs::" + doc.DocumentID,
+		Docwsid:      doc.DocumentID,
+		Itemid:       doc.ItemID,
+		Etag:         doc.Etag,
+		ParentID:     doc.ParentID,
+		DateModified: time.Unix(r.Mtime, 0),
+		DateCreated:  time.Unix(r.Mtime, 0),
+		Type:         doc.Type,
+		Name:         doc.Name,
+		Size:         doc.Size,
+	}
+
+	return &item, resp, err
+}
+
+// UpdateFileInfo represents the information for an update to a file in the DriveService.
+type UpdateFileInfo struct {
+	AllowConflict   bool   `json:"allow_conflict"`
+	Btime           int64  `json:"btime"`
+	Command         string `json:"command"`
+	CreateShortGUID bool   `json:"create_short_guid"`
+	Data            struct {
+		Receipt            string `json:"receipt,omitempty"`
+		ReferenceSignature string `json:"reference_signature,omitempty"`
+		Signature          string `json:"signature,omitempty"`
+		Size               int64  `json:"size,omitempty"`
+		WrappingKey        string `json:"wrapping_key,omitempty"`
+	} `json:"data,omitempty"`
+	DocumentID string    `json:"document_id"`
+	FileFlags  FileFlags `json:"file_flags"`
+	Mtime      int64     `json:"mtime"`
+	Path       struct {
+		Path               string `json:"path"`
+		StartingDocumentID string `json:"starting_document_id"`
+	} `json:"path"`
+}
+
+// FileFlags defines the file flags for a document.
+type FileFlags struct {
+	IsExecutable bool `json:"is_executable"`
+	IsHidden     bool `json:"is_hidden"`
+	IsWritable   bool `json:"is_writable"`
+}
+
+// NewUpdateFileInfo creates a new UpdateFileInfo object with default values.
+//
+// Returns an UpdateFileInfo object.
+func NewUpdateFileInfo() UpdateFileInfo {
+	return UpdateFileInfo{
+		Command:         "add_file",
+		CreateShortGUID: true,
+		AllowConflict:   true,
+		FileFlags: FileFlags{
+			IsExecutable: true,
+			IsHidden:     false,
+			IsWritable:   false,
+		},
+	}
+}
+
+// DriveItemRaw is a raw drive item.
+// not suure what to call this but there seems to be a "unified" and non "unified" drive item response. This is the non unified.
+type DriveItemRaw struct {
+	ItemID   string            `json:"item_id"`
+	ItemInfo *DriveItemRawInfo `json:"item_info"`
+}
+
+// SplitName splits the name of a DriveItemRaw into its name and extension.
+//
+// It returns the name and extension as separate strings. If the name ends with a dot,
+// it means there is no extension, so an empty string is returned for the extension.
+// If the name does not contain a dot, it means
+func (d *DriveItemRaw) SplitName() (string, string) {
+	name := d.ItemInfo.Name
+	// ends with a dot, no extension
+	if strings.HasSuffix(name, ".") {
+		return name, ""
+	}
+	lastInd := strings.LastIndex(name, ".")
+
+	if lastInd == -1 {
+		return name, ""
+	}
+	return name[:lastInd], name[lastInd+1:]
+}
+
+// ModTime returns the modification time of the DriveItemRaw.
+//
+// It parses the ModifiedAt field of the ItemInfo struct and converts it to a time.Time value.
+// If the parsing fails, it returns the zero value of time.Time.
+// The returned time.Time value represents the modification time of the DriveItemRaw.
+func (d *DriveItemRaw) ModTime() time.Time {
+	i, err := strconv.ParseInt(d.ItemInfo.ModifiedAt, 10, 64)
+	if err != nil {
+		return time.Time{}
+	}
+	return time.UnixMilli(i)
+}
+
+// CreatedTime returns the creation time of the DriveItemRaw.
+//
+// It parses the CreatedAt field of the ItemInfo struct and converts it to a time.Time value.
+// If the parsing fails, it returns the zero value of time.Time.
+// The returned time.Time
+func (d *DriveItemRaw) CreatedTime() time.Time {
+	i, err := strconv.ParseInt(d.ItemInfo.CreatedAt, 10, 64)
+	if err != nil {
+		return time.Time{}
+	}
+	return time.UnixMilli(i)
+}
+
+// DriveItemRawInfo is the raw information about a drive item.
+type DriveItemRawInfo struct {
+	Name string `json:"name"`
+	// Extension is absolutely borked on endpoints so dont use it.
+	Extension  string `json:"extension"`
+	Size       int64  `json:"size,string"`
+	Type       string `json:"type"`
+	Version    string `json:"version"`
+	ModifiedAt string `json:"modified_at"`
+	CreatedAt  string `json:"created_at"`
+	Urls       struct {
+		URLDownload string `json:"url_download"`
+	} `json:"urls"`
+}
+
+// IntoDriveItem converts a DriveItemRaw into a DriveItem.
+//
+// It takes no parameters.
+// It returns a pointer to a DriveItem.
+func (d *DriveItemRaw) IntoDriveItem() *DriveItem {
+	name, extension := d.SplitName()
+	return &DriveItem{
+		Itemid:       d.ItemID,
+		Name:         name,
+		Extension:    extension,
+		Type:         d.ItemInfo.Type,
+		Etag:         d.ItemInfo.Version,
+		DateModified: d.ModTime(),
+		DateCreated:  d.CreatedTime(),
+		Size:         d.ItemInfo.Size,
+		Urls:         d.ItemInfo.Urls,
+	}
+}
+
+// DocumentUpdateResponse is the response of a document update request.
+type DocumentUpdateResponse struct {
+	Status struct {
+		StatusCode   int    `json:"status_code"`
+		ErrorMessage string `json:"error_message"`
+	} `json:"status"`
+	Results []struct {
+		Status struct {
+			StatusCode   int    `json:"status_code"`
+			ErrorMessage string `json:"error_message"`
+		} `json:"status"`
+		OperationID interface{} `json:"operation_id"`
+		Document    *Document   `json:"document"`
+	} `json:"results"`
+}
+
+// Document represents a document on iCloud.
+type Document struct {
+	Status struct {
+		StatusCode   int    `json:"status_code"`
+		ErrorMessage string `json:"error_message"`
+	} `json:"status"`
+	DocumentID string `json:"document_id"`
+	ItemID     string `json:"item_id"`
+	Urls       struct {
+		URLDownload string `json:"url_download"`
+	} `json:"urls"`
+	Etag           string       `json:"etag"`
+	ParentID       string       `json:"parent_id"`
+	Name           string       `json:"name"`
+	Type           string       `json:"type"`
+	Deleted        bool         `json:"deleted"`
+	Mtime          int64        `json:"mtime"`
+	LastEditorName string       `json:"last_editor_name"`
+	Data           DocumentData `json:"data"`
+	Size           int64        `json:"size"`
+	Btime          int64        `json:"btime"`
+	Zone           string       `json:"zone"`
+	FileFlags      struct {
+		IsExecutable bool `json:"is_executable"`
+		IsWritable   bool `json:"is_writable"`
+		IsHidden     bool `json:"is_hidden"`
+	} `json:"file_flags"`
+	LastOpenedTime   int64       `json:"lastOpenedTime"`
+	RestorePath      interface{} `json:"restorePath"`
+	HasChainedParent bool        `json:"hasChainedParent"`
+}
+
+// DriveID returns the drive ID of the Document.
+func (d *Document) DriveID() string {
+	if d.Zone == "" {
+		d.Zone = defaultZone
+	}
+	return d.Type + "::" + d.Zone + "::" + d.DocumentID
+}
+
+// DocumentData represents the data of a document.
+type DocumentData struct {
+	Signature          string `json:"signature"`
+	Owner              string `json:"owner"`
+	Size               int64  `json:"size"`
+	ReferenceSignature string `json:"reference_signature"`
+	WrappingKey        string `json:"wrapping_key"`
+	PcsInfo            string `json:"pcsInfo"`
+}
+
+// SingleFileResponse is the response of a single file request.
+type SingleFileResponse struct {
+	SingleFile *SingleFileInfo `json:"singleFile"`
+}
+
+// SingleFileInfo represents the information of a single file.
+type SingleFileInfo struct {
+	ReferenceSignature string `json:"referenceChecksum"`
+	Size               int64  `json:"size"`
+	Signature          string `json:"fileChecksum"`
+	WrappingKey        string `json:"wrappingKey"`
+	Receipt            string `json:"receipt"`
+}
+
+// UploadResponse is the response of an upload request.
+type UploadResponse struct {
+	URL        string `json:"url"`
+	DocumentID string `json:"document_id"`
+}
+
+// FileRequestToken represents the token of a file request.
+type FileRequestToken struct {
+	URL                string `json:"url"`
+	Token              string `json:"token"`
+	Signature          string `json:"signature"`
+	WrappingKey        string `json:"wrapping_key"`
+	ReferenceSignature string `json:"reference_signature"`
+}
+
+// FileRequest represents the request of a file.
+type FileRequest struct {
+	DocumentID   string            `json:"document_id"`
+	ItemID       string            `json:"item_id"`
+	OwnerDsid    int64             `json:"owner_dsid"`
+	DataToken    *FileRequestToken `json:"data_token,omitempty"`
+	PackageToken *FileRequestToken `json:"package_token,omitempty"`
+	DoubleEtag   string            `json:"double_etag"`
+}
+
+// CreateFoldersResponse is the response of a create folders request.
+type CreateFoldersResponse struct {
+	Folders []*DriveItem `json:"folders"`
+}
+
+// DriveItem represents an item on iCloud.
+type DriveItem struct {
+	DateCreated         time.Time    `json:"dateCreated"`
+	Drivewsid           string       `json:"drivewsid"`
+	Docwsid             string       `json:"docwsid"`
+	Itemid              string       `json:"item_id"`
+	Zone                string       `json:"zone"`
+	Name                string       `json:"name"`
+	ParentID            string       `json:"parentId"`
+	Hierarchy           []DriveItem  `json:"hierarchy"`
+	Etag                string       `json:"etag"`
+	Type                string       `json:"type"`
+	AssetQuota          int64        `json:"assetQuota"`
+	FileCount           int64        `json:"fileCount"`
+	ShareCount          int64        `json:"shareCount"`
+	ShareAliasCount     int64        `json:"shareAliasCount"`
+	DirectChildrenCount int64        `json:"directChildrenCount"`
+	Items               []*DriveItem `json:"items"`
+	NumberOfItems       int64        `json:"numberOfItems"`
+	Status              string       `json:"status"`
+	Extension           string       `json:"extension,omitempty"`
+	DateModified        time.Time    `json:"dateModified,omitempty"`
+	DateChanged         time.Time    `json:"dateChanged,omitempty"`
+	Size                int64        `json:"size,omitempty"`
+	LastOpenTime        time.Time    `json:"lastOpenTime,omitempty"`
+	Urls                struct {
+		URLDownload string `json:"url_download"`
+	} `json:"urls"`
+}
+
+// IsFolder returns true if the item is a folder.
+func (d *DriveItem) IsFolder() bool {
+	return d.Type == "FOLDER" || d.Type == "APP_CONTAINER" || d.Type == "APP_LIBRARY"
+}
+
+// DownloadURL returns the download URL of the item.
+func (d *DriveItem) DownloadURL() string {
+	return d.Urls.URLDownload
+}
+
+// FullName returns the full name of the item.
+// name + extension
+func (d *DriveItem) FullName() string {
+	if d.Extension != "" {
+		return d.Name + "." + d.Extension
+	}
+	return d.Name
+}
+
+// GetDocIDFromDriveID returns the DocumentID from the drive ID.
+func GetDocIDFromDriveID(id string) string {
+	split := strings.Split(id, "::")
+	return split[len(split)-1]
+}
+
+// DeconstructDriveID returns the document type, zone, and document ID from the drive ID.
+func DeconstructDriveID(id string) (docType, zone, docid string) {
+	split := strings.Split(id, "::")
+	if len(split) < 3 {
+		return "", "", id
+	}
+	return split[0], split[1], split[2]
+}
+
+// ConstructDriveID constructs a drive ID from the given components.
+func ConstructDriveID(id string, zone string, t string) string {
+	return strings.Join([]string{t, zone, id}, "::")
+}
+
+// GetContentTypeForFile detects content type for given file name.
+func GetContentTypeForFile(name string) string {
+	// detect MIME type by looking at the filename only
+	mimeType := mime.TypeByExtension(filepath.Ext(name))
+	if mimeType == "" {
+		// api requires a mime type passed in
+		mimeType = "text/plain"
+	}
+	return strings.Split(mimeType, ";")[0]
+}
diff --git a/backend/iclouddrive/api/session.go b/backend/iclouddrive/api/session.go
new file mode 100644
index 000000000..7ee350675
--- /dev/null
+++ b/backend/iclouddrive/api/session.go
@@ -0,0 +1,412 @@
+package api
+
+import (
+	"context"
+	"fmt"
+	"net/http"
+	"net/url"
+	"slices"
+	"strings"
+
+	"github.com/oracle/oci-go-sdk/v65/common"
+
+	"github.com/rclone/rclone/fs/fshttp"
+	"github.com/rclone/rclone/lib/rest"
+)
+
+// Session represents an iCloud session
+type Session struct {
+	SessionToken   string         `json:"session_token"`
+	Scnt           string         `json:"scnt"`
+	SessionID      string         `json:"session_id"`
+	AccountCountry string         `json:"account_country"`
+	TrustToken     string         `json:"trust_token"`
+	ClientID       string         `json:"client_id"`
+	Cookies        []*http.Cookie `json:"cookies"`
+	AccountInfo    AccountInfo    `json:"account_info"`
+
+	srv *rest.Client `json:"-"`
+}
+
+// String returns the session as a string
+// func (s *Session) String() string {
+// 	jsession, _ := json.Marshal(s)
+// 	return string(jsession)
+// }
+
+// Request makes a request
+func (s *Session) Request(ctx context.Context, opts rest.Opts, request interface{}, response interface{}) (*http.Response, error) {
+	resp, err := s.srv.CallJSON(ctx, &opts, &request, &response)
+
+	if err != nil {
+		return resp, err
+	}
+
+	if val := resp.Header.Get("X-Apple-ID-Account-Country"); val != "" {
+		s.AccountCountry = val
+	}
+	if val := resp.Header.Get("X-Apple-ID-Session-Id"); val != "" {
+		s.SessionID = val
+	}
+	if val := resp.Header.Get("X-Apple-Session-Token"); val != "" {
+		s.SessionToken = val
+	}
+	if val := resp.Header.Get("X-Apple-TwoSV-Trust-Token"); val != "" {
+		s.TrustToken = val
+	}
+	if val := resp.Header.Get("scnt"); val != "" {
+		s.Scnt = val
+	}
+
+	return resp, nil
+}
+
+// Requires2FA returns true if the session requires 2FA
+func (s *Session) Requires2FA() bool {
+	return s.AccountInfo.DsInfo.HsaVersion == 2 && s.AccountInfo.HsaChallengeRequired
+}
+
+// SignIn signs in the session
+func (s *Session) SignIn(ctx context.Context, appleID, password string) error {
+	trustTokens := []string{}
+	if s.TrustToken != "" {
+		trustTokens = []string{s.TrustToken}
+	}
+	values := map[string]any{
+		"accountName": appleID,
+		"password":    password,
+		"rememberMe":  true,
+		"trustTokens": trustTokens,
+	}
+	body, err := IntoReader(values)
+	if err != nil {
+		return err
+	}
+	opts := rest.Opts{
+		Method:       "POST",
+		Path:         "/signin",
+		Parameters:   url.Values{},
+		ExtraHeaders: s.GetAuthHeaders(map[string]string{}),
+		RootURL:      authEndpoint,
+		IgnoreStatus: true, // need to handle 409 for hsa2
+		NoResponse:   true,
+		Body:         body,
+	}
+	opts.Parameters.Set("isRememberMeEnabled", "true")
+	_, err = s.Request(ctx, opts, nil, nil)
+
+	return err
+
+}
+
+// AuthWithToken authenticates the session
+func (s *Session) AuthWithToken(ctx context.Context) error {
+	values := map[string]any{
+		"accountCountryCode": s.AccountCountry,
+		"dsWebAuthToken":     s.SessionToken,
+		"extended_login":     true,
+		"trustToken":         s.TrustToken,
+	}
+	body, err := IntoReader(values)
+	if err != nil {
+		return err
+	}
+	opts := rest.Opts{
+		Method:       "POST",
+		Path:         "/accountLogin",
+		ExtraHeaders: GetCommonHeaders(map[string]string{}),
+		RootURL:      setupEndpoint,
+		Body:         body,
+	}
+
+	resp, err := s.Request(ctx, opts, nil, &s.AccountInfo)
+	if err == nil {
+		s.Cookies = resp.Cookies()
+	}
+
+	return err
+}
+
+// Validate2FACode validates the 2FA code
+func (s *Session) Validate2FACode(ctx context.Context, code string) error {
+	values := map[string]interface{}{"securityCode": map[string]string{"code": code}}
+	body, err := IntoReader(values)
+	if err != nil {
+		return err
+	}
+
+	headers := s.GetAuthHeaders(map[string]string{})
+	headers["scnt"] = s.Scnt
+	headers["X-Apple-ID-Session-Id"] = s.SessionID
+
+	opts := rest.Opts{
+		Method:       "POST",
+		Path:         "/verify/trusteddevice/securitycode",
+		ExtraHeaders: headers,
+		RootURL:      authEndpoint,
+		Body:         body,
+		NoResponse:   true,
+	}
+
+	_, err = s.Request(ctx, opts, nil, nil)
+	if err == nil {
+		if err := s.TrustSession(ctx); err != nil {
+			return err
+		}
+
+		return nil
+	}
+
+	return fmt.Errorf("validate2FACode failed: %w", err)
+}
+
+// TrustSession trusts the session
+func (s *Session) TrustSession(ctx context.Context) error {
+	headers := s.GetAuthHeaders(map[string]string{})
+	headers["scnt"] = s.Scnt
+	headers["X-Apple-ID-Session-Id"] = s.SessionID
+
+	opts := rest.Opts{
+		Method:        "GET",
+		Path:          "/2sv/trust",
+		ExtraHeaders:  headers,
+		RootURL:       authEndpoint,
+		NoResponse:    true,
+		ContentLength: common.Int64(0),
+	}
+
+	_, err := s.Request(ctx, opts, nil, nil)
+	if err != nil {
+		return fmt.Errorf("trustSession failed: %w", err)
+	}
+
+	return s.AuthWithToken(ctx)
+}
+
+// ValidateSession validates the session
+func (s *Session) ValidateSession(ctx context.Context) error {
+	opts := rest.Opts{
+		Method:        "POST",
+		Path:          "/validate",
+		ExtraHeaders:  s.GetHeaders(map[string]string{}),
+		RootURL:       setupEndpoint,
+		ContentLength: common.Int64(0),
+	}
+	_, err := s.Request(ctx, opts, nil, &s.AccountInfo)
+	if err != nil {
+		return fmt.Errorf("validateSession failed: %w", err)
+	}
+
+	return nil
+}
+
+// GetAuthHeaders returns the authentication headers for the session.
+//
+// It takes an `overwrite` map[string]string parameter which allows
+// overwriting the default headers. It returns a map[string]string.
+func (s *Session) GetAuthHeaders(overwrite map[string]string) map[string]string {
+	headers := map[string]string{
+		"Accept":                           "application/json",
+		"Content-Type":                     "application/json",
+		"X-Apple-OAuth-Client-Id":          s.ClientID,
+		"X-Apple-OAuth-Client-Type":        "firstPartyAuth",
+		"X-Apple-OAuth-Redirect-URI":       "https://www.icloud.com",
+		"X-Apple-OAuth-Require-Grant-Code": "true",
+		"X-Apple-OAuth-Response-Mode":      "web_message",
+		"X-Apple-OAuth-Response-Type":      "code",
+		"X-Apple-OAuth-State":              s.ClientID,
+		"X-Apple-Widget-Key":               s.ClientID,
+		"Origin":                           homeEndpoint,
+		"Referer":                          fmt.Sprintf("%s/", homeEndpoint),
+		"User-Agent":                       "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0",
+	}
+	for k, v := range overwrite {
+		headers[k] = v
+	}
+	return headers
+}
+
+// GetHeaders Gets the authentication headers required for a request
+func (s *Session) GetHeaders(overwrite map[string]string) map[string]string {
+	headers := GetCommonHeaders(map[string]string{})
+	headers["Cookie"] = s.GetCookieString()
+	for k, v := range overwrite {
+		headers[k] = v
+	}
+	return headers
+}
+
+// GetCookieString returns the cookie header string for the session.
+func (s *Session) GetCookieString() string {
+	cookieHeader := ""
+	// we only care about name and value.
+	for _, cookie := range s.Cookies {
+		cookieHeader = cookieHeader + cookie.Name + "=" + cookie.Value + ";"
+	}
+	return cookieHeader
+}
+
+// GetCommonHeaders generates common HTTP headers with optional overwrite.
+func GetCommonHeaders(overwrite map[string]string) map[string]string {
+	headers := map[string]string{
+		"Content-Type": "application/json",
+		"Origin":       baseEndpoint,
+		"Referer":      fmt.Sprintf("%s/", baseEndpoint),
+		"User-Agent":   "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:103.0) Gecko/20100101 Firefox/103.0",
+	}
+	for k, v := range overwrite {
+		headers[k] = v
+	}
+	return headers
+}
+
+// MergeCookies merges two slices of http.Cookies, ensuring no duplicates are added.
+func MergeCookies(left []*http.Cookie, right []*http.Cookie) ([]*http.Cookie, error) {
+	var hashes []string
+	for _, cookie := range right {
+		hashes = append(hashes, cookie.Raw)
+	}
+	for _, cookie := range left {
+		if !slices.Contains(hashes, cookie.Raw) {
+			right = append(right, cookie)
+		}
+	}
+	return right, nil
+}
+
+// GetCookiesForDomain filters the provided cookies based on the domain of the given URL.
+func GetCookiesForDomain(url *url.URL, cookies []*http.Cookie) ([]*http.Cookie, error) {
+	var domainCookies []*http.Cookie
+	for _, cookie := range cookies {
+		if strings.HasSuffix(url.Host, cookie.Domain) {
+			domainCookies = append(domainCookies, cookie)
+		}
+	}
+	return domainCookies, nil
+}
+
+// NewSession creates a new Session instance with default values.
+func NewSession() *Session {
+	session := &Session{}
+	session.srv = rest.NewClient(fshttp.NewClient(context.Background())).SetRoot(baseEndpoint)
+	//session.ClientID = "auth-" + uuid.New().String()
+	return session
+}
+
+// AccountInfo represents an account info
+type AccountInfo struct {
+	DsInfo                       *ValidateDataDsInfo    `json:"dsInfo"`
+	HasMinimumDeviceForPhotosWeb bool                   `json:"hasMinimumDeviceForPhotosWeb"`
+	ICDPEnabled                  bool                   `json:"iCDPEnabled"`
+	Webservices                  map[string]*webService `json:"webservices"`
+	PcsEnabled                   bool                   `json:"pcsEnabled"`
+	TermsUpdateNeeded            bool                   `json:"termsUpdateNeeded"`
+	ConfigBag                    struct {
+		Urls struct {
+			AccountCreateUI     string `json:"accountCreateUI"`
+			AccountLoginUI      string `json:"accountLoginUI"`
+			AccountLogin        string `json:"accountLogin"`
+			AccountRepairUI     string `json:"accountRepairUI"`
+			DownloadICloudTerms string `json:"downloadICloudTerms"`
+			RepairDone          string `json:"repairDone"`
+			AccountAuthorizeUI  string `json:"accountAuthorizeUI"`
+			VettingURLForEmail  string `json:"vettingUrlForEmail"`
+			AccountCreate       string `json:"accountCreate"`
+			GetICloudTerms      string `json:"getICloudTerms"`
+			VettingURLForPhone  string `json:"vettingUrlForPhone"`
+		} `json:"urls"`
+		AccountCreateEnabled bool `json:"accountCreateEnabled"`
+	} `json:"configBag"`
+	HsaTrustedBrowser            bool     `json:"hsaTrustedBrowser"`
+	AppsOrder                    []string `json:"appsOrder"`
+	Version                      int      `json:"version"`
+	IsExtendedLogin              bool     `json:"isExtendedLogin"`
+	PcsServiceIdentitiesIncluded bool     `json:"pcsServiceIdentitiesIncluded"`
+	IsRepairNeeded               bool     `json:"isRepairNeeded"`
+	HsaChallengeRequired         bool     `json:"hsaChallengeRequired"`
+	RequestInfo                  struct {
+		Country  string `json:"country"`
+		TimeZone string `json:"timeZone"`
+		Region   string `json:"region"`
+	} `json:"requestInfo"`
+	PcsDeleted bool `json:"pcsDeleted"`
+	ICloudInfo struct {
+		SafariBookmarksHasMigratedToCloudKit bool `json:"SafariBookmarksHasMigratedToCloudKit"`
+	} `json:"iCloudInfo"`
+	Apps map[string]*ValidateDataApp `json:"apps"`
+}
+
+// ValidateDataDsInfo represents an validation info
+type ValidateDataDsInfo struct {
+	HsaVersion                         int           `json:"hsaVersion"`
+	LastName                           string        `json:"lastName"`
+	ICDPEnabled                        bool          `json:"iCDPEnabled"`
+	TantorMigrated                     bool          `json:"tantorMigrated"`
+	Dsid                               string        `json:"dsid"`
+	HsaEnabled                         bool          `json:"hsaEnabled"`
+	IsHideMyEmailSubscriptionActive    bool          `json:"isHideMyEmailSubscriptionActive"`
+	IroncadeMigrated                   bool          `json:"ironcadeMigrated"`
+	Locale                             string        `json:"locale"`
+	BrZoneConsolidated                 bool          `json:"brZoneConsolidated"`
+	ICDRSCapableDeviceList             string        `json:"ICDRSCapableDeviceList"`
+	IsManagedAppleID                   bool          `json:"isManagedAppleID"`
+	IsCustomDomainsFeatureAvailable    bool          `json:"isCustomDomainsFeatureAvailable"`
+	IsHideMyEmailFeatureAvailable      bool          `json:"isHideMyEmailFeatureAvailable"`
+	ContinueOnDeviceEligibleDeviceInfo []string      `json:"ContinueOnDeviceEligibleDeviceInfo"`
+	Gilligvited                        bool          `json:"gilligvited"`
+	AppleIDAliases                     []interface{} `json:"appleIdAliases"`
+	UbiquityEOLEnabled                 bool          `json:"ubiquityEOLEnabled"`
+	IsPaidDeveloper                    bool          `json:"isPaidDeveloper"`
+	CountryCode                        string        `json:"countryCode"`
+	NotificationID                     string        `json:"notificationId"`
+	PrimaryEmailVerified               bool          `json:"primaryEmailVerified"`
+	ADsID                              string        `json:"aDsID"`
+	Locked                             bool          `json:"locked"`
+	ICDRSCapableDeviceCount            int           `json:"ICDRSCapableDeviceCount"`
+	HasICloudQualifyingDevice          bool          `json:"hasICloudQualifyingDevice"`
+	PrimaryEmail                       string        `json:"primaryEmail"`
+	AppleIDEntries                     []struct {
+		IsPrimary bool   `json:"isPrimary"`
+		Type      string `json:"type"`
+		Value     string `json:"value"`
+	} `json:"appleIdEntries"`
+	GilliganEnabled    bool   `json:"gilligan-enabled"`
+	IsWebAccessAllowed bool   `json:"isWebAccessAllowed"`
+	FullName           string `json:"fullName"`
+	MailFlags          struct {
+		IsThreadingAvailable           bool `json:"isThreadingAvailable"`
+		IsSearchV2Provisioned          bool `json:"isSearchV2Provisioned"`
+		SCKMail                        bool `json:"sCKMail"`
+		IsMppSupportedInCurrentCountry bool `json:"isMppSupportedInCurrentCountry"`
+	} `json:"mailFlags"`
+	LanguageCode         string `json:"languageCode"`
+	AppleID              string `json:"appleId"`
+	HasUnreleasedOS      bool   `json:"hasUnreleasedOS"`
+	AnalyticsOptInStatus bool   `json:"analyticsOptInStatus"`
+	FirstName            string `json:"firstName"`
+	ICloudAppleIDAlias   string `json:"iCloudAppleIdAlias"`
+	NotesMigrated        bool   `json:"notesMigrated"`
+	BeneficiaryInfo      struct {
+		IsBeneficiary bool `json:"isBeneficiary"`
+	} `json:"beneficiaryInfo"`
+	HasPaymentInfo bool   `json:"hasPaymentInfo"`
+	PcsDelet       bool   `json:"pcsDelet"`
+	AppleIDAlias   string `json:"appleIdAlias"`
+	BrMigrated     bool   `json:"brMigrated"`
+	StatusCode     int    `json:"statusCode"`
+	FamilyEligible bool   `json:"familyEligible"`
+}
+
+// ValidateDataApp represents an app
+type ValidateDataApp struct {
+	CanLaunchWithOneFactor bool `json:"canLaunchWithOneFactor"`
+	IsQualifiedForBeta     bool `json:"isQualifiedForBeta"`
+}
+
+// WebService represents a web service
+type webService struct {
+	PcsRequired bool   `json:"pcsRequired"`
+	URL         string `json:"url"`
+	UploadURL   string `json:"uploadUrl"`
+	Status      string `json:"status"`
+}
diff --git a/backend/iclouddrive/iclouddrive.go b/backend/iclouddrive/iclouddrive.go
new file mode 100644
index 000000000..591d81718
--- /dev/null
+++ b/backend/iclouddrive/iclouddrive.go
@@ -0,0 +1,1174 @@
+//go:build !plan9 && !solaris
+
+// Package iclouddrive implements the iCloud Drive backend
+package iclouddrive
+
+import (
+	"bytes"
+	"context"
+	"path"
+
+	"errors"
+	"fmt"
+	"io"
+	"net/http"
+	"strings"
+	"time"
+
+	"github.com/rclone/rclone/fs"
+	"github.com/rclone/rclone/fs/config"
+	"github.com/rclone/rclone/fs/config/configmap"
+	"github.com/rclone/rclone/fs/config/configstruct"
+	"github.com/rclone/rclone/fs/config/obscure"
+	"github.com/rclone/rclone/fs/fserrors"
+
+	"github.com/rclone/rclone/backend/iclouddrive/api"
+	"github.com/rclone/rclone/fs/hash"
+	"github.com/rclone/rclone/lib/dircache"
+	"github.com/rclone/rclone/lib/encoder"
+	"github.com/rclone/rclone/lib/pacer"
+)
+
+/*
+- dirCache operates on relative path to root
+- path sanitization
+	- rule of thumb: sanitize before use, but store things as-is
+	- the paths cached in dirCache are after sanitizing
+	- the remote/dir passed in aren't, and are stored as-is
+*/
+
+const (
+	configAppleID    = "apple_id"
+	configPassword   = "password"
+	configClientID   = "client_id"
+	configCookies    = "cookies"
+	configTrustToken = "trust_token"
+
+	minSleep      = 10 * time.Millisecond
+	maxSleep      = 2 * time.Second
+	decayConstant = 2
+)
+
+// Register with Fs
+func init() {
+	fs.Register(&fs.RegInfo{
+		Name:        "iclouddrive",
+		Description: "iCloud Drive",
+		Config:      Config,
+		NewFs:       NewFs,
+		Options: []fs.Option{{
+			Name:      configAppleID,
+			Help:      "Apple ID.",
+			Required:  true,
+			Sensitive: true,
+		}, {
+			Name:       configPassword,
+			Help:       "Password.",
+			Required:   true,
+			IsPassword: true,
+			Sensitive:  true,
+		}, {
+			Name:       configTrustToken,
+			Help:       "Trust token (internal use)",
+			IsPassword: false,
+			Required:   false,
+			Sensitive:  true,
+			Hide:       fs.OptionHideBoth,
+		}, {
+			Name:      configCookies,
+			Help:      "cookies (internal use only)",
+			Required:  false,
+			Advanced:  false,
+			Sensitive: true,
+			Hide:      fs.OptionHideBoth,
+		}, {
+			Name:     configClientID,
+			Help:     "Client id",
+			Required: false,
+			Advanced: true,
+			Default:  "d39ba9916b7251055b22c7f910e2ea796ee65e98b2ddecea8f5dde8d9d1a815d",
+		}, {
+			Name:     config.ConfigEncoding,
+			Help:     config.ConfigEncodingHelp,
+			Advanced: true,
+			Default: (encoder.Display |
+				//encoder.EncodeDot |
+				encoder.EncodeBackSlash |
+				encoder.EncodeInvalidUtf8),
+		}},
+	})
+}
+
+// Options defines the configuration for this backend
+type Options struct {
+	AppleID    string               `config:"apple_id"`
+	Password   string               `config:"password"`
+	Photos     bool                 `config:"photos"`
+	TrustToken string               `config:"trust_token"`
+	Cookies    string               `config:"cookies"`
+	ClientID   string               `config:"client_id"`
+	Enc        encoder.MultiEncoder `config:"encoding"`
+}
+
+// Fs represents a remote icloud drive
+type Fs struct {
+	name     string // name of this remote
+	root     string // the path we are working on.
+	rootID   string
+	opt      Options            // parsed config options
+	features *fs.Features       // optional features
+	dirCache *dircache.DirCache // Map of directory path to directory id
+	icloud   *api.Client
+	service  *api.DriveService
+	pacer    *fs.Pacer // pacer for API calls
+}
+
+// Object describes an icloud drive object
+type Object struct {
+	fs          *Fs       // what this object is part of
+	remote      string    // The remote path (relative to the fs.root)
+	size        int64     // size of the object (on server, after encryption)
+	modTime     time.Time // modification time of the object
+	createdTime time.Time // creation time of the object
+	driveID     string    // item ID of the object
+	docID       string    // document ID of the object
+	itemID      string    // item ID of the object
+	etag        string
+	downloadURL string
+}
+
+// Config configures the iCloud remote.
+func Config(ctx context.Context, name string, m configmap.Mapper, config fs.ConfigIn) (*fs.ConfigOut, error) {
+	var err error
+	appleid, _ := m.Get(configAppleID)
+	if appleid == "" {
+		return nil, errors.New("a apple ID is required")
+	}
+
+	password, _ := m.Get(configPassword)
+	if password != "" {
+		password, err = obscure.Reveal(password)
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	trustToken, _ := m.Get(configTrustToken)
+	cookieRaw, _ := m.Get(configCookies)
+	clientID, _ := m.Get(configClientID)
+	cookies := ReadCookies(cookieRaw)
+
+	switch config.State {
+	case "":
+		icloud, err := api.New(appleid, password, trustToken, clientID, cookies, nil)
+		if err != nil {
+			return nil, err
+		}
+		if err := icloud.Authenticate(ctx); err != nil {
+			return nil, err
+		}
+		m.Set(configCookies, icloud.Session.GetCookieString())
+		if icloud.Session.Requires2FA() {
+			return fs.ConfigInput("2fa_do", "config_2fa", "Two-factor authentication: please enter your 2FA code")
+		}
+		return nil, nil
+	case "2fa_do":
+		code := config.Result
+		if code == "" {
+			return fs.ConfigError("authenticate", "2FA codes can't be blank")
+		}
+
+		icloud, err := api.New(appleid, password, trustToken, clientID, cookies, nil)
+		if err != nil {
+			return nil, err
+		}
+		if err := icloud.SignIn(ctx); err != nil {
+			return nil, err
+		}
+
+		if err := icloud.Session.Validate2FACode(ctx, code); err != nil {
+			return nil, err
+		}
+
+		m.Set(configTrustToken, icloud.Session.TrustToken)
+		m.Set(configCookies, icloud.Session.GetCookieString())
+		return nil, nil
+
+	case "2fa_error":
+		if config.Result == "true" {
+			return fs.ConfigGoto("2fa")
+		}
+		return nil, errors.New("2fa authentication failed")
+	}
+	return nil, fmt.Errorf("unknown state %q", config.State)
+}
+
+// find item by path. Will not return any children for the item
+func (f *Fs) findItem(ctx context.Context, dir string) (item *api.DriveItem, found bool, err error) {
+	var resp *http.Response
+	if err = f.pacer.Call(func() (bool, error) {
+		item, resp, err = f.service.GetItemByPath(ctx, path.Join(f.root, dir))
+		return shouldRetry(ctx, resp, err)
+	}); err != nil {
+		if item == nil && resp.StatusCode == 404 {
+			return nil, false, nil
+		}
+		return nil, false, err
+	}
+
+	return item, true, nil
+}
+
+func (f *Fs) findLeafItem(ctx context.Context, pathID string, leaf string) (item *api.DriveItem, found bool, err error) {
+	items, err := f.listAll(ctx, pathID)
+	if err != nil {
+		return nil, false, err
+	}
+	for _, item := range items {
+		if strings.EqualFold(item.FullName(), leaf) {
+			return item, true, nil
+		}
+	}
+
+	return nil, false, nil
+
+}
+
+// FindLeaf finds a directory of name leaf in the folder with ID pathID
+func (f *Fs) FindLeaf(ctx context.Context, pathID string, leaf string) (pathIDOut string, found bool, err error) {
+	item, found, err := f.findLeafItem(ctx, pathID, leaf)
+
+	if err != nil {
+		return "", found, err
+	}
+
+	if !found {
+		return "", false, err
+	}
+
+	if !item.IsFolder() {
+		return "", false, fs.ErrorIsFile
+	}
+
+	return f.IDJoin(item.Drivewsid, item.Etag), true, nil
+}
+
+// Features implements fs.Fs.
+func (f *Fs) Features() *fs.Features {
+	return f.features
+}
+
+// Hashes are not exposed anywhere
+func (f *Fs) Hashes() hash.Set {
+	return hash.Set(hash.None)
+}
+
+func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) error {
+	root := path.Join(f.root, dir)
+	if root == "" {
+		return errors.New("can't purge root directory")
+	}
+
+	directoryID, etag, err := f.FindDir(ctx, dir, false)
+	if err != nil {
+		return err
+	}
+
+	if check {
+		item, found, err := f.findItem(ctx, dir)
+		if err != nil {
+			return err
+		}
+
+		if found && item.DirectChildrenCount > 0 {
+			return fs.ErrorDirectoryNotEmpty
+		}
+	}
+
+	var _ *api.DriveItem
+	var resp *http.Response
+	if err = f.pacer.Call(func() (bool, error) {
+		_, resp, err = f.service.MoveItemToTrashByID(ctx, directoryID, etag, true)
+		return retryResultUnknown(ctx, resp, err)
+	}); err != nil {
+		return err
+	}
+
+	// flush everything from the left of the dir
+	f.dirCache.FlushDir(dir)
+
+	return nil
+}
+
+// Purge all files in the directory specified
+//
+// Implement this if you have a way of deleting all the files
+// quicker than just running Remove() on the result of List()
+//
+// Return an error if it doesn't exist
+func (f *Fs) Purge(ctx context.Context, dir string) error {
+	if dir == "" {
+		return fs.ErrorCantPurge
+	}
+	return f.purgeCheck(ctx, dir, false)
+}
+
+func (f *Fs) listAll(ctx context.Context, dirID string) (items []*api.DriveItem, err error) {
+	var item *api.DriveItem
+	var resp *http.Response
+
+	if err = f.pacer.Call(func() (bool, error) {
+		id, _ := f.parseNormalizedID(dirID)
+		item, resp, err = f.service.GetItemByDriveID(ctx, id, true)
+		return shouldRetry(ctx, resp, err)
+	}); err != nil {
+		return nil, err
+	}
+
+	items = item.Items
+
+	for i, item := range items {
+		item.Name = f.opt.Enc.ToStandardName(item.Name)
+		item.Extension = f.opt.Enc.ToStandardName(item.Extension)
+		items[i] = item
+	}
+
+	return items, nil
+}
+
+// List implements fs.Fs.
+func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err error) {
+	dirRemoteID, err := f.dirCache.FindDir(ctx, dir, false)
+	if err != nil {
+		return nil, err
+	}
+
+	entries = make(fs.DirEntries, 0)
+	items, err := f.listAll(ctx, dirRemoteID)
+
+	if err != nil {
+		return nil, err
+	}
+
+	for _, item := range items {
+		id := item.Drivewsid
+		name := item.FullName()
+		remote := path.Join(dir, name)
+		if item.IsFolder() {
+			jid := f.putFolderCache(id, item.Etag, remote)
+			d := fs.NewDir(remote, item.DateModified).SetID(jid).SetSize(item.AssetQuota)
+			entries = append(entries, d)
+		} else {
+			o, err := f.NewObjectFromDriveItem(ctx, remote, item)
+			if err != nil {
+				return nil, err
+			}
+			entries = append(entries, o)
+		}
+	}
+
+	return entries, nil
+}
+
+// Mkdir implements fs.Fs.
+func (f *Fs) Mkdir(ctx context.Context, dir string) error {
+	_, _, err := f.FindDir(ctx, dir, true)
+	return err
+}
+
+// Name implements fs.Fs.
+func (f *Fs) Name() string {
+	return f.name
+}
+
+// Precision implements fs.Fs.
+func (f *Fs) Precision() time.Duration {
+	return time.Second
+}
+
+// Copy src to this remote using server-side copy operations.
+//
+// This is stored with the remote path given.
+//
+// It returns the destination Object and a possible error.
+//
+// Will only be called if src.Fs().Name() == f.Name()
+//
+// If it isn't possible then return fs.ErrorCantCopy
+//
+//nolint:all
+func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
+	// ICloud cooy endpoint is broken. Once they fixed it this can be re-enabled.
+	return nil, fs.ErrorCantCopy
+
+	// note: so many calls its only just faster then a reupload for big files.
+	srcObj, ok := src.(*Object)
+	if !ok {
+		fs.Debugf(src, "Can't copy - not same remote type")
+		return nil, fs.ErrorCantCopy
+	}
+
+	file, pathID, _, err := f.FindPath(ctx, remote, true)
+	if err != nil {
+		return nil, err
+	}
+
+	var resp *http.Response
+	var info *api.DriveItemRaw
+
+	// make a copy
+	if err = f.pacer.Call(func() (bool, error) {
+		info, resp, err = f.service.CopyDocByItemID(ctx, srcObj.itemID)
+		return retryResultUnknown(ctx, resp, err)
+	}); err != nil {
+		return nil, err
+	}
+
+	// renaming in CopyDocByID endpoint does not work :/ so do it the hard way
+
+	// get new document
+	var doc *api.Document
+	if err = f.pacer.Call(func() (bool, error) {
+		doc, resp, err = f.service.GetDocByItemID(ctx, info.ItemID)
+		return shouldRetry(ctx, resp, err)
+	}); err != nil {
+		return nil, err
+	}
+
+	// get parentdrive id
+	var dirDoc *api.Document
+	if err = f.pacer.Call(func() (bool, error) {
+		dirDoc, resp, err = f.service.GetDocByItemID(ctx, pathID)
+		return shouldRetry(ctx, resp, err)
+	}); err != nil {
+		return nil, err
+	}
+
+	// build request
+	// cant use normal rename as file needs to be "activated" first
+
+	r := api.NewUpdateFileInfo()
+	r.DocumentID = doc.DocumentID
+	r.Path.Path = file
+	r.Path.StartingDocumentID = dirDoc.DocumentID
+	r.Data.Signature = doc.Data.Signature
+	r.Data.ReferenceSignature = doc.Data.ReferenceSignature
+	r.Data.WrappingKey = doc.Data.WrappingKey
+	r.Data.Size = doc.Data.Size
+	r.Mtime = srcObj.modTime.UnixMilli()
+	r.Btime = srcObj.modTime.UnixMilli()
+
+	var item *api.DriveItem
+	if err = f.pacer.Call(func() (bool, error) {
+		item, resp, err = f.service.UpdateFile(ctx, &r)
+		return retryResultUnknown(ctx, resp, err)
+	}); err != nil {
+		return nil, err
+	}
+
+	o, err := f.NewObjectFromDriveItem(ctx, remote, item)
+	if err != nil {
+		return nil, err
+	}
+	obj := o.(*Object)
+
+	// cheat unit tests
+	obj.modTime = srcObj.modTime
+	obj.createdTime = srcObj.createdTime
+
+	return obj, nil
+}
+
+// Put in to the remote path with the modTime given of the given size
+//
+// When called from outside an Fs by rclone, src.Size() will always be >= 0.
+// But for unknown-sized objects (indicated by src.Size() == -1), Put should either
+// return an error or upload it properly (rather than e.g. calling panic).
+//
+// May create the object even if it returns an error - if so
+// will return the object and the error, otherwise will return
+// nil and the error
+func (f *Fs) Put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
+	size := src.Size()
+	if size < 0 {
+		return nil, errors.New("file size unknown")
+	}
+	existingObj, err := f.NewObject(ctx, src.Remote())
+	switch err {
+	case nil:
+		// object is found
+		return existingObj, existingObj.Update(ctx, in, src, options...)
+	case fs.ErrorObjectNotFound:
+		// object not found, so we need to create it
+		remote := src.Remote()
+		size := src.Size()
+		modTime := src.ModTime(ctx)
+
+		obj, err := f.createObject(ctx, remote, modTime, size)
+		if err != nil {
+			return nil, err
+		}
+		return obj, obj.Update(ctx, in, src, options...)
+	default:
+		// real error caught
+		return nil, err
+	}
+}
+
+// DirCacheFlush resets the directory cache - used in testing as an
+// optional interface
+func (f *Fs) DirCacheFlush() {
+	f.dirCache.ResetRoot()
+}
+
+// parseNormalizedID parses a normalized ID (may be in the form `driveID#itemID` or just `itemID`)
+// and returns itemID, driveID, rootURL.
+// Such a normalized ID can come from (*Item).GetID()
+//
+// Parameters:
+// - rid: the normalized ID to be parsed
+//
+// Returns:
+// - id: the itemID extracted from the normalized ID
+// - etag: the driveID extracted from the normalized ID, or an empty string if not present
+func (f *Fs) parseNormalizedID(rid string) (id string, etag string) {
+	split := strings.Split(rid, "#")
+	if len(split) == 1 {
+		return split[0], ""
+	}
+	return split[0], split[1]
+}
+
+// FindPath finds the leaf and directoryID from a normalized path
+func (f *Fs) FindPath(ctx context.Context, remote string, create bool) (leaf, directoryID, etag string, err error) {
+	leaf, jDirectoryID, err := f.dirCache.FindPath(ctx, remote, create)
+	if err != nil {
+		return "", "", "", err
+	}
+	directoryID, etag = f.parseNormalizedID(jDirectoryID)
+	return leaf, directoryID, etag, nil
+}
+
+// FindDir finds the directory passed in returning the directory ID
+// starting from pathID
+func (f *Fs) FindDir(ctx context.Context, path string, create bool) (pathID string, etag string, err error) {
+	jDirectoryID, err := f.dirCache.FindDir(ctx, path, create)
+	if err != nil {
+		return "", "", err
+	}
+	directoryID, etag := f.parseNormalizedID(jDirectoryID)
+	return directoryID, etag, nil
+}
+
+// IDJoin joins the given ID and ETag into a single string with a "#" delimiter.
+func (f *Fs) IDJoin(id string, etag string) string {
+	if strings.Contains(id, "#") {
+		// already contains an etag, replace
+		id, _ = f.parseNormalizedID(id)
+	}
+
+	return strings.Join([]string{id, etag}, "#")
+}
+
+func (f *Fs) putFolderCache(id, etag, remote string) string {
+	jid := f.IDJoin(id, etag)
+	f.dirCache.Put(remote, f.IDJoin(id, etag))
+	return jid
+}
+
+// Rmdir implements fs.Fs.
+func (f *Fs) Rmdir(ctx context.Context, dir string) error {
+	return f.purgeCheck(ctx, dir, true)
+}
+
+// Root implements fs.Fs.
+func (f *Fs) Root() string {
+	return f.opt.Enc.ToStandardPath(f.root)
+}
+
+// String implements fs.Fs.
+func (f *Fs) String() string {
+	return f.root
+}
+
+// CreateDir makes a directory with pathID as parent and name leaf
+//
+// This should be implemented by the backend and will be called by the
+// dircache package when appropriate.
+func (f *Fs) CreateDir(ctx context.Context, pathID, leaf string) (string, error) {
+	var item *api.DriveItem
+	var err error
+	var found bool
+	var resp *http.Response
+	if err = f.pacer.Call(func() (bool, error) {
+		id, _ := f.parseNormalizedID(pathID)
+		item, resp, err = f.service.CreateNewFolderByDriveID(ctx, id, f.opt.Enc.FromStandardName(leaf))
+
+		// check if it went oke
+		if requestError, ok := err.(*api.RequestError); ok {
+			if requestError.Status == "unknown" {
+				fs.Debugf(requestError, " checking if dir is created with separate call.")
+				time.Sleep(1 * time.Second) // sleep to give icloud time to clear up its mind
+				item, found, err = f.findLeafItem(ctx, pathID, leaf)
+				if err != nil {
+					return false, err
+				}
+
+				if !found {
+					// lets assume it failed and retry
+					return true, err
+				}
+
+				// success, clear err
+				err = nil
+			}
+		}
+
+		return ignoreResultUnknown(ctx, resp, err)
+	}); err != nil {
+		return "", err
+	}
+
+	return f.IDJoin(item.Drivewsid, item.Etag), err
+}
+
+// DirMove moves src, srcRemote to this remote at dstRemote
+// using server-side move operations.
+//
+// Will only be called if src.Fs().Name() == f.Name()
+//
+// If it isn't possible then return fs.ErrorCantDirMove
+//
+// If destination exists then return fs.ErrorDirExists
+func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string) error {
+	srcFs, ok := src.(*Fs)
+	if !ok {
+		fs.Debugf(srcFs, "Can't move directory - not same remote type")
+		return fs.ErrorCantDirMove
+	}
+
+	srcID, jsrcDirectoryID, srcLeaf, jdstDirectoryID, dstLeaf, err := f.dirCache.DirMove(ctx, srcFs.dirCache, srcFs.root, srcRemote, f.root, dstRemote)
+	if err != nil {
+		return err
+	}
+
+	srcDirectoryID, srcEtag := f.parseNormalizedID(jsrcDirectoryID)
+	dstDirectoryID, _ := f.parseNormalizedID(jdstDirectoryID)
+
+	_, err = f.move(ctx, srcID, srcDirectoryID, srcLeaf, srcEtag, dstDirectoryID, dstLeaf)
+	if err != nil {
+		return err
+	}
+
+	srcFs.dirCache.FlushDir(srcRemote)
+
+	return nil
+}
+
+func (f *Fs) move(ctx context.Context, ID, srcDirectoryID, srcLeaf, srcEtag, dstDirectoryID, dstLeaf string) (*api.DriveItem, error) {
+	var resp *http.Response
+	var item *api.DriveItem
+	var err error
+
+	// move
+	if srcDirectoryID != dstDirectoryID {
+		if err = f.pacer.Call(func() (bool, error) {
+			id, _ := f.parseNormalizedID(ID)
+			item, resp, err = f.service.MoveItemByDriveID(ctx, id, srcEtag, dstDirectoryID, true)
+			return ignoreResultUnknown(ctx, resp, err)
+		}); err != nil {
+			return nil, err
+		}
+		ID = item.Drivewsid
+		srcEtag = item.Etag
+	}
+
+	// rename
+	if srcLeaf != dstLeaf {
+		if err = f.pacer.Call(func() (bool, error) {
+			id, _ := f.parseNormalizedID(ID)
+			item, resp, err = f.service.RenameItemByDriveID(ctx, id, srcEtag, dstLeaf, true)
+			return ignoreResultUnknown(ctx, resp, err)
+		}); err != nil {
+			return item, err
+		}
+	}
+
+	return item, err
+}
+
+// Move moves the src object to the specified remote.
+func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
+	srcObj, ok := src.(*Object)
+	if !ok {
+		fs.Debugf(src, "Can't move - not same remote type")
+		return nil, fs.ErrorCantMove
+	}
+
+	srcLeaf, srcDirectoryID, _, err := srcObj.fs.FindPath(ctx, srcObj.remote, true)
+	if err != nil {
+		return nil, err
+	}
+
+	dstLeaf, dstDirectoryID, _, err := f.FindPath(ctx, remote, true)
+	if err != nil {
+		return nil, err
+	}
+
+	item, err := f.move(ctx, srcObj.driveID, srcDirectoryID, srcLeaf, srcObj.etag, dstDirectoryID, dstLeaf)
+	if err != nil {
+		return src, err
+	}
+
+	return f.NewObjectFromDriveItem(ctx, remote, item)
+}
+
+// Creates from the parameters passed in a half finished Object which
+// must have setMetaData called on it
+//
+// Returns the object, leaf, directoryID and error.
+//
+// Used to create new objects
+func (f *Fs) createObject(ctx context.Context, remote string, modTime time.Time, size int64) (o *Object, err error) {
+	// Create the directory for the object if it doesn't exist
+	_, _, _, err = f.FindPath(ctx, remote, true)
+	if err != nil {
+		return
+	}
+	// Temporary Object under construction
+	o = &Object{
+		fs:      f,
+		remote:  remote,
+		modTime: modTime,
+		size:    size,
+	}
+	return o, nil
+}
+
+// ReadCookies parses the raw cookie string and returns an array of http.Cookie objects.
+func ReadCookies(raw string) []*http.Cookie {
+	header := http.Header{}
+	header.Add("Cookie", raw)
+	request := http.Request{Header: header}
+	return request.Cookies()
+}
+
+var retryErrorCodes = []int{
+	400, // icloud is a mess, sometimes returns 400 on a perfectly fine request. So just retry
+	408, // Request Timeout
+	409, // Conflict, retry could fix it.
+	429, // Rate exceeded.
+	500, // Get occasional 500 Internal Server Error
+	502, // Server overload
+	503, // Service Unavailable
+	504, // Gateway Time-out
+}
+
+func shouldRetry(ctx context.Context, resp *http.Response, err error) (bool, error) {
+	if fserrors.ContextError(ctx, &err) {
+		return false, err
+	}
+
+	return fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
+}
+
+func ignoreResultUnknown(ctx context.Context, resp *http.Response, err error) (bool, error) {
+	if requestError, ok := err.(*api.RequestError); ok {
+		if requestError.Status == "unknown" {
+			fs.Debugf(requestError, " ignoring.")
+			return false, nil
+		}
+	}
+	return shouldRetry(ctx, resp, err)
+}
+
+func retryResultUnknown(ctx context.Context, resp *http.Response, err error) (bool, error) {
+	if requestError, ok := err.(*api.RequestError); ok {
+		if requestError.Status == "unknown" {
+			fs.Debugf(requestError, " retrying.")
+			return true, err
+		}
+	}
+	return shouldRetry(ctx, resp, err)
+}
+
+// NewFs constructs an Fs from the path, container:path
+func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, error) {
+	// Parse config into Options struct
+	opt := new(Options)
+	err := configstruct.Set(m, opt)
+	if err != nil {
+		return nil, err
+	}
+
+	if opt.Password != "" {
+		var err error
+		opt.Password, err = obscure.Reveal(opt.Password)
+		if err != nil {
+			return nil, fmt.Errorf("couldn't decrypt user password: %w", err)
+		}
+	}
+
+	if opt.TrustToken == "" {
+		return nil, fmt.Errorf("missing icloud trust token: try refreshing it with \"rclone config reconnect %s:\"", name)
+	}
+
+	cookies := ReadCookies(opt.Cookies)
+
+	callback := func(session *api.Session) {
+		m.Set(configCookies, session.GetCookieString())
+	}
+
+	icloud, err := api.New(
+		opt.AppleID,
+		opt.Password,
+		opt.TrustToken,
+		opt.ClientID,
+		cookies,
+		callback,
+	)
+	if err != nil {
+		return nil, err
+	}
+
+	if err := icloud.Authenticate(ctx); err != nil {
+		return nil, err
+	}
+
+	if icloud.Session.Requires2FA() {
+		return nil, errors.New("trust token expired, please reauth")
+	}
+
+	root = strings.Trim(root, "/")
+
+	f := &Fs{
+		name:   name,
+		root:   root,
+		icloud: icloud,
+		rootID: "FOLDER::com.apple.CloudDocs::root",
+		opt:    *opt,
+		pacer:  fs.NewPacer(ctx, pacer.NewDefault(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant))),
+	}
+	f.features = (&fs.Features{
+		CanHaveEmptyDirectories: true,
+		PartialUploads:          false,
+	}).Fill(ctx, f)
+
+	rootID := f.rootID
+	f.service, err = icloud.DriveService()
+	if err != nil {
+		return nil, err
+	}
+
+	f.dirCache = dircache.New(
+		root,
+		rootID,
+		f,
+	)
+
+	err = f.dirCache.FindRoot(ctx, false)
+	if err != nil {
+		// Assume it is a file
+		newRoot, remote := dircache.SplitPath(root)
+		tempF := *f
+		tempF.dirCache = dircache.New(newRoot, rootID, &tempF)
+		tempF.root = newRoot
+		// Make new Fs which is the parent
+		err = tempF.dirCache.FindRoot(ctx, false)
+		if err != nil {
+			// No root so return old f
+			return f, nil
+		}
+
+		_, err := tempF.NewObject(ctx, remote)
+		if err != nil {
+			if err == fs.ErrorObjectNotFound {
+				// File doesn't exist so return old f
+				return f, nil
+			}
+
+			return nil, err
+		}
+
+		f.dirCache = tempF.dirCache
+		f.root = tempF.root
+		// return an error with an fs which points to the parent
+		return f, fs.ErrorIsFile
+	}
+
+	return f, nil
+}
+
+// NewObject creates a new fs.Object from a given remote string.
+//
+// ctx: The context.Context for the function.
+// remote: The remote string representing the object's location.
+// Returns an fs.Object and an error.
+func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
+	return f.NewObjectFromDriveItem(ctx, remote, nil)
+}
+
+// NewObjectFromDriveItem creates a new fs.Object from a given remote string and DriveItem.
+//
+// ctx: The context.Context for the function.
+// remote: The remote string representing the object's location.
+// item: The optional DriveItem to use for initializing the Object. If nil, the function will read the metadata from the remote location.
+// Returns an fs.Object and an error.
+func (f *Fs) NewObjectFromDriveItem(ctx context.Context, remote string, item *api.DriveItem) (fs.Object, error) {
+	o := &Object{
+		fs:     f,
+		remote: remote,
+	}
+	if item != nil {
+		err := o.setMetaData(item)
+		if err != nil {
+			return nil, err
+		}
+	} else {
+		item, err := f.readMetaData(ctx, remote)
+
+		if err != nil {
+			return nil, err
+		}
+
+		err = o.setMetaData(item)
+		if err != nil {
+			return nil, err
+		}
+	}
+
+	return o, nil
+}
+
+func (f *Fs) readMetaData(ctx context.Context, path string) (item *api.DriveItem, err error) {
+	leaf, ID, _, err := f.FindPath(ctx, path, false)
+
+	if err != nil {
+		if err == fs.ErrorDirNotFound {
+			return nil, fs.ErrorObjectNotFound
+		}
+		return nil, err
+	}
+
+	item, found, err := f.findLeafItem(ctx, ID, leaf)
+
+	if err != nil {
+		return nil, err
+	}
+
+	if !found {
+		return nil, fs.ErrorObjectNotFound
+	}
+
+	return item, nil
+}
+
+func (o *Object) setMetaData(item *api.DriveItem) (err error) {
+	if item.IsFolder() {
+		return fs.ErrorIsDir
+	}
+	o.size = item.Size
+	o.modTime = item.DateModified
+	o.createdTime = item.DateCreated
+	o.driveID = item.Drivewsid
+	o.docID = item.Docwsid
+	o.itemID = item.Itemid
+	o.etag = item.Etag
+	o.downloadURL = item.DownloadURL()
+	return nil
+}
+
+// ID returns the ID of the Object if known, or "" if not
+func (o *Object) ID() string {
+	return o.driveID
+}
+
+// Fs implements fs.Object.
+func (o *Object) Fs() fs.Info {
+	return o.fs
+}
+
+// Hash implements fs.Object.
+func (o *Object) Hash(ctx context.Context, t hash.Type) (string, error) {
+	return "", hash.ErrUnsupported
+}
+
+// ModTime implements fs.Object.
+func (o *Object) ModTime(context.Context) time.Time {
+	return o.modTime
+}
+
+// Open implements fs.Object.
+func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (io.ReadCloser, error) {
+	fs.FixRangeOption(options, o.size)
+
+	// Drive does not support empty files, so we cheat
+	if o.size == 0 {
+		return io.NopCloser(bytes.NewBufferString("")), nil
+	}
+
+	var resp *http.Response
+	var err error
+
+	if err = o.fs.pacer.Call(func() (bool, error) {
+		var url string
+
+		//var doc *api.Document
+		//if o.docID == "" {
+		//doc, resp, err = o.fs.service.GetDocByItemID(ctx, o.itemID)
+		//}
+
+		// Can not get the download url on a item to work, so do it the hard way.
+		url, _, err = o.fs.service.GetDownloadURLByDriveID(ctx, o.driveID)
+
+		resp, err = o.fs.service.DownloadFile(ctx, url, options)
+		return shouldRetry(ctx, resp, err)
+	}); err != nil {
+		return nil, err
+	}
+
+	return resp.Body, err
+}
+
+// Remote implements fs.Object.
+func (o *Object) Remote() string {
+	return o.remote
+}
+
+// Remove implements fs.Object.
+func (o *Object) Remove(ctx context.Context) error {
+	if o.itemID == "" {
+		return nil
+	}
+
+	var resp *http.Response
+	var err error
+	if err = o.fs.pacer.Call(func() (bool, error) {
+		_, resp, err = o.fs.service.MoveItemToTrashByID(ctx, o.driveID, o.etag, true)
+		return retryResultUnknown(ctx, resp, err)
+	}); err != nil {
+		return err
+	}
+
+	return nil
+}
+
+// SetModTime implements fs.Object.
+func (o *Object) SetModTime(ctx context.Context, t time.Time) error {
+	return fs.ErrorCantSetModTime
+}
+
+// Size implements fs.Object.
+func (o *Object) Size() int64 {
+	return o.size
+}
+
+// Storable implements fs.Object.
+func (o *Object) Storable() bool {
+	return true
+}
+
+// String implements fs.Object.
+func (o *Object) String() string {
+	if o == nil {
+		return "<nil>"
+	}
+	return o.remote
+}
+
+// Update implements fs.Object.
+func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) error {
+	size := src.Size()
+	if size < 0 {
+		return errors.New("file size unknown")
+	}
+
+	remote := o.Remote()
+	modTime := src.ModTime(ctx)
+
+	leaf, dirID, _, err := o.fs.FindPath(ctx, path.Clean(remote), true)
+	if err != nil {
+		return err
+	}
+
+	// Move current file to trash
+	if o.driveID != "" {
+		err = o.Remove(ctx)
+		if err != nil {
+			return err
+		}
+	}
+
+	name := o.fs.opt.Enc.FromStandardName(leaf)
+	var resp *http.Response
+
+	// Create document
+	var uploadInfo *api.UploadResponse
+	if err = o.fs.pacer.Call(func() (bool, error) {
+		uploadInfo, resp, err = o.fs.service.CreateUpload(ctx, size, name)
+		return ignoreResultUnknown(ctx, resp, err)
+	}); err != nil {
+		return err
+	}
+
+	// Upload content
+	var upload *api.SingleFileResponse
+	if err = o.fs.pacer.Call(func() (bool, error) {
+		upload, resp, err = o.fs.service.Upload(ctx, in, size, name, uploadInfo.URL)
+		return ignoreResultUnknown(ctx, resp, err)
+	}); err != nil {
+		return err
+	}
+
+	//var doc *api.Document
+	//if err = o.fs.pacer.Call(func() (bool, error) {
+	//	doc, resp, err = o.fs.service.GetDocByItemID(ctx, dirID)
+	//	return ignoreResultUnknown(ctx, resp, err)
+	//}); err != nil {
+	//	return err
+	//}
+
+	r := api.NewUpdateFileInfo()
+	r.DocumentID = uploadInfo.DocumentID
+	r.Path.Path = name
+	r.Path.StartingDocumentID = api.GetDocIDFromDriveID(dirID)
+	//r.Path.StartingDocumentID = doc.DocumentID
+	r.Data.Receipt = upload.SingleFile.Receipt
+	r.Data.Signature = upload.SingleFile.Signature
+	r.Data.ReferenceSignature = upload.SingleFile.ReferenceSignature
+	r.Data.WrappingKey = upload.SingleFile.WrappingKey
+	r.Data.Size = upload.SingleFile.Size
+	r.Mtime = modTime.Unix() * 1000
+	r.Btime = modTime.Unix() * 1000
+
+	// Update metadata
+	var item *api.DriveItem
+	if err = o.fs.pacer.Call(func() (bool, error) {
+		item, resp, err = o.fs.service.UpdateFile(ctx, &r)
+		return ignoreResultUnknown(ctx, resp, err)
+	}); err != nil {
+		return err
+	}
+
+	err = o.setMetaData(item)
+	if err != nil {
+		return err
+	}
+
+	o.modTime = modTime
+	o.size = src.Size()
+
+	return nil
+}
+
+// Check interfaces are satisfied
+var (
+	_ fs.Fs              = &Fs{}
+	_ fs.Mover           = (*Fs)(nil)
+	_ fs.Purger          = (*Fs)(nil)
+	_ fs.DirMover        = (*Fs)(nil)
+	_ fs.DirCacheFlusher = (*Fs)(nil)
+	_ fs.Copier          = (*Fs)(nil)
+	_ fs.Object          = &Object{}
+	_ fs.IDer            = (*Object)(nil)
+)
diff --git a/backend/iclouddrive/iclouddrive_unsupported.go b/backend/iclouddrive/iclouddrive_unsupported.go
new file mode 100644
index 000000000..2eeb6b639
--- /dev/null
+++ b/backend/iclouddrive/iclouddrive_unsupported.go
@@ -0,0 +1,7 @@
+// Build for iclouddrive for unsupported platforms to stop go complaining
+// about "no buildable Go source files "
+
+//go:build plan9 || solaris
+
+// Package iclouddrive implements the iCloud Drive backend
+package iclouddrive
\ No newline at end of file
diff --git a/backend/onedrive/onedrive.go b/backend/onedrive/onedrive.go
index 7db269c2f..3ceac3f55 100644
--- a/backend/onedrive/onedrive.go
+++ b/backend/onedrive/onedrive.go
@@ -1545,9 +1545,12 @@ func (f *Fs) Rmdir(ctx context.Context, dir string) error {
 
 // Precision return the precision of this Fs
 func (f *Fs) Precision() time.Duration {
-	if f.driveType == driveTypePersonal {
-		return time.Millisecond
-	}
+	// While this is true for some OneDrive personal accounts, it
+	// isn't true for all of them. See #8101 for details
+	//
+	// if f.driveType == driveTypePersonal {
+	// 	return time.Millisecond
+	// }
 	return time.Second
 }
 
diff --git a/bin/make_manual.py b/bin/make_manual.py
index c28398dfd..990ab7130 100755
--- a/bin/make_manual.py
+++ b/bin/make_manual.py
@@ -52,6 +52,7 @@ docs = [
     "hidrive.md",
     "http.md",
     "imagekit.md",
+    "iclouddrive.md",
     "internetarchive.md",
     "jottacloud.md",
     "koofr.md",
diff --git a/cmd/cmd.go b/cmd/cmd.go
index 37a7d6f72..8f5f5f093 100644
--- a/cmd/cmd.go
+++ b/cmd/cmd.go
@@ -83,12 +83,13 @@ func ShowVersion() {
 // It returns a string with the file name if points to a file
 // otherwise "".
 func NewFsFile(remote string) (fs.Fs, string) {
+	ctx := context.Background()
 	_, fsPath, err := fspath.SplitFs(remote)
 	if err != nil {
-		err = fs.CountError(err)
+		err = fs.CountError(ctx, err)
 		fs.Fatalf(nil, "Failed to create file system for %q: %v", remote, err)
 	}
-	f, err := cache.Get(context.Background(), remote)
+	f, err := cache.Get(ctx, remote)
 	switch err {
 	case fs.ErrorIsFile:
 		cache.Pin(f) // pin indefinitely since it was on the CLI
@@ -97,7 +98,7 @@ func NewFsFile(remote string) (fs.Fs, string) {
 		cache.Pin(f) // pin indefinitely since it was on the CLI
 		return f, ""
 	default:
-		err = fs.CountError(err)
+		err = fs.CountError(ctx, err)
 		fs.Fatalf(nil, "Failed to create file system for %q: %v", remote, err)
 	}
 	return nil, ""
@@ -108,18 +109,19 @@ func NewFsFile(remote string) (fs.Fs, string) {
 // This works the same as NewFsFile however it adds filters to the Fs
 // to limit it to a single file if the remote pointed to a file.
 func newFsFileAddFilter(remote string) (fs.Fs, string) {
-	fi := filter.GetConfig(context.Background())
+	ctx := context.Background()
+	fi := filter.GetConfig(ctx)
 	f, fileName := NewFsFile(remote)
 	if fileName != "" {
 		if !fi.InActive() {
 			err := fmt.Errorf("can't limit to single files when using filters: %v", remote)
-			err = fs.CountError(err)
+			err = fs.CountError(ctx, err)
 			fs.Fatal(nil, err.Error())
 		}
 		// Limit transfers to this file
 		err := fi.AddFile(fileName)
 		if err != nil {
-			err = fs.CountError(err)
+			err = fs.CountError(ctx, err)
 			fs.Fatalf(nil, "Failed to limit to single file %q: %v", remote, err)
 		}
 	}
@@ -139,9 +141,10 @@ func NewFsSrc(args []string) fs.Fs {
 //
 // This must point to a directory
 func newFsDir(remote string) fs.Fs {
-	f, err := cache.Get(context.Background(), remote)
+	ctx := context.Background()
+	f, err := cache.Get(ctx, remote)
 	if err != nil {
-		err = fs.CountError(err)
+		err = fs.CountError(ctx, err)
 		fs.Fatalf(nil, "Failed to create file system for %q: %v", remote, err)
 	}
 	cache.Pin(f) // pin indefinitely since it was on the CLI
@@ -175,6 +178,7 @@ func NewFsSrcFileDst(args []string) (fsrc fs.Fs, srcFileName string, fdst fs.Fs)
 // NewFsSrcDstFiles creates a new src and dst fs from the arguments
 // If src is a file then srcFileName and dstFileName will be non-empty
 func NewFsSrcDstFiles(args []string) (fsrc fs.Fs, srcFileName string, fdst fs.Fs, dstFileName string) {
+	ctx := context.Background()
 	fsrc, srcFileName = newFsFileAddFilter(args[0])
 	// If copying a file...
 	dstRemote := args[1]
@@ -193,14 +197,14 @@ func NewFsSrcDstFiles(args []string) (fsrc fs.Fs, srcFileName string, fdst fs.Fs
 			fs.Fatalf(nil, "%q is a directory", args[1])
 		}
 	}
-	fdst, err := cache.Get(context.Background(), dstRemote)
+	fdst, err := cache.Get(ctx, dstRemote)
 	switch err {
 	case fs.ErrorIsFile:
-		_ = fs.CountError(err)
+		_ = fs.CountError(ctx, err)
 		fs.Fatalf(nil, "Source doesn't exist or is a directory and destination is a file")
 	case nil:
 	default:
-		_ = fs.CountError(err)
+		_ = fs.CountError(ctx, err)
 		fs.Fatalf(nil, "Failed to create file system for destination %q: %v", dstRemote, err)
 	}
 	cache.Pin(fdst) // pin indefinitely since it was on the CLI
@@ -234,7 +238,8 @@ func ShowStats() bool {
 
 // Run the function with stats and retries if required
 func Run(Retry bool, showStats bool, cmd *cobra.Command, f func() error) {
-	ci := fs.GetConfig(context.Background())
+	ctx := context.Background()
+	ci := fs.GetConfig(ctx)
 	var cmdErr error
 	stopStats := func() {}
 	if !showStats && ShowStats() {
@@ -248,7 +253,7 @@ func Run(Retry bool, showStats bool, cmd *cobra.Command, f func() error) {
 	SigInfoHandler()
 	for try := 1; try <= ci.Retries; try++ {
 		cmdErr = f()
-		cmdErr = fs.CountError(cmdErr)
+		cmdErr = fs.CountError(ctx, cmdErr)
 		lastErr := accounting.GlobalStats().GetLastError()
 		if cmdErr == nil {
 			cmdErr = lastErr
@@ -436,19 +441,19 @@ func initConfig() {
 		fs.Infof(nil, "Creating CPU profile %q\n", *cpuProfile)
 		f, err := os.Create(*cpuProfile)
 		if err != nil {
-			err = fs.CountError(err)
+			err = fs.CountError(ctx, err)
 			fs.Fatal(nil, fmt.Sprint(err))
 		}
 		err = pprof.StartCPUProfile(f)
 		if err != nil {
-			err = fs.CountError(err)
+			err = fs.CountError(ctx, err)
 			fs.Fatal(nil, fmt.Sprint(err))
 		}
 		atexit.Register(func() {
 			pprof.StopCPUProfile()
 			err := f.Close()
 			if err != nil {
-				err = fs.CountError(err)
+				err = fs.CountError(ctx, err)
 				fs.Fatal(nil, fmt.Sprint(err))
 			}
 		})
@@ -460,17 +465,17 @@ func initConfig() {
 			fs.Infof(nil, "Saving Memory profile %q\n", *memProfile)
 			f, err := os.Create(*memProfile)
 			if err != nil {
-				err = fs.CountError(err)
+				err = fs.CountError(ctx, err)
 				fs.Fatal(nil, fmt.Sprint(err))
 			}
 			err = pprof.WriteHeapProfile(f)
 			if err != nil {
-				err = fs.CountError(err)
+				err = fs.CountError(ctx, err)
 				fs.Fatal(nil, fmt.Sprint(err))
 			}
 			err = f.Close()
 			if err != nil {
-				err = fs.CountError(err)
+				err = fs.CountError(ctx, err)
 				fs.Fatal(nil, fmt.Sprint(err))
 			}
 		})
@@ -478,7 +483,8 @@ func initConfig() {
 }
 
 func resolveExitCode(err error) {
-	ci := fs.GetConfig(context.Background())
+	ctx := context.Background()
+	ci := fs.GetConfig(ctx)
 	atexit.Run()
 	if err == nil {
 		if ci.ErrorOnNoTransfer {
diff --git a/cmd/serve/dlna/dlna.go b/cmd/serve/dlna/dlna.go
index 29af87b4a..199350877 100644
--- a/cmd/serve/dlna/dlna.go
+++ b/cmd/serve/dlna/dlna.go
@@ -190,16 +190,17 @@ func (s *server) ModelNumber() string {
 
 // Renders the root device descriptor.
 func (s *server) rootDescHandler(w http.ResponseWriter, r *http.Request) {
+	ctx := r.Context()
 	tmpl, err := data.GetTemplate()
 	if err != nil {
-		serveError(s, w, "Failed to load root descriptor template", err)
+		serveError(ctx, s, w, "Failed to load root descriptor template", err)
 		return
 	}
 
 	buffer := new(bytes.Buffer)
 	err = tmpl.Execute(buffer, s)
 	if err != nil {
-		serveError(s, w, "Failed to render root descriptor XML", err)
+		serveError(ctx, s, w, "Failed to render root descriptor XML", err)
 		return
 	}
 
@@ -215,15 +216,16 @@ func (s *server) rootDescHandler(w http.ResponseWriter, r *http.Request) {
 
 // Handle a service control HTTP request.
 func (s *server) serviceControlHandler(w http.ResponseWriter, r *http.Request) {
+	ctx := r.Context()
 	soapActionString := r.Header.Get("SOAPACTION")
 	soapAction, err := upnp.ParseActionHTTPHeader(soapActionString)
 	if err != nil {
-		serveError(s, w, "Could not parse SOAPACTION header", err)
+		serveError(ctx, s, w, "Could not parse SOAPACTION header", err)
 		return
 	}
 	var env soap.Envelope
 	if err := xml.NewDecoder(r.Body).Decode(&env); err != nil {
-		serveError(s, w, "Could not parse SOAP request body", err)
+		serveError(ctx, s, w, "Could not parse SOAP request body", err)
 		return
 	}
 
@@ -257,6 +259,7 @@ func (s *server) soapActionResponse(sa upnp.SoapAction, actionRequestXML []byte,
 
 // Serves actual resources (media files).
 func (s *server) resourceHandler(w http.ResponseWriter, r *http.Request) {
+	ctx := r.Context()
 	remotePath := r.URL.Path
 	node, err := s.vfs.Stat(r.URL.Path)
 	if err != nil {
@@ -277,7 +280,7 @@ func (s *server) resourceHandler(w http.ResponseWriter, r *http.Request) {
 	file := node.(*vfs.File)
 	in, err := file.Open(os.O_RDONLY)
 	if err != nil {
-		serveError(node, w, "Could not open resource", err)
+		serveError(ctx, node, w, "Could not open resource", err)
 		return
 	}
 	defer fs.CheckClose(in, &err)
diff --git a/cmd/serve/dlna/dlna_util.go b/cmd/serve/dlna/dlna_util.go
index a63f31f5b..ca61488af 100644
--- a/cmd/serve/dlna/dlna_util.go
+++ b/cmd/serve/dlna/dlna_util.go
@@ -1,6 +1,7 @@
 package dlna
 
 import (
+	"context"
 	"crypto/md5"
 	"encoding/xml"
 	"fmt"
@@ -142,9 +143,10 @@ func logging(next http.Handler) http.Handler {
 // Error recovery and general request logging are left to logging().
 func traceLogging(next http.Handler) http.Handler {
 	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+		ctx := r.Context()
 		dump, err := httputil.DumpRequest(r, true)
 		if err != nil {
-			serveError(nil, w, "error dumping request", err)
+			serveError(ctx, nil, w, "error dumping request", err)
 			return
 		}
 		fs.Debugf(nil, "%s", dump)
@@ -182,8 +184,8 @@ func withHeader(name string, value string, next http.Handler) http.Handler {
 }
 
 // serveError returns an http.StatusInternalServerError and logs the error
-func serveError(what interface{}, w http.ResponseWriter, text string, err error) {
-	err = fs.CountError(err)
+func serveError(ctx context.Context, what interface{}, w http.ResponseWriter, text string, err error) {
+	err = fs.CountError(ctx, err)
 	fs.Errorf(what, "%s: %v", text, err)
 	http.Error(w, text+".", http.StatusInternalServerError)
 }
diff --git a/cmd/serve/http/http.go b/cmd/serve/http/http.go
index 9ace8e663..7aba1e598 100644
--- a/cmd/serve/http/http.go
+++ b/cmd/serve/http/http.go
@@ -186,6 +186,7 @@ func (s *HTTP) handler(w http.ResponseWriter, r *http.Request) {
 
 // serveDir serves a directory index at dirRemote
 func (s *HTTP) serveDir(w http.ResponseWriter, r *http.Request, dirRemote string) {
+	ctx := r.Context()
 	VFS, err := s.getVFS(r.Context())
 	if err != nil {
 		http.Error(w, "Root directory not found", http.StatusNotFound)
@@ -198,7 +199,7 @@ func (s *HTTP) serveDir(w http.ResponseWriter, r *http.Request, dirRemote string
 		http.Error(w, "Directory not found", http.StatusNotFound)
 		return
 	} else if err != nil {
-		serve.Error(dirRemote, w, "Failed to list directory", err)
+		serve.Error(ctx, dirRemote, w, "Failed to list directory", err)
 		return
 	}
 	if !node.IsDir() {
@@ -208,7 +209,7 @@ func (s *HTTP) serveDir(w http.ResponseWriter, r *http.Request, dirRemote string
 	dir := node.(*vfs.Dir)
 	dirEntries, err := dir.ReadDirAll()
 	if err != nil {
-		serve.Error(dirRemote, w, "Failed to list directory", err)
+		serve.Error(ctx, dirRemote, w, "Failed to list directory", err)
 		return
 	}
 
@@ -234,6 +235,7 @@ func (s *HTTP) serveDir(w http.ResponseWriter, r *http.Request, dirRemote string
 
 // serveFile serves a file object at remote
 func (s *HTTP) serveFile(w http.ResponseWriter, r *http.Request, remote string) {
+	ctx := r.Context()
 	VFS, err := s.getVFS(r.Context())
 	if err != nil {
 		http.Error(w, "File not found", http.StatusNotFound)
@@ -247,7 +249,7 @@ func (s *HTTP) serveFile(w http.ResponseWriter, r *http.Request, remote string)
 		http.Error(w, "File not found", http.StatusNotFound)
 		return
 	} else if err != nil {
-		serve.Error(remote, w, "Failed to find file", err)
+		serve.Error(ctx, remote, w, "Failed to find file", err)
 		return
 	}
 	if !node.IsFile() {
@@ -287,7 +289,7 @@ func (s *HTTP) serveFile(w http.ResponseWriter, r *http.Request, remote string)
 	// open the object
 	in, err := file.Open(os.O_RDONLY)
 	if err != nil {
-		serve.Error(remote, w, "Failed to open file", err)
+		serve.Error(ctx, remote, w, "Failed to open file", err)
 		return
 	}
 	defer func() {
diff --git a/cmd/serve/webdav/webdav.go b/cmd/serve/webdav/webdav.go
index 622575e8d..0c5c16416 100644
--- a/cmd/serve/webdav/webdav.go
+++ b/cmd/serve/webdav/webdav.go
@@ -349,6 +349,7 @@ func (w *WebDAV) ServeHTTP(rw http.ResponseWriter, r *http.Request) {
 // serveDir serves a directory index at dirRemote
 // This is similar to serveDir in serve http.
 func (w *WebDAV) serveDir(rw http.ResponseWriter, r *http.Request, dirRemote string) {
+	ctx := r.Context()
 	VFS, err := w.getVFS(r.Context())
 	if err != nil {
 		http.Error(rw, "Root directory not found", http.StatusNotFound)
@@ -361,7 +362,7 @@ func (w *WebDAV) serveDir(rw http.ResponseWriter, r *http.Request, dirRemote str
 		http.Error(rw, "Directory not found", http.StatusNotFound)
 		return
 	} else if err != nil {
-		serve.Error(dirRemote, rw, "Failed to list directory", err)
+		serve.Error(ctx, dirRemote, rw, "Failed to list directory", err)
 		return
 	}
 	if !node.IsDir() {
@@ -372,7 +373,7 @@ func (w *WebDAV) serveDir(rw http.ResponseWriter, r *http.Request, dirRemote str
 	dirEntries, err := dir.ReadDirAll()
 
 	if err != nil {
-		serve.Error(dirRemote, rw, "Failed to list directory", err)
+		serve.Error(ctx, dirRemote, rw, "Failed to list directory", err)
 		return
 	}
 
diff --git a/docs/content/_index.md b/docs/content/_index.md
index 2fa03fb06..5bb813a0b 100644
--- a/docs/content/_index.md
+++ b/docs/content/_index.md
@@ -132,6 +132,7 @@ WebDAV or S3, that work out of the box.)
 {{< provider name="Hetzner Storage Box" home="https://www.hetzner.com/storage/storage-box" config="/sftp/#hetzner-storage-box" >}}
 {{< provider name="HiDrive" home="https://www.strato.de/cloud-speicher/" config="/hidrive/" >}}
 {{< provider name="HTTP" home="https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol" config="/http/" >}}
+{{< provider name="iCloud Drive" home="https://icloud.com/" config="/iclouddrive/" >}}
 {{< provider name="ImageKit" home="https://imagekit.io" config="/imagekit/" >}}
 {{< provider name="Internet Archive" home="https://archive.org/" config="/internetarchive/" >}}
 {{< provider name="Jottacloud" home="https://www.jottacloud.com/en/" config="/jottacloud/" >}}
diff --git a/docs/content/authors.md b/docs/content/authors.md
index ae30d47f2..99c53a48a 100644
--- a/docs/content/authors.md
+++ b/docs/content/authors.md
@@ -894,3 +894,4 @@ put them back in again.` >}}
   * quiescens <quiescens@gmail.com>
   * rishi.sridhar <rishi.sridhar@zohocorp.com>
   * Lawrence Murray <lawrence@indii.org>
+  * Leandro Piccilli <leandro.piccilli@thalesgroup.com>
diff --git a/docs/content/docs.md b/docs/content/docs.md
index 1d1bdaa76..cb05455b6 100644
--- a/docs/content/docs.md
+++ b/docs/content/docs.md
@@ -53,6 +53,7 @@ See the following for detailed instructions for
   * [Hetzner Storage Box](/sftp/#hetzner-storage-box)
   * [HiDrive](/hidrive/)
   * [HTTP](/http/)
+  * [iCloud Drive](/iclouddrive/)
   * [Internet Archive](/internetarchive/)
   * [Jottacloud](/jottacloud/)
   * [Koofr](/koofr/)
diff --git a/docs/content/drive.md b/docs/content/drive.md
index 73f6b61a3..444886677 100644
--- a/docs/content/drive.md
+++ b/docs/content/drive.md
@@ -536,6 +536,7 @@ represent the currently available conversions.
 | html | text/html | An HTML Document |
 | jpg  | image/jpeg | A JPEG Image File |
 | json | application/vnd.google-apps.script+json | JSON Text Format for Google Apps scripts |
+| md   | text/markdown | Markdown Text Format |
 | odp  | application/vnd.oasis.opendocument.presentation | Openoffice Presentation |
 | ods  | application/vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet |
 | ods  | application/x-vnd.oasis.opendocument.spreadsheet | Openoffice Spreadsheet |
diff --git a/docs/content/iclouddrive.md b/docs/content/iclouddrive.md
new file mode 100644
index 000000000..e4982cc4b
--- /dev/null
+++ b/docs/content/iclouddrive.md
@@ -0,0 +1,156 @@
+---
+title: "iCloud Drive"
+description: "Rclone docs for iCloud Drive"
+versionIntroduced: "v1.69"
+---
+
+# {{< icon "fa fa-cloud" >}} iCloud Drive
+
+
+## Configuration
+
+The initial setup for an iCloud Drive backend involves getting a trust token/session.
+`rclone config` walks you through it. The trust token is valid for 30 days. After which you will have to reauthenticate with rclone reconnect or rclone config.
+
+Here is an example of how to make a remote called `iclouddrive`.  First run:
+
+     rclone config
+
+This will guide you through an interactive setup process:
+
+```
+No remotes found, make a new one?
+n) New remote
+s) Set configuration password
+q) Quit config
+n/s/q> n
+name> iclouddrive
+Option Storage.
+Type of storage to configure.
+Choose a number from below, or type in your own value.
+[snip]
+XX / iCloud Drive
+   \ (iclouddrive)
+[snip]
+Storage> iclouddrive
+Option apple_id.
+Apple ID.
+Enter a value.
+apple_id> APPLEID  
+Option password.
+Password.
+Choose an alternative below.
+y) Yes, type in my own password
+g) Generate random password
+y/g> y
+Enter the password:
+password:
+Confirm the password:
+password:
+Edit advanced config?
+y) Yes
+n) No (default)
+y/n> n
+Option config_2fa.
+Two-factor authentication: please enter your 2FA code
+Enter a value.
+config_2fa> 2FACODE
+Remote config
+--------------------
+[koofr]
+- type: iclouddrive
+- apple_id: APPLEID
+- password: *** ENCRYPTED ***
+- cookies: ****************************
+- trust_token: ****************************
+--------------------
+y) Yes this is OK (default)
+e) Edit this remote
+d) Delete this remote
+y/e/d> y
+```
+
+## Advanced Data Protection
+
+ADP is currently unsupported and need to be disabled
+
+{{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/iclouddrive/iclouddrive.go then run make backenddocs" >}}
+### Standard options
+
+Here are the Standard options specific to iclouddrive (iCloud Drive).
+
+#### --iclouddrive-apple-id
+
+Apple ID.
+
+Properties:
+
+- Config:      apple_id
+- Env Var:     RCLONE_ICLOUDDRIVE_APPLE_ID
+- Type:        string
+- Required:    true
+
+#### --iclouddrive-password
+
+Password.
+
+**NB** Input to this must be obscured - see [rclone obscure](/commands/rclone_obscure/).
+
+Properties:
+
+- Config:      password
+- Env Var:     RCLONE_ICLOUDDRIVE_PASSWORD
+- Type:        string
+- Required:    true
+
+#### --iclouddrive-trust-token
+
+trust token (internal use)
+
+Properties:
+
+- Config:      trust_token
+- Env Var:     RCLONE_ICLOUDDRIVE_TRUST_TOKEN
+- Type:        string
+- Required:    false
+
+#### --iclouddrive-cookies
+
+cookies (internal use only)
+
+Properties:
+
+- Config:      cookies
+- Env Var:     RCLONE_ICLOUDDRIVE_COOKIES
+- Type:        string
+- Required:    false
+
+### Advanced options
+
+Here are the Advanced options specific to iclouddrive (iCloud Drive).
+
+#### --iclouddrive-encoding
+
+The encoding for the backend.
+
+See the [encoding section in the overview](/overview/#encoding) for more info.
+
+Properties:
+
+- Config:      encoding
+- Env Var:     RCLONE_ICLOUDDRIVE_ENCODING
+- Type:        Encoding
+- Default:     Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
+
+#### --iclouddrive-description
+
+Description of the remote.
+
+Properties:
+
+- Config:      description
+- Env Var:     RCLONE_ICLOUDDRIVE_DESCRIPTION
+- Type:        string
+- Required:    false
+
+{{< rem autogenerated options stop >}}
diff --git a/docs/content/overview.md b/docs/content/overview.md
index 089889540..6131ff438 100644
--- a/docs/content/overview.md
+++ b/docs/content/overview.md
@@ -33,6 +33,7 @@ Here is an overview of the major features of each cloud storage system.
 | HDFS                         | -                 | R/W     | No               | No              | -         | -        |
 | HiDrive                      | HiDrive ¹²        | R/W     | No               | No              | -         | -        |
 | HTTP                         | -                 | R       | No               | No              | R         | -        |
+| iCloud Drive                 | -                 | R       | No               | No              | -         | -        |
 | Internet Archive             | MD5, SHA1, CRC32  | R/W ¹¹  | No               | No              | -         | RWU      |
 | Jottacloud                   | MD5               | R/W     | Yes              | No              | R         | RW       |
 | Koofr                        | MD5               | -       | Yes              | No              | -         | -        |
@@ -505,12 +506,13 @@ upon backend-specific capabilities.
 | Files.com                    | Yes   | Yes  | Yes  | Yes     | No      | No    | Yes          | No                | Yes          | No    | Yes      |
 | FTP                          | No    | No   | Yes  | Yes     | No      | No    | Yes          | No                | No           | No    | Yes      |
 | Gofile                       | Yes   | Yes  | Yes  | Yes     | No      | No    | Yes          | No                | Yes          | Yes   | Yes      |
-| Google Cloud Storage         | Yes   | Yes  | No   | No      | No      | Yes   | Yes          | No                | No           | No    | No       |
+| Google Cloud Storage         | Yes   | Yes  | No   | No      | No      | No    | Yes          | No                | No           | No    | No       |
 | Google Drive                 | Yes   | Yes  | Yes  | Yes     | Yes     | Yes   | Yes          | No                | Yes          | Yes   | Yes      |
 | Google Photos                | No    | No   | No   | No      | No      | No    | No           | No                | No           | No    | No       |
 | HDFS                         | Yes   | No   | Yes  | Yes     | No      | No    | Yes          | No                | No           | Yes   | Yes      |
 | HiDrive                      | Yes   | Yes  | Yes  | Yes     | No      | No    | Yes          | No                | No           | No    | Yes      |
 | HTTP                         | No    | No   | No   | No      | No      | No    | No           | No                | No           | No    | Yes      |
+| iCloud Drive                 | Yes   | Yes  | Yes  | Yes     | No      | No    | No           | No                | No           | No    | Yes      |
 | ImageKit                     | Yes   | Yes  | Yes  | No      | No      | No    | No           | No                | No           | No    | Yes      |
 | Internet Archive             | No    | Yes  | No   | No      | Yes     | Yes   | No           | No                | Yes          | Yes   | No       |
 | Jottacloud                   | Yes   | Yes  | Yes  | Yes     | Yes     | Yes   | No           | No                | Yes          | Yes   | Yes      |
diff --git a/docs/content/sponsor.md b/docs/content/sponsor.md
index f6638a4fc..429127980 100644
--- a/docs/content/sponsor.md
+++ b/docs/content/sponsor.md
@@ -61,3 +61,4 @@ Thank you very much to our sponsors:
 {{< sponsor src="/img/logos/warp.svg" width="300" height="200" title="Visit our sponsor warp.dev" link="https://www.warp.dev/?utm_source=rclone&utm_medium=referral&utm_campaign=rclone_20231103">}}
 {{< sponsor src="/img/logos/sia.svg" width="200" height="200" title="Visit our sponsor sia" link="https://sia.tech">}}
 {{< sponsor src="/img/logos/route4me.svg" width="400" height="200" title="Visit our sponsor Route4Me" link="https://route4me.com/">}}
+{{< sponsor src="/img/logos/rcloneview.svg" width="300" height="200" title="Visit our sponsor RcloneView" link="https://rcloneview.com/">}}
diff --git a/docs/layouts/chrome/navbar.html b/docs/layouts/chrome/navbar.html
index 6b2938c75..c6fa0db6f 100644
--- a/docs/layouts/chrome/navbar.html
+++ b/docs/layouts/chrome/navbar.html
@@ -75,6 +75,7 @@
           <a class="dropdown-item" href="/hdfs/"><i class="fa fa-globe fa-fw"></i> HDFS (Hadoop Distributed Filesystem)</a>
           <a class="dropdown-item" href="/hidrive/"><i class="fa fa-cloud fa-fw"></i> HiDrive</a>
           <a class="dropdown-item" href="/http/"><i class="fa fa-globe fa-fw"></i> HTTP</a>
+          <a class="dropdown-item" href="/iclouddrive/"><i class="fa fa-archive fa-fw"></i> iCloud Drive</a>
           <a class="dropdown-item" href="/imagekit/"><i class="fa fa-cloud fa-fw"></i> ImageKit</a>
           <a class="dropdown-item" href="/internetarchive/"><i class="fa fa-archive fa-fw"></i> Internet Archive</a>
           <a class="dropdown-item" href="/jottacloud/"><i class="fa fa-cloud fa-fw"></i> Jottacloud</a>
diff --git a/fs/accounting/accounting.go b/fs/accounting/accounting.go
index d7d7e02dd..5a6be7df4 100644
--- a/fs/accounting/accounting.go
+++ b/fs/accounting/accounting.go
@@ -44,7 +44,9 @@ func Start(ctx context.Context) {
 	//
 	// We can't do this in an init() method as it uses fs.Config
 	// and that isn't set up then.
-	fs.CountError = GlobalStats().Error
+	fs.CountError = func(ctx context.Context, err error) error {
+		return Stats(ctx).Error(err)
+	}
 }
 
 // Account limits and accounts for one transfer
diff --git a/fs/cache/cache.go b/fs/cache/cache.go
index ae86ac082..63afa94e8 100644
--- a/fs/cache/cache.go
+++ b/fs/cache/cache.go
@@ -4,6 +4,7 @@ package cache
 import (
 	"context"
 	"runtime"
+	"strings"
 	"sync"
 
 	"github.com/rclone/rclone/fs"
@@ -12,10 +13,11 @@ import (
 )
 
 var (
-	once  sync.Once // creation
-	c     *cache.Cache
-	mu    sync.Mutex            // mutex to protect remap
-	remap = map[string]string{} // map user supplied names to canonical names
+	once           sync.Once // creation
+	c              *cache.Cache
+	mu             sync.Mutex            // mutex to protect remap
+	remap          = map[string]string{} // map user supplied names to canonical names - [fsString]canonicalName
+	childParentMap = map[string]string{} // tracks a one-to-many relationship between parent dirs and their direct children files - [child]parent
 )
 
 // Create the cache just once
@@ -27,7 +29,7 @@ func createOnFirstUse() {
 		c.SetExpireInterval(ci.FsCacheExpireInterval)
 		c.SetFinalizer(func(value interface{}) {
 			if s, ok := value.(fs.Shutdowner); ok {
-				_ = fs.CountError(s.Shutdown(context.Background()))
+				_ = fs.CountError(context.Background(), s.Shutdown(context.Background()))
 			}
 		})
 	})
@@ -57,6 +59,39 @@ func addMapping(fsString, canonicalName string) {
 	mu.Unlock()
 }
 
+// addChild tracks known file (child) to directory (parent) relationships.
+// Note that the canonicalName of a child will always equal that of its parent,
+// but not everything with an equal canonicalName is a child.
+// It could be an alias or overridden version of a directory.
+func addChild(child, parent string) {
+	if child == parent {
+		return
+	}
+	mu.Lock()
+	childParentMap[child] = parent
+	mu.Unlock()
+}
+
+// returns true if name is definitely known to be a child (i.e. a file, not a dir).
+// returns false if name is a dir or if we don't know.
+func isChild(child string) bool {
+	mu.Lock()
+	_, found := childParentMap[child]
+	mu.Unlock()
+	return found
+}
+
+// ensures that we return fs.ErrorIsFile when necessary
+func getError(fsString string, err error) error {
+	if err != nil && err != fs.ErrorIsFile {
+		return err
+	}
+	if isChild(fsString) {
+		return fs.ErrorIsFile
+	}
+	return nil
+}
+
 // GetFn gets an fs.Fs named fsString either from the cache or creates
 // it afresh with the create function
 func GetFn(ctx context.Context, fsString string, create func(ctx context.Context, fsString string) (fs.Fs, error)) (f fs.Fs, err error) {
@@ -69,31 +104,39 @@ func GetFn(ctx context.Context, fsString string, create func(ctx context.Context
 		created = ok
 		return f, ok, err
 	})
+	f, ok := value.(fs.Fs)
 	if err != nil && err != fs.ErrorIsFile {
+		if ok {
+			return f, err // for possible future uses of PutErr
+		}
 		return nil, err
 	}
-	f = value.(fs.Fs)
 	// Check we stored the Fs at the canonical name
 	if created {
 		canonicalName := fs.ConfigString(f)
 		if canonicalName != canonicalFsString {
-			// Note that if err == fs.ErrorIsFile at this moment
-			// then we can't rename the remote as it will have the
-			// wrong error status, we need to add a new one.
-			if err == nil {
+			if err == nil { // it's a dir
 				fs.Debugf(nil, "fs cache: renaming cache item %q to be canonical %q", canonicalFsString, canonicalName)
 				value, found := c.Rename(canonicalFsString, canonicalName)
 				if found {
 					f = value.(fs.Fs)
 				}
 				addMapping(canonicalFsString, canonicalName)
-			} else {
-				fs.Debugf(nil, "fs cache: adding new entry for parent of %q, %q", canonicalFsString, canonicalName)
-				Put(canonicalName, f)
+			} else { // it's a file
+				// the fs we cache is always the file's parent, never the file,
+				// but we use the childParentMap to return the correct error status based on the fsString passed in.
+				fs.Debugf(nil, "fs cache: renaming child cache item %q to be canonical for parent %q", canonicalFsString, canonicalName)
+				value, found := c.Rename(canonicalFsString, canonicalName) // rename the file entry to parent
+				if found {
+					f = value.(fs.Fs) // if parent already exists, use it
+				}
+				Put(canonicalName, f)                        // force err == nil for the cache
+				addMapping(canonicalFsString, canonicalName) // note the fsString-canonicalName connection for future lookups
+				addChild(fsString, canonicalName)            // note the file-directory connection for future lookups
 			}
 		}
 	}
-	return f, err
+	return f, getError(fsString, err) // ensure fs.ErrorIsFile is returned when necessary
 }
 
 // Pin f into the cache until Unpin is called
@@ -111,7 +154,6 @@ func PinUntilFinalized(f fs.Fs, x interface{}) {
 	runtime.SetFinalizer(x, func(_ interface{}) {
 		Unpin(f)
 	})
-
 }
 
 // Unpin f from the cache
@@ -174,6 +216,9 @@ func PutErr(fsString string, f fs.Fs, err error) {
 	canonicalName := fs.ConfigString(f)
 	c.PutErr(canonicalName, f, err)
 	addMapping(fsString, canonicalName)
+	if err == fs.ErrorIsFile {
+		addChild(fsString, canonicalName)
+	}
 }
 
 // Put puts an fs.Fs named fsString into the cache
@@ -186,6 +231,7 @@ func Put(fsString string, f fs.Fs) {
 // Returns number of entries deleted
 func ClearConfig(name string) (deleted int) {
 	createOnFirstUse()
+	ClearMappingsPrefix(name)
 	return c.DeletePrefix(name + ":")
 }
 
@@ -193,6 +239,7 @@ func ClearConfig(name string) (deleted int) {
 func Clear() {
 	createOnFirstUse()
 	c.Clear()
+	ClearMappings()
 }
 
 // Entries returns the number of entries in the cache
@@ -200,3 +247,39 @@ func Entries() int {
 	createOnFirstUse()
 	return c.Entries()
 }
+
+// ClearMappings removes everything from remap and childParentMap
+func ClearMappings() {
+	mu.Lock()
+	defer mu.Unlock()
+	remap = map[string]string{}
+	childParentMap = map[string]string{}
+}
+
+// ClearMappingsPrefix deletes all mappings to parents with given prefix
+//
+// Returns number of entries deleted
+func ClearMappingsPrefix(prefix string) (deleted int) {
+	mu.Lock()
+	do := func(mapping map[string]string) {
+		for key, val := range mapping {
+			if !strings.HasPrefix(val, prefix) {
+				continue
+			}
+			delete(mapping, key)
+			deleted++
+		}
+	}
+	do(remap)
+	do(childParentMap)
+	mu.Unlock()
+	return deleted
+}
+
+// EntriesWithPinCount returns the number of pinned and unpinned entries in the cache
+//
+// Each entry is counted only once, regardless of entry.pinCount
+func EntriesWithPinCount() (pinned, unpinned int) {
+	createOnFirstUse()
+	return c.EntriesWithPinCount()
+}
diff --git a/fs/config.go b/fs/config.go
index caf242fdf..1b11da821 100644
--- a/fs/config.go
+++ b/fs/config.go
@@ -41,7 +41,7 @@ var (
 	//
 	// This is a function pointer to decouple the config
 	// implementation from the fs
-	CountError = func(err error) error { return err }
+	CountError = func(ctx context.Context, err error) error { return err }
 
 	// ConfigProvider is the config key used for provider options
 	ConfigProvider = "provider"
diff --git a/fs/march/march.go b/fs/march/march.go
index 856be0e7c..1116220db 100644
--- a/fs/march/march.go
+++ b/fs/march/march.go
@@ -415,7 +415,7 @@ func (m *March) processJob(job listDirJob) ([]listDirJob, error) {
 		} else {
 			fs.Errorf(m.Fsrc, "error reading source root directory: %v", srcListErr)
 		}
-		srcListErr = fs.CountError(srcListErr)
+		srcListErr = fs.CountError(m.Ctx, srcListErr)
 		return nil, srcListErr
 	}
 	if dstListErr == fs.ErrorDirNotFound {
@@ -426,7 +426,7 @@ func (m *March) processJob(job listDirJob) ([]listDirJob, error) {
 		} else {
 			fs.Errorf(m.Fdst, "error reading destination root directory: %v", dstListErr)
 		}
-		dstListErr = fs.CountError(dstListErr)
+		dstListErr = fs.CountError(m.Ctx, dstListErr)
 		return nil, dstListErr
 	}
 
diff --git a/fs/operations/check.go b/fs/operations/check.go
index b9fe36aab..c3c747071 100644
--- a/fs/operations/check.go
+++ b/fs/operations/check.go
@@ -49,6 +49,7 @@ type CheckOpt struct {
 // checkMarch is used to march over two Fses in the same way as
 // sync/copy
 type checkMarch struct {
+	ctx             context.Context
 	ioMu            sync.Mutex
 	wg              sync.WaitGroup
 	tokens          chan struct{}
@@ -83,7 +84,7 @@ func (c *checkMarch) DstOnly(dst fs.DirEntry) (recurse bool) {
 		}
 		err := fmt.Errorf("file not in %v", c.opt.Fsrc)
 		fs.Errorf(dst, "%v", err)
-		_ = fs.CountError(err)
+		_ = fs.CountError(c.ctx, err)
 		c.differences.Add(1)
 		c.srcFilesMissing.Add(1)
 		c.report(dst, c.opt.MissingOnSrc, '-')
@@ -105,7 +106,7 @@ func (c *checkMarch) SrcOnly(src fs.DirEntry) (recurse bool) {
 	case fs.Object:
 		err := fmt.Errorf("file not in %v", c.opt.Fdst)
 		fs.Errorf(src, "%v", err)
-		_ = fs.CountError(err)
+		_ = fs.CountError(c.ctx, err)
 		c.differences.Add(1)
 		c.dstFilesMissing.Add(1)
 		c.report(src, c.opt.MissingOnDst, '+')
@@ -155,13 +156,13 @@ func (c *checkMarch) Match(ctx context.Context, dst, src fs.DirEntry) (recurse b
 				differ, noHash, err := c.checkIdentical(ctx, dstX, srcX)
 				if err != nil {
 					fs.Errorf(src, "%v", err)
-					_ = fs.CountError(err)
+					_ = fs.CountError(ctx, err)
 					c.report(src, c.opt.Error, '!')
 				} else if differ {
 					c.differences.Add(1)
 					err := errors.New("files differ")
 					// the checkFn has already logged the reason
-					_ = fs.CountError(err)
+					_ = fs.CountError(ctx, err)
 					c.report(src, c.opt.Differ, '*')
 				} else {
 					c.matches.Add(1)
@@ -177,7 +178,7 @@ func (c *checkMarch) Match(ctx context.Context, dst, src fs.DirEntry) (recurse b
 		} else {
 			err := fmt.Errorf("is file on %v but directory on %v", c.opt.Fsrc, c.opt.Fdst)
 			fs.Errorf(src, "%v", err)
-			_ = fs.CountError(err)
+			_ = fs.CountError(ctx, err)
 			c.differences.Add(1)
 			c.dstFilesMissing.Add(1)
 			c.report(src, c.opt.MissingOnDst, '+')
@@ -190,7 +191,7 @@ func (c *checkMarch) Match(ctx context.Context, dst, src fs.DirEntry) (recurse b
 		}
 		err := fmt.Errorf("is file on %v but directory on %v", c.opt.Fdst, c.opt.Fsrc)
 		fs.Errorf(dst, "%v", err)
-		_ = fs.CountError(err)
+		_ = fs.CountError(ctx, err)
 		c.differences.Add(1)
 		c.srcFilesMissing.Add(1)
 		c.report(dst, c.opt.MissingOnSrc, '-')
@@ -214,6 +215,7 @@ func CheckFn(ctx context.Context, opt *CheckOpt) error {
 		return errors.New("internal error: nil check function")
 	}
 	c := &checkMarch{
+		ctx:    ctx,
 		tokens: make(chan struct{}, ci.Checkers),
 		opt:    *opt,
 	}
@@ -430,6 +432,7 @@ func CheckSum(ctx context.Context, fsrc, fsum fs.Fs, sumFile string, hashType ha
 
 	ci := fs.GetConfig(ctx)
 	c := &checkMarch{
+		ctx:    ctx,
 		tokens: make(chan struct{}, ci.Checkers),
 		opt:    *opt,
 	}
@@ -450,7 +453,7 @@ func CheckSum(ctx context.Context, fsrc, fsum fs.Fs, sumFile string, hashType ha
 		// filesystem missed the file, sum wasn't consumed
 		err := fmt.Errorf("file not in %v", opt.Fdst)
 		fs.Errorf(filename, "%v", err)
-		_ = fs.CountError(err)
+		_ = fs.CountError(ctx, err)
 		if lastErr == nil {
 			lastErr = err
 		}
@@ -479,7 +482,7 @@ func (c *checkMarch) checkSum(ctx context.Context, obj fs.Object, download bool,
 
 	if !sumFound {
 		err = errors.New("sum not found")
-		_ = fs.CountError(err)
+		_ = fs.CountError(ctx, err)
 		fs.Errorf(obj, "%v", err)
 		c.differences.Add(1)
 		c.srcFilesMissing.Add(1)
@@ -528,12 +531,12 @@ func (c *checkMarch) checkSum(ctx context.Context, obj fs.Object, download bool,
 func (c *checkMarch) matchSum(ctx context.Context, sumHash, objHash string, obj fs.Object, err error, hashType hash.Type) {
 	switch {
 	case err != nil:
-		_ = fs.CountError(err)
+		_ = fs.CountError(ctx, err)
 		fs.Errorf(obj, "Failed to calculate hash: %v", err)
 		c.report(obj, c.opt.Error, '!')
 	case sumHash == "":
 		err = errors.New("duplicate file")
-		_ = fs.CountError(err)
+		_ = fs.CountError(ctx, err)
 		fs.Errorf(obj, "%v", err)
 		c.report(obj, c.opt.Error, '!')
 	case objHash == "":
@@ -548,7 +551,7 @@ func (c *checkMarch) matchSum(ctx context.Context, sumHash, objHash string, obj
 		c.report(obj, c.opt.Match, '=')
 	default:
 		err = errors.New("files differ")
-		_ = fs.CountError(err)
+		_ = fs.CountError(ctx, err)
 		fs.Debugf(nil, "%v = %s (sum)", hashType, sumHash)
 		fs.Debugf(obj, "%v = %s (%v)", hashType, objHash, c.opt.Fdst)
 		fs.Errorf(obj, "%v", err)
diff --git a/fs/operations/copy.go b/fs/operations/copy.go
index 4e2f09762..8228a14c7 100644
--- a/fs/operations/copy.go
+++ b/fs/operations/copy.go
@@ -331,7 +331,7 @@ func (c *copy) copy(ctx context.Context) (newDst fs.Object, err error) {
 		}
 	}
 	if err != nil {
-		err = fs.CountError(err)
+		err = fs.CountError(ctx, err)
 		fs.Errorf(c.src, "Failed to copy: %v", err)
 		if !c.inplace {
 			c.removeFailedPartialCopy(ctx, c.f, c.remoteForCopy)
@@ -343,7 +343,7 @@ func (c *copy) copy(ctx context.Context) (newDst fs.Object, err error) {
 	err = c.verify(ctx, newDst)
 	if err != nil {
 		fs.Errorf(newDst, "%v", err)
-		err = fs.CountError(err)
+		err = fs.CountError(ctx, err)
 		c.removeFailedCopy(ctx, newDst)
 		return nil, err
 	}
@@ -353,7 +353,7 @@ func (c *copy) copy(ctx context.Context) (newDst fs.Object, err error) {
 		movedNewDst, err := c.dstFeatures.Move(ctx, newDst, c.remote)
 		if err != nil {
 			fs.Errorf(newDst, "partial file rename failed: %v", err)
-			err = fs.CountError(err)
+			err = fs.CountError(ctx, err)
 			c.removeFailedCopy(ctx, newDst)
 			return nil, err
 		}
diff --git a/fs/operations/dedupe.go b/fs/operations/dedupe.go
index 3e0f72cd5..60c66c4a3 100644
--- a/fs/operations/dedupe.go
+++ b/fs/operations/dedupe.go
@@ -32,7 +32,7 @@ outer:
 		_, err := f.NewObject(ctx, newName)
 		for ; err != fs.ErrorObjectNotFound; suffix++ {
 			if err != nil {
-				err = fs.CountError(err)
+				err = fs.CountError(ctx, err)
 				fs.Errorf(o, "Failed to check for existing object: %v", err)
 				continue outer
 			}
@@ -46,7 +46,7 @@ outer:
 		if !SkipDestructive(ctx, o, "rename") {
 			newObj, err := doMove(ctx, o, newName)
 			if err != nil {
-				err = fs.CountError(err)
+				err = fs.CountError(ctx, err)
 				fs.Errorf(o, "Failed to rename: %v", err)
 				continue
 			}
@@ -374,7 +374,7 @@ func dedupeMergeDuplicateDirs(ctx context.Context, f fs.Fs, duplicateDirs [][]*d
 		fs.Infof(fsDirs[0], "Merging contents of duplicate directories")
 		err := mergeDirs(ctx, fsDirs)
 		if err != nil {
-			err = fs.CountError(err)
+			err = fs.CountError(ctx, err)
 			fs.Errorf(nil, "merge duplicate dirs: %v", err)
 		}
 	}
diff --git a/fs/operations/operations.go b/fs/operations/operations.go
index f14ecb61e..53d846525 100644
--- a/fs/operations/operations.go
+++ b/fs/operations/operations.go
@@ -101,11 +101,11 @@ func checkHashes(ctx context.Context, src fs.ObjectInfo, dst fs.Object, ht hash.
 		return true, hash.None, srcHash, dstHash, nil
 	}
 	if srcErr != nil {
-		err = fs.CountError(srcErr)
+		err = fs.CountError(ctx, srcErr)
 		fs.Errorf(src, "Failed to calculate src hash: %v", err)
 	}
 	if dstErr != nil {
-		err = fs.CountError(dstErr)
+		err = fs.CountError(ctx, dstErr)
 		fs.Errorf(dst, "Failed to calculate dst hash: %v", err)
 	}
 	if err != nil {
@@ -340,7 +340,7 @@ func equal(ctx context.Context, src fs.ObjectInfo, dst fs.Object, opt equalOpt)
 				logger(ctx, Differ, src, dst, nil)
 				return false
 			} else if err != nil {
-				err = fs.CountError(err)
+				err = fs.CountError(ctx, err)
 				fs.Errorf(dst, "Failed to set modification time: %v", err)
 			} else {
 				fs.Infof(src, "Updated modification time in destination")
@@ -481,7 +481,7 @@ func move(ctx context.Context, fdst fs.Fs, dst fs.Object, remote string, src fs.
 			fs.Debugf(src, "Can't move, switching to copy")
 			_ = in.Close()
 		default:
-			err = fs.CountError(err)
+			err = fs.CountError(ctx, err)
 			fs.Errorf(src, "Couldn't move: %v", err)
 			_ = in.Close()
 			return newDst, err
@@ -566,7 +566,7 @@ func DeleteFileWithBackupDir(ctx context.Context, dst fs.Object, backupDir fs.Fs
 	}
 	if err != nil {
 		fs.Errorf(dst, "Couldn't %s: %v", action, err)
-		err = fs.CountError(err)
+		err = fs.CountError(ctx, err)
 	} else if !skip {
 		fs.Infof(dst, "%s", actioned)
 	}
@@ -974,7 +974,7 @@ func HashLister(ctx context.Context, ht hash.Type, outputBase64 bool, downloadFl
 			}()
 			sum, err := HashSum(ctx, ht, outputBase64, downloadFlag, o)
 			if err != nil {
-				fs.Errorf(o, "%v", fs.CountError(err))
+				fs.Errorf(o, "%v", fs.CountError(ctx, err))
 				return
 			}
 			SyncFprintf(w, "%*s  %s\n", width, sum, o.Remote())
@@ -1053,7 +1053,7 @@ func Mkdir(ctx context.Context, f fs.Fs, dir string) error {
 	fs.Debugf(fs.LogDirName(f, dir), "Making directory")
 	err := f.Mkdir(ctx, dir)
 	if err != nil {
-		err = fs.CountError(err)
+		err = fs.CountError(ctx, err)
 		return err
 	}
 	return nil
@@ -1075,7 +1075,7 @@ func MkdirMetadata(ctx context.Context, f fs.Fs, dir string, metadata fs.Metadat
 	fs.Debugf(fs.LogDirName(f, dir), "Making directory with metadata")
 	newDst, err = do(ctx, dir, metadata)
 	if err != nil {
-		err = fs.CountError(err)
+		err = fs.CountError(ctx, err)
 		return nil, err
 	}
 	if mtime, ok := metadata["mtime"]; ok {
@@ -1133,7 +1133,7 @@ func TryRmdir(ctx context.Context, f fs.Fs, dir string) error {
 func Rmdir(ctx context.Context, f fs.Fs, dir string) error {
 	err := TryRmdir(ctx, f, dir)
 	if err != nil {
-		err = fs.CountError(err)
+		err = fs.CountError(ctx, err)
 		return err
 	}
 	return err
@@ -1162,7 +1162,7 @@ func Purge(ctx context.Context, f fs.Fs, dir string) (err error) {
 		err = Rmdirs(ctx, f, dir, false)
 	}
 	if err != nil {
-		err = fs.CountError(err)
+		err = fs.CountError(ctx, err)
 		return err
 	}
 	return nil
@@ -1207,7 +1207,7 @@ func listToChan(ctx context.Context, f fs.Fs, dir string) fs.ObjectsChan {
 		})
 		if err != nil && err != fs.ErrorDirNotFound {
 			err = fmt.Errorf("failed to list: %w", err)
-			err = fs.CountError(err)
+			err = fs.CountError(ctx, err)
 			fs.Errorf(nil, "%v", err)
 		}
 	}()
@@ -1267,7 +1267,7 @@ func Cat(ctx context.Context, f fs.Fs, w io.Writer, offset, count int64, sep []b
 		var in io.ReadCloser
 		in, err = Open(ctx, o, options...)
 		if err != nil {
-			err = fs.CountError(err)
+			err = fs.CountError(ctx, err)
 			fs.Errorf(o, "Failed to open: %v", err)
 			return
 		}
@@ -1280,13 +1280,13 @@ func Cat(ctx context.Context, f fs.Fs, w io.Writer, offset, count int64, sep []b
 		defer mu.Unlock()
 		_, err = io.Copy(w, in)
 		if err != nil {
-			err = fs.CountError(err)
+			err = fs.CountError(ctx, err)
 			fs.Errorf(o, "Failed to send to output: %v", err)
 		}
 		if len(sep) > 0 {
 			_, err = w.Write(sep)
 			if err != nil {
-				err = fs.CountError(err)
+				err = fs.CountError(ctx, err)
 				fs.Errorf(o, "Failed to send separator to output: %v", err)
 			}
 		}
@@ -1423,7 +1423,7 @@ func rcatSrc(ctx context.Context, fdst fs.Fs, dstFileName string, in io.ReadClos
 	src := object.NewStaticObjectInfo(dstFileName, modTime, int64(readCounter.BytesRead()), false, sums, fdst).WithMetadata(meta)
 	if !equal(ctx, src, dst, opt) {
 		err = fmt.Errorf("corrupted on transfer")
-		err = fs.CountError(err)
+		err = fs.CountError(ctx, err)
 		fs.Errorf(dst, "%v", err)
 		return dst, err
 	}
@@ -1450,7 +1450,7 @@ func Rmdirs(ctx context.Context, f fs.Fs, dir string, leaveRoot bool) error {
 	dirEmpty[dir] = !leaveRoot
 	err := walk.Walk(ctx, f, dir, false, ci.MaxDepth, func(dirPath string, entries fs.DirEntries, err error) error {
 		if err != nil {
-			err = fs.CountError(err)
+			err = fs.CountError(ctx, err)
 			fs.Errorf(f, "Failed to list %q: %v", dirPath, err)
 			return nil
 		}
@@ -1526,7 +1526,7 @@ func Rmdirs(ctx context.Context, f fs.Fs, dir string, leaveRoot bool) error {
 			g.Go(func() error {
 				err := TryRmdir(gCtx, f, dir)
 				if err != nil {
-					err = fs.CountError(err)
+					err = fs.CountError(ctx, err)
 					fs.Errorf(dir, "Failed to rmdir: %v", err)
 					errCount.Add(err)
 				}
@@ -2096,7 +2096,7 @@ func TouchDir(ctx context.Context, f fs.Fs, remote string, t time.Time, recursiv
 				err := o.SetModTime(ctx, t)
 				if err != nil {
 					err = fmt.Errorf("failed to touch: %w", err)
-					err = fs.CountError(err)
+					err = fs.CountError(ctx, err)
 					fs.Errorf(o, "%v", err)
 				}
 			}
diff --git a/fs/sync/sync.go b/fs/sync/sync.go
index 67998268f..e0a3e23ac 100644
--- a/fs/sync/sync.go
+++ b/fs/sync/sync.go
@@ -400,7 +400,7 @@ func (s *syncCopyMove) pairChecker(in *pipe, out *pipe, fraction int, wg *sync.W
 			if needTransfer {
 				// If files are treated as immutable, fail if destination exists and does not match
 				if s.ci.Immutable && pair.Dst != nil {
-					err := fs.CountError(fserrors.NoRetryError(fs.ErrorImmutableModified))
+					err := fs.CountError(s.ctx, fserrors.NoRetryError(fs.ErrorImmutableModified))
 					fs.Errorf(pair.Dst, "Source and destination exist but do not match: %v", err)
 					s.processError(err)
 				} else {
@@ -1204,7 +1204,7 @@ func (s *syncCopyMove) setDelayedDirModTimes(ctx context.Context) error {
 					_, err = operations.SetDirModTime(gCtx, s.fdst, item.dst, item.dir, item.modTime)
 				}
 				if err != nil {
-					err = fs.CountError(err)
+					err = fs.CountError(ctx, err)
 					fs.Errorf(item.dir, "Failed to update directory timestamp or metadata: %v", err)
 					errCount.Add(err)
 				}
@@ -1399,7 +1399,7 @@ func MoveDir(ctx context.Context, fdst, fsrc fs.Fs, deleteEmptySrcDirs bool, cop
 			fs.Infof(fdst, "Server side directory move succeeded")
 			return nil
 		default:
-			err = fs.CountError(err)
+			err = fs.CountError(ctx, err)
 			fs.Errorf(fdst, "Server side directory move failed: %v", err)
 			return err
 		}
diff --git a/fs/walk/walk.go b/fs/walk/walk.go
index ada7a284d..51e567a4f 100644
--- a/fs/walk/walk.go
+++ b/fs/walk/walk.go
@@ -170,7 +170,7 @@ func listRwalk(ctx context.Context, f fs.Fs, path string, includeAll bool, maxLe
 		// Carry on listing but return the error at the end
 		if err != nil {
 			listErr = err
-			err = fs.CountError(err)
+			err = fs.CountError(ctx, err)
 			fs.Errorf(path, "error listing: %v", err)
 			return nil
 		}
@@ -415,7 +415,7 @@ func walk(ctx context.Context, f fs.Fs, path string, includeAll bool, maxLevel i
 					// NB once we have passed entries to fn we mustn't touch it again
 					if err != nil && err != ErrorSkipDir {
 						traversing.Done()
-						err = fs.CountError(err)
+						err = fs.CountError(ctx, err)
 						fs.Errorf(job.remote, "error listing: %v", err)
 						closeQuit()
 						// Send error to error channel if space
diff --git a/lib/cache/cache.go b/lib/cache/cache.go
index fc73eab68..6f841d182 100644
--- a/lib/cache/cache.go
+++ b/lib/cache/cache.go
@@ -260,3 +260,19 @@ func (c *Cache) SetFinalizer(finalize func(interface{})) {
 	c.finalize = finalize
 	c.mu.Unlock()
 }
+
+// EntriesWithPinCount returns the number of pinned and unpinned entries in the cache
+//
+// Each entry is counted only once, regardless of entry.pinCount
+func (c *Cache) EntriesWithPinCount() (pinned, unpinned int) {
+	c.mu.Lock()
+	for _, entry := range c.cache {
+		if entry.pinCount <= 0 {
+			unpinned++
+		} else {
+			pinned++
+		}
+	}
+	c.mu.Unlock()
+	return pinned, unpinned
+}
diff --git a/lib/http/serve/dir.go b/lib/http/serve/dir.go
index 664f81cf2..5a92ddeb8 100644
--- a/lib/http/serve/dir.go
+++ b/lib/http/serve/dir.go
@@ -2,6 +2,7 @@ package serve
 
 import (
 	"bytes"
+	"context"
 	"fmt"
 	"html/template"
 	"net/http"
@@ -124,8 +125,8 @@ func (d *Directory) AddEntry(remote string, isDir bool) {
 }
 
 // Error logs the error and if a ResponseWriter is given it writes an http.StatusInternalServerError
-func Error(what interface{}, w http.ResponseWriter, text string, err error) {
-	err = fs.CountError(err)
+func Error(ctx context.Context, what interface{}, w http.ResponseWriter, text string, err error) {
+	err = fs.CountError(ctx, err)
 	fs.Errorf(what, "%s: %v", text, err)
 	if w != nil {
 		http.Error(w, text+".", http.StatusInternalServerError)
@@ -223,6 +224,7 @@ const (
 
 // Serve serves a directory
 func (d *Directory) Serve(w http.ResponseWriter, r *http.Request) {
+	ctx := r.Context()
 	// Account the transfer
 	tr := accounting.Stats(r.Context()).NewTransferRemoteSize(d.DirRemote, -1, nil, nil)
 	defer tr.Done(r.Context(), nil)
@@ -232,12 +234,12 @@ func (d *Directory) Serve(w http.ResponseWriter, r *http.Request) {
 	buf := &bytes.Buffer{}
 	err := d.HTMLTemplate.Execute(buf, d)
 	if err != nil {
-		Error(d.DirRemote, w, "Failed to render template", err)
+		Error(ctx, d.DirRemote, w, "Failed to render template", err)
 		return
 	}
 	w.Header().Set("Content-Length", fmt.Sprintf("%d", buf.Len()))
 	_, err = buf.WriteTo(w)
 	if err != nil {
-		Error(d.DirRemote, nil, "Failed to drain template buffer", err)
+		Error(ctx, d.DirRemote, nil, "Failed to drain template buffer", err)
 	}
 }


Test patch:
diff --git a/backend/drive/drive_internal_test.go b/backend/drive/drive_internal_test.go
index 998d9ba43..023dd42da 100644
--- a/backend/drive/drive_internal_test.go
+++ b/backend/drive/drive_internal_test.go
@@ -95,7 +95,7 @@ func TestInternalParseExtensions(t *testing.T) {
 		wantErr error
 	}{
 		{"doc", []string{".doc"}, nil},
-		{" docx ,XLSX, 	pptx,svg", []string{".docx", ".xlsx", ".pptx", ".svg"}, nil},
+		{" docx ,XLSX, 	pptx,svg,md", []string{".docx", ".xlsx", ".pptx", ".svg", ".md"}, nil},
 		{"docx,svg,Docx", []string{".docx", ".svg"}, nil},
 		{"docx,potato,docx", []string{".docx"}, errors.New(`couldn't find MIME type for extension ".potato"`)},
 	} {
diff --git a/backend/iclouddrive/iclouddrive_test.go b/backend/iclouddrive/iclouddrive_test.go
new file mode 100644
index 000000000..13db708ea
--- /dev/null
+++ b/backend/iclouddrive/iclouddrive_test.go
@@ -0,0 +1,18 @@
+//go:build !plan9 && !solaris
+
+package iclouddrive_test
+
+import (
+	"testing"
+
+	"github.com/rclone/rclone/backend/iclouddrive"
+	"github.com/rclone/rclone/fstest/fstests"
+)
+
+// TestIntegration runs integration tests against the remote
+func TestIntegration(t *testing.T) {
+	fstests.Run(t, &fstests.Opt{
+		RemoteName: "TestICloudDrive:",
+		NilObject:  (*iclouddrive.Object)(nil),
+	})
+}
diff --git a/cmd/bisync/bisync_test.go b/cmd/bisync/bisync_test.go
index bc359bbd3..da89c3e20 100644
--- a/cmd/bisync/bisync_test.go
+++ b/cmd/bisync/bisync_test.go
@@ -15,6 +15,7 @@ import (
 	"path/filepath"
 	"regexp"
 	"runtime"
+	"slices"
 	"sort"
 	"strconv"
 	"strings"
@@ -207,15 +208,16 @@ type bisyncTest struct {
 	parent1  fs.Fs
 	parent2  fs.Fs
 	// global flags
-	argRemote1    string
-	argRemote2    string
-	noCompare     bool
-	noCleanup     bool
-	golden        bool
-	debug         bool
-	stopAt        int
-	TestFn        bisync.TestFunc
-	ignoreModtime bool // ignore modtimes when comparing final listings, for backends without support
+	argRemote1      string
+	argRemote2      string
+	noCompare       bool
+	noCleanup       bool
+	golden          bool
+	debug           bool
+	stopAt          int
+	TestFn          bisync.TestFunc
+	ignoreModtime   bool // ignore modtimes when comparing final listings, for backends without support
+	ignoreBlankHash bool // ignore blank hashes for backends where we allow them to be blank
 }
 
 var color = bisync.Color
@@ -946,6 +948,10 @@ func (b *bisyncTest) checkPreReqs(ctx context.Context, opt *bisync.Options) (con
 	if (!b.fs1.Features().CanHaveEmptyDirectories || !b.fs2.Features().CanHaveEmptyDirectories) && (b.testCase == "createemptysrcdirs" || b.testCase == "rmdirs") {
 		b.t.Skip("skipping test as remote does not support empty dirs")
 	}
+	ignoreHashBackends := []string{"TestWebdavNextcloud", "TestWebdavOwncloud", "TestAzureFiles"} // backends that support hashes but allow them to be blank
+	if slices.ContainsFunc(ignoreHashBackends, func(prefix string) bool { return strings.HasPrefix(b.fs1.Name(), prefix) }) || slices.ContainsFunc(ignoreHashBackends, func(prefix string) bool { return strings.HasPrefix(b.fs2.Name(), prefix) }) {
+		b.ignoreBlankHash = true
+	}
 	if b.fs1.Precision() == fs.ModTimeNotSupported || b.fs2.Precision() == fs.ModTimeNotSupported {
 		if b.testCase != "nomodtime" {
 			b.t.Skip("skipping test as at least one remote does not support setting modtime")
@@ -1551,6 +1557,12 @@ func (b *bisyncTest) mangleResult(dir, file string, golden bool) string {
 	if b.fs1.Hashes() == hash.Set(hash.None) || b.fs2.Hashes() == hash.Set(hash.None) {
 		logReplacements = append(logReplacements, `^.*{hashtype} differ.*$`, dropMe)
 	}
+	if b.ignoreBlankHash {
+		logReplacements = append(logReplacements,
+			`^.*hash is missing.*$`, dropMe,
+			`^.*not equal on recheck.*$`, dropMe,
+		)
+	}
 	rep := logReplacements
 	if b.testCase == "dry_run" {
 		rep = append(rep, dryrunReplacements...)
diff --git a/fs/accounting/stats_groups_test.go b/fs/accounting/stats_groups_test.go
index dda9dccef..8c02229e7 100644
--- a/fs/accounting/stats_groups_test.go
+++ b/fs/accounting/stats_groups_test.go
@@ -7,6 +7,8 @@ import (
 	"testing"
 	"time"
 
+	"github.com/rclone/rclone/fs"
+	"github.com/rclone/rclone/fs/fserrors"
 	"github.com/rclone/rclone/fs/rc"
 	"github.com/rclone/rclone/fstest/testy"
 	"github.com/stretchr/testify/assert"
@@ -206,6 +208,34 @@ func TestStatsGroupOperations(t *testing.T) {
 	})
 }
 
+func TestCountError(t *testing.T) {
+	ctx := context.Background()
+	Start(ctx)
+	defer func() {
+		groups = newStatsGroups()
+	}()
+	t.Run("global stats", func(t *testing.T) {
+		GlobalStats().ResetCounters()
+		err := fs.CountError(ctx, fmt.Errorf("global err"))
+		assert.Equal(t, int64(1), GlobalStats().errors)
+
+		assert.True(t, fserrors.IsCounted(err))
+	})
+	t.Run("group stats", func(t *testing.T) {
+		statGroupName := fmt.Sprintf("%s-error_group", t.Name())
+		GlobalStats().ResetCounters()
+		stCtx := WithStatsGroup(ctx, statGroupName)
+		st := StatsGroup(stCtx, statGroupName)
+
+		err := fs.CountError(stCtx, fmt.Errorf("group err"))
+
+		assert.Equal(t, int64(0), GlobalStats().errors)
+		assert.Equal(t, int64(1), st.errors)
+		assert.True(t, fserrors.IsCounted(err))
+	})
+
+}
+
 func percentDiff(start, end uint64) uint64 {
 	return (start - end) * 100 / start
 }
diff --git a/fs/cache/cache_test.go b/fs/cache/cache_test.go
index acddac80c..d9d54fa2e 100644
--- a/fs/cache/cache_test.go
+++ b/fs/cache/cache_test.go
@@ -24,7 +24,7 @@ func mockNewFs(t *testing.T) func(ctx context.Context, path string) (fs.Fs, erro
 		switch path {
 		case "mock:/":
 			return mockfs.NewFs(ctx, "mock", "/", nil)
-		case "mock:/file.txt", "mock:file.txt":
+		case "mock:/file.txt", "mock:file.txt", "mock:/file2.txt", "mock:file2.txt":
 			fMock, err := mockfs.NewFs(ctx, "mock", "/", nil)
 			require.NoError(t, err)
 			return fMock, fs.ErrorIsFile
@@ -55,6 +55,7 @@ func TestGet(t *testing.T) {
 }
 
 func TestGetFile(t *testing.T) {
+	defer ClearMappings()
 	create := mockNewFs(t)
 
 	assert.Equal(t, 0, Entries())
@@ -63,7 +64,7 @@ func TestGetFile(t *testing.T) {
 	require.Equal(t, fs.ErrorIsFile, err)
 	require.NotNil(t, f)
 
-	assert.Equal(t, 2, Entries())
+	assert.Equal(t, 1, Entries())
 
 	f2, err := GetFn(context.Background(), "mock:/file.txt", create)
 	require.Equal(t, fs.ErrorIsFile, err)
@@ -71,7 +72,7 @@ func TestGetFile(t *testing.T) {
 
 	assert.Equal(t, f, f2)
 
-	// check parent is there too
+	// check it is also found when referred to by parent name
 	f2, err = GetFn(context.Background(), "mock:/", create)
 	require.Nil(t, err)
 	require.NotNil(t, f2)
@@ -80,6 +81,7 @@ func TestGetFile(t *testing.T) {
 }
 
 func TestGetFile2(t *testing.T) {
+	defer ClearMappings()
 	create := mockNewFs(t)
 
 	assert.Equal(t, 0, Entries())
@@ -88,7 +90,7 @@ func TestGetFile2(t *testing.T) {
 	require.Equal(t, fs.ErrorIsFile, err)
 	require.NotNil(t, f)
 
-	assert.Equal(t, 2, Entries())
+	assert.Equal(t, 1, Entries())
 
 	f2, err := GetFn(context.Background(), "mock:file.txt", create)
 	require.Equal(t, fs.ErrorIsFile, err)
@@ -96,7 +98,7 @@ func TestGetFile2(t *testing.T) {
 
 	assert.Equal(t, f, f2)
 
-	// check parent is there too
+	// check it is also found when referred to by parent name
 	f2, err = GetFn(context.Background(), "mock:/", create)
 	require.Nil(t, err)
 	require.NotNil(t, f2)
@@ -124,22 +126,22 @@ func TestPutErr(t *testing.T) {
 
 	assert.Equal(t, 0, Entries())
 
-	PutErr("mock:file.txt", f, fs.ErrorIsFile)
+	PutErr("mock:/", f, fs.ErrorNotFoundInConfigFile)
 
 	assert.Equal(t, 1, Entries())
 
-	fNew, err := GetFn(context.Background(), "mock:file.txt", create)
-	require.Equal(t, fs.ErrorIsFile, err)
+	fNew, err := GetFn(context.Background(), "mock:/", create)
+	require.Equal(t, fs.ErrorNotFoundInConfigFile, err)
 	require.Equal(t, f, fNew)
 
 	assert.Equal(t, 1, Entries())
 
 	// Check canonicalisation
 
-	PutErr("mock:/file.txt", f, fs.ErrorIsFile)
+	PutErr("mock:/file.txt", f, fs.ErrorNotFoundInConfigFile)
 
 	fNew, err = GetFn(context.Background(), "mock:/file.txt", create)
-	require.Equal(t, fs.ErrorIsFile, err)
+	require.Equal(t, fs.ErrorNotFoundInConfigFile, err)
 	require.Equal(t, f, fNew)
 
 	assert.Equal(t, 1, Entries())
@@ -190,6 +192,75 @@ func TestPin(t *testing.T) {
 	Unpin(f2)
 }
 
+func TestPinFile(t *testing.T) {
+	defer ClearMappings()
+	create := mockNewFs(t)
+
+	// Test pinning and unpinning nonexistent
+	f, err := mockfs.NewFs(context.Background(), "mock", "/file.txt", nil)
+	require.NoError(t, err)
+	Pin(f)
+	Unpin(f)
+
+	// Now test pinning an existing
+	f2, err := GetFn(context.Background(), "mock:/file.txt", create)
+	require.Equal(t, fs.ErrorIsFile, err)
+	assert.Equal(t, 1, len(childParentMap))
+
+	Pin(f2)
+	assert.Equal(t, 1, Entries())
+	pinned, unpinned := EntriesWithPinCount()
+	assert.Equal(t, 1, pinned)
+	assert.Equal(t, 0, unpinned)
+
+	Unpin(f2)
+	assert.Equal(t, 1, Entries())
+	pinned, unpinned = EntriesWithPinCount()
+	assert.Equal(t, 0, pinned)
+	assert.Equal(t, 1, unpinned)
+
+	// try a different child of the same parent, and parent
+	// should not add additional cache items
+	called = 0 // this one does create() because we haven't seen it before and don't yet know it's a file
+	f3, err := GetFn(context.Background(), "mock:/file2.txt", create)
+	assert.Equal(t, fs.ErrorIsFile, err)
+	assert.Equal(t, 1, Entries())
+	assert.Equal(t, 2, len(childParentMap))
+
+	parent, err := GetFn(context.Background(), "mock:/", create)
+	assert.NoError(t, err)
+	assert.Equal(t, 1, Entries())
+	assert.Equal(t, 2, len(childParentMap))
+
+	Pin(f3)
+	assert.Equal(t, 1, Entries())
+	pinned, unpinned = EntriesWithPinCount()
+	assert.Equal(t, 1, pinned)
+	assert.Equal(t, 0, unpinned)
+
+	Unpin(f3)
+	assert.Equal(t, 1, Entries())
+	pinned, unpinned = EntriesWithPinCount()
+	assert.Equal(t, 0, pinned)
+	assert.Equal(t, 1, unpinned)
+
+	Pin(parent)
+	assert.Equal(t, 1, Entries())
+	pinned, unpinned = EntriesWithPinCount()
+	assert.Equal(t, 1, pinned)
+	assert.Equal(t, 0, unpinned)
+
+	Unpin(parent)
+	assert.Equal(t, 1, Entries())
+	pinned, unpinned = EntriesWithPinCount()
+	assert.Equal(t, 0, pinned)
+	assert.Equal(t, 1, unpinned)
+
+	// all 3 should have equal configstrings
+	assert.Equal(t, fs.ConfigString(f2), fs.ConfigString(f3))
+	assert.Equal(t, fs.ConfigString(f2), fs.ConfigString(parent))
+}
+
 func TestClearConfig(t *testing.T) {
 	create := mockNewFs(t)
 
@@ -198,9 +269,9 @@ func TestClearConfig(t *testing.T) {
 	_, err := GetFn(context.Background(), "mock:/file.txt", create)
 	require.Equal(t, fs.ErrorIsFile, err)
 
-	assert.Equal(t, 2, Entries()) // file + parent
+	assert.Equal(t, 1, Entries())
 
-	assert.Equal(t, 2, ClearConfig("mock"))
+	assert.Equal(t, 1, ClearConfig("mock"))
 
 	assert.Equal(t, 0, Entries())
 }
diff --git a/fs/rc/cache_test.go b/fs/rc/cache_test.go
index 0a351cee7..15952fa04 100644
--- a/fs/rc/cache_test.go
+++ b/fs/rc/cache_test.go
@@ -22,7 +22,7 @@ func mockNewFs(t *testing.T) func() {
 	require.NoError(t, err)
 	cache.Put("mock:/", f)
 	cache.Put(":mock:/", f)
-	f, err = mockfs.NewFs(ctx, "mock", "dir/file.txt", nil)
+	f, err = mockfs.NewFs(ctx, "mock", "dir/", nil)
 	require.NoError(t, err)
 	cache.PutErr("mock:dir/file.txt", f, fs.ErrorIsFile)
 	return func() {
diff --git a/fs/sync/sync_test.go b/fs/sync/sync_test.go
index f1c07921e..f4f1c556a 100644
--- a/fs/sync/sync_test.go
+++ b/fs/sync/sync_test.go
@@ -597,6 +597,108 @@ func TestServerSideCopy(t *testing.T) {
 	fstest.CheckItems(t, FremoteCopy, file1)
 }
 
+// Test copying a file over itself
+func TestCopyOverSelf(t *testing.T) {
+	ctx := context.Background()
+	r := fstest.NewRun(t)
+	file1 := r.WriteObject(ctx, "sub dir/hello world", "hello world", t1)
+	r.CheckRemoteItems(t, file1)
+	file2 := r.WriteFile("sub dir/hello world", "hello world again", t2)
+	r.CheckLocalItems(t, file2)
+
+	ctx = predictDstFromLogger(ctx)
+	err := CopyDir(ctx, r.Fremote, r.Flocal, false)
+	require.NoError(t, err)
+	testLoggerVsLsf(ctx, r.Fremote, operations.GetLoggerOpt(ctx).JSON, t)
+	r.CheckRemoteItems(t, file2)
+}
+
+// Test server-side copying a file over itself
+func TestServerSideCopyOverSelf(t *testing.T) {
+	ctx := context.Background()
+	r := fstest.NewRun(t)
+	file1 := r.WriteObject(ctx, "sub dir/hello world", "hello world", t1)
+	r.CheckRemoteItems(t, file1)
+
+	FremoteCopy, _, finaliseCopy, err := fstest.RandomRemote()
+	require.NoError(t, err)
+	defer finaliseCopy()
+	t.Logf("Server side copy (if possible) %v -> %v", r.Fremote, FremoteCopy)
+
+	ctx = predictDstFromLogger(ctx)
+	err = CopyDir(ctx, FremoteCopy, r.Fremote, false)
+	require.NoError(t, err)
+	testLoggerVsLsf(ctx, r.Fremote, operations.GetLoggerOpt(ctx).JSON, t)
+	fstest.CheckItems(t, FremoteCopy, file1)
+
+	file2 := r.WriteObject(ctx, "sub dir/hello world", "hello world again", t2)
+	r.CheckRemoteItems(t, file2)
+
+	ctx = predictDstFromLogger(ctx)
+	err = CopyDir(ctx, FremoteCopy, r.Fremote, false)
+	require.NoError(t, err)
+	testLoggerVsLsf(ctx, r.Fremote, operations.GetLoggerOpt(ctx).JSON, t)
+	fstest.CheckItems(t, FremoteCopy, file2)
+}
+
+// Test moving a file over itself
+func TestMoveOverSelf(t *testing.T) {
+	ctx := context.Background()
+	r := fstest.NewRun(t)
+	file1 := r.WriteObject(ctx, "sub dir/hello world", "hello world", t1)
+	r.CheckRemoteItems(t, file1)
+	file2 := r.WriteFile("sub dir/hello world", "hello world again", t2)
+	r.CheckLocalItems(t, file2)
+
+	ctx = predictDstFromLogger(ctx)
+	err := MoveDir(ctx, r.Fremote, r.Flocal, false, false)
+	require.NoError(t, err)
+	testLoggerVsLsf(ctx, r.Fremote, operations.GetLoggerOpt(ctx).JSON, t)
+	r.CheckLocalItems(t)
+	r.CheckRemoteItems(t, file2)
+}
+
+// Test server-side moving a file over itself
+func TestServerSideMoveOverSelf(t *testing.T) {
+	ctx := context.Background()
+	r := fstest.NewRun(t)
+	file1 := r.WriteObject(ctx, "sub dir/hello world", "hello world", t1)
+	r.CheckRemoteItems(t, file1)
+
+	FremoteCopy, _, finaliseCopy, err := fstest.RandomRemote()
+	require.NoError(t, err)
+	defer finaliseCopy()
+	t.Logf("Server side copy (if possible) %v -> %v", r.Fremote, FremoteCopy)
+
+	ctx = predictDstFromLogger(ctx)
+	err = CopyDir(ctx, FremoteCopy, r.Fremote, false)
+	require.NoError(t, err)
+	testLoggerVsLsf(ctx, r.Fremote, operations.GetLoggerOpt(ctx).JSON, t)
+	fstest.CheckItems(t, FremoteCopy, file1)
+
+	file2 := r.WriteObject(ctx, "sub dir/hello world", "hello world again", t2)
+	r.CheckRemoteItems(t, file2)
+
+	// ctx = predictDstFromLogger(ctx)
+	err = MoveDir(ctx, FremoteCopy, r.Fremote, false, false)
+	require.NoError(t, err)
+	// testLoggerVsLsf(ctx, r.Fremote, operations.GetLoggerOpt(ctx).JSON, t) // not currently supported
+	r.CheckRemoteItems(t)
+	fstest.CheckItems(t, FremoteCopy, file2)
+
+	// check that individual file moves also work without MoveDir
+	file3 := r.WriteObject(ctx, "sub dir/hello world", "hello world a third time", t3)
+	r.CheckRemoteItems(t, file3)
+
+	ctx = predictDstFromLogger(ctx)
+	fs.Debugf(nil, "testing file moves")
+	err = moveDir(ctx, FremoteCopy, r.Fremote, false, false)
+	require.NoError(t, err)
+	testLoggerVsLsf(ctx, FremoteCopy, operations.GetLoggerOpt(ctx).JSON, t)
+	r.CheckRemoteItems(t)
+	fstest.CheckItems(t, FremoteCopy, file3)
+}
+
 // Check that if the local file doesn't exist when we copy it up,
 // nothing happens to the remote file
 func TestCopyAfterDelete(t *testing.T) {
@@ -844,7 +946,7 @@ func TestSyncIgnoreErrors(t *testing.T) {
 
 	accounting.GlobalStats().ResetCounters()
 	ctx = predictDstFromLogger(ctx)
-	_ = fs.CountError(errors.New("boom"))
+	_ = fs.CountError(ctx, errors.New("boom"))
 	assert.NoError(t, Sync(ctx, r.Fremote, r.Flocal, false))
 	testLoggerVsLsf(ctx, r.Fremote, operations.GetLoggerOpt(ctx).JSON, t)
 
@@ -1165,7 +1267,7 @@ func TestSyncAfterRemovingAFileAndAddingAFileSubDirWithErrors(t *testing.T) {
 
 	ctx = predictDstFromLogger(ctx)
 	accounting.GlobalStats().ResetCounters()
-	_ = fs.CountError(errors.New("boom"))
+	_ = fs.CountError(ctx, errors.New("boom"))
 	err := Sync(ctx, r.Fremote, r.Flocal, false)
 	assert.Equal(t, fs.ErrorNotDeleting, err)
 	testLoggerVsLsf(ctx, r.Fremote, operations.GetLoggerOpt(ctx).JSON, t)
@@ -2320,15 +2422,19 @@ func testSyncBackupDir(t *testing.T, backupDir string, suffix string, suffixKeep
 
 	r.CheckRemoteItems(t, file1b, file2, file3a, file1a)
 }
+
 func TestSyncBackupDir(t *testing.T) {
 	testSyncBackupDir(t, "backup", "", false)
 }
+
 func TestSyncBackupDirWithSuffix(t *testing.T) {
 	testSyncBackupDir(t, "backup", ".bak", false)
 }
+
 func TestSyncBackupDirWithSuffixKeepExtension(t *testing.T) {
 	testSyncBackupDir(t, "backup", "-2019-01-01", true)
 }
+
 func TestSyncBackupDirSuffixOnly(t *testing.T) {
 	testSyncBackupDir(t, "", ".bak", false)
 }
@@ -2806,7 +2912,7 @@ func predictDstFromLogger(ctx context.Context) context.Context {
 }
 
 func DstLsf(ctx context.Context, Fremote fs.Fs) *bytes.Buffer {
-	var opt = operations.ListJSONOpt{
+	opt := operations.ListJSONOpt{
 		NoModTime:  false,
 		NoMimeType: true,
 		DirsOnly:   false,
diff --git a/fstest/test_all/config.yaml b/fstest/test_all/config.yaml
index 66d60d993..3aa68afab 100644
--- a/fstest/test_all/config.yaml
+++ b/fstest/test_all/config.yaml
@@ -395,6 +395,10 @@ backends:
  - backend:  "cache"
    remote:   "TestCache:"
    fastlist: false
+   ignoretests:
+     - TestBisyncLocalRemote
+     - TestBisyncRemoteLocal
+     - TestBisyncRemoteRemote
  - backend:  "mega"
    remote:   "TestMega:"
    fastlist: false
@@ -499,3 +503,6 @@ backends:
  - backend:  "ulozto"
    remote:   "TestUlozto:"
    fastlist: false
+ - backend:  "iclouddrive"
+   remote:   "TestICloudDrive:"
+   fastlist: false
diff --git a/lib/http/serve/dir_test.go b/lib/http/serve/dir_test.go
index 7bedc0471..19ab22f57 100644
--- a/lib/http/serve/dir_test.go
+++ b/lib/http/serve/dir_test.go
@@ -1,6 +1,7 @@
 package serve
 
 import (
+	"context"
 	"errors"
 	"html/template"
 	"io"
@@ -88,9 +89,10 @@ func TestAddEntry(t *testing.T) {
 }
 
 func TestError(t *testing.T) {
+	ctx := context.Background()
 	w := httptest.NewRecorder()
 	err := errors.New("help")
-	Error("potato", w, "sausage", err)
+	Error(ctx, "potato", w, "sausage", err)
 	resp := w.Result()
 	assert.Equal(t, http.StatusInternalServerError, resp.StatusCode)
 	body, _ := io.ReadAll(resp.Body)