Repository: rclone/rclone (language: go)

Issue:
WebDAV partly unusable when CORS applied
This is a follow-up to #5078. First of all, thank you very much for implementing that request!
As my main goal was to access WebDAV via rclone from the browser (a PWA), that is unfortunately still not completely possible. Let's try to dig into details.

It is still not 100% clear to me what/where the problem lies, as the messages are different in different browsers.
The test case can be accessed from https://cashier-ii.alensiljak.eu.org/webdav-demo
This is the (PWA) page that issues the requests.

I'll provide here some general info that does not fit into the answers below.

As noted in #5078, these headers are included in the Response and I'm wondering if that is the source of the issue:
```
Access-Control-Allow-Origin  | *
Access-Control-Request-Method | POST, OPTIONS, GET, HEAD
```
This would explain the failure of PROPFIND. PUT also does not work.
There's a failure of POST, as well, but I suspect there is something wrong with how I'm using it (?).

#### The associated forum post URL from `https://forum.rclone.org`

N/A

#### What is the problem you are having with rclone?

Accessing the Rclone WebDAV server from a browser does not work as expected.

Initially I've tried a few WebDAV-specific libraries. These all use PROPFIND request, which seems to be blocked in this scenario.

After that, I've used the HttpClient directly, to issue HTTP commands. For the following commands:
- simple get (root)
- get a file
- propfind (root)
- post a file

```
GET http://localhost:8080

GET http://localhost:8080/rates.ledger

PROPFIND http://localhost:8080

POST http://localhost:8080/upload-test.txt
with string stream attached.
```
only the first two succeed. PROPFIND and POST fail. 
I may be using POST wrong. I'll adjust the request as we troubleshoot.

However, when other tools are used (non-browser, Total Commander on Android, curl, Windows Explorer), things seem fine. I believe the reason for this is that these do not care about CORS.

Let me know if you have questions or need clarifications. I'd be happy to help to try to narrow this down.

#### What is your rclone version (output from `rclone version`)

rclone v1.64.2
- os/version: Microsoft Windows 11 Pro for Workstations 22H2 (64 bit)
- os/kernel: 10.0.23570.1000 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.21.3
- go/linking: static
- go/tags: cmount

#### Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Windows 11, 64-bit

#### Which cloud storage system are you using? (e.g. Google Drive)

Local drive/folder, for this case.

#### The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`)

```sh
rclone serve webdav . --allow-origin "*" -vv --log-file bug.log
```



#### A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`)

```
2023/11/21 13:57:29 DEBUG : Creating backend with remote "."
2023/11/21 13:57:29 DEBUG : Using config file from "C:\\Users\\Alen\\AppData\\Roaming\\rclone\\rclone.conf"
2023/11/21 13:57:29 DEBUG : fs cache: renaming cache item "." to be canonical "//?/D:/tmp"
2023/11/21 13:57:29 INFO  : Local file system at //?/D:/tmp: poll-interval is not supported by this remote
2023/11/21 13:57:29 NOTICE: Warning: Allow origin set to *. This can cause serious security problems.
2023/11/21 13:57:29 NOTICE: Local file system at //?/D:/tmp: WebDav Server started on [http://127.0.0.1:8080/]
2023/11/21 13:57:43 INFO  : : 127.0.0.1:57907: Serving directory
2023/11/21 13:57:44 DEBUG : /rates.ledger: OpenFile: flags=O_RDONLY, perm=----------
2023/11/21 13:57:44 DEBUG : rates.ledger: Open: flags=O_RDONLY
2023/11/21 13:57:44 DEBUG : rates.ledger: >Open: fd=rates.ledger (r), err=<nil>
2023/11/21 13:57:44 DEBUG : /rates.ledger: >OpenFile: fd=rates.ledger (r), err=<nil>
2023/11/21 13:57:44 DEBUG : rates.ledger: ChunkedReader.openRange at 0 length 134217728
2023/11/21 13:57:44 DEBUG : rates.ledger: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
2023/11/21 13:57:44 DEBUG : rates.ledger: ReadFileHandle.seek from 512 to 0 (fs.RangeSeeker)
2023/11/21 13:57:44 DEBUG : rates.ledger: ChunkedReader.RangeSeek from 512 to 0 length -1
2023/11/21 13:57:44 DEBUG : rates.ledger: ChunkedReader.Read at -1 length 541 chunkOffset 0 chunkSize 134217728
2023/11/21 13:57:44 DEBUG : rates.ledger: ChunkedReader.openRange at 0 length 134217728
2023/11/21 13:57:44 INFO  : /rates.ledger: GET from 127.0.0.1:57907
2023/11/21 13:57:46 INFO  : /: OPTIONS from 127.0.0.1:57907
2023/11/21 13:57:48 DEBUG : /upload-test.txt: OpenFile: flags=O_RDONLY, perm=----------
2023/11/21 13:57:48 DEBUG : /upload-test.txt: >OpenFile: fd=<nil>, err=file does not exist
2023/11/21 13:57:48 INFO  : /upload-test.txt: POST from 127.0.0.1:57907
2023/11/21 13:57:50 INFO  : Signal received: interrupt
2023/11/21 13:57:50 DEBUG : 7 go routines active
2023/11/21 13:57:50 INFO  : Exiting...
```

The log seems to indicate that there was no PROPFIND request at all. This may be blocked by the browser itself. It hints that this method might need to be explicitly allowed by the server.

I also don't see the GET request for "/".

#### How to use GitHub

* Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue.
* Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
* Subscribe to receive notifications on status change and new comments.

WebDAV partly unusable when CORS applied
This is a follow-up to #5078. First of all, thank you very much for implementing that request!
As my main goal was to access WebDAV via rclone from the browser (a PWA), that is unfortunately still not completely possible. Let's try to dig into details.

It is still not 100% clear to me what/where the problem lies, as the messages are different in different browsers.
The test case can be accessed from https://cashier-ii.alensiljak.eu.org/webdav-demo
This is the (PWA) page that issues the requests.

I'll provide here some general info that does not fit into the answers below.

As noted in #5078, these headers are included in the Response and I'm wondering if that is the source of the issue:
```
Access-Control-Allow-Origin  | *
Access-Control-Request-Method | POST, OPTIONS, GET, HEAD
```
This would explain the failure of PROPFIND. PUT also does not work.
There's a failure of POST, as well, but I suspect there is something wrong with how I'm using it (?).

#### The associated forum post URL from `https://forum.rclone.org`

N/A

#### What is the problem you are having with rclone?

Accessing the Rclone WebDAV server from a browser does not work as expected.

Initially I've tried a few WebDAV-specific libraries. These all use PROPFIND request, which seems to be blocked in this scenario.

After that, I've used the HttpClient directly, to issue HTTP commands. For the following commands:
- simple get (root)
- get a file
- propfind (root)
- post a file

```
GET http://localhost:8080

GET http://localhost:8080/rates.ledger

PROPFIND http://localhost:8080

POST http://localhost:8080/upload-test.txt
with string stream attached.
```
only the first two succeed. PROPFIND and POST fail. 
I may be using POST wrong. I'll adjust the request as we troubleshoot.

However, when other tools are used (non-browser, Total Commander on Android, curl, Windows Explorer), things seem fine. I believe the reason for this is that these do not care about CORS.

Let me know if you have questions or need clarifications. I'd be happy to help to try to narrow this down.

#### What is your rclone version (output from `rclone version`)

rclone v1.64.2
- os/version: Microsoft Windows 11 Pro for Workstations 22H2 (64 bit)
- os/kernel: 10.0.23570.1000 (x86_64)
- os/type: windows
- os/arch: amd64
- go/version: go1.21.3
- go/linking: static
- go/tags: cmount

#### Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Windows 11, 64-bit

#### Which cloud storage system are you using? (e.g. Google Drive)

Local drive/folder, for this case.

#### The command you were trying to run (e.g. `rclone copy /tmp remote:tmp`)

```sh
rclone serve webdav . --allow-origin "*" -vv --log-file bug.log
```



#### A log from the command with the `-vv` flag (e.g. output from `rclone -vv copy /tmp remote:tmp`)

```
2023/11/21 13:57:29 DEBUG : Creating backend with remote "."
2023/11/21 13:57:29 DEBUG : Using config file from "C:\\Users\\Alen\\AppData\\Roaming\\rclone\\rclone.conf"
2023/11/21 13:57:29 DEBUG : fs cache: renaming cache item "." to be canonical "//?/D:/tmp"
2023/11/21 13:57:29 INFO  : Local file system at //?/D:/tmp: poll-interval is not supported by this remote
2023/11/21 13:57:29 NOTICE: Warning: Allow origin set to *. This can cause serious security problems.
2023/11/21 13:57:29 NOTICE: Local file system at //?/D:/tmp: WebDav Server started on [http://127.0.0.1:8080/]
2023/11/21 13:57:43 INFO  : : 127.0.0.1:57907: Serving directory
2023/11/21 13:57:44 DEBUG : /rates.ledger: OpenFile: flags=O_RDONLY, perm=----------
2023/11/21 13:57:44 DEBUG : rates.ledger: Open: flags=O_RDONLY
2023/11/21 13:57:44 DEBUG : rates.ledger: >Open: fd=rates.ledger (r), err=<nil>
2023/11/21 13:57:44 DEBUG : /rates.ledger: >OpenFile: fd=rates.ledger (r), err=<nil>
2023/11/21 13:57:44 DEBUG : rates.ledger: ChunkedReader.openRange at 0 length 134217728
2023/11/21 13:57:44 DEBUG : rates.ledger: ChunkedReader.Read at 0 length 512 chunkOffset 0 chunkSize 134217728
2023/11/21 13:57:44 DEBUG : rates.ledger: ReadFileHandle.seek from 512 to 0 (fs.RangeSeeker)
2023/11/21 13:57:44 DEBUG : rates.ledger: ChunkedReader.RangeSeek from 512 to 0 length -1
2023/11/21 13:57:44 DEBUG : rates.ledger: ChunkedReader.Read at -1 length 541 chunkOffset 0 chunkSize 134217728
2023/11/21 13:57:44 DEBUG : rates.ledger: ChunkedReader.openRange at 0 length 134217728
2023/11/21 13:57:44 INFO  : /rates.ledger: GET from 127.0.0.1:57907
2023/11/21 13:57:46 INFO  : /: OPTIONS from 127.0.0.1:57907
2023/11/21 13:57:48 DEBUG : /upload-test.txt: OpenFile: flags=O_RDONLY, perm=----------
2023/11/21 13:57:48 DEBUG : /upload-test.txt: >OpenFile: fd=<nil>, err=file does not exist
2023/11/21 13:57:48 INFO  : /upload-test.txt: POST from 127.0.0.1:57907
2023/11/21 13:57:50 INFO  : Signal received: interrupt
2023/11/21 13:57:50 DEBUG : 7 go routines active
2023/11/21 13:57:50 INFO  : Exiting...
```

The log seems to indicate that there was no PROPFIND request at all. This may be blocked by the browser itself. It hints that this method might need to be explicitly allowed by the server.

I also don't see the GET request for "/".

#### How to use GitHub

* Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue.
* Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
* Subscribe to receive notifications on status change and new comments.



Discussion / hints:
# PROPFIND

For the PROPFIND request, Firefox is very verbose but I don't understand fully:
```
Failed to load ‘http://localhost:8080/’. A ServiceWorker passed a promise to FetchEvent.respondWith() that rejected with ‘TypeError: NetworkError when attempting to fetch resource.’. 

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:8080/. (Reason: CORS request did not succeed). Status code: (null).
```

Brave/Chrome reports just
```
PROPFIND http://localhost:8080/ net::ERR_BLOCKED_BY_CLIENT
```

It is possible that the The Same Origin Policy is really the cause but I'm not sure if I can adjust it.
If I run dufs as the WebDAV server in the same directory:

```
dufs --enable-cors
```
the PROPFIND request returns XML, as expected:

```
<?xml version="1.0" encoding="utf-8" ?>
<D:multistatus xmlns:D="DAV:">
<D:response>
<D:href>/</D:href>
<D:propstat>
<D:prop>
<D:displayname></D:displayname>
```
etc.

The log is below. 8080 is rclone, 5000 is dufs.
![image](https://github.com/rclone/rclone/assets/462445/f5240e03-2643-4e8d-8e23-d40bad331749)

Note that dufs does not have anything specified for `Access-Control-Request-Method`.
> As noted in #5078, these headers are included in the Response and I'm wondering if that is the source of the issue:
> 
> ```
> Access-Control-Allow-Origin  | *
> Access-Control-Request-Method | POST, OPTIONS, GET, HEAD
> ```

Webdav requires quite a few special HTTP methods. Here is [the full list](https://www.ibm.com/docs/en/i/7.1?topic=concepts-webdav)


|Method|Specifications|Description|
| --- | --- | --- |
|COPY|WebDAV|Copies the resource.|
|DELETE|HTTP 1.1/WebDAV|Deletes the resource.|
|GET|HTTP 1.1|Gets the contents of the resource.|
|HEAD|HTTP 1.1|Returns the message headers from a message sent to the server.|
|LOCK|WebDAV|Locks the resource.|
|MKCOL|WebDAV|Creates the collection specified.|
|MOVE|WebDAV|Moves the resource.|
|OPTIONS|HTTP 1.1|Performs an option call to the server.|
|POST|HTTP 1.1|Action defined by the server.|
|PROPFIND|WebDAV|Performs a property find on the server.|
|PROPPATCH|WebDAV|Sets or removes properties on the server.|
|PUT|HTTP 1.1/WebDAV|Puts the contents of the resource to the server in the specified location.|
|TRACE|HTTP 1.1|Does a trace call to the server.|
|UNLOCK|WebDAV|Unlocks the resource.|

Rclone doesn't actually use all of these methods though.

According to the [Mozilla docs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Request-Method) `Access-Control-Request-Method` is something the client sends the server, not the other way round so I don't know why rclone is sending that.

Try commenting out, or changing this line to see if you can get it to work

https://github.com/rclone/rclone/blob/668711e43213bc9464c0b25c88621cd987898f08/lib/http/middleware.go#L186
>Try commenting out, or changing this line to see if you can get it to work

Good tip. Unfortunately, that doesn't seem to do it. :'(

![image](https://github.com/rclone/rclone/assets/462445/e07450b4-b4db-42ab-a8c9-10bb5e0b6561)

![image](https://github.com/rclone/rclone/assets/462445/921a43ff-bd7b-4466-b6de-c597f8768014)

Which is confusing considering that a request made from Windows Explorer lists the folder regularly:

![image](https://github.com/rclone/rclone/assets/462445/2fae4c1c-fbe3-4769-aea2-4aadce98c34c)

Obviously it's some combination with the browser. 

In Brave, all of the requests now fail. Chrome reports 

> Cross-origin resource sharing error: Method Disallowed by Preflight response

When using `dufs` as a server, everything works:

![image](https://github.com/rclone/rclone/assets/462445/d634cf81-ee86-4312-9e98-e69fb63016f3)

I also tried adding PROPFIND to the list at line 186 but the results are the same.
> rclone v1.65.0-DEV

The interesting thing is that, when issuing the same requests (GET, PROPFIND) from the REST Client vscode extension, the results are the same: GET works, PROPFIND doesn't.
> RequestError: getaddrinfo ENOTFOUND propfind

But when doing it from THUNDER Client, both work.

😱
So far, it is clear that the browser blocks these requests. 
The errors happen when the requests are issues from JavaScript and Wasm (which, I guess use the same fetch).
The question, however, is why? Why does this happen with rclone but not duf?
> Try commenting out, or changing this line to see if you can get it to work
> 
> https://github.com/rclone/rclone/blob/668711e43213bc9464c0b25c88621cd987898f08/lib/http/middleware.go#L186

I think this like is wrong...

I think it should be

    w.Header().Add("Access-Control-Allow-Methods", "POST, OPTIONS, GET, HEAD") 

And if you want webdav to work add some extra methods - this should do it.

    w.Header().Add("Access-Control-Allow-Methods", "COPY, DELETE, GET, HEAD, LOCK, MKCOL, MOVE, OPTIONS, POST, PROPFIND, PROPPATCH, PUT, TRACE, UNLOCK") 

Please try both of those!
> Access-Control-Allow-Methods

Yes!!!
I'll try a few variations and report back. Should I submit a pull request with that line modified?
# PROPFIND

For the PROPFIND request, Firefox is very verbose but I don't understand fully:
```
Failed to load ‘http://localhost:8080/’. A ServiceWorker passed a promise to FetchEvent.respondWith() that rejected with ‘TypeError: NetworkError when attempting to fetch resource.’. 

Cross-Origin Request Blocked: The Same Origin Policy disallows reading the remote resource at http://localhost:8080/. (Reason: CORS request did not succeed). Status code: (null).
```

Brave/Chrome reports just
```
PROPFIND http://localhost:8080/ net::ERR_BLOCKED_BY_CLIENT
```

It is possible that the The Same Origin Policy is really the cause but I'm not sure if I can adjust it.
If I run dufs as the WebDAV server in the same directory:

```
dufs --enable-cors
```
the PROPFIND request returns XML, as expected:

```
<?xml version="1.0" encoding="utf-8" ?>
<D:multistatus xmlns:D="DAV:">
<D:response>
<D:href>/</D:href>
<D:propstat>
<D:prop>
<D:displayname></D:displayname>
```
etc.

The log is below. 8080 is rclone, 5000 is dufs.
![image](https://github.com/rclone/rclone/assets/462445/f5240e03-2643-4e8d-8e23-d40bad331749)

Note that dufs does not have anything specified for `Access-Control-Request-Method`.
> As noted in #5078, these headers are included in the Response and I'm wondering if that is the source of the issue:
> 
> ```
> Access-Control-Allow-Origin  | *
> Access-Control-Request-Method | POST, OPTIONS, GET, HEAD
> ```

Webdav requires quite a few special HTTP methods. Here is [the full list](https://www.ibm.com/docs/en/i/7.1?topic=concepts-webdav)


|Method|Specifications|Description|
| --- | --- | --- |
|COPY|WebDAV|Copies the resource.|
|DELETE|HTTP 1.1/WebDAV|Deletes the resource.|
|GET|HTTP 1.1|Gets the contents of the resource.|
|HEAD|HTTP 1.1|Returns the message headers from a message sent to the server.|
|LOCK|WebDAV|Locks the resource.|
|MKCOL|WebDAV|Creates the collection specified.|
|MOVE|WebDAV|Moves the resource.|
|OPTIONS|HTTP 1.1|Performs an option call to the server.|
|POST|HTTP 1.1|Action defined by the server.|
|PROPFIND|WebDAV|Performs a property find on the server.|
|PROPPATCH|WebDAV|Sets or removes properties on the server.|
|PUT|HTTP 1.1/WebDAV|Puts the contents of the resource to the server in the specified location.|
|TRACE|HTTP 1.1|Does a trace call to the server.|
|UNLOCK|WebDAV|Unlocks the resource.|

Rclone doesn't actually use all of these methods though.

According to the [Mozilla docs](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Request-Method) `Access-Control-Request-Method` is something the client sends the server, not the other way round so I don't know why rclone is sending that.

Try commenting out, or changing this line to see if you can get it to work

https://github.com/rclone/rclone/blob/668711e43213bc9464c0b25c88621cd987898f08/lib/http/middleware.go#L186
>Try commenting out, or changing this line to see if you can get it to work

Good tip. Unfortunately, that doesn't seem to do it. :'(

![image](https://github.com/rclone/rclone/assets/462445/e07450b4-b4db-42ab-a8c9-10bb5e0b6561)

![image](https://github.com/rclone/rclone/assets/462445/921a43ff-bd7b-4466-b6de-c597f8768014)

Which is confusing considering that a request made from Windows Explorer lists the folder regularly:

![image](https://github.com/rclone/rclone/assets/462445/2fae4c1c-fbe3-4769-aea2-4aadce98c34c)

Obviously it's some combination with the browser. 

In Brave, all of the requests now fail. Chrome reports 

> Cross-origin resource sharing error: Method Disallowed by Preflight response

When using `dufs` as a server, everything works:

![image](https://github.com/rclone/rclone/assets/462445/d634cf81-ee86-4312-9e98-e69fb63016f3)

I also tried adding PROPFIND to the list at line 186 but the results are the same.
> rclone v1.65.0-DEV

The interesting thing is that, when issuing the same requests (GET, PROPFIND) from the REST Client vscode extension, the results are the same: GET works, PROPFIND doesn't.
> RequestError: getaddrinfo ENOTFOUND propfind

But when doing it from THUNDER Client, both work.

😱
So far, it is clear that the browser blocks these requests. 
The errors happen when the requests are issues from JavaScript and Wasm (which, I guess use the same fetch).
The question, however, is why? Why does this happen with rclone but not duf?
> Try commenting out, or changing this line to see if you can get it to work
> 
> https://github.com/rclone/rclone/blob/668711e43213bc9464c0b25c88621cd987898f08/lib/http/middleware.go#L186

I think this like is wrong...

I think it should be

    w.Header().Add("Access-Control-Allow-Methods", "POST, OPTIONS, GET, HEAD") 

And if you want webdav to work add some extra methods - this should do it.

    w.Header().Add("Access-Control-Allow-Methods", "COPY, DELETE, GET, HEAD, LOCK, MKCOL, MOVE, OPTIONS, POST, PROPFIND, PROPPATCH, PUT, TRACE, UNLOCK") 

Please try both of those!
> Access-Control-Allow-Methods

Yes!!!
I'll try a few variations and report back. Should I submit a pull request with that line modified?

Fix patch:
diff --git a/lib/http/middleware.go b/lib/http/middleware.go
index e7e4582b3c757..3ee5c0cdc2020 100644
--- a/lib/http/middleware.go
+++ b/lib/http/middleware.go
@@ -183,8 +183,8 @@ func MiddlewareCORS(allowOrigin string) Middleware {
 
 			if allowOrigin != "" {
 				w.Header().Add("Access-Control-Allow-Origin", allowOrigin)
-				w.Header().Add("Access-Control-Request-Method", "POST, OPTIONS, GET, HEAD")
 				w.Header().Add("Access-Control-Allow-Headers", "authorization, Content-Type")
+				w.Header().Add("Access-Control-Allow-Methods", "COPY, DELETE, GET, HEAD, LOCK, MKCOL, MOVE, OPTIONS, POST, PROPFIND, PROPPATCH, PUT, TRACE, UNLOCK")
 			}
 
 			next.ServeHTTP(w, r)


Test patch:
diff --git a/lib/http/middleware_test.go b/lib/http/middleware_test.go
index 283848f94c46c..15d0b0525d663 100644
--- a/lib/http/middleware_test.go
+++ b/lib/http/middleware_test.go
@@ -323,8 +323,8 @@ func TestMiddlewareAuthCertificateUser(t *testing.T) {
 
 var _testCORSHeaderKeys = []string{
 	"Access-Control-Allow-Origin",
-	"Access-Control-Request-Method",
 	"Access-Control-Allow-Headers",
+	"Access-Control-Allow-Methods",
 }
 
 func TestMiddlewareCORS(t *testing.T) {