Repository: rclone/rclone
Pull request #2921 (2019-01-17 12:46:45)

webdav: add support for sharepoint NTLM auth
fix for issue #2171

<!--
Thank you very much for contributing code or documentation to rclone! Please
fill out the following questions to make it easier for us to review your
changes.

You do not need to check all the boxes below all at once, feel free to take
your time and add more commits. If you're done and ready for review, please
check the last box.
-->

#### What is the purpose of this change?

<!--
Describe the changes here
-->

#### Was the change discussed in an issue or in the forum before?

<!--
Link issues and relevant forum posts here.
-->

#### Checklist

- [x] I have read the [contribution guidelines](https://github.com/ncw/rclone/blob/master/CONTRIBUTING.md#submitting-a-pull-request).
- [x] I have added tests for all changes in this PR if appropriate.
- [x] I have added documentation for the changes if appropriate.
- [x] All commit messages are in [house style](https://github.com/ncw/rclone/blob/master/CONTRIBUTING.md#commit-messages).
- [x] I'm done, this Pull Request is ready for review :-)


ivandeex: This PR needs a maintainer.
ncw: It is a simple enough patch as it is basically adding support for github.com/Azure/go-ntlmssp - however we need some documentation to go with it...
Cnly: I think a big difficulty for this PR is finding a Sharepoint instance to test the NTLM auth? Can we do that with Azure online instead of on-premise setups?
ncw: > I think a big difficulty for this PR is finding a Sharepoint instance to test the NTLM auth? Can we do that with Azure online instead of on-premise setups?

I don't know... It sounds promising... I don't relish setting it up though! Do you want to have a go @Cnly ?
ivandeex: Rebased on master
ivandeex: @Cnly Have you had any success with Azure approach?
Cnly: > @Cnly Have you had any success with Azure approach?

Nope. 😕 I found that doesn't really look like a plausible way to go...
ncw: > > @Cnly Have you had any success with Azure approach?
> 
> Nope.  I found that doesn't really look like a plausible way to go...

:-(

Shall we merge it anyway @Cnly ? It has the advantage of being relatively self contained so the blast radius is small.

What do you think about the docs?
Rots: There are many tests failing, but the basic operations seem to work I can list and download files.
georne: Hi, thank you all for your work on this! I thought I'd help with testing and tried this PR with an on-premises Sharepoint Server 2019 site.

When I run `rclone ls` on the remote, this fails with a `stream error: stream ID 39; HTTP_1_1_REQUIRED`.

I have not been able to look deeper into the issue, so all that I can say for now is that NTLM authentication indeed requires HTTP/1.1 (rather than HTTP/2). Do you have a Sharepoint site that I can access and test against so I can make sure that the error is not on my side?
ivandeex: @georne 
Thank you very much for help.
Tomorrow I will make a binary with this PR on top of latest rclone master for you to try.
georne: While waiting for @ivandeex 's binary, I tried again with the binary that I had built myself. Surprisingly, `rclone ls` on the _same SharePoint remote_ where the _same binary_ had failed 2 days ago **worked fine** today! I am planning to do more testing on Monday.
ivandeex: @georne I'm being working on the staff. Sorry for delay.
georne: I can now consistently reproduce the error:

- When I am connecting to the SharePoint server _over the internet_, everything is _fine_. (I do get spurious `401 Unauthorized` errors, but the automatic retries work around the issue: `Attempt 2/3 succeeded`)

- When I am in the same _local network_ with the SP server, everything _fails_ with `stream error: stream ID 39; HTTP_1_1_REQUIRED`.

The SharePoint server, rclone binary, config, and invocation are of course the same in both cases. I suspect that a reverse proxy is in the path (and somehow fixes things) when I connect over the internet.

Is there an easy way to log the full, unencrypted HTTP traffic from an rclone session? 
ivandeex: rclone -vv --dump headers,bodies ....
ncw: > * When I am in the same _local network_ with the SP server, everything _fails_ with `stream error: stream ID 39; HTTP_1_1_REQUIRED`.

Ah, this looks like rclone is attempting HTTP/2 but something is complaining about that.

I don't know if you saw this issue: https://github.com/Azure/go-ntlmssp/issues/15

We had to disable HTTP/2 for the drive backend - here is how we did it

https://github.com/rclone/rclone/blob/41127965b036a12645243f3ccd0a150581daba9b/backend/drive/drive.go#L991-L1001


georne: Thanks a lot for the helpful pointers @ivandeex and @ncw ! TL;DR: Applying the fix suggested by @ncw should solve the issue.


Comparing the logs (`-vv --dump headers,bodies` - only including the relevant parts below), I see exactly the same first client request:

```
DEBUG : HTTP REQUEST
DEBUG : PROPFIND /projects/myproject/fileshare/ HTTP/1.1
```

But there is a crucial difference in the very first server response.

Working:
```
DEBUG : HTTP RESPONSE
DEBUG : HTTP/1.1 401 Unauthorized
Server: Apache
Www-Authenticate: NTLM
```

Not working:
```
DEBUG : HTTP RESPONSE
DEBUG : HTTP/2.0 401 Unauthorized
Server: Microsoft-IIS/10.0
Www-Authenticate: NTLM
```

Although this leads to the question where the HTTP/2.0 comes from (really IIS or maybe something to do with the http package?), it at least confirms my suspicion that the reverse proxy is masking the issue.

I attempted to apply the suggested fix to line 341 of `rclone/backend/webdav/webdav.go` but ran out of time, so I set `GODEBUG=http2client=0` as documented in https://golang.org/pkg/net/http/ . The client request still looks identical, but the first server response now reads:

```
DEBUG : HTTP RESPONSE
DEBUG : HTTP/1.1 401 Unauthorized
Server: Microsoft-IIS/10.0
Www-Authenticate: NTLM
```

... and everything works fine.

Of course there might be a configuration issue with the IIS that I am talking to (I couldn't say), but disabling HTTP/2 as suggested seems like a good idea to me for sharepoint-ntlm WebDAV remotes.

georne: Here is a patch that disables HTTP/2 as suggested and solves the issue that occurs when talking to IIS 10 directly. I have not done any systematic testing, but it works fine in my setup. 

Could someone be so kind to check if I've done anything obviously dumb and - provided I didn't - add this to this PR? It's only 3 lines basically, so it should be quick to review. 

[disable_http2.patch.txt](https://github.com/rclone/rclone/files/5920481/disable_http2.patch.txt)

ncw: @georne 
> Could someone be so kind to have a look and add this to this PR if appropriate?

That code looks good!

If you push your branch up to your github then I or @ivandeex can cherry-pick the commit and push it into this branch.
ivandeex: I'm still working on https://github.com/ivandeex/sharepoint-vm
Currently tuning timeouts between installation reboots
Hope it can help reproduce the issue and debug the code
Next I will need advice from @georne to reproduce his setup
I'm not sure yet what is the appropriate method of exporting folders/resources via Sharepoint
georne: Thanks for the review @ncw ! Here's the commit: b626612bf669519e1f4e3d2c66b6372042cfaaa7

@ivandeex , I am happy to help where I can, but I am only a SP _end-user_ I'm afraid.

On another note, I am also regularly seeing `corrupted on transfer: sizes differ` errors besides the already mentioned `couldn't list files: 401 Unauthorized` and `failed to open source object: 401 Unauthorized`, but so far either the 1st or the 2nd automatic retry attempt have made them all disappear (except for the Forms folder, where I would not expect anything else).
ivandeex: @georne 
I have force-pushed your patch into this PR.

I have also made https://github.com/ivandeex/sharepoint-vm sufficiently stable and created a test lab.
You can try to play with it using instructions from https://github.com/ivandeex/sharepoint-vm#testing
and credentials I sent to your mail.

I am still working to make tests pass.
ivandeex: @ncw
How can one fix the "xgo cross compile" error?
```
error writing go.mod: open /ext-go/1/src/github.com/rclone/rclone/go.mod399753607.tmp: read-only file system
```
Assuming that github runner keeps working directory read-only I modeled that and remounted rclone source directory read-only (so that even root inside container cannot write). I see cross-go in Docker attempting to write temporary _go.mod_ but I don't undestand why. You have probably met this issue before. Please help.
georne: I can now consistently reproduce the `401 Unauthorized` errors and have narrowed down their cause.

They always occur when a sub-tree exceeds a certain number of files and folders. I can't pinpoint the number exactly, but here is an example [rclonetest.zip](https://github.com/rclone/rclone/files/5975351/rclonetest.zip) with 65 files in 21 folders (no deep nesting) that always triggers `error listing: couldn't list files: 401 Unauthorized` for me on multiple folders, different ones each time. While `lsd`and `lsf` work, `ls` and `size` _always fail_ on this example tree.

The _errors disappear_ and everything works fine when I set `--checkers 1`.

I'm afraid that I won't have the time to debug this myself in the near future.
georne: To be more precise: 
- I can reproduce/eliminate the 401s as above in "my" environment (SharePoint server 2019, IIS 10, HTTPS), but 
- I _cannnot_ reproduce them in @ivandeex ' current test lab (SharePoint server 2016, IIS 8.5, HTTP).

(I am not saying that the different software versions and SSL are the reason for that, just pointing it out because it might have an influence.)
ivandeex: @ncw

While @georne investigates bugs in his _IE10 / Sharepoint 2019_ setup, I'm working on making integration tests pass for my _IE8.5 / Sharepoint 2016_ testbed. I see many enoding related bugs. The webdav backend currently lacks a onedrive-like encoding layer. I will add one with defaults depending on the `type`: IIS-relevant translations for `sharepoint-ntlm`, identity otherwise,

I remember you had a program that can be run against a backend (you sent it to me w.r.t. mailru backend) and it print the correct encoder settings. Could you point me at the relevant howto or comment, please?

Also, please take a look at https://github.com/rclone/rclone/pull/2921#issuecomment-778010515
I'm digging through the billziss's container but can't understand why it would need to write a temporary `go.mod`.
Moreover, if it needed to add something in go.mod for e.g. Android, why don't I see any changes if I permit writes?
I'm stuck at this point. I think you might have encountered this quirk already.

Thank you!
ivandeex: @georne 

> I can now consistently reproduce the `401 Unauthorized` errors and have narrowed down their cause.
> They always occur when a sub-tree exceeds a certain number of files and folders. I can't pinpoint the number exactly, but here is an example [rclonetest.zip](https://github.com/rclone/rclone/files/5975351/rclonetest.zip) with 65 files in 21 folders (no deep nesting) that always triggers `error listing: couldn't list files: 401 Unauthorized` for me on multiple folders, different ones each time. While `lsd`and `lsf` work, `ls` and `size` _always fail_ on this example tree.

Does it depend on sizes or contents of test files?
We could add a new test case in _webdav_test.go_ that generates files according to your pattern using test helpers from `fstests.go`, then calls `List` and `Size`.

Generally, the more automated tests you have, the less fragile your program is.
georne: @ivandeex

No, the 401 errors occur independently of the file size and content. (This actually makes sense, considering that they also occur with `ls` and `size`, where I would not expect file contents to be involved.) It seems that as soon as multiple checkers are running in parallel, most of them fail to properly handle the NTLM authentication at the first attempt, and the 401 responses which are just a normal part of the protocol are raised as errors.
georne: I have continued to look into the errors that I am getting on "my" setup (IIS 10, SP 2019).

* For `copy` and `sync` operations, the _401 errors disappear_ when I specify `--transfers 1` in addition to `--checkers 1`.
* The 401 errors occur no matter if rclone is running on Windows or on Linux.
* I have _not been able to reproduce_ the `corrupted on transfer: sizes differ` errors that I encountered previously. I have not changed anything, but the same operations on the same remote now run completely without showing this kind of error. Maybe there was some kind of network issue.
* I also tested transferring large files (> 1 GB), which worked without issues.

I wonder if forcing the number of checkers and transfers to 1 (or at least setting their default number to 1) specifically for sharepoint-ntlm remotes - until someone finds the time to have a deeper look into the issue - could be an option. A copy operation with 400 MB total in 170 objects slows down from 70 s to 100 s for me this way, which would not be unreasonable.
ivandeex: Assigning on @ncw too to have some attention as this thread has questions for him
ivandeex: @georne 
I found why you see 401 errors with NTLM auth enabled. After increasing size of the test tree sufficiently I can reproduce your situation and tell what's going on.
If you look at the code of NTLM authenticator (https://github.com/Azure/go-ntlmssp/blob/master/negotiator.go#L30), you will see that negotiation can involve several roundtrips (look for the calls `rt.RoundTrip(req)`). The negotiator tries various authentication methods one after another and analyzes replies from Sharepoint. Intermediate responses from Sharepoint have status code 401 (i.e. authentication _still_ required). The final response returns something else, e.g. "directory created", "listing returned" or maybe some kind of error (not 401 now).
When rclone decides to run a large operation in several threads/goroutines according to the _--checkers_ or _--transfers_ flags, the threads will all use a common http client from the filesystem descriptor AFAICS. The request-response loops from them can interleave with a probability proportional to the amount of requested data and one of threads can receive _intermediate 401 response_ targeted for another thread thinking that its own request failed with authentication error.
The _go-ntlmssp_ has a bug request https://github.com/Azure/go-ntlmssp/issues/14 describing this very issue. It's hanging since 2018 without any attention from developers.
I think that other authentication methods supported by the `webdav` backend do not hit this situation because those methods involve less roundtrips.
I have pushed a commit that guards http requests by a mutex effectively serializing roundtrips from multiple threads. This approach made the `size` and `ls` commands work without errors in my testbed (IE/8.5 with pure HTTP).
Please pull the latest source and try it out.
Thanks, Ivan
georne: @ivandeex

Great find, thanks a lot! That goes exactly into the direction that I was thinking of. Good to know that the IIS/SP versions have nothing to do with the issue.

I can confirm that the 401s during `size` and `ls` are gone indeed. However, I still see 401s at the beginning of `copy` and `sync` actions - not every time, but most of the time. I still have to set `--transfers 1` to make these go away. 

ivandeex: The commit "_Check that purged directory really exists_" (https://github.com/rclone/rclone/pull/2921/commits/d717aa6493e8bab46adf84ebab608f4cfbb69122) fixes another integration test failure.
Now integration suite passes in my test environment (Windows Server 2012 R2, Sharepoint 2016, IIS/8.5, plain HTTP, rclone client on ubuntu 20.04 with go 1.14):
```
go run ./fstest/test_all -verbose -backends webdav -remotes ntlm:
......
PASS: All tests finished OK in 1m40..s
```

rclone config:
```
[ntlm]
type = webdav
url = http://spap.redacted.com/sites/test/Shared%20Documents
vendor = sharepoint-ntlm
user = redacted\spadmin
pass = redacted
```
I am going to enable HTTPS now.

@georne
> I still see 401s at the beginning of copy and sync actions - not every time, but most of the time.

I will look into it. Gotta reproduce.
ivandeex: @georne 
I can't reproduce 401 with copy/move and _current_ branch. Testing repeatedly with a tree of 50 dirs and 2500 files.
`copy` never failed. `sync` fails regularly with _409 CONFLICT_ (each time at a random file at the beginning) but rclone every time retried it to success:
```
ubuntu@unix:~/rclone$ ./rclone purge ntlm:xxx >&/dev/null; time ./rclone sync ntlm:auth401test ntlm:xxx; echo retval $?
2021/02/22 20:19:45 ERROR : 01/01.txt: Failed to copy: Copy call failed: 409 CONFLICT
2021/02/22 20:23:26 ERROR : webdav root 'xxx': not deleting files as there were IO errors
2021/02/22 20:23:26 ERROR : webdav root 'xxx': not deleting directories as there were IO errors
2021/02/22 20:23:26 ERROR : Attempt 1/3 failed with 1 errors and: Copy call failed: 409 CONFLICT
2021/02/22 20:23:27 ERROR : Attempt 2/3 succeeded
real    3m52.387s
retval 0
```
Please confirm that you are on the _latest_ branch (https://github.com/rclone/rclone/pull/2921/commits/d717aa6493e8bab46adf84ebab608f4cfbb69122).
BTW, Can you try the full suite in your env?
(`go run ./fstest/test_all -verbose -backends webdav -remotes YourTestRemoteName:`)
ivandeex: Google returns a number of useful advices regarfing 409. One example from https://stackoverflow.com/questions/6785036/reasons-for-a-409-conflict-http-error-when-uploading-a-file-to-sharepoint-using
> We are also getting the same error while we are trying to access a same resource within milliseconds.

Sounds fair to me. Now as we have authorization phases of our web requests serialized by mutex, data exchange phases follow each other within milliseconds. IIS and application server behind it (from looking at the IIS management I believe it's based on .NET CLR) need some time to check permissions for each request, query database etc. In my test environment the Sharepoint farms consists of a single 1-CPU VPS server that queries a SQL Server database on another 1-CPU VPS server.

Sometimes IIS will experience contention, especially when client sends 2500 files in 10 parallel streams. While the test Linux box is a small 1-CPU VPS, rclone is a golang application that can use multiple parallel goroutines even on a 1-CPU box quickly switching goroutine in a single Linux thread on a single CPU when someone pauses waiting on network I/O.

I think I need not fix the 409 issue. rclone's retry machinery handles it pretty gracefully. Generally the result will depend on characteristics of your Sharepoint server hardware. Either add CPUs or fall back to the `--transfers 1` hack. Does this sound reasonable?
ivandeex: **I enabled SSL for the test Sharepoint IIS site.**
I don't have time now to add this possibility in the https://github.com/ivandeex/sharepoint-vm scripts. Just manually recreated the Sharepoint web application using their `Central Administration`. One minor trick: you will have to access the site once with a browser for Sharepoint to create its initial content from templates and do the warm-up.

In the process Sharepoint created a self-signed certificate named `spap.redacted.com`. I export-imported it to the `Control Panel > Internet Properties > Content > Certificates > Trusted Root Certificate Authorities`, and Windows explorer successfully mounted the site document library as a web folder.

Linux...
For starters, the site responded to `curl -v -k https://spap.redacted.com/sites/test/Shared%20Documents` (the `-k` flag tells curl to accept self-signedness).
After editing a single line in rclone.conf (note the extra `s` in _https_, I kept the same remote name `ntlm`):
```
url = https://spap.redacted.com/sites/test/Shared%20Documents
```
the command `rclone lsd --no-check-certificate ntlm:` worked for me.

I then exported the cert from the Windows control panel (it's formatted as CER/DER), copied it to Ubuntu and installed as a trusted certificate:
```
rclone copy --no-check-certificate ntlm:self-signed.cer .
sudo apt-get install -y ca-certificates
openssl x509 -in self-signed.cer -inform der -out self-signed.pem
sudo cp ./self-signed.pem /usr/local/share/ca-certificates/self-signed.crt
sudo update-ca-certificates
```
This "blessed" the certificate for the whole Linux and I could use rclone as simply as `rclone ls ntlm:`

The `stream error HTTP_1_1_REQUIRED` never hurt me thanks to the patch provided by gnädiger @georne.

The size test (`rclone size ntlm:auth401test`) worked without errors on 2500 files in 50 folders, repeatedly. Uploading 2500 files from local disk to Sharepoint (`./rclone sync ./xxx ntlm:xxx`) worked smoothly in a loop. CPU load on the Sharepoint server kept around 60% during the upload. File transfer from Sharepoint to Sharepoint still prints `409 CONFLICT` which is succssfully retried by rclone:
```
$ ./rclone purge ntlm:xxx &>/dev/null; time ./rclone sync ntlm:auth401test ntlm:xxx
2021/02/22 22:56:48 ERROR : 01/01.txt: Failed to copy: Copy call failed: 409 CONFLICT
2021/02/22 23:00:16 ERROR : webdav root 'xxx': not deleting files as there were IO errors
2021/02/22 23:00:16 ERROR : webdav root 'xxx': not deleting directories as there were IO errors
2021/02/22 23:00:16 ERROR : Attempt 1/3 failed with 1 errors and: Copy call failed: 409 CONFLICT
2021/02/22 23:00:18 ERROR : Attempt 2/3 succeeded
real    3m38.913s
$ echo $?
0
```

Integration tests pass with SSL too:
```
$ go run ./fstest/test_all -verbose -backends webdav -remotes ntlm:
2021/02/22 23:22:27 Remote "ntlm:" not found - inserting with default flags
2021/02/22 23:22:27 Testing remotes: ntlm:
...snip...
2021/02/22 23:23:45 PASS: All tests finished OK in 1m17..s
```

cc @georne
ivandeex: If no more requests bubble up here for a while, I am going to shutdown the testbed (anyone interested can recreate it for themselves using https://github.com/ivandeex/sharepoint-vm and terraform) and switch to other tasks.
ncw: Thank you for your excellent testing @ivandeex - that gives great confidence in the changes.

I think we should merge this now.

Let's keep the history as it has several authors contributing and the individual commits are relatively self contained.

Can you
- add `webdav: ` to the front of the first commit lines that don't have them
- remove the (author)

I'm happy for you to merge then

Thank you :-)
ivandeex: Waiting for @georne to run integration tests in his env and confirm the 401 fix. https://github.com/rclone/rclone/pull/2921#issuecomment-783653633
georne: @ivandeex

The 401 errors on `copy` and `sync` have completely disappeared. I have to admit that I have no idea why; it is just as mysterious to me as it was with the `corrupted on transfer` errors. Anyway, everything looks fine now in my environment! I also was not able to provoke a single 409 error. 

> BTW, Can you try the full suite in your env?
> (`go run ./fstest/test_all -verbose -backends webdav -remotes YourTestRemoteName:`)

On Windows, `TestRWFileModTimeWithOpenWriters`, `TestRWCacheRename`, `TestCaseSensitivity`, and `TestVFSNew` (all from the `vfs` test) fail. All other tests (i.e. `backend/webdav`, `fs/operations`, `fs/sync`) pass, sometimes only on the first retry though. 

On Linux, _all tests pass_ (at roughly twice the speed btw).

I made sure to run everything directly against the IIS 10 (and not the Apache proxy) at all times.

One final thing that I noted is that file timestamps are not preserved on upload (but they are preserved on download).

All in all, this looks more than perfectly usable to me. Thank you very much again for everything!
ivandeex: Vielen dank @georne !

Patch:
diff --git a/backend/webdav/webdav.go b/backend/webdav/webdav.go
index e441b06a5..dfc490947 100644
--- a/backend/webdav/webdav.go
+++ b/backend/webdav/webdav.go
@@ -10,6 +10,7 @@ package webdav
 import (
 	"bytes"
 	"context"
+	"crypto/tls"
 	"encoding/xml"
 	"fmt"
 	"io"
@@ -19,20 +20,25 @@ import (
 	"path"
 	"strconv"
 	"strings"
+	"sync"
 	"time"
 
 	"github.com/pkg/errors"
 	"github.com/rclone/rclone/backend/webdav/api"
 	"github.com/rclone/rclone/backend/webdav/odrvcookie"
 	"github.com/rclone/rclone/fs"
+	"github.com/rclone/rclone/fs/config"
 	"github.com/rclone/rclone/fs/config/configmap"
 	"github.com/rclone/rclone/fs/config/configstruct"
 	"github.com/rclone/rclone/fs/config/obscure"
 	"github.com/rclone/rclone/fs/fserrors"
 	"github.com/rclone/rclone/fs/fshttp"
 	"github.com/rclone/rclone/fs/hash"
+	"github.com/rclone/rclone/lib/encoder"
 	"github.com/rclone/rclone/lib/pacer"
 	"github.com/rclone/rclone/lib/rest"
+
+	ntlmssp "github.com/Azure/go-ntlmssp"
 )
 
 const (
@@ -42,8 +48,22 @@ const (
 	defaultDepth  = "1" // depth for PROPFIND
 )
 
+const defaultEncodingSharepointNTLM = (encoder.EncodeWin |
+	encoder.EncodeHashPercent | // required by IIS/8.5 in contrast with onedrive which doesn't need it
+	(encoder.Display &^ encoder.EncodeDot) | // test with IIS/8.5 shows that EncodeDot is not needed
+	encoder.EncodeBackSlash |
+	encoder.EncodeLeftSpace |
+	encoder.EncodeLeftTilde |
+	encoder.EncodeRightPeriod |
+	encoder.EncodeRightSpace |
+	encoder.EncodeInvalidUtf8)
+
 // Register with Fs
 func init() {
+	configEncodingHelp := fmt.Sprintf(
+		"%s\n\nDefault encoding is %s for sharepoint-ntlm or identity otherwise.",
+		config.ConfigEncodingHelp, defaultEncodingSharepointNTLM)
+
 	fs.Register(&fs.RegInfo{
 		Name:        "webdav",
 		Description: "Webdav",
@@ -67,14 +87,17 @@ func init() {
 				Help:  "Owncloud",
 			}, {
 				Value: "sharepoint",
-				Help:  "Sharepoint",
+				Help:  "Sharepoint Online, authenticated by Microsoft account.",
+			}, {
+				Value: "sharepoint-ntlm",
+				Help:  "Sharepoint with NTLM authentication. Usually self-hosted or on-premises.",
 			}, {
 				Value: "other",
 				Help:  "Other site/service or software",
 			}},
 		}, {
 			Name: "user",
-			Help: "User name",
+			Help: "User name. In case NTLM authentication is used, the username should be in the format 'Domain\\User'.",
 		}, {
 			Name:       "pass",
 			Help:       "Password.",
@@ -86,18 +109,23 @@ func init() {
 			Name:     "bearer_token_command",
 			Help:     "Command to run to get a bearer token",
 			Advanced: true,
+		}, {
+			Name:     config.ConfigEncoding,
+			Help:     configEncodingHelp,
+			Advanced: true,
 		}},
 	})
 }
 
 // Options defines the configuration for this backend
 type Options struct {
-	URL                string `config:"url"`
-	Vendor             string `config:"vendor"`
-	User               string `config:"user"`
-	Pass               string `config:"pass"`
-	BearerToken        string `config:"bearer_token"`
-	BearerTokenCommand string `config:"bearer_token_command"`
+	URL                string               `config:"url"`
+	Vendor             string               `config:"vendor"`
+	User               string               `config:"user"`
+	Pass               string               `config:"pass"`
+	BearerToken        string               `config:"bearer_token"`
+	BearerTokenCommand string               `config:"bearer_token_command"`
+	Enc                encoder.MultiEncoder `config:"encoding"`
 }
 
 // Fs represents a remote webdav
@@ -114,8 +142,10 @@ type Fs struct {
 	canStream          bool          // set if can stream
 	useOCMtime         bool          // set if can use X-OC-Mtime
 	retryWithZeroDepth bool          // some vendors (sharepoint) won't list files when Depth is 1 (our default)
+	checkBeforePurge   bool          // enables extra check that directory to purge really exists
 	hasMD5             bool          // set if can use owncloud style checksums for MD5
 	hasSHA1            bool          // set if can use owncloud style checksums for SHA1
+	ntlmAuthMu         sync.Mutex    // mutex to serialize NTLM auth roundtrips
 }
 
 // Object describes a webdav object
@@ -179,6 +209,22 @@ func (f *Fs) shouldRetry(resp *http.Response, err error) (bool, error) {
 	return fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
 }
 
+// safeRoundTripper is a wrapper for http.RoundTripper that serializes
+// http roundtrips. NTLM authentication sequence can involve up to four
+// rounds of negotiations and might fail due to concurrency.
+// This wrapper allows to use ntlmssp.Negotiator safely with goroutines.
+type safeRoundTripper struct {
+	fs *Fs
+	rt http.RoundTripper
+}
+
+// RoundTrip guards wrapped RoundTripper by a mutex.
+func (srt *safeRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
+	srt.fs.ntlmAuthMu.Lock()
+	defer srt.fs.ntlmAuthMu.Unlock()
+	return srt.rt.RoundTrip(req)
+}
+
 // itemIsDir returns true if the item is a directory
 //
 // When a client sees a resourcetype it doesn't recognize it should
@@ -285,7 +331,11 @@ func addSlash(s string) string {
 
 // filePath returns a file path (f.root, file)
 func (f *Fs) filePath(file string) string {
-	return rest.URLPathEscape(path.Join(f.root, file))
+	subPath := path.Join(f.root, file)
+	if f.opt.Enc != encoder.EncodeZero {
+		subPath = f.opt.Enc.FromStandardPath(subPath)
+	}
+	return rest.URLPathEscape(subPath)
 }
 
 // dirPath returns a directory path (f.root, dir)
@@ -324,6 +374,10 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 	}
 	root = strings.Trim(root, "/")
 
+	if opt.Enc == encoder.EncodeZero && opt.Vendor == "sharepoint-ntlm" {
+		opt.Enc = defaultEncodingSharepointNTLM
+	}
+
 	// Parse the endpoint
 	u, err := url.Parse(opt.URL)
 	if err != nil {
@@ -336,10 +390,28 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 		opt:         *opt,
 		endpoint:    u,
 		endpointURL: u.String(),
-		srv:         rest.NewClient(fshttp.NewClient(ctx)).SetRoot(u.String()),
 		pacer:       fs.NewPacer(ctx, pacer.NewDefault(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant))),
 		precision:   fs.ModTimeNotSupported,
 	}
+
+	client := fshttp.NewClient(ctx)
+	if opt.Vendor == "sharepoint-ntlm" {
+		// Disable transparent HTTP/2 support as per https://golang.org/pkg/net/http/ ,
+		// otherwise any connection to IIS 10.0 fails with 'stream error: stream ID 39; HTTP_1_1_REQUIRED'
+		// https://docs.microsoft.com/en-us/iis/get-started/whats-new-in-iis-10/http2-on-iis says:
+		// 'Windows authentication (NTLM/Kerberos/Negotiate) is not supported with HTTP/2.'
+		t := fshttp.NewTransportCustom(ctx, func(t *http.Transport) {
+			t.TLSNextProto = map[string]func(string, *tls.Conn) http.RoundTripper{}
+		})
+
+		// Add NTLM layer
+		client.Transport = &safeRoundTripper{
+			fs: f,
+			rt: ntlmssp.Negotiator{RoundTripper: t},
+		}
+	}
+	f.srv = rest.NewClient(client).SetRoot(u.String())
+
 	f.features = (&fs.Features{
 		CanHaveEmptyDirectories: true,
 	}).Fill(ctx, f)
@@ -465,6 +537,16 @@ func (f *Fs) setQuirks(ctx context.Context, vendor string) error {
 		// to determine if we may have found a file, the request has to be resent
 		// with the depth set to 0
 		f.retryWithZeroDepth = true
+	case "sharepoint-ntlm":
+		// Sharepoint with NTLM authentication
+		// See comment above
+		f.retryWithZeroDepth = true
+
+		// Sharepoint 2016 returns status 204 to the purge request
+		// even if the directory to purge does not really exist
+		// so we must perform an extra check to detect this
+		// condition and return a proper error code.
+		f.checkBeforePurge = true
 	case "other":
 	default:
 		fs.Debugf(f, "Unknown vendor %q", vendor)
@@ -583,7 +665,11 @@ func (f *Fs) listAll(ctx context.Context, dir string, directoriesOnly bool, file
 			fs.Debugf(nil, "Item with unknown path received: %q, %q", u.Path, baseURL.Path)
 			continue
 		}
-		remote := path.Join(dir, u.Path[len(baseURL.Path):])
+		subPath := u.Path[len(baseURL.Path):]
+		if f.opt.Enc != encoder.EncodeZero {
+			subPath = f.opt.Enc.ToStandardPath(subPath)
+		}
+		remote := path.Join(dir, subPath)
 		if strings.HasSuffix(remote, "/") {
 			remote = remote[:len(remote)-1]
 		}
@@ -800,6 +886,21 @@ func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) error {
 		if notEmpty {
 			return fs.ErrorDirectoryNotEmpty
 		}
+	} else if f.checkBeforePurge {
+		// We are doing purge as the `check` argument is unset.
+		// The quirk says that we are working with Sharepoint 2016.
+		// This provider returns status 204 even if the purged directory
+		// does not really exist so we perform an extra check here.
+		// Only the existence is checked, all other errors must be
+		// ignored here to make the rclone test suite pass.
+		depth := defaultDepth
+		if f.retryWithZeroDepth {
+			depth = "0"
+		}
+		_, err := f.readMetaDataForPath(ctx, dir, depth)
+		if err == fs.ErrorObjectNotFound {
+			return fs.ErrorDirNotFound
+		}
 	}
 	opts := rest.Opts{
 		Method:     "DELETE",
diff --git a/docs/content/webdav.md b/docs/content/webdav.md
index b5c65a073..557e00aec 100644
--- a/docs/content/webdav.md
+++ b/docs/content/webdav.md
@@ -45,9 +45,11 @@ Choose a number from below, or type in your own value
    \ "nextcloud"
  2 / Owncloud
    \ "owncloud"
- 3 / Sharepoint
+ 3 / Sharepoint Online, authenticated by Microsoft account.
    \ "sharepoint"
- 4 / Other site/service or software
+ 4 / Sharepoint with NTLM authentication. Usually self-hosted or on-premises.
+   \ "sharepoint-ntlm"
+ 5 / Other site/service or software
    \ "other"
 vendor> 1
 User name
@@ -136,6 +138,8 @@ Name of the Webdav site/service/software you are using
         - Owncloud
     - "sharepoint"
         - Sharepoint
+    - "sharepoint-ntlm"
+        - Sharepoint with NTLM authentication
     - "other"
         - Other site/service or software
 
@@ -148,6 +152,8 @@ User name
 - Type:        string
 - Default:     ""
 
+In case vendor mode `sharepoint-ntlm` is used, the user name is in the form `DOMAIN\user`
+
 #### --webdav-pass
 
 Password.
@@ -201,7 +207,7 @@ This is configured in an identical way to Owncloud.  Note that
 Nextcloud initially did not support streaming of files (`rcat`) whereas
 Owncloud did, but [this](https://github.com/nextcloud/nextcloud-snap/issues/365) seems to be fixed as of 2020-11-27 (tested with rclone v1.53.1 and Nextcloud Server v19).
 
-### Sharepoint ###
+### Sharepoint Online ###
 
 Rclone can be used with Sharepoint provided by OneDrive for Business
 or Office365 Education Accounts.
@@ -237,11 +243,40 @@ Your config file should look like this:
 [sharepoint]
 type = webdav
 url = https://[YOUR-DOMAIN]-my.sharepoint.com/personal/[YOUR-EMAIL]/Documents
-vendor = other
+vendor = sharepoint
 user = YourEmailAddress
 pass = encryptedpassword
 ```
 
+### Sharepoint with NTLM Authentication ###
+
+Use this option in case your (hosted) Sharepoint is not tied to OneDrive accounts and uses NTLM authentication.
+
+To get the `url` configuration, similarly to the above, first navigate to the desired directory in your browser to get the URL,
+then strip everything after the name of the opened directory.
+
+Example:
+If the URL is:
+https://example.sharepoint.com/sites/12345/Documents/Forms/AllItems.aspx
+
+The configuration to use would be:
+https://example.sharepoint.com/sites/12345/Documents
+
+Set the `vendor` to `sharepoint-ntlm`.
+
+NTLM uses domain and user name combination for authentication,
+set `user` to `DOMAIN\username`.
+
+Your config file should look like this:
+
+```
+[sharepoint]
+type = webdav
+url = https://[YOUR-DOMAIN]/some-path-to/Documents
+vendor = sharepoint-ntlm
+user = DOMAIN\user
+pass = encryptedpassword
+```
 #### Required Flags for SharePoint ####
 As SharePoint does some special things with uploaded documents, you won't be able to use the documents size or the documents hash to compare if a file has been changed since the upload / which file is newer.
 
diff --git a/go.mod b/go.mod
index 13c94287a..61958e4ed 100644
--- a/go.mod
+++ b/go.mod
@@ -8,6 +8,7 @@ require (
 	github.com/Azure/azure-pipeline-go v0.2.3
 	github.com/Azure/azure-storage-blob-go v0.13.0
 	github.com/Azure/go-autorest/autorest/adal v0.9.10
+	github.com/Azure/go-ntlmssp v0.0.0-20200615164410-66371956d46c
 	github.com/Microsoft/go-winio v0.4.16 // indirect
 	github.com/Unknwon/goconfig v0.0.0-20200908083735-df7de6a44db8
 	github.com/a8m/tree v0.0.0-20210115125333-10a5fd5b637d
diff --git a/go.sum b/go.sum
index 135d8930f..3ca16fe04 100644
--- a/go.sum
+++ b/go.sum
@@ -55,6 +55,8 @@ github.com/Azure/go-autorest/autorest/mocks v0.4.1 h1:K0laFcLE6VLTOwNgSxaGbUcLPu
 github.com/Azure/go-autorest/autorest/mocks v0.4.1/go.mod h1:LTp+uSrOhSkaKrUy935gNZuuIPPVsHlr9DSOxSayd+k=
 github.com/Azure/go-autorest/tracing v0.6.0 h1:TYi4+3m5t6K48TGI9AUdb+IzbnSxvnvUMfuitfgcfuo=
 github.com/Azure/go-autorest/tracing v0.6.0/go.mod h1:+vhtPC754Xsa23ID7GlGsrdKBpUA79WCAKPPZVC2DeU=
+github.com/Azure/go-ntlmssp v0.0.0-20200615164410-66371956d46c h1:/IBSNwUN8+eKzUzbJPqhK839ygXJ82sde8x3ogr6R28=
+github.com/Azure/go-ntlmssp v0.0.0-20200615164410-66371956d46c/go.mod h1:chxPXzSsl7ZWRAuOIE23GDNzjWuZquvFlgA8xmpunjU=
 github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
 github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=


Test patch:
diff --git a/backend/webdav/webdav_test.go b/backend/webdav/webdav_test.go
index 3b4edcb2d..e23176afe 100644
--- a/backend/webdav/webdav_test.go
+++ b/backend/webdav/webdav_test.go
@@ -38,3 +38,14 @@ func TestIntegration3(t *testing.T) {
 		NilObject:  (*webdav.Object)(nil),
 	})
 }
+
+// TestIntegration runs integration tests against the remote
+func TestIntegration4(t *testing.T) {
+	if *fstest.RemoteName != "" {
+		t.Skip("skipping as -remote is set")
+	}
+	fstests.Run(t, &fstests.Opt{
+		RemoteName: "TestWebdavNTLM:",
+		NilObject:  (*webdav.Object)(nil),
+	})
+}