Repository: rclone/rclone (language: go)

Issue:
Breaking behavior change for sync/copy commands in `1.69.0`
#### The associated forum post URL from `https://forum.rclone.org`

None

#### What is the problem you are having with rclone?

I have encountered this problem specifically with `http` remote, but other could be affected too. Given `sync`/`copy` command with specific source filename `1.68.2` used to sync only the single specified file. In `1.69.0` however, it copies entire file's directory instead (or nothing if HTTP server doesn't allow file listing).

Using `git bisect` I have identified the breaking change to introduced by https://github.com/rclone/rclone/commit/9b4b3033da25ec41aabb35b858a765dd734a5f09 (PR: https://github.com/rclone/rclone/pull/7668).

#### What is your rclone version (output from `rclone version`)

```
rclone v1.69.0
- os/version: arch 25.0.0 (64 bit)
- os/kernel: 5.10.230-1-MANJARO (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.4
- go/linking: dynamic
- go/tags: none
```

Also tested with `1.69.0` and `master` (`2a1e28f5f5683eaba22d67a0e253c95b934f8c41`)

#### Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Linux Manjaro, 64 bit

#### Which cloud storage system are you using? (e.g. Google Drive)

HTTP

#### Setup

```sh
mkdir rclone-server
echo 111 > rclone-server/file1.txt
echo 222 > rclone-server/file2.md

# Configure the remote
rclone config create http_sync http url=http://localhost:9999

# Run a server (sever version is probably irrelevant)
rclone serve http ./rclone-server --addr 127.0.0.1:9999 -vv
```

#### The command you were trying to run

```
rclone sync http_sync:/file1.txt ./rclone-sync -vv
```

Weirdly enough, when using `--http-url` argument instead of configured HTTP remote, the behavior stays the same even for the newer versions.
```
rclone sync :http:/file1.txt ./rclone-sync --http-url http://localhost:9999 -vv
````

#### A log from the command with the `-vv` flag [version: `1.69.0`]

```
2025/03/05 23:55:18 DEBUG : rclone: Version "v1.69.0" starting with parameters ["./rclone" "sync" "http_sync:/file1.txt" "./rclone-sync" "-vv"]
2025/03/05 23:55:18 DEBUG : Creating backend with remote "http_sync:/file1.txt"
2025/03/05 23:55:18 DEBUG : Using config file from "/home/janosik/.config/rclone/rclone.conf"
2025/03/05 23:55:18 DEBUG : Assuming path is a file as HEAD response is success (200 OK)
2025/03/05 23:55:18 DEBUG : If path is a directory you must add a trailing '/'
2025/03/05 23:55:18 DEBUG : Root: http://localhost:9999/
2025/03/05 23:55:18 DEBUG : Creating backend with remote "./rclone-sync"
2025/03/05 23:55:18 DEBUG : fs cache: renaming cache item "./rclone-sync" to be canonical "/home/janosik/Downloads/rclone-v1.69.0-linux-amd64/rclone-sync"
2025/03/05 23:55:18 DEBUG : file1.txt: Need to transfer - File not found at Destination
2025/03/05 23:55:18 DEBUG : file2.md: Need to transfer - File not found at Destination
2025/03/05 23:55:18 DEBUG : Local file system at /home/janosik/Downloads/rclone-v1.69.0-linux-amd64/rclone-sync: Waiting for checks to finish
2025/03/05 23:55:18 DEBUG : Local file system at /home/janosik/Downloads/rclone-v1.69.0-linux-amd64/rclone-sync: Waiting for transfers to finish
2025/03/05 23:55:18 DEBUG : file1.txt.f170f1d5.partial: renamed to: file1.txt
2025/03/05 23:55:18 DEBUG : file2.md.d66d5580.partial: renamed to: file2.md
2025/03/05 23:55:18 INFO  : file1.txt: Copied (new)
2025/03/05 23:55:18 INFO  : file2.md: Copied (new)
2025/03/05 23:55:18 DEBUG : Waiting for deletions to finish
2025/03/05 23:55:18 INFO  : 
Transferred:   	        675 B / 675 B, 100%, 0 B/s, ETA -
Transferred:            2 / 2, 100%
Elapsed time:         0.0s
```

#### A log from the command with the `-vv` flag [version: `1.68.2`]

```
2025/03/05 23:53:34 DEBUG : rclone: Version "v1.68.2" starting with parameters ["./rclone" "sync" "http_sync:/file1.txt" "./rclone-sync" "-vv"]
2025/03/05 23:53:34 DEBUG : Creating backend with remote "http_sync:/file1.txt"
2025/03/05 23:53:34 DEBUG : Using config file from "/home/janosik/.config/rclone/rclone.conf"
2025/03/05 23:53:34 DEBUG : Assuming path is a file as HEAD response is success (200 OK)
2025/03/05 23:53:34 DEBUG : If path is a directory you must add a trailing '/'
2025/03/05 23:53:34 DEBUG : Root: http://localhost:9999/
2025/03/05 23:53:34 DEBUG : Creating backend with remote "./rclone-sync"
2025/03/05 23:53:34 DEBUG : fs cache: renaming cache item "./rclone-sync" to be canonical "/home/janosik/Downloads/rclone-v1.68.2-linux-amd64/rclone-sync"
2025/03/05 23:53:34 DEBUG : file1.txt: Need to transfer - File not found at Destination
2025/03/05 23:53:34 DEBUG : file1.txt.f170f1d5.partial: renamed to: file1.txt
2025/03/05 23:53:34 INFO  : file1.txt: Copied (new)
2025/03/05 23:53:34 INFO  : 
Transferred:   	          8 B / 8 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.0s
```

#### Used `bisect-rclone.sh` test script

```sh
rm -rf test-sync || true
rclone sync -vv http_sync:/file1.txt ./test-sync/
test -f test-sync/file1.txt
test ! -f test-sync/file2.md
```

<!--- Please keep the note below for others who read your bug report. -->

#### How to use GitHub

* Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue.
* Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
* Subscribe to receive notifications on status change and new comments.

Breaking behavior change for sync/copy commands in `1.69.0`
#### The associated forum post URL from `https://forum.rclone.org`

None

#### What is the problem you are having with rclone?

I have encountered this problem specifically with `http` remote, but other could be affected too. Given `sync`/`copy` command with specific source filename `1.68.2` used to sync only the single specified file. In `1.69.0` however, it copies entire file's directory instead (or nothing if HTTP server doesn't allow file listing).

Using `git bisect` I have identified the breaking change to introduced by https://github.com/rclone/rclone/commit/9b4b3033da25ec41aabb35b858a765dd734a5f09 (PR: https://github.com/rclone/rclone/pull/7668).

#### What is your rclone version (output from `rclone version`)

```
rclone v1.69.0
- os/version: arch 25.0.0 (64 bit)
- os/kernel: 5.10.230-1-MANJARO (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.4
- go/linking: dynamic
- go/tags: none
```

Also tested with `1.69.0` and `master` (`2a1e28f5f5683eaba22d67a0e253c95b934f8c41`)

#### Which OS you are using and how many bits (e.g. Windows 7, 64 bit)

Linux Manjaro, 64 bit

#### Which cloud storage system are you using? (e.g. Google Drive)

HTTP

#### Setup

```sh
mkdir rclone-server
echo 111 > rclone-server/file1.txt
echo 222 > rclone-server/file2.md

# Configure the remote
rclone config create http_sync http url=http://localhost:9999

# Run a server (sever version is probably irrelevant)
rclone serve http ./rclone-server --addr 127.0.0.1:9999 -vv
```

#### The command you were trying to run

```
rclone sync http_sync:/file1.txt ./rclone-sync -vv
```

Weirdly enough, when using `--http-url` argument instead of configured HTTP remote, the behavior stays the same even for the newer versions.
```
rclone sync :http:/file1.txt ./rclone-sync --http-url http://localhost:9999 -vv
````

#### A log from the command with the `-vv` flag [version: `1.69.0`]

```
2025/03/05 23:55:18 DEBUG : rclone: Version "v1.69.0" starting with parameters ["./rclone" "sync" "http_sync:/file1.txt" "./rclone-sync" "-vv"]
2025/03/05 23:55:18 DEBUG : Creating backend with remote "http_sync:/file1.txt"
2025/03/05 23:55:18 DEBUG : Using config file from "/home/janosik/.config/rclone/rclone.conf"
2025/03/05 23:55:18 DEBUG : Assuming path is a file as HEAD response is success (200 OK)
2025/03/05 23:55:18 DEBUG : If path is a directory you must add a trailing '/'
2025/03/05 23:55:18 DEBUG : Root: http://localhost:9999/
2025/03/05 23:55:18 DEBUG : Creating backend with remote "./rclone-sync"
2025/03/05 23:55:18 DEBUG : fs cache: renaming cache item "./rclone-sync" to be canonical "/home/janosik/Downloads/rclone-v1.69.0-linux-amd64/rclone-sync"
2025/03/05 23:55:18 DEBUG : file1.txt: Need to transfer - File not found at Destination
2025/03/05 23:55:18 DEBUG : file2.md: Need to transfer - File not found at Destination
2025/03/05 23:55:18 DEBUG : Local file system at /home/janosik/Downloads/rclone-v1.69.0-linux-amd64/rclone-sync: Waiting for checks to finish
2025/03/05 23:55:18 DEBUG : Local file system at /home/janosik/Downloads/rclone-v1.69.0-linux-amd64/rclone-sync: Waiting for transfers to finish
2025/03/05 23:55:18 DEBUG : file1.txt.f170f1d5.partial: renamed to: file1.txt
2025/03/05 23:55:18 DEBUG : file2.md.d66d5580.partial: renamed to: file2.md
2025/03/05 23:55:18 INFO  : file1.txt: Copied (new)
2025/03/05 23:55:18 INFO  : file2.md: Copied (new)
2025/03/05 23:55:18 DEBUG : Waiting for deletions to finish
2025/03/05 23:55:18 INFO  : 
Transferred:   	        675 B / 675 B, 100%, 0 B/s, ETA -
Transferred:            2 / 2, 100%
Elapsed time:         0.0s
```

#### A log from the command with the `-vv` flag [version: `1.68.2`]

```
2025/03/05 23:53:34 DEBUG : rclone: Version "v1.68.2" starting with parameters ["./rclone" "sync" "http_sync:/file1.txt" "./rclone-sync" "-vv"]
2025/03/05 23:53:34 DEBUG : Creating backend with remote "http_sync:/file1.txt"
2025/03/05 23:53:34 DEBUG : Using config file from "/home/janosik/.config/rclone/rclone.conf"
2025/03/05 23:53:34 DEBUG : Assuming path is a file as HEAD response is success (200 OK)
2025/03/05 23:53:34 DEBUG : If path is a directory you must add a trailing '/'
2025/03/05 23:53:34 DEBUG : Root: http://localhost:9999/
2025/03/05 23:53:34 DEBUG : Creating backend with remote "./rclone-sync"
2025/03/05 23:53:34 DEBUG : fs cache: renaming cache item "./rclone-sync" to be canonical "/home/janosik/Downloads/rclone-v1.68.2-linux-amd64/rclone-sync"
2025/03/05 23:53:34 DEBUG : file1.txt: Need to transfer - File not found at Destination
2025/03/05 23:53:34 DEBUG : file1.txt.f170f1d5.partial: renamed to: file1.txt
2025/03/05 23:53:34 INFO  : file1.txt: Copied (new)
2025/03/05 23:53:34 INFO  : 
Transferred:   	          8 B / 8 B, 100%, 0 B/s, ETA -
Transferred:            1 / 1, 100%
Elapsed time:         0.0s
```

#### Used `bisect-rclone.sh` test script

```sh
rm -rf test-sync || true
rclone sync -vv http_sync:/file1.txt ./test-sync/
test -f test-sync/file1.txt
test ! -f test-sync/file2.md
```

<!--- Please keep the note below for others who read your bug report. -->

#### How to use GitHub

* Please use the 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to show that you are affected by the same issue.
* Please don't comment if you have no relevant information to add. It's just extra noise for everyone subscribed to this issue.
* Subscribe to receive notifications on status change and new comments.



Discussion / hints:
Can you try without the leading / in front of :http:/file1.txt ?

That commit is by @nielash - thoughts?
Leading `/` in file path, nor trailing slash in the remote's `url` has no effect.
So far, I have identified the reason is that in `NewFsFile`, the call to `cache.Get(ctx, remote)` now provides incorrect result. This makes the `NewFsFile` return empty string as a file name, which causes the difference between copy-file and sync-directory behaviors (https://github.com/rclone/rclone/blob/master/cmd/sync/sync.go#L242).

Debug statement I've added
```
	f, err := cache.Get(ctx, remote)
	fs.Debugf(nil, "NewFsFile - cache.Get: remote=%q f=%v err=%v" , remote, f, err)
```

#### Debug message without any options

```
DEBUG: NewFsFile - cache.Get: remote="local9:file1.txt" f=http://localhost:9999/ err=<nil>
```

#### Debug message  with `--http-no-head` option (provides the same results as `--http-url ...`)

```
DEBUG: NewFsFile - cache.Get: remote="local9:file1.txt" f=http://localhost:9999/ err=is a file not a directory
```

That just confirms that it's somehow related to cache. Don't understand how yet.
What bugs me is that `f=http://localhost:9999/` and not `f=http://localhost:9999/file1.txt` (but I'm not sure whether it's even related).

---

Update: Okay, found out what the issue is and why. But I have no idea how the cache is supposed to work, so I'm not sure what the correct fix should be.

What causes different behavior between no command line options and having some command line overrides (e.g. `--http-no-head`) is the added suffix when override is detected.

> DEBUG : http_sync: detected overridden config - adding "{NBTUO}" suffix to name
> DEBUG : fs cache: renaming child cache item "http_sync:file1.txt" to be canonical for parent "http_sync{NBTUO}:file1.txt"

That affects the `canonicalName != canonicalFsString` condition: https://github.com/rclone/rclone/blob/2a1e28f5f5683eaba22d67a0e253c95b934f8c41/fs/cache/cache.go#L117

Which then calls `addChild`: https://github.com/rclone/rclone/blob/2a1e28f5f5683eaba22d67a0e253c95b934f8c41/fs/cache/cache.go#L135

And finally the difference shows as result of `getError(fsString, err)`:
https://github.com/rclone/rclone/blob/2a1e28f5f5683eaba22d67a0e253c95b934f8c41/fs/cache/cache.go#L139

That's because the `getError` treats `ErrorIsFile` somehow specially and if the file is not found by `isChild` (it won't because `addChild` was never called on it), the error is dropped.
https://github.com/rclone/rclone/blob/2a1e28f5f5683eaba22d67a0e253c95b934f8c41/fs/cache/cache.go#L85-L93

---

After looking at the code even more, I think following patch should do it.

```diff
diff --git a/fs/cache/cache.go b/fs/cache/cache.go
index 0ab2492ce..1764d31d1 100644
--- a/fs/cache/cache.go
+++ b/fs/cache/cache.go
@@ -83,13 +83,10 @@ func isChild(child string) bool {
 
 // ensures that we return fs.ErrorIsFile when necessary
 func getError(fsString string, err error) error {
-	if err != nil && err != fs.ErrorIsFile {
-		return err
-	}
-	if isChild(fsString) {
+	if err == nil && isChild(fsString) {
 		return fs.ErrorIsFile
 	}
-	return nil
+	return err
 }
 
 // GetFn gets an fs.Fs named fsString either from the cache or creates
```

`getError` is only called in scope of `GetFn` and at that point, all errors except `ErrorIsFile` are already handled and returned early by https://github.com/rclone/rclone/blob/2a1e28f5f5683eaba22d67a0e253c95b934f8c41/fs/cache/cache.go#L108-L113

But at that point, it would be probably better to just get rid of `getError` and inline the error handling logic.
Can you try without the leading / in front of :http:/file1.txt ?

That commit is by @nielash - thoughts?
Leading `/` in file path, nor trailing slash in the remote's `url` has no effect.
So far, I have identified the reason is that in `NewFsFile`, the call to `cache.Get(ctx, remote)` now provides incorrect result. This makes the `NewFsFile` return empty string as a file name, which causes the difference between copy-file and sync-directory behaviors (https://github.com/rclone/rclone/blob/master/cmd/sync/sync.go#L242).

Debug statement I've added
```
	f, err := cache.Get(ctx, remote)
	fs.Debugf(nil, "NewFsFile - cache.Get: remote=%q f=%v err=%v" , remote, f, err)
```

#### Debug message without any options

```
DEBUG: NewFsFile - cache.Get: remote="local9:file1.txt" f=http://localhost:9999/ err=<nil>
```

#### Debug message  with `--http-no-head` option (provides the same results as `--http-url ...`)

```
DEBUG: NewFsFile - cache.Get: remote="local9:file1.txt" f=http://localhost:9999/ err=is a file not a directory
```

That just confirms that it's somehow related to cache. Don't understand how yet.
What bugs me is that `f=http://localhost:9999/` and not `f=http://localhost:9999/file1.txt` (but I'm not sure whether it's even related).

---

Update: Okay, found out what the issue is and why. But I have no idea how the cache is supposed to work, so I'm not sure what the correct fix should be.

What causes different behavior between no command line options and having some command line overrides (e.g. `--http-no-head`) is the added suffix when override is detected.

> DEBUG : http_sync: detected overridden config - adding "{NBTUO}" suffix to name
> DEBUG : fs cache: renaming child cache item "http_sync:file1.txt" to be canonical for parent "http_sync{NBTUO}:file1.txt"

That affects the `canonicalName != canonicalFsString` condition: https://github.com/rclone/rclone/blob/2a1e28f5f5683eaba22d67a0e253c95b934f8c41/fs/cache/cache.go#L117

Which then calls `addChild`: https://github.com/rclone/rclone/blob/2a1e28f5f5683eaba22d67a0e253c95b934f8c41/fs/cache/cache.go#L135

And finally the difference shows as result of `getError(fsString, err)`:
https://github.com/rclone/rclone/blob/2a1e28f5f5683eaba22d67a0e253c95b934f8c41/fs/cache/cache.go#L139

That's because the `getError` treats `ErrorIsFile` somehow specially and if the file is not found by `isChild` (it won't because `addChild` was never called on it), the error is dropped.
https://github.com/rclone/rclone/blob/2a1e28f5f5683eaba22d67a0e253c95b934f8c41/fs/cache/cache.go#L85-L93

---

After looking at the code even more, I think following patch should do it.

```diff
diff --git a/fs/cache/cache.go b/fs/cache/cache.go
index 0ab2492ce..1764d31d1 100644
--- a/fs/cache/cache.go
+++ b/fs/cache/cache.go
@@ -83,13 +83,10 @@ func isChild(child string) bool {
 
 // ensures that we return fs.ErrorIsFile when necessary
 func getError(fsString string, err error) error {
-	if err != nil && err != fs.ErrorIsFile {
-		return err
-	}
-	if isChild(fsString) {
+	if err == nil && isChild(fsString) {
 		return fs.ErrorIsFile
 	}
-	return nil
+	return err
 }
 
 // GetFn gets an fs.Fs named fsString either from the cache or creates
```

`getError` is only called in scope of `GetFn` and at that point, all errors except `ErrorIsFile` are already handled and returned early by https://github.com/rclone/rclone/blob/2a1e28f5f5683eaba22d67a0e253c95b934f8c41/fs/cache/cache.go#L108-L113

But at that point, it would be probably better to just get rid of `getError` and inline the error handling logic.

Fix patch:
diff --git a/backend/http/http.go b/backend/http/http.go
index 4b64ec3042dfc..d4b426a7fabe0 100644
--- a/backend/http/http.go
+++ b/backend/http/http.go
@@ -180,7 +180,6 @@ func getFsEndpoint(ctx context.Context, client *http.Client, url string, opt *Op
 	}
 	addHeaders(req, opt)
 	res, err := noRedir.Do(req)
-
 	if err != nil {
 		fs.Debugf(nil, "Assuming path is a file as HEAD request could not be sent: %v", err)
 		return createFileResult()
@@ -249,6 +248,14 @@ func (f *Fs) httpConnection(ctx context.Context, opt *Options) (isFile bool, err
 	f.httpClient = client
 	f.endpoint = u
 	f.endpointURL = u.String()
+
+	if isFile {
+		// Correct root if definitely pointing to a file
+		f.root = path.Dir(f.root)
+		if f.root == "." || f.root == "/" {
+			f.root = ""
+		}
+	}
 	return isFile, nil
 }
 


Test patch: