Repository: syncthing/syncthing (language: go)

Issue:
Scanning on Linux ext4 very slow in version 1.22.0
 - which version of Syncthing and what operating system you are using
 1.22.0

 - what happened,
 I have 3 servers syncing. 1 is MacOs M1 that has no issues. The other 2 are Ubuntu x86_64 on kernel 4.15.0-192 and 5.15.0-362206031516 and both of them are constantly thrashing the disk and never completing syncing. 
I have not enabled any of the new extended attribute settings in 1.22.0

 - what you expected to happen instead, and
 Expect it to scan as fast as 1.21.0 and/or the MacOs with 1.22.0

 - any steps to reproduce the problem.
upgrade from 1.21.0 to 1.22.0 on an Ubuntu os with and ext4 file system



Discussion / hints:
"constantly thrashing the disk and never completing syncing" doesn't sound like scanning, in Syncthing terminology.

Do you have logs of what it's doing? I'm not aware of any relevant changes in 1.22.0, apart from obviously the xattr stuff that you didn't enable.
Here are logs of pressing "rescan" on a small folder on both macos and ubuntu. I only enabled the scanner log. 
It looks like that on ubuntu it for some reason believes some, but not all, files have changed and it re-hashes them
On some of my larger directories it just continues re-hashing and never finishes 

Working -- macos
[macos.txt](https://github.com/syncthing/syncthing/files/9736954/macos.txt)

Not working -- ununtu
[ubuntu.txt](https://github.com/syncthing/syncthing/files/9736955/ubuntu.txt)

So, the one new thing that would cause that is we now record the inode change time and use it when comparing for changes. Unfortunately that value isn't printed in the debug output (adding that now...) but you can inspect it using the CLI for a known file;

```
syncthing cli debug file $folderID $filePath
```

If you could paste the output for a file that keeps getting rescanned so we can see which attributes change that would be helpful. If it's just the inode change time, that's ... odd, I don't know why it would keep changing, but it would explain what's going on.

Are you running anything else that might be relevant at the same time -- some dedup thing, or some backup program that sets attributes on the files to mark the processed, or something like that?

(FTR I'm not seeing this at all on Linux, but my files are on ZFS.)
```
{
  "availability": [
    "<redacted>",
    "<redacted>"
  ],
  "global": {
    "deleted": false,
    "ignored": false,
    "inodeChange": "1969-12-31T18:00:00-06:00",
    "invalid": false,
    "localFlags": 0,
    "modified": "2022-09-22T22:23:25.549513628-05:00",
    "modifiedBy": "XI3MV2B",
    "mustRescan": false,
    "name": "<redacted>",
    "noPermissions": true,
    "numBlocks": 1525,
    "platform": {
      "darwin": null,
      "freebsd": null,
      "linux": null,
      "netbsd": null,
      "unix": null,
      "windows": null
    },
    "sequence": 414,
    "size": 3196579256,
    "type": "FILE_INFO_TYPE_FILE",
    "version": [
      "XI3MV2B:1665164603"
    ]
  },
  "globalVersions": "{{Version:{[{XI3MV2B 1665164603}]}, Deleted:false, Devices:{ZMBI6TA, 7777777}, Invalid:{}}}",
  "local": {
    "deleted": false,
    "ignored": false,
    "inodeChange": "2022-10-07T12:45:11.156467433-05:00",
    "invalid": false,
    "localFlags": 0,
    "modified": "2022-09-22T22:23:25.549513628-05:00",
    "modifiedBy": "XI3MV2B",
    "mustRescan": false,
    "name": "<redacted>",
    "noPermissions": true,
    "numBlocks": 1525,
    "platform": {
      "darwin": null,
      "freebsd": null,
      "linux": null,
      "netbsd": null,
      "unix": null,
      "windows": null
    },
    "sequence": 1641,
    "size": 3196579256,
    "type": "FILE_INFO_TYPE_FILE",
    "version": [
      "XI3MV2B:1665164603"
    ]
  },
  "mtime": {
    "err": null,
    "value": {
      "real": "0001-01-01T00:00:00Z",
      "virtual": "0001-01-01T00:00:00Z"
    }
  }
}
```
The ones I see rescanning all have `1969-12-31T18:00:00-06:00` in one or the other inodeChange fields and a current date in the other. Ones that work either have `1969-12-31T18:00:00-06:00` in both or a current date in both. 
Yeah both of those situations are normal, the question is if the ones that have a current date in the `local` part have that date change from pass to pass.
I can confirm they are not changing between scans yet that file is scanned every time
I even did a json diff of the output and confirmed no values are changing
Also confirmed the directory the same way and it is also not changing
Could this be caused by an overflow? The date obviously corresponds to the start of the unix epoch, but if it ignores the timezone part, we'd get a negative number. If it then interprets that as an unsigned int, it will be larger than the current time.
No, there's no overflow, the comparison looks at the timestamp as nanoseconds-since-the-epoch, the rest is just formatting for the current time zone. I have no idea what would cause the behaviour in question, but the debug output from the scanner is kind of crappy -- both ridiculously verbose and lacking relevant details at the same time, so I'll improve that and then we can test this with a dev build.
Something a little interesting to add to this. On the smaller folder I've been testing on I ran `find . -type f -exec touch {} +` and after it scanned everything once it is no longer reproducible on that folder. 
I do still have more folders I can reproduce this on when you get a build that you want me to try.
So when scanned "locally" they're fine... if you touch one of those files from another host, so that the changed gets synced to the problematic one, does that trigger the problem for that file?
Now that I have touched all the files in that folder it is no longer reproducible. Touching on any of the remote hosts causes them to rescan but does not trigger the issue on the local problematic host.
It's worth noting that for the folders that this is still reproducible on, if I downgrade to 1.21.0 it is fine again. Upgrading to 1.22.0 triggers it again but not for the folder that I have touched all the files in.
I wonder what's special then, given that it wasn't that the new attribute we look at keeps changing. Why isn't everyone seeing this on their computers...
I can certainly just touch all the files and let it rescan them to solve for myself but it would be good to understand why this is happening. 
Another interesting test...
On the MacOs host running 1.22.0 I touched all the files in a problematic directory and let that sync to the problematic Ubuntu hosts while they were running 1.21.0. I then upgraded them to 1.22.0 and that directory was fine after that everywhere.

Fix patch:
diff --git a/lib/model/folder.go b/lib/model/folder.go
index 555de4ac69c..52638a2dfb6 100644
--- a/lib/model/folder.go
+++ b/lib/model/folder.go
@@ -601,14 +601,15 @@ func (b *scanBatch) Update(fi protocol.FileInfo, snap *db.Snapshot) bool {
 			b.Remove(fi.Name)
 			return true
 		}
-	case gf.IsEquivalentOptional(fi, protocol.FileInfoComparison{
-		ModTimeWindow:   b.f.modTimeWindow,
-		IgnorePerms:     b.f.IgnorePerms,
-		IgnoreBlocks:    true,
-		IgnoreFlags:     protocol.FlagLocalReceiveOnly,
-		IgnoreOwnership: !b.f.SyncOwnership,
-		IgnoreXattrs:    !b.f.SyncXattrs,
-	}):
+	case (b.f.Type == config.FolderTypeReceiveOnly || b.f.Type == config.FolderTypeReceiveEncrypted) &&
+		gf.IsEquivalentOptional(fi, protocol.FileInfoComparison{
+			ModTimeWindow:   b.f.modTimeWindow,
+			IgnorePerms:     b.f.IgnorePerms,
+			IgnoreBlocks:    true,
+			IgnoreFlags:     protocol.FlagLocalReceiveOnly,
+			IgnoreOwnership: !b.f.SyncOwnership && !b.f.SendOwnership,
+			IgnoreXattrs:    !b.f.SyncXattrs && !b.f.SendXattrs,
+		}):
 		// What we have locally is equivalent to the global file.
 		l.Debugf("%v scanning: Merging identical locally changed item with global", b.f, fi)
 		fi = gf
diff --git a/lib/protocol/bep_extensions.go b/lib/protocol/bep_extensions.go
index d2a0cad35c4..3d3eb437c36 100644
--- a/lib/protocol/bep_extensions.go
+++ b/lib/protocol/bep_extensions.go
@@ -246,7 +246,7 @@ func (f FileInfo) isEquivalent(other FileInfo, comp FileInfoComparison) bool {
 
 	// If we are recording inode change times and it changed, they are not
 	// equal.
-	if f.InodeChangeNs != 0 && other.InodeChangeNs != 0 && f.InodeChangeNs != other.InodeChangeNs {
+	if (f.InodeChangeNs != 0 && other.InodeChangeNs != 0) && f.InodeChangeNs != other.InodeChangeNs {
 		return false
 	}
 
@@ -258,21 +258,12 @@ func (f FileInfo) isEquivalent(other FileInfo, comp FileInfoComparison) bool {
 		return false
 	}
 
-	// OS data comparison is special: we consider a difference only if an
-	// entry for the same OS exists on both sides and they are different.
-	// Otherwise a file would become different as soon as it's synced from
-	// Windows to Linux, as Linux would add a new POSIX entry for the file.
 	if !comp.IgnoreOwnership && f.Platform != other.Platform {
-		if f.Platform.Unix != nil && other.Platform.Unix != nil {
-			if *f.Platform.Unix != *other.Platform.Unix {
-				return false
-			}
+		if !unixOwnershipEqual(f.Platform.Unix, other.Platform.Unix) {
+			return false
 		}
-		if f.Platform.Windows != nil && other.Platform.Windows != nil {
-			if f.Platform.Windows.OwnerName != other.Platform.Windows.OwnerName ||
-				f.Platform.Windows.OwnerIsGroup != other.Platform.Windows.OwnerIsGroup {
-				return false
-			}
+		if !windowsOwnershipEqual(f.Platform.Windows, other.Platform.Windows) {
+			return false
 		}
 	}
 	if !comp.IgnoreXattrs && f.Platform != other.Platform {
@@ -542,11 +533,15 @@ func (x *FileInfoType) UnmarshalJSON(data []byte) error {
 }
 
 func xattrsEqual(a, b *XattrData) bool {
-	if a == nil || b == nil {
-		// Having no data on either side means we have nothing to compare
-		// to, and we consider that equal.
+	aEmpty := a == nil || len(a.Xattrs) == 0
+	bEmpty := b == nil || len(b.Xattrs) == 0
+	if aEmpty && bEmpty {
 		return true
 	}
+	if aEmpty || bEmpty {
+		// Only one side is empty, so they can't be equal.
+		return false
+	}
 	if len(a.Xattrs) != len(b.Xattrs) {
 		return false
 	}
@@ -560,3 +555,23 @@ func xattrsEqual(a, b *XattrData) bool {
 	}
 	return true
 }
+
+func unixOwnershipEqual(a, b *UnixData) bool {
+	if a == nil && b == nil {
+		return true
+	}
+	if a == nil || b == nil {
+		return false
+	}
+	return a.UID == b.UID && a.GID == b.GID && a.OwnerName == b.OwnerName && a.GroupName == b.GroupName
+}
+
+func windowsOwnershipEqual(a, b *WindowsData) bool {
+	if a == nil && b == nil {
+		return true
+	}
+	if a == nil || b == nil {
+		return false
+	}
+	return a.OwnerName == b.OwnerName && a.OwnerIsGroup == b.OwnerIsGroup
+}


Test patch: