Repository: syncthing/syncthing (language: go)

Issue:
panic: counter cannot decrease in value
I am running syncthing on two devices. Both run syncthing v1.25.0-rc.1 "Gold Grasshopper" (go1.21.1 linux-amd64) debian@github.syncthing.net 2023-09-12 12:48:15 UTC [noupgrade]

Machine A is a Ubuntu 23.04 virtual machine on a Windows host with NAT network adapter, machine B runs Ubuntu 20.04.6 native.

Both machines were syncing reliably until a couple of weeks ago. Now, syncthing crashes on machine A after running successfully for few seconds. I have deleted syncthing and its config in ~/.config/syncthing/ from machine A and reinstalled via `sudo apt install syncthing`. The following is the output when running syncthing from the terminal and starting a new session.

The machine crashes briefly after I attempt to share a (new but non-empty) folder from machine A with machine B. The command line output is the following

```
[start] 2023/09/19 16:59:57 INFO: syncthing v1.25.0-rc.1 "Gold Grasshopper" (go1.21.1 linux-amd64) debian@github.syncthing.net 2023-09-12 12:48:15 UTC [noupgrade]
[start] 2023/09/19 16:59:57 INFO: Generating ECDSA key and certificate for syncthing...
[start] 2023/09/19 16:59:57 INFO: Default folder created and/or linked to new config
[start] 2023/09/19 16:59:57 INFO: Default config saved. Edit /home/harold/.config/syncthing/config.xml to taste (with Syncthing stopped) or use the GUI
[start] 2023/09/19 16:59:57 INFO: Archiving a copy of old config file format at: /home/harold/.config/syncthing/config.xml.v0
[VVQ5H] 2023/09/19 16:59:57 INFO: My ID: VVQ5HPR-Z5OBEIC-RTADZXP-CH7L6FU-3FIXUDT-PH7GXVJ-OZX326J-MJNJZQK
[VVQ5H] 2023/09/19 16:59:58 INFO: Single thread SHA256 performance is 496 MB/s using crypto/sha256 (491 MB/s using minio/sha256-simd).
[VVQ5H] 2023/09/19 16:59:59 INFO: Hashing performance is 423.29 MB/s
[VVQ5H] 2023/09/19 16:59:59 INFO: Running database migration 1...
[VVQ5H] 2023/09/19 16:59:59 INFO: Running database migration 2...
[VVQ5H] 2023/09/19 16:59:59 INFO: Running database migration 3...
[VVQ5H] 2023/09/19 16:59:59 INFO: Running database migration 5...
[VVQ5H] 2023/09/19 16:59:59 INFO: Running database migration 6...
[VVQ5H] 2023/09/19 16:59:59 INFO: Running database migration 7...
[VVQ5H] 2023/09/19 16:59:59 INFO: Running database migration 9...
[VVQ5H] 2023/09/19 16:59:59 INFO: Running database migration 10...
[VVQ5H] 2023/09/19 16:59:59 INFO: Running database migration 11...
[VVQ5H] 2023/09/19 16:59:59 INFO: Running database migration 13...
[VVQ5H] 2023/09/19 16:59:59 INFO: Running database migration 14...
[VVQ5H] 2023/09/19 16:59:59 INFO: Running database migration 16...
[VVQ5H] 2023/09/19 16:59:59 INFO: Running database migration 17...
[VVQ5H] 2023/09/19 16:59:59 INFO: Running database migration 19...
[VVQ5H] 2023/09/19 16:59:59 INFO: Compacting database after migration...
[VVQ5H] 2023/09/19 16:59:59 INFO: Starting deadlock detector with 20m0s timeout
[VVQ5H] 2023/09/19 16:59:59 INFO: Overall send rate is unlimited, receive rate is unlimited
[VVQ5H] 2023/09/19 16:59:59 INFO: No stored folder metadata for "default"; recalculating
[VVQ5H] 2023/09/19 16:59:59 INFO: Anonymous usage reporting is always enabled for candidate releases.
[VVQ5H] 2023/09/19 16:59:59 INFO: Using discovery mechanism: global discovery server https://discovery.syncthing.net/v2/?noannounce&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VVQ5H] 2023/09/19 16:59:59 INFO: Using discovery mechanism: global discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VVQ5H] 2023/09/19 16:59:59 INFO: Using discovery mechanism: global discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VVQ5H] 2023/09/19 16:59:59 INFO: Using discovery mechanism: IPv4 local broadcast discovery on port 21027
[VVQ5H] 2023/09/19 16:59:59 INFO: Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
[VVQ5H] 2023/09/19 16:59:59 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
[VVQ5H] 2023/09/19 16:59:59 INFO: Ready to synchronize "Default Folder" (default) (sendreceive)
[VVQ5H] 2023/09/19 16:59:59 INFO: TCP listener ([::]:22000) starting
[VVQ5H] 2023/09/19 16:59:59 INFO: QUIC listener ([::]:22000) starting
[VVQ5H] 2023/09/19 16:59:59 INFO: Completed initial scan of sendreceive folder "Default Folder" (default)
[VVQ5H] 2023/09/19 16:59:59 INFO: Loading HTTPS certificate: open /home/harold/.config/syncthing/https-cert.pem: no such file or directory
[VVQ5H] 2023/09/19 16:59:59 INFO: Creating new HTTPS certificate
[VVQ5H] 2023/09/19 16:59:59 INFO: GUI and API listening on 127.0.0.1:8384
[VVQ5H] 2023/09/19 16:59:59 INFO: Access the GUI via the following URL: http://127.0.0.1:8384/
[VVQ5H] 2023/09/19 16:59:59 INFO: My name is "nanolinux"
[VVQ5H] 2023/09/19 17:00:10 INFO: Detected 1 NAT service
[VVQ5H] 2023/09/19 17:00:19 INFO: quic://0.0.0.0:22000 detected NAT type: Port restricted NAT
[VVQ5H] 2023/09/19 17:00:19 INFO: quic://0.0.0.0:22000 resolved external address quic://31.22.1.26:59065 (via stun.syncthing.net:3478)
[VVQ5H] 2023/09/19 17:00:46 INFO: Joined relay relay://31.208.128.28:22067
[VVQ5H] 2023/09/19 17:01:32 INFO: Connection from NUMHC6S-IKZP7I4-G3FWEEE-2UMAS74-PJ5XZKT-PCDGGA3-SJYHZGN-GD575AA at 31.208.128.28:22067 (relay-server) rejected: unknown device
[VVQ5H] 2023/09/19 17:01:43 INFO: Adding folder "nanovery" (juhkv-dhmlj)
[VVQ5H] 2023/09/19 17:01:43 INFO: No stored folder metadata for "juhkv-dhmlj"; recalculating
[VVQ5H] 2023/09/19 17:01:43 INFO: Ready to synchronize "nanovery" (juhkv-dhmlj) (sendreceive)
[VVQ5H] 2023/09/19 17:01:51 INFO: Established secure connection to NUMHC6S at 10.0.2.15:22000-10.111.19.174:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-5S6B0SGIBHS204GUGAS9AOCDRE
[VVQ5H] 2023/09/19 17:01:51 INFO: Device NUMHC6S client is "syncthing v1.25.0-rc.1" named "harolds" at 10.0.2.15:22000-10.111.19.174:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-5S6B0SGIBHS204GUGAS9AOCDRE
panic: counter cannot decrease in value
[monitor] 2023/09/19 17:02:15 WARNING: Panic detected, writing to "/home/harold/.config/syncthing/panic-20230919-170215.log"
[monitor] 2023/09/19 17:02:15 WARNING: Please check for existing issues with similar panic message at https://github.com/syncthing/syncthing/issues/
[monitor] 2023/09/19 17:02:15 WARNING: If no issue with similar panic message exists, please create a new issue with the panic log attached
[monitor] 2023/09/19 17:02:15 INFO: Reporting crash found in panic-20230919-170215.log (report ID b0e664c4) ...
[monitor] 2023/09/19 17:02:16 INFO: Syncthing exited: exit status 2
[start] 2023/09/19 17:02:17 INFO: syncthing v1.25.0-rc.1 "Gold Grasshopper" (go1.21.1 linux-amd64) debian@github.syncthing.net 2023-09-12 12:48:15 UTC [noupgrade]
[VVQ5H] 2023/09/19 17:02:18 INFO: My ID: VVQ5HPR-Z5OBEIC-RTADZXP-CH7L6FU-3FIXUDT-PH7GXVJ-OZX326J-MJNJZQK
[VVQ5H] 2023/09/19 17:02:18 INFO: Single thread SHA256 performance is 466 MB/s using minio/sha256-simd (462 MB/s using crypto/sha256).
[VVQ5H] 2023/09/19 17:02:19 INFO: Hashing performance is 412.74 MB/s
[VVQ5H] 2023/09/19 17:02:19 INFO: Starting deadlock detector with 20m0s timeout
[VVQ5H] 2023/09/19 17:02:19 INFO: Overall send rate is unlimited, receive rate is unlimited
[VVQ5H] 2023/09/19 17:02:19 INFO: Anonymous usage reporting is always enabled for candidate releases.
[VVQ5H] 2023/09/19 17:02:19 INFO: Ready to synchronize "Default Folder" (default) (sendreceive)
[VVQ5H] 2023/09/19 17:02:19 INFO: Using discovery mechanism: global discovery server https://discovery.syncthing.net/v2/?noannounce&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VVQ5H] 2023/09/19 17:02:19 INFO: Using discovery mechanism: global discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VVQ5H] 2023/09/19 17:02:19 INFO: Using discovery mechanism: global discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VVQ5H] 2023/09/19 17:02:19 INFO: Using discovery mechanism: IPv4 local broadcast discovery on port 21027
[VVQ5H] 2023/09/19 17:02:19 INFO: Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
[VVQ5H] 2023/09/19 17:02:19 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
[VVQ5H] 2023/09/19 17:02:19 INFO: Ready to synchronize "nanovery" (juhkv-dhmlj) (sendreceive)
[VVQ5H] 2023/09/19 17:02:19 INFO: TCP listener ([::]:22000) starting
[VVQ5H] 2023/09/19 17:02:19 INFO: Completed initial scan of sendreceive folder "Default Folder" (default)
[VVQ5H] 2023/09/19 17:02:19 INFO: QUIC listener ([::]:22000) starting
[VVQ5H] 2023/09/19 17:02:19 INFO: GUI and API listening on 127.0.0.1:8384
[VVQ5H] 2023/09/19 17:02:19 INFO: Access the GUI via the following URL: http://127.0.0.1:8384/
[VVQ5H] 2023/09/19 17:02:19 INFO: My name is "nanolinux"
[VVQ5H] 2023/09/19 17:02:19 INFO: Device NUMHC6S-IKZP7I4-G3FWEEE-2UMAS74-PJ5XZKT-PCDGGA3-SJYHZGN-GD575AA is "harolds" at [dynamic]
[VVQ5H] 2023/09/19 17:02:20 INFO: Established secure connection to NUMHC6S at 10.0.2.15:22000-10.111.19.174:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-5S6B0VR1F1DLIIUB17DHBG444E
[VVQ5H] 2023/09/19 17:02:20 INFO: Device NUMHC6S client is "syncthing v1.25.0-rc.1" named "harolds" at 10.0.2.15:22000-10.111.19.174:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-5S6B0VR1F1DLIIUB17DHBG444E
panic: counter cannot decrease in value
[monitor] 2023/09/19 17:02:28 WARNING: Panic detected, writing to "/home/harold/.config/syncthing/panic-20230919-170228.log"
[monitor] 2023/09/19 17:02:28 WARNING: Please check for existing issues with similar panic message at https://github.com/syncthing/syncthing/issues/
[monitor] 2023/09/19 17:02:28 WARNING: If no issue with similar panic message exists, please create a new issue with the panic log attached
[monitor] 2023/09/19 17:02:28 INFO: Reporting crash found in panic-20230919-170228.log (report ID 045e2f34) ...
[monitor] 2023/09/19 17:02:28 INFO: Syncthing exited: exit status 2
[start] 2023/09/19 17:02:29 INFO: syncthing v1.25.0-rc.1 "Gold Grasshopper" (go1.21.1 linux-amd64) debian@github.syncthing.net 2023-09-12 12:48:15 UTC [noupgrade]
[VVQ5H] 2023/09/19 17:02:29 INFO: My ID: VVQ5HPR-Z5OBEIC-RTADZXP-CH7L6FU-3FIXUDT-PH7GXVJ-OZX326J-MJNJZQK
[VVQ5H] 2023/09/19 17:02:30 INFO: Single thread SHA256 performance is 472 MB/s using crypto/sha256 (460 MB/s using minio/sha256-simd).
[VVQ5H] 2023/09/19 17:02:31 INFO: Hashing performance is 414.01 MB/s
[VVQ5H] 2023/09/19 17:02:31 INFO: Starting deadlock detector with 20m0s timeout
[VVQ5H] 2023/09/19 17:02:31 INFO: Overall send rate is unlimited, receive rate is unlimited
[VVQ5H] 2023/09/19 17:02:31 INFO: Anonymous usage reporting is always enabled for candidate releases.
[VVQ5H] 2023/09/19 17:02:31 INFO: Using discovery mechanism: global discovery server https://discovery.syncthing.net/v2/?noannounce&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VVQ5H] 2023/09/19 17:02:31 INFO: Using discovery mechanism: global discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VVQ5H] 2023/09/19 17:02:31 INFO: Using discovery mechanism: global discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VVQ5H] 2023/09/19 17:02:31 INFO: Using discovery mechanism: IPv4 local broadcast discovery on port 21027
[VVQ5H] 2023/09/19 17:02:31 INFO: Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
[VVQ5H] 2023/09/19 17:02:31 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
[VVQ5H] 2023/09/19 17:02:31 INFO: QUIC listener ([::]:22000) starting
[VVQ5H] 2023/09/19 17:02:31 INFO: Ready to synchronize "Default Folder" (default) (sendreceive)
[VVQ5H] 2023/09/19 17:02:31 INFO: GUI and API listening on 127.0.0.1:8384
[VVQ5H] 2023/09/19 17:02:31 INFO: Access the GUI via the following URL: http://127.0.0.1:8384/
[VVQ5H] 2023/09/19 17:02:31 INFO: My name is "nanolinux"
[VVQ5H] 2023/09/19 17:02:31 INFO: Device NUMHC6S-IKZP7I4-G3FWEEE-2UMAS74-PJ5XZKT-PCDGGA3-SJYHZGN-GD575AA is "harolds" at [dynamic]
[VVQ5H] 2023/09/19 17:02:31 INFO: Ready to synchronize "nanovery" (juhkv-dhmlj) (sendreceive)
[VVQ5H] 2023/09/19 17:02:31 INFO: Completed initial scan of sendreceive folder "Default Folder" (default)
[VVQ5H] 2023/09/19 17:02:31 INFO: TCP listener ([::]:22000) starting
[VVQ5H] 2023/09/19 17:02:31 INFO: Established secure connection to NUMHC6S at 10.0.2.15:22000-10.111.19.174:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-5S6B115EFK2HQDSCLDHS3T2DLU
[VVQ5H] 2023/09/19 17:02:31 INFO: Device NUMHC6S client is "syncthing v1.25.0-rc.1" named "harolds" at 10.0.2.15:22000-10.111.19.174:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-5S6B115EFK2HQDSCLDHS3T2DLU
[VVQ5H] 2023/09/19 17:02:41 INFO: Detected 1 NAT service
panic: counter cannot decrease in value
[monitor] 2023/09/19 17:02:41 WARNING: Panic detected, writing to "/home/harold/.config/syncthing/panic-20230919-170241.log"
[monitor] 2023/09/19 17:02:41 WARNING: Please check for existing issues with similar panic message at https://github.com/syncthing/syncthing/issues/
[monitor] 2023/09/19 17:02:41 WARNING: If no issue with similar panic message exists, please create a new issue with the panic log attached
[monitor] 2023/09/19 17:02:41 INFO: Reporting crash found in panic-20230919-170241.log (report ID 0022f4f0) ...
[monitor] 2023/09/19 17:02:42 INFO: Syncthing exited: exit status 2
[start] 2023/09/19 17:02:43 INFO: syncthing v1.25.0-rc.1 "Gold Grasshopper" (go1.21.1 linux-amd64) debian@github.syncthing.net 2023-09-12 12:48:15 UTC [noupgrade]
[VVQ5H] 2023/09/19 17:02:43 INFO: My ID: VVQ5HPR-Z5OBEIC-RTADZXP-CH7L6FU-3FIXUDT-PH7GXVJ-OZX326J-MJNJZQK
[VVQ5H] 2023/09/19 17:02:44 INFO: Single thread SHA256 performance is 459 MB/s using crypto/sha256 (449 MB/s using minio/sha256-simd).
[VVQ5H] 2023/09/19 17:02:44 INFO: Hashing performance is 405.62 MB/s
[VVQ5H] 2023/09/19 17:02:44 INFO: Starting deadlock detector with 20m0s timeout
[VVQ5H] 2023/09/19 17:02:44 INFO: Overall send rate is unlimited, receive rate is unlimited
[VVQ5H] 2023/09/19 17:02:44 INFO: Anonymous usage reporting is always enabled for candidate releases.
[VVQ5H] 2023/09/19 17:02:44 INFO: Ready to synchronize "Default Folder" (default) (sendreceive)
[VVQ5H] 2023/09/19 17:02:44 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
[VVQ5H] 2023/09/19 17:02:44 INFO: Ready to synchronize "nanovery" (juhkv-dhmlj) (sendreceive)
[VVQ5H] 2023/09/19 17:02:44 INFO: Using discovery mechanism: global discovery server https://discovery.syncthing.net/v2/?noannounce&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VVQ5H] 2023/09/19 17:02:44 INFO: Using discovery mechanism: global discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VVQ5H] 2023/09/19 17:02:44 INFO: Using discovery mechanism: global discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VVQ5H] 2023/09/19 17:02:44 INFO: TCP listener ([::]:22000) starting
[VVQ5H] 2023/09/19 17:02:44 INFO: QUIC listener ([::]:22000) starting
[VVQ5H] 2023/09/19 17:02:44 INFO: Using discovery mechanism: IPv4 local broadcast discovery on port 21027
[VVQ5H] 2023/09/19 17:02:44 INFO: Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
[VVQ5H] 2023/09/19 17:02:44 INFO: GUI and API listening on 127.0.0.1:8384
[VVQ5H] 2023/09/19 17:02:44 INFO: Access the GUI via the following URL: http://127.0.0.1:8384/
[VVQ5H] 2023/09/19 17:02:44 INFO: My name is "nanolinux"
[VVQ5H] 2023/09/19 17:02:44 INFO: Completed initial scan of sendreceive folder "Default Folder" (default)
[VVQ5H] 2023/09/19 17:02:44 INFO: Device NUMHC6S-IKZP7I4-G3FWEEE-2UMAS74-PJ5XZKT-PCDGGA3-SJYHZGN-GD575AA is "harolds" at [dynamic]
[VVQ5H] 2023/09/19 17:02:45 INFO: Established secure connection to NUMHC6S at 10.0.2.15:22000-10.111.19.174:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-5S6B12OOELUGUA3JVS4AK4OUL2
[VVQ5H] 2023/09/19 17:02:45 INFO: Device NUMHC6S client is "syncthing v1.25.0-rc.1" named "harolds" at 10.0.2.15:22000-10.111.19.174:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-5S6B12OOELUGUA3JVS4AK4OUL2
panic: counter cannot decrease in value
[monitor] 2023/09/19 17:02:48 WARNING: Panic detected, writing to "/home/harold/.config/syncthing/panic-20230919-170248.log"
[monitor] 2023/09/19 17:02:48 WARNING: Please check for existing issues with similar panic message at https://github.com/syncthing/syncthing/issues/
[monitor] 2023/09/19 17:02:48 WARNING: If no issue with similar panic message exists, please create a new issue with the panic log attached
[monitor] 2023/09/19 17:02:48 INFO: Reporting crash found in panic-20230919-170248.log (report ID cabebdc9) ...
[monitor] 2023/09/19 17:02:49 INFO: Syncthing exited: exit status 2
[start] 2023/09/19 17:02:50 INFO: syncthing v1.25.0-rc.1 "Gold Grasshopper" (go1.21.1 linux-amd64) debian@github.syncthing.net 2023-09-12 12:48:15 UTC [noupgrade]
[VVQ5H] 2023/09/19 17:02:50 INFO: My ID: VVQ5HPR-Z5OBEIC-RTADZXP-CH7L6FU-3FIXUDT-PH7GXVJ-OZX326J-MJNJZQK
[VVQ5H] 2023/09/19 17:02:51 INFO: Single thread SHA256 performance is 460 MB/s using crypto/sha256 (460 MB/s using minio/sha256-simd).
[VVQ5H] 2023/09/19 17:02:51 INFO: Hashing performance is 412.04 MB/s
[VVQ5H] 2023/09/19 17:02:51 INFO: Starting deadlock detector with 20m0s timeout
[VVQ5H] 2023/09/19 17:02:51 INFO: Overall send rate is unlimited, receive rate is unlimited
[VVQ5H] 2023/09/19 17:02:51 INFO: Anonymous usage reporting is always enabled for candidate releases.
[VVQ5H] 2023/09/19 17:02:51 INFO: Using discovery mechanism: global discovery server https://discovery.syncthing.net/v2/?noannounce&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VVQ5H] 2023/09/19 17:02:51 INFO: Using discovery mechanism: global discovery server https://discovery-v4.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VVQ5H] 2023/09/19 17:02:51 INFO: Using discovery mechanism: global discovery server https://discovery-v6.syncthing.net/v2/?nolookup&id=LYXKCHX-VI3NYZR-ALCJBHF-WMZYSPK-QG6QJA3-MPFYMSO-U56GTUK-NA2MIAW
[VVQ5H] 2023/09/19 17:02:51 INFO: Using discovery mechanism: IPv4 local broadcast discovery on port 21027
[VVQ5H] 2023/09/19 17:02:51 INFO: Using discovery mechanism: IPv6 local multicast discovery on address [ff12::8384]:21027
[VVQ5H] 2023/09/19 17:02:51 INFO: Relay listener (dynamic+https://relays.syncthing.net/endpoint) starting
[VVQ5H] 2023/09/19 17:02:51 INFO: Ready to synchronize "Default Folder" (default) (sendreceive)
[VVQ5H] 2023/09/19 17:02:51 INFO: Ready to synchronize "nanovery" (juhkv-dhmlj) (sendreceive)
[VVQ5H] 2023/09/19 17:02:51 INFO: TCP listener ([::]:22000) starting
[VVQ5H] 2023/09/19 17:02:51 INFO: Completed initial scan of sendreceive folder "Default Folder" (default)
[VVQ5H] 2023/09/19 17:02:51 INFO: GUI and API listening on 127.0.0.1:8384
[VVQ5H] 2023/09/19 17:02:51 INFO: Access the GUI via the following URL: http://127.0.0.1:8384/
[VVQ5H] 2023/09/19 17:02:51 INFO: My name is "nanolinux"
[VVQ5H] 2023/09/19 17:02:51 INFO: Device NUMHC6S-IKZP7I4-G3FWEEE-2UMAS74-PJ5XZKT-PCDGGA3-SJYHZGN-GD575AA is "harolds" at [dynamic]
[VVQ5H] 2023/09/19 17:02:51 INFO: QUIC listener ([::]:22000) starting
[VVQ5H] 2023/09/19 17:02:52 INFO: quic://0.0.0.0:22000 detected NAT type: Full cone NAT
[VVQ5H] 2023/09/19 17:02:52 INFO: quic://0.0.0.0:22000 resolved external address quic://31.22.1.26:59065 (via stun.syncthing.net:3478)
[VVQ5H] 2023/09/19 17:02:52 INFO: Established secure connection to NUMHC6S at 10.0.2.15:22000-10.111.19.174:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-5S6B13IR4ITFOCI52KEDKR8H3U
[VVQ5H] 2023/09/19 17:02:52 INFO: Device NUMHC6S client is "syncthing v1.25.0-rc.1" named "harolds" at 10.0.2.15:22000-10.111.19.174:22000/tcp-client/TLS1.3-TLS_AES_128_GCM_SHA256/WAN-P30-5S6B13IR4ITFOCI52KEDKR8H3U
panic: counter cannot decrease in value
[monitor] 2023/09/19 17:02:56 WARNING: Panic detected, writing to "/home/harold/.config/syncthing/panic-20230919-170256.log"
[monitor] 2023/09/19 17:02:56 WARNING: Please check for existing issues with similar panic message at https://github.com/syncthing/syncthing/issues/
[monitor] 2023/09/19 17:02:56 WARNING: If no issue with similar panic message exists, please create a new issue with the panic log attached
[monitor] 2023/09/19 17:02:56 INFO: Reporting crash found in panic-20230919-170256.log (report ID 9b4230be) ...
[monitor] 2023/09/19 17:02:56 INFO: Syncthing exited: exit status 2
[monitor] 2023/09/19 17:02:57 WARNING: 4 restarts in 39.878358056s; not retrying further

```



Discussion / hints:
I see a lot of these in Sentry too, all seemingly caused by the pattern

```
t0 := time.Now()
...
metricTotalOperationSeconds.WithLabelValues(root, op).Add(time.Since(t0).Seconds())
```

which to my understanding should never be negative, but here we are...

Fix patch:
diff --git a/lib/fs/metrics.go b/lib/fs/metrics.go
index ffde7772150..a97660ce09c 100644
--- a/lib/fs/metrics.go
+++ b/lib/fs/metrics.go
@@ -92,7 +92,9 @@ func (m *metricsFS) account(op string) func(bytes int) {
 	t0 := time.Now()
 	root := m.next.URI()
 	return func(bytes int) {
-		metricTotalOperationSeconds.WithLabelValues(root, op).Add(time.Since(t0).Seconds())
+		if dur := time.Since(t0).Seconds(); dur > 0 {
+			metricTotalOperationSeconds.WithLabelValues(root, op).Add(dur)
+		}
 		metricTotalOperationsCount.WithLabelValues(root, op).Inc()
 		if bytes >= 0 {
 			metricTotalBytesCount.WithLabelValues(root, op).Add(float64(bytes))
diff --git a/lib/model/util.go b/lib/model/util.go
index 2364c5e80ca..54d33c5808c 100644
--- a/lib/model/util.go
+++ b/lib/model/util.go
@@ -158,7 +158,9 @@ func inWritableDir(fn func(string) error, targetFs fs.Filesystem, path string, i
 func addTimeUntilCancelled(ctx context.Context, counter prometheus.Counter) {
 	t0 := time.Now()
 	defer func() {
-		counter.Add(time.Since(t0).Seconds())
+		if dur := time.Since(t0).Seconds(); dur > 0 {
+			counter.Add(dur)
+		}
 	}()
 
 	ticker := time.NewTicker(time.Second)
@@ -167,7 +169,9 @@ func addTimeUntilCancelled(ctx context.Context, counter prometheus.Counter) {
 	for {
 		select {
 		case t := <-ticker.C:
-			counter.Add(t.Sub(t0).Seconds())
+			if dur := t.Sub(t0).Seconds(); dur > 0 {
+				counter.Add(dur)
+			}
 			t0 = t
 		case <-ctx.Done():
 			return


Test patch: