Repository: lightningnetwork/lnd
Pull request #9801: peer+lnd: add new CLI option to control if we D/C on slow pongs
Author: login: Roasbeef
Created: 2025-05-09T23:16:28Z | Merged: 2025-05-15T17:25:30Z

In this commit, we add a new CLI option to control if we D/C on slow pongs or not. Due to the existence of head-of-the-line blocking at various levels of abstraction (app buffer, slow processing, TCP kernel buffers, etc), if there's a flurry of gossip messages (eg: 1K channel updates), then even with a reasonable processing latency, a peer may still not read our ping in time.

To combat this, we change the default behavior to just logging for slow pongs, and add a new CLI option to re-enable the old behavior.

Along the way, we also add some more enhanced logging, so we can tell when the last successful ping was, and also the deadline reached.




Diff:
diff --git a/config.go b/config.go
index 824bcc55f67..a0bfe02b782 100644
--- a/config.go
+++ b/config.go
@@ -251,6 +251,11 @@ const (
 
 	defaultPrunedNodeMaxPeers = 4
 	defaultNeutrinoMaxPeers   = 8
+
+	// defaultNoDisconnectOnPongFailure is the default value for whether we
+	// should *not* disconnect from a peer if we don't receive a pong
+	// response in time after we send a ping.
+	defaultNoDisconnectOnPongFailure = false
 )
 
 var (
@@ -527,6 +532,10 @@ type Config struct {
 	// NumRestrictedSlots is the number of restricted slots we'll allocate
 	// in the server.
 	NumRestrictedSlots uint64 `long:"num-restricted-slots" description:"The number of restricted slots we'll allocate in the server."`
+
+	// NoDisconnectOnPongFailure controls if we'll disconnect if a peer
+	// doesn't respond to a pong in time.
+	NoDisconnectOnPongFailure bool `long:"no-disconnect-on-pong-failure" description:"If true, a peer will *not* be disconnected if a pong is not received in time or is mismatched. Defaults to false, meaning peers *will* be disconnected on pong failure."`
 }
 
 // GRPCConfig holds the configuration options for the gRPC server.
@@ -747,10 +756,11 @@ func DefaultConfig() Config {
 			ServerPingTimeout: defaultGrpcServerPingTimeout,
 			ClientPingMinWait: defaultGrpcClientPingMinWait,
 		},
-		LogConfig:          build.DefaultLogConfig(),
-		WtClient:           lncfg.DefaultWtClientCfg(),
-		HTTPHeaderTimeout:  DefaultHTTPHeaderTimeout,
-		NumRestrictedSlots: DefaultNumRestrictedSlots,
+		LogConfig:                 build.DefaultLogConfig(),
+		WtClient:                  lncfg.DefaultWtClientCfg(),
+		HTTPHeaderTimeout:         DefaultHTTPHeaderTimeout,
+		NumRestrictedSlots:        DefaultNumRestrictedSlots,
+		NoDisconnectOnPongFailure: defaultNoDisconnectOnPongFailure,
 	}
 }
 
diff --git a/docs/release-notes/release-notes-0.19.0.md b/docs/release-notes/release-notes-0.19.0.md
index 3b66186925f..d44ef9f6ea0 100644
--- a/docs/release-notes/release-notes-0.19.0.md
+++ b/docs/release-notes/release-notes-0.19.0.md
@@ -305,6 +305,10 @@ when running LND with an aux component injected (custom channels).
 
 * [remove x/exp/maps dependency](https://github.com/lightningnetwork/lnd/pull/9621)
 
+* [Add a new configuration option](https://github.com/lightningnetwork/lnd/pull/9801)
+  `--no-disconnect-on-pong-failure` (defaulting to false) to control whether a
+  peer is disconnected if a pong message is not received in time or is mismatched.
+
 ## RPC Updates
 
 * Some RPCs that previously just returned an empty response message now at least
diff --git a/peer/brontide.go b/peer/brontide.go
index bfc603ae8af..da4aa610a44 100644
--- a/peer/brontide.go
+++ b/peer/brontide.go
@@ -94,7 +94,7 @@ const (
 	torTimeoutMultiplier = 3
 
 	// msgStreamSize is the size of the message streams.
-	msgStreamSize = 5
+	msgStreamSize = 50
 )
 
 var (
@@ -455,6 +455,10 @@ type Config struct {
 	// experimental endorsement signals should be set.
 	ShouldFwdExpEndorsement func() bool
 
+	// NoDisconnectOnPongFailure indicates whether the peer should *not* be
+	// disconnected if a pong is not received in time or is mismatched.
+	NoDisconnectOnPongFailure bool
+
 	// Quit is the server's quit channel. If this is closed, we halt operation.
 	Quit chan struct{}
 }
@@ -735,11 +739,27 @@ func NewBrontide(cfg Config) *Brontide {
 		SendPing: func(ping *lnwire.Ping) {
 			p.queueMsg(ping, nil)
 		},
-		OnPongFailure: func(err error) {
-			eStr := "pong response failure for %s: %v " +
-				"-- disconnecting"
-			p.log.Warnf(eStr, p, err)
-			go p.Disconnect(fmt.Errorf(eStr, p, err))
+		OnPongFailure: func(reason error,
+			timeWaitedForPong time.Duration,
+			lastKnownRTT time.Duration) {
+
+			logMsg := fmt.Sprintf("pong response "+
+				"failure for %s: %v. Time waited for this "+
+				"pong: %v. Last successful RTT: %v.",
+				p, reason, timeWaitedForPong, lastKnownRTT)
+
+			// If NoDisconnectOnPongFailure is true, we don't
+			// disconnect. Otherwise (if it's false, the default),
+			// we disconnect.
+			if p.cfg.NoDisconnectOnPongFailure {
+				p.log.Warnf("%s -- not disconnecting "+
+					"due to config", logMsg)
+				return
+			}
+
+			p.log.Warnf("%s -- disconnecting", logMsg)
+
+			go p.Disconnect(fmt.Errorf("pong failure: %w", reason))
 		},
 	})
 
diff --git a/peer/ping_manager.go b/peer/ping_manager.go
index f5c6180be13..7686cbe8b01 100644
--- a/peer/ping_manager.go
+++ b/peer/ping_manager.go
@@ -7,6 +7,7 @@ import (
 	"sync/atomic"
 	"time"
 
+	"github.com/lightningnetwork/lnd/fn/v2"
 	"github.com/lightningnetwork/lnd/lnwire"
 )
 
@@ -36,7 +37,8 @@ type PingManagerConfig struct {
 	// OnPongFailure is a closure that is responsible for executing the
 	// logic when a Pong message is either late or does not match our
 	// expectations for that Pong
-	OnPongFailure func(error)
+	OnPongFailure func(failureReason error, timeWaitedForPong time.Duration,
+		lastKnownRTT time.Duration)
 }
 
 // PingManager is a structure that is designed to manage the internal state
@@ -108,6 +110,26 @@ func (m *PingManager) Start() error {
 	return err
 }
 
+// getLastRTT safely retrieves the last known RTT, returning 0 if none exists.
+func (m *PingManager) getLastRTT() time.Duration {
+	rttPtr := m.pingTime.Load()
+	if rttPtr == nil {
+		return 0
+	}
+
+	return *rttPtr
+}
+
+// pendingPingWait calculates the time waited since the last ping was sent. If
+// no ping time is reported, None is returned. defaultDuration.
+func (m *PingManager) pendingPingWait() fn.Option[time.Duration] {
+	if m.pingLastSend != nil {
+		return fn.Some(time.Since(*m.pingLastSend))
+	}
+
+	return fn.None[time.Duration]()
+}
+
 // pingHandler is the main goroutine responsible for enforcing the ping/pong
 // protocol.
 func (m *PingManager) pingHandler() {
@@ -119,6 +141,10 @@ func (m *PingManager) pingHandler() {
 		<-m.pingTimeout.C
 	}
 
+	// Because we don't know if the OnPingFailure callback actually
+	// disconnects a peer (dependent on user config), we should never return
+	// from this loop unless the ping manager is stopped explicitly (which
+	// happens on disconnect).
 	for {
 		select {
 		case <-m.pingTicker.C:
@@ -127,12 +153,20 @@ func (m *PingManager) pingHandler() {
 			// awaiting a pong response.  This should never occur,
 			// but if it does, it implies a timeout.
 			if m.outstandingPongSize >= 0 {
-				e := errors.New("impossible: new ping" +
-					"in unclean state",
+				// Ping was outstanding, meaning it timed out by
+				// the arrival of the next ping interval.
+				timeWaited := m.pendingPingWait().UnwrapOr(
+					m.cfg.IntervalDuration,
+				)
+				lastRTT := m.getLastRTT()
+
+				m.cfg.OnPongFailure(
+					errors.New("ping timed "+
+						"out by next interval"),
+					timeWaited, lastRTT,
 				)
-				m.cfg.OnPongFailure(e)
 
-				return
+				m.resetPingState()
 			}
 
 			pongSize := m.cfg.NewPongSize()
@@ -143,53 +177,67 @@ func (m *PingManager) pingHandler() {
 
 			// Set up our bookkeeping for the new Ping.
 			if err := m.setPingState(pongSize); err != nil {
-				m.cfg.OnPongFailure(err)
+				// This is an internal error related to timer
+				// reset. Pass it to OnPongFailure as it's
+				// critical. Current and last RTT are not
+				// directly applicable here.
+				m.cfg.OnPongFailure(err, 0, 0)
+
+				m.resetPingState()
 
-				return
+				continue
 			}
 
 			m.cfg.SendPing(ping)
 
 		case <-m.pingTimeout.C:
-			m.resetPingState()
-
-			e := errors.New("timeout while waiting for " +
-				"pong response",
+			timeWaited := m.pendingPingWait().UnwrapOr(
+				m.cfg.TimeoutDuration,
 			)
+			lastRTT := m.getLastRTT()
 
-			m.cfg.OnPongFailure(e)
+			m.cfg.OnPongFailure(
+				errors.New("timeout while waiting for "+
+					"pong response"),
+				timeWaited, lastRTT,
+			)
 
-			return
+			m.resetPingState()
 
 		case pong := <-m.pongChan:
 			pongSize := int32(len(pong.PongBytes))
 
-			// Save off values we are about to override when we
-			// call resetPingState.
+			// Save off values we are about to override when we call
+			// resetPingState.
 			expected := m.outstandingPongSize
-			lastPing := m.pingLastSend
+			lastPingTime := m.pingLastSend
 
-			m.resetPingState()
+			// This is an unexpected pong, we'll continue.
+			if lastPingTime == nil {
+				continue
+			}
 
-			// If the pong we receive doesn't match the ping we
-			// sent out, then we fail out.
+			actualRTT := time.Since(*lastPingTime)
+
+			// If the pong we receive doesn't match the ping we sent
+			// out, then we fail out.
 			if pongSize != expected {
-				e := errors.New("pong response does " +
-					"not match expected size",
-				)
+				e := fmt.Errorf("pong response does not match "+
+					"expected size. Expected: %d, Got: %d",
+					expected, pongSize)
 
-				m.cfg.OnPongFailure(e)
+				lastRTT := m.getLastRTT()
+				m.cfg.OnPongFailure(e, actualRTT, lastRTT)
 
-				return
-			}
+				m.resetPingState()
 
-			// Compute RTT of ping and save that for future
-			// querying.
-			if lastPing != nil {
-				rtt := time.Since(*lastPing)
-				m.pingTime.Store(&rtt)
+				continue
 			}
 
+			// Pong is good, update RTT and reset state.
+			m.pingTime.Store(&actualRTT)
+			m.resetPingState()
+
 		case <-m.quit:
 			return
 		}
@@ -231,6 +279,7 @@ func (m *PingManager) setPingState(pongSize uint16) error {
 func (m *PingManager) resetPingState() {
 	m.pingLastSend = nil
 	m.outstandingPongSize = -1
+
 	if !m.pingTimeout.Stop() {
 		select {
 		case <-m.pingTimeout.C:
diff --git a/peer/ping_manager_test.go b/peer/ping_manager_test.go
index 0f4c3be4982..d0d96bb9432 100644
--- a/peer/ping_manager_test.go
+++ b/peer/ping_manager_test.go
@@ -1,6 +1,7 @@
 package peer
 
 import (
+	"sync"
 	"testing"
 	"time"
 
@@ -23,19 +24,19 @@ func TestPingManager(t *testing.T) {
 		result   bool
 	}{
 		{
-			name:     "Happy Path",
+			name:     "happy Path",
 			delay:    0,
 			pongSize: 4,
 			result:   true,
 		},
 		{
-			name:     "Bad Pong",
+			name:     "bad Pong",
 			delay:    0,
 			pongSize: 3,
 			result:   false,
 		},
 		{
-			name:     "Timeout",
+			name:     "timeout",
 			delay:    2,
 			pongSize: 4,
 			result:   false,
@@ -44,45 +45,56 @@ func TestPingManager(t *testing.T) {
 
 	payload := make([]byte, 4)
 	for _, test := range testCases {
-		// Set up PingManager.
-		pingSent := make(chan struct{})
-		disconnected := make(chan struct{})
-		mgr := NewPingManager(&PingManagerConfig{
-			NewPingPayload: func() []byte {
-				return payload
-			},
-			NewPongSize: func() uint16 {
-				return 4
-			},
-			IntervalDuration: time.Second * 2,
-			TimeoutDuration:  time.Second,
-			SendPing: func(ping *lnwire.Ping) {
-				close(pingSent)
-			},
-			OnPongFailure: func(err error) {
-				close(disconnected)
-			},
-		})
-		require.NoError(t, mgr.Start(), "Could not start pingManager")
+		t.Run(test.name, func(t *testing.T) {
+			// Set up PingManager.
+			var pingOnce sync.Once
+			pingSent := make(chan struct{})
+			disconnected := make(chan struct{})
+			mgr := NewPingManager(&PingManagerConfig{
+				NewPingPayload: func() []byte {
+					return payload
+				},
+				NewPongSize: func() uint16 {
+					return 4
+				},
+				IntervalDuration: time.Second * 2,
+				TimeoutDuration:  time.Second,
+				SendPing: func(ping *lnwire.Ping) {
+					pingOnce.Do(func() {
+						close(pingSent)
+					})
+				},
+				OnPongFailure: func(err error,
+					_ time.Duration, _ time.Duration) {
+
+					close(disconnected)
+				},
+			})
+			require.NoError(
+				t, mgr.Start(), "Could not start pingManager",
+			)
 
-		// Wait for initial Ping.
-		<-pingSent
+			// Wait for initial Ping.
+			<-pingSent
 
-		// Wait for pre-determined time before sending Pong response.
-		time.Sleep(time.Duration(test.delay) * time.Second)
+			// Wait for pre-determined time before sending Pong
+			// response.
+			time.Sleep(time.Duration(test.delay) * time.Second)
 
-		// Send Pong back.
-		res := lnwire.Pong{PongBytes: make([]byte, test.pongSize)}
-		mgr.ReceivedPong(&res)
+			// Send Pong back.
+			res := lnwire.Pong{
+				PongBytes: make([]byte, test.pongSize),
+			}
+			mgr.ReceivedPong(&res)
 
-		// Evaluate result
-		select {
-		case <-time.NewTimer(time.Second / 2).C:
-			require.True(t, test.result)
-		case <-disconnected:
-			require.False(t, test.result)
-		}
+			select {
+			case <-time.NewTimer(time.Second / 2).C:
+				require.True(t, test.result)
+			case <-disconnected:
+				require.False(t, test.result)
+			}
 
-		mgr.Stop()
+			mgr.Stop()
+		})
 	}
 }
diff --git a/sample-lnd.conf b/sample-lnd.conf
index af4923846dc..ae929d8e6cf 100644
--- a/sample-lnd.conf
+++ b/sample-lnd.conf
@@ -569,6 +569,11 @@
 ; The number of restricted slots the server will allocate for peers.
 ; num-restricted-slots=30
 
+; If true, a peer will *not* be disconnected if a pong is not received in time
+; or is mismatched. Defaults to false, meaning peers *will* be disconnected on
+; pong failure.
+; no-disconnect-on-pong-failure=false
+
 [fee]
 
 ; Optional URL for external fee estimation. If no URL is specified, the method
diff --git a/server.go b/server.go
index e816c3ca4cd..3a077a28f46 100644
--- a/server.go
+++ b/server.go
@@ -4493,6 +4493,7 @@ func (s *server) peerConnected(conn net.Conn, connReq *connmgr.ConnReq,
 				EndorsementExperimentEnd,
 			)
 		},
+		NoDisconnectOnPongFailure: s.cfg.NoDisconnectOnPongFailure,
 	}
 
 	copy(pCfg.PubKeyBytes[:], peerAddr.IdentityKey.SerializeCompressed())


Reviews:
user: login: guggero
body: I think the flag could be useful!
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-12T04:55:39Z
user: login: Roasbeef
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-12T22:54:43Z
user: login: Roasbeef
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-12T22:58:33Z
user: login: Roasbeef
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-12T22:59:11Z
user: login: Roasbeef
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-12T23:03:22Z
user: login: ellemouton
body: the last commit does not change existing behaviour afaict
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-13T13:50:06Z
user: login: Roasbeef
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-13T23:25:43Z
user: login: Roasbeef
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-05-14T19:58:10Z
user: login: guggero
body: Pushed up a fix for the sample config and also made the behavior of `return` vs. `continue` in the ping manager more clear.

With that, LGTM :tada: 
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-15T07:51:35Z
user: login: ellemouton
body: just one comment - otherwise lgtm!
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-15T12:29:09Z
user: login: guggero
state: COMMENTED
author_association: COLLABORATOR
submitted_at: 2025-05-15T14:37:42Z
user: login: ziggie1984
body: LGTM
state: APPROVED
author_association: COLLABORATOR
submitted_at: 2025-05-15T15:57:57Z

Review comments:
diff_hunk: @@ -108,6 +109,29 @@ func (m *PingManager) Start() error {
 	return err
 }
 
+// getLastRTT safely retrieves the last known RTT, returning 0 if none exists.
+func (m *PingManager) getLastRTT() time.Duration {
+	rttPtr := m.pingTime.Load()
+	if rttPtr == nil {
+		return 0
+	}
+
+	return *rttPtr
+}
+
+// getTimeWaitedOrDefault calculates the time waited since the last ping was
+// sent. If no ping was sent (pingLastSend is nil), it returns the provided
+// defaultDuration.
+func (m *PingManager) getTimeWaitedOrDefault(defaultDuration time.Duration,
path: peer/ping_manager.go
user: login: guggero
body: nit: wrap before argument instead.
created_at: 2025-05-12T04:47:57Z
author_association: COLLABORATOR
diff_hunk: @@ -143,53 +175,64 @@ func (m *PingManager) pingHandler() {
 
 			// Set up our bookkeeping for the new Ping.
 			if err := m.setPingState(pongSize); err != nil {
-				m.cfg.OnPongFailure(err)
-
+				// This is an internal error related to timer
+				// reset. Pass it to OnPongFailure as it's
+				// critical. Current and last RTT are not
+				// directly applicable here.
+				m.cfg.OnPongFailure(err, 0, 0)
 				return
 			}
 
 			m.cfg.SendPing(ping)
 
 		case <-m.pingTimeout.C:
-			m.resetPingState()
-
-			e := errors.New("timeout while waiting for " +
-				"pong response",
+			timeWaited := m.getTimeWaitedOrDefault(
+				m.cfg.TimeoutDuration,
 			)
+			lastRTT := m.getLastRTT()
 
-			m.cfg.OnPongFailure(e)
+			m.cfg.OnPongFailure(
+				errors.New("timeout while waiting for "+
+					"pong response"),
+				timeWaited, lastRTT,
+			)
 
-			return
+			m.resetPingState()
path: peer/ping_manager.go
user: login: guggero
body: We previously returned here. Maybe we need to signal from `OnPongFailure` whether we actually disconnected. And if we did, we should still return here? Otherwise the ping handler will continue running, even if we disconnect.
created_at: 2025-05-12T04:53:36Z
author_association: COLLABORATOR
diff_hunk: @@ -143,53 +175,64 @@ func (m *PingManager) pingHandler() {
 
 			// Set up our bookkeeping for the new Ping.
 			if err := m.setPingState(pongSize); err != nil {
-				m.cfg.OnPongFailure(err)
-
+				// This is an internal error related to timer
+				// reset. Pass it to OnPongFailure as it's
+				// critical. Current and last RTT are not
+				// directly applicable here.
+				m.cfg.OnPongFailure(err, 0, 0)
 				return
 			}
 
 			m.cfg.SendPing(ping)
 
 		case <-m.pingTimeout.C:
-			m.resetPingState()
-
-			e := errors.New("timeout while waiting for " +
-				"pong response",
+			timeWaited := m.getTimeWaitedOrDefault(
+				m.cfg.TimeoutDuration,
 			)
+			lastRTT := m.getLastRTT()
 
-			m.cfg.OnPongFailure(e)
+			m.cfg.OnPongFailure(
+				errors.New("timeout while waiting for "+
+					"pong response"),
+				timeWaited, lastRTT,
+			)
 
-			return
+			m.resetPingState()
 
 		case pong := <-m.pongChan:
 			pongSize := int32(len(pong.PongBytes))
 
-			// Save off values we are about to override when we
-			// call resetPingState.
+			// Save off values we are about to override when we call
+			// resetPingState.
 			expected := m.outstandingPongSize
-			lastPing := m.pingLastSend
+			lastPingTime := m.pingLastSend
 
-			m.resetPingState()
+			if lastPingTime == nil {
+				// This is an unexpected pong, we'll continue.
+				continue
+			}
+
+			actualRTT := time.Since(*lastPingTime)
 
-			// If the pong we receive doesn't match the ping we
-			// sent out, then we fail out.
+			// If the pong we receive doesn't match the ping we sent
+			// out, then we fail out.
 			if pongSize != expected {
-				e := errors.New("pong response does " +
-					"not match expected size",
-				)
+				e := fmt.Errorf("pong response does not match "+
+					"expected size. Expected: %d, Got: %d",
+					expected, pongSize)
 
-				m.cfg.OnPongFailure(e)
+				lastRTT := m.getLastRTT()
+				m.cfg.OnPongFailure(e, actualRTT, lastRTT)
path: peer/ping_manager.go
user: login: guggero
body: Same here re continuing to run on potential disconnect.
created_at: 2025-05-12T04:54:00Z
author_association: COLLABORATOR
diff_hunk: @@ -527,6 +532,8 @@ type Config struct {
 	// NumRestrictedSlots is the number of restricted slots we'll allocate
 	// in the server.
 	NumRestrictedSlots uint64 `long:"num-restricted-slots" description:"The number of restricted slots we'll allocate in the server."`
+
+	DisconnectOnPongFailure bool `long:"disconnect-on-pong-failure" description:"Whether to disconnect a peer if a pong is not received in time or is mismatched. Defaults to false."`
path: config.go
user: login: guggero
body: Not sure if we need the "default to false" sentence here, since all our boolean flags behave that way?
created_at: 2025-05-12T04:55:17Z
author_association: COLLABORATOR
diff_hunk: @@ -143,53 +175,64 @@ func (m *PingManager) pingHandler() {
 
 			// Set up our bookkeeping for the new Ping.
 			if err := m.setPingState(pongSize); err != nil {
-				m.cfg.OnPongFailure(err)
-
+				// This is an internal error related to timer
+				// reset. Pass it to OnPongFailure as it's
+				// critical. Current and last RTT are not
+				// directly applicable here.
+				m.cfg.OnPongFailure(err, 0, 0)
 				return
 			}
 
 			m.cfg.SendPing(ping)
 
 		case <-m.pingTimeout.C:
-			m.resetPingState()
-
-			e := errors.New("timeout while waiting for " +
-				"pong response",
+			timeWaited := m.getTimeWaitedOrDefault(
+				m.cfg.TimeoutDuration,
 			)
+			lastRTT := m.getLastRTT()
 
-			m.cfg.OnPongFailure(e)
+			m.cfg.OnPongFailure(
+				errors.New("timeout while waiting for "+
+					"pong response"),
+				timeWaited, lastRTT,
+			)
 
-			return
+			m.resetPingState()
path: peer/ping_manager.go
user: login: Roasbeef
body: Hmm, good point. I think to resolve this, we can pass in the actual peer context into the ping manager. Then this way, the ping manager will _always_ shutdown when/if the peer disconnects. 
created_at: 2025-05-12T22:54:42Z
author_association: MEMBER
diff_hunk: @@ -143,53 +175,64 @@ func (m *PingManager) pingHandler() {
 
 			// Set up our bookkeeping for the new Ping.
 			if err := m.setPingState(pongSize); err != nil {
-				m.cfg.OnPongFailure(err)
-
+				// This is an internal error related to timer
+				// reset. Pass it to OnPongFailure as it's
+				// critical. Current and last RTT are not
+				// directly applicable here.
+				m.cfg.OnPongFailure(err, 0, 0)
 				return
 			}
 
 			m.cfg.SendPing(ping)
 
 		case <-m.pingTimeout.C:
-			m.resetPingState()
-
-			e := errors.New("timeout while waiting for " +
-				"pong response",
+			timeWaited := m.getTimeWaitedOrDefault(
+				m.cfg.TimeoutDuration,
 			)
+			lastRTT := m.getLastRTT()
 
-			m.cfg.OnPongFailure(e)
+			m.cfg.OnPongFailure(
+				errors.New("timeout while waiting for "+
+					"pong response"),
+				timeWaited, lastRTT,
+			)
 
-			return
+			m.resetPingState()
path: peer/ping_manager.go
user: login: Roasbeef
body: Pushed this up as a new commit. 
created_at: 2025-05-12T22:58:33Z
author_association: MEMBER
diff_hunk: @@ -527,6 +532,8 @@ type Config struct {
 	// NumRestrictedSlots is the number of restricted slots we'll allocate
 	// in the server.
 	NumRestrictedSlots uint64 `long:"num-restricted-slots" description:"The number of restricted slots we'll allocate in the server."`
+
+	DisconnectOnPongFailure bool `long:"disconnect-on-pong-failure" description:"Whether to disconnect a peer if a pong is not received in time or is mismatched. Defaults to false."`
path: config.go
user: login: Roasbeef
body: Removed, was sort second guessing myself re some of the quirks w/ the parser and bool args. 
created_at: 2025-05-12T22:59:11Z
author_association: MEMBER
diff_hunk: @@ -108,6 +109,29 @@ func (m *PingManager) Start() error {
 	return err
 }
 
+// getLastRTT safely retrieves the last known RTT, returning 0 if none exists.
+func (m *PingManager) getLastRTT() time.Duration {
+	rttPtr := m.pingTime.Load()
+	if rttPtr == nil {
+		return 0
+	}
+
+	return *rttPtr
+}
+
+// getTimeWaitedOrDefault calculates the time waited since the last ping was
+// sent. If no ping was sent (pingLastSend is nil), it returns the provided
+// defaultDuration.
+func (m *PingManager) getTimeWaitedOrDefault(defaultDuration time.Duration,
path: peer/ping_manager.go
user: login: Roasbeef
body: Ended up re-working things slightly to use `UnwrapOr` instead of passing in a default. 
created_at: 2025-05-12T23:03:22Z
author_association: MEMBER
diff_hunk: @@ -731,6 +731,8 @@ func NewBrontide(cfg Config) *Brontide {
 		)
 	}
 
+	peerCtx, _ := p.cg.Create(context.Background())
+
 	p.pingManager = NewPingManager(&PingManagerConfig{
path: peer/brontide.go
user: login: ellemouton
body: ideally we dont pass contexts via constructors but instead explicitly via "Start" calls. ctxs should only be passed to a constructor if the actual constructor makes calls that require a ctx. 
created_at: 2025-05-13T13:44:19Z
author_association: COLLABORATOR
diff_hunk: @@ -39,6 +40,11 @@ type PingManagerConfig struct {
 	// expectations for that Pong
 	OnPongFailure func(failureReason error, timeWaitedForPong time.Duration,
 		lastKnownRTT time.Duration)
+
+	// PeerCtx is the context of the peer that owns this ping manager. We'll
+	// use this to ensure that if the peer is shutting down, then we will as
+	// well.
+	PeerCtx context.Context
path: peer/ping_manager.go
user: login: ellemouton
body: we should instead call this PingManager's Stop method if the peer is exiting
created_at: 2025-05-13T13:47:47Z
author_association: COLLABORATOR
diff_hunk: @@ -231,6 +237,9 @@ func (m *PingManager) pingHandler() {
 			m.pingTime.Store(&actualRTT)
 			m.resetPingState()
 
+		case <-m.cfg.PeerCtx.Done():
+			return
+
 		case <-m.quit:
 			return
path: peer/ping_manager.go
user: login: ellemouton
body: we should not need to do this:

the `m.quit` channel is already closed if the PingManager's `Stop` method is called. And today it is already called in ` (p *Brontide) Disconnect`. 

And this is done just before the Brontide's ContextGuard's `Quit` method is called - which would cause this `PeerCtx` to be cancelled anyways. 

So this commit is not needed
created_at: 2025-05-13T13:49:45Z
author_association: COLLABORATOR
diff_hunk: @@ -231,6 +237,9 @@ func (m *PingManager) pingHandler() {
 			m.pingTime.Store(&actualRTT)
 			m.resetPingState()
 
+		case <-m.cfg.PeerCtx.Done():
+			return
+
 		case <-m.quit:
 			return
path: peer/ping_manager.go
user: login: Roasbeef
body: Great observation! I tacked this on in response to an earlier comment, but should've cross referenced with the over all goroutine life cycle. 

I think we may go with https://github.com/lightningnetwork/lnd/pull/9805 instead after all. 
created_at: 2025-05-13T23:25:43Z
author_association: MEMBER
diff_hunk: @@ -231,6 +237,9 @@ func (m *PingManager) pingHandler() {
 			m.pingTime.Store(&actualRTT)
 			m.resetPingState()
 
+		case <-m.cfg.PeerCtx.Done():
+			return
+
 		case <-m.quit:
 			return
path: peer/ping_manager.go
user: login: Roasbeef
body: Dropped the last commit all together. 
created_at: 2025-05-14T19:58:10Z
author_association: MEMBER
diff_hunk: @@ -143,53 +177,65 @@ func (m *PingManager) pingHandler() {
 
 			// Set up our bookkeeping for the new Ping.
 			if err := m.setPingState(pongSize); err != nil {
-				m.cfg.OnPongFailure(err)
+				// This is an internal error related to timer
+				// reset. Pass it to OnPongFailure as it's
+				// critical. Current and last RTT are not
+				// directly applicable here.
+				m.cfg.OnPongFailure(err, 0, 0)
 
-				return
+				continue
path: peer/ping_manager.go
user: login: guggero
body: I turned this into a `continue` as well, since we don't necessarily disconnect. Otherwise if we ran into this but the user has configured to not disconnect, then the ping manager won't be running anymore but the peer is still connected (which could have weird side effects).
created_at: 2025-05-15T07:50:12Z
author_association: COLLABORATOR
diff_hunk: @@ -143,53 +177,65 @@ func (m *PingManager) pingHandler() {
 
 			// Set up our bookkeeping for the new Ping.
 			if err := m.setPingState(pongSize); err != nil {
-				m.cfg.OnPongFailure(err)
+				// This is an internal error related to timer
+				// reset. Pass it to OnPongFailure as it's
+				// critical. Current and last RTT are not
+				// directly applicable here.
+				m.cfg.OnPongFailure(err, 0, 0)
 
-				return
+				continue
path: peer/ping_manager.go
user: login: ellemouton
body: do we need to reset the state here too?
created_at: 2025-05-15T12:26:06Z
author_association: COLLABORATOR
diff_hunk: @@ -143,53 +177,65 @@ func (m *PingManager) pingHandler() {
 
 			// Set up our bookkeeping for the new Ping.
 			if err := m.setPingState(pongSize); err != nil {
-				m.cfg.OnPongFailure(err)
+				// This is an internal error related to timer
+				// reset. Pass it to OnPongFailure as it's
+				// critical. Current and last RTT are not
+				// directly applicable here.
+				m.cfg.OnPongFailure(err, 0, 0)
 
-				return
+				continue
path: peer/ping_manager.go
user: login: guggero
body: Good point. I think so, yes. Because we only get into the error case if the ping timer was already active. So we do want to stop/reset it in that case.
created_at: 2025-05-15T14:37:42Z
author_association: COLLABORATOR
diff_hunk: @@ -735,11 +739,27 @@ func NewBrontide(cfg Config) *Brontide {
 		SendPing: func(ping *lnwire.Ping) {
 			p.queueMsg(ping, nil)
 		},
-		OnPongFailure: func(err error) {
-			eStr := "pong response failure for %s: %v " +
-				"-- disconnecting"
-			p.log.Warnf(eStr, p, err)
-			go p.Disconnect(fmt.Errorf(eStr, p, err))
+		OnPongFailure: func(reason error,
+			timeWaitedForPong time.Duration,
+			lastKnownRTT time.Duration) {
+
+			logMsg := fmt.Sprintf("pong response "+
+				"failure for %s: %v. Time waited for this "+
+				"pong: %v. Last successful RTT: %v.",
+				p, reason, timeWaitedForPong, lastKnownRTT)
+
+			// If NoDisconnectOnPongFailure is true, we don't
+			// disconnect. Otherwise (if it's false, the default),
+			// we disconnect.
path: peer/brontide.go
user: login: ziggie1984
body: Nit: Comment can be removed, describes the code.
created_at: 2025-05-15T15:16:03Z
author_association: COLLABORATOR
diff_hunk: @@ -108,6 +110,26 @@ func (m *PingManager) Start() error {
 	return err
 }
 
+// getLastRTT safely retrieves the last known RTT, returning 0 if none exists.
+func (m *PingManager) getLastRTT() time.Duration {
path: peer/ping_manager.go
user: login: ziggie1984
body: Nit: Use fn.Option here as well similar as below `pendingPingWait` ?
created_at: 2025-05-15T15:18:42Z
author_association: COLLABORATOR
diff_hunk: @@ -44,45 +45,56 @@ func TestPingManager(t *testing.T) {
 
 	payload := make([]byte, 4)
 	for _, test := range testCases {
-		// Set up PingManager.
-		pingSent := make(chan struct{})
-		disconnected := make(chan struct{})
-		mgr := NewPingManager(&PingManagerConfig{
-			NewPingPayload: func() []byte {
-				return payload
-			},
-			NewPongSize: func() uint16 {
-				return 4
-			},
-			IntervalDuration: time.Second * 2,
-			TimeoutDuration:  time.Second,
-			SendPing: func(ping *lnwire.Ping) {
-				close(pingSent)
-			},
-			OnPongFailure: func(err error) {
-				close(disconnected)
-			},
-		})
-		require.NoError(t, mgr.Start(), "Could not start pingManager")
+		t.Run(test.name, func(t *testing.T) {
+			// Set up PingManager.
+			var pingOnce sync.Once
+			pingSent := make(chan struct{})
+			disconnected := make(chan struct{})
+			mgr := NewPingManager(&PingManagerConfig{
+				NewPingPayload: func() []byte {
+					return payload
+				},
+				NewPongSize: func() uint16 {
+					return 4
+				},
+				IntervalDuration: time.Second * 2,
+				TimeoutDuration:  time.Second,
+				SendPing: func(ping *lnwire.Ping) {
+					pingOnce.Do(func() {
+						close(pingSent)
+					})
+				},
+				OnPongFailure: func(err error,
path: peer/ping_manager_test.go
user: login: ziggie1984
body: Nit: should be protected by a Once struct maybe as well, because it is closing a channel ?
created_at: 2025-05-15T15:57:12Z
author_association: COLLABORATOR

PR comments:
user: login: coderabbitai[bot]
created_at: 2025-05-09T23:16:34Z
body: <!-- This is an auto-generated comment: summarize by coderabbit.ai -->
<!-- This is an auto-generated comment: skip review by coderabbit.ai -->

> [!IMPORTANT]
> ## Review skipped
> 
> Auto reviews are limited to specific labels.
> 
> <details>
> <summary>:label: Labels to auto review (1)</summary>
> 
> * llm-review
> 
> </details>
> 
> Please check the settings in the CodeRabbit UI or the `.coderabbit.yaml` file in this repository. To trigger a single review, invoke the `@coderabbitai review` command.
> 
> You can disable this status message by setting the `reviews.review_status` to `false` in the CodeRabbit configuration file.

<!-- end of auto-generated comment: skip review by coderabbit.ai -->
<!-- tips_start -->

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

<details>
<summary>❤️ Share</summary>

- [X](https://twitter.com/intent/tweet?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A&url=https%3A//coderabbit.ai)
- [Mastodon](https://mastodon.social/share?text=I%20just%20used%20%40coderabbitai%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20the%20proprietary%20code.%20Check%20it%20out%3A%20https%3A%2F%2Fcoderabbit.ai)
- [Reddit](https://www.reddit.com/submit?title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&text=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code.%20Check%20it%20out%3A%20https%3A//coderabbit.ai)
- [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fcoderabbit.ai&mini=true&title=Great%20tool%20for%20code%20review%20-%20CodeRabbit&summary=I%20just%20used%20CodeRabbit%20for%20my%20code%20review%2C%20and%20it%27s%20fantastic%21%20It%27s%20free%20for%20OSS%20and%20offers%20a%20free%20trial%20for%20proprietary%20code)

</details>

<details>
<summary>🪧 Tips</summary>

### Chat

There are 3 ways to chat with [CodeRabbit](https://coderabbit.ai?utm_source=oss&utm_medium=github&utm_campaign=lightningnetwork/lnd&utm_content=9801):

- Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  - `I pushed a fix in commit <commit_id>, please review it.`
  - `Explain this complex logic.`
  - `Open a follow-up GitHub issue for this discussion.`
- Files and specific lines of code (under the "Files changed" tab): Tag `@coderabbitai` in a new review comment at the desired location with your query. Examples:
  - `@coderabbitai explain this code block.`
  -	`@coderabbitai modularize this function.`
- PR comments: Tag `@coderabbitai` in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  - `@coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.`
  - `@coderabbitai read src/utils.ts and explain its main purpose.`
  - `@coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.`
  - `@coderabbitai help me debug CodeRabbit configuration file.`

### Support

Need help? Create a ticket on our [support page](https://www.coderabbit.ai/contact-us/support) for assistance with any issues or questions.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

### CodeRabbit Commands (Invoked using PR comments)

- `@coderabbitai pause` to pause the reviews on a PR.
- `@coderabbitai resume` to resume the paused reviews.
- `@coderabbitai review` to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
- `@coderabbitai full review` to do a full review from scratch and review all the files again.
- `@coderabbitai summary` to regenerate the summary of the PR.
- `@coderabbitai generate docstrings` to [generate docstrings](https://docs.coderabbit.ai/finishing-touches/docstrings) for this PR.
- `@coderabbitai generate sequence diagram` to generate a sequence diagram of the changes in this PR.
- `@coderabbitai resolve` resolve all the CodeRabbit review comments.
- `@coderabbitai configuration` to show the current CodeRabbit configuration for the repository.
- `@coderabbitai help` to get help.

### Other keywords and placeholders

- Add `@coderabbitai ignore` anywhere in the PR description to prevent this PR from being reviewed.
- Add `@coderabbitai summary` to generate the high-level summary at a specific location in the PR description.
- Add `@coderabbitai` anywhere in the PR title to generate the title automatically.

### CodeRabbit Configuration File (`.coderabbit.yaml`)

- You can programmatically configure CodeRabbit by adding a `.coderabbit.yaml` file to the root of your repository.
- Please see the [configuration documentation](https://docs.coderabbit.ai/guides/configure-coderabbit) for more information.
- If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: `# yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json`

### Documentation and Community

- Visit our [Documentation](https://docs.coderabbit.ai) for detailed information on how to use CodeRabbit.
- Join our [Discord Community](http://discord.gg/coderabbit) to get help, request features, and share feedback.
- Follow us on [X/Twitter](https://twitter.com/coderabbitai) for updates and announcements.

</details>

<!-- tips_end -->
author_association: CONTRIBUTOR
user: login: djkazic
created_at: 2025-05-10T22:50:36Z
body: tACK, this PR improves connection reliability with my peers compared to just `0.19.0-beta-rc4`.

Some slight weirdness that I figured would be good to document:

Pong size did not match expected size:
```
2025-05-10 12:15:51.125 [WRN] PEER: Peer(03f311f6bb92443e93ee75af7b8cabe0b8bd49f7eaa2757086124c1a75cbe2bfcd): pong response failure for 03f311f6bb92443e93ee75af7b8cabe0b8bd49f7eaa2757086124c1a75cbe2bfcd@96.230.252.205:9734: pong response does not match expected size. Expected: 1486, Got: 788. Time waited for this pong: 28.401370887s. Last successful RTT: 209.450171ms. -- not disconnecting due to config
2025-05-10 13:04:19.667 [WRN] PEER: Peer(0206dfc7fa5f56f3ce752c6621b1d03223c11b16cffb05835b67481c12c6d87031): pong response failure for 0206dfc7fa5f56f3ce752c6621b1d03223c11b16cffb05835b67481c12c6d87031@96.230.252.205:16079: pong response does not match expected size. Expected: 1402, Got: 1846. Time waited for this pong: 23.744974397s. Last successful RTT: 2.675905ms. -- not disconnecting due to config
2025-05-10 13:55:26.524 [WRN] PEER: Peer(021d2436cab847373a4212bf6d754ead5304f5d0791479643893a837b295f3441c): pong response failure for 021d2436cab847373a4212bf6d754ead5304f5d0791479643893a837b295f3441c@10.21.21.11:56890: pong response does not match expected size. Expected: 1164, Got: 3340. Time waited for this pong: 1.071593118s. Last successful RTT: 882.700042ms. -- not disconnecting due to config
```

Back-to-back repeated Pongs (remote is CLN):
```
2025-05-10 14:59:23.851 [DBG] PEER: Peer(03f311f6bb92443e93ee75af7b8cabe0b8bd49f7eaa2757086124c1a75cbe2bfcd): Received Pong(len(pong_bytes)=2146) from 03f311f6bb92443e93ee75af7b8cabe0b8bd49f7eaa2757086124c1a75cbe2bfcd@96.230.252.205:9734
2025-05-10 14:59:23.851 [DBG] PEER: Peer(03f311f6bb92443e93ee75af7b8cabe0b8bd49f7eaa2757086124c1a75cbe2bfcd): Received Pong(len(pong_bytes)=1267) from 03f311f6bb92443e93ee75af7b8cabe0b8bd49f7eaa2757086124c1a75cbe2bfcd@96.230.252.205:9734
2025-05-10 14:59:23.851 [DBG] PEER: Peer(03f311f6bb92443e93ee75af7b8cabe0b8bd49f7eaa2757086124c1a75cbe2bfcd): Received Pong(len(pong_bytes)=1052) from 03f311f6bb92443e93ee75af7b8cabe0b8bd49f7eaa2757086124c1a75cbe2bfcd@96.230.252.205:9734
2025-05-10 14:59:23.851 [WRN] PEER: Peer(03f311f6bb92443e93ee75af7b8cabe0b8bd49f7eaa2757086124c1a75cbe2bfcd): pong response failure for 03f311f6bb92443e93ee75af7b8cabe0b8bd49f7eaa2757086124c1a75cbe2bfcd@96.230.252.205:9734: pong response does not match expected size. Expected: 3013, Got: 2146. Time waited for this pong: 1.127449045s. Last successful RTT: 209.450171ms. -- not disconnecting due to config
```
author_association: CONTRIBUTOR
user: login: ziggie1984
created_at: 2025-05-12T16:24:01Z
body: I propose adding a Connection-Failure-Treshold (N) which only disconnects after several failed ping/pongs in a row rather than just adding a connection flag wdyt ?
author_association: COLLABORATOR
user: login: Roasbeef
created_at: 2025-05-12T23:02:26Z
body: > I propose adding a Connection-Failure-Treshold (N) which only disconnects after several failed ping/pongs in a row rather than just adding a connection flag wdyt ?

I had also considered making it a sort of peer level healthcheck, to inherit that threshold logic, but instead went in this direction as I started to second guess the design rationale in disconnecting in the first place. I think if we add a threshold flag, then we'd also want to add a flag to tune what the timeout value should be. 

When I started to run this on my node (even w/ the super prio queue), I noticed some nodes that were just persistently slow  in replying. 

Ultimately slow nodes _do_ affect payment latency e2e. There's a credible design direction here where we start to factor it in at the first hop level, but then also have the link sample the ping RTT of a peer, and decide if the link is even _eligible_ to send based on that. 
author_association: MEMBER
user: login: Roasbeef
created_at: 2025-05-14T21:14:19Z
body: The latest version _inverts_ the original PR: default stays, but users have an option to turn off the disconnect behavior. 
author_association: MEMBER