Repository: metallb/metallb
Pull request #2386 (2024-04-26 11:40:37)

Add graceful-restart support
/kind feature

**What this PR does / why we need it**:
Issue covers #2368 

**Special notes for your reviewer**:

**Release note**:


```release-note
TODO
```


karampok: Note to discuss:
```
Graceful restart configuration changed, reset this peer to take effect
% The Graceful Restart command used is not valid at this moment.
line 76: Failure to communicate[13] to bgpd, line:  neighbor 11.11.11.1 graceful-restart

[311|bfdd] done
[294|zebra] done
[300|bgpd] Configuration file[/etc/frr/frr.conf] processing failure: 13
2024-04-26 13:18:54,515 WARNING: frr-reload.py failed due to
vtysh (exec file) exited with status 13
Failed to fully apply configuration file 1 seconds
Caught SIGHUP and acquired lock! Reloading FRR..
Checking the configuration file syntax
```
it seems it fails, and retries to apply that the speaker/frr-reloader.
karampok: The above might be related to https://github.com/FRRouting/frr/issues/8403 
karampok: In my test: I achieved zero downtime if I have graceful restart either in global `bgp graceful restarst` or in neighbor mode `neighbor x.y.z.k graceful restart` and having that in  `bgp graceful-restart preserve-fw-state`. 

(Test= in runtime I delete the speaker and see if the client loses connectivity)

The second command that enables the F -bit is only available in global mode, not available per neighbor. Therefore a solution which we add graceful-restart in neighbor level and f-bit in global (because is not safe to just have that there by default) seem not optimal.

@fedepaol How much do you object to have it global (one for enable gracefull restart/one to on/off f-bit), and also have the option to opt out per neighbor? Any other ideas?
karampok: https://github.com/FRRouting/frr/issues/15880 created this issue upstream, even though we are not really blocked on that (well we will get an error but doe not look like it has an impact)
fedepaol: > The second command that enables the F -bit is only available in global mode, not available per neighbor. Therefore a solution which we add graceful-restart in neighbor level and f-bit in global (because is not safe to just have that there by default) seem not optimal.
> 
> @fedepaol How much do you object to have it global (one for enable gracefull restart/one to on/off f-bit), and also have the option to opt out per neighbor? Any other ideas?

Won't the global one affect only peers with gr enabled? If so, doesn't sound too terrible to always enable it.

karampok: My preference would be to be configurable to be in the safe side. I cannot find argument against always having it (or being default on) but no idea for which case this option exists.
karampok: @fedepaol I have completed the last commit with adding the e2e test (plus the reverse to see a failure without graceful restart). I think I addressed all your comments. Let me know if I miss something. Thanks
karampok: Another version pushed that does make the function not depend on external variable
fedepaol: few nits, almost there!
fedepaol: > What about now :) let me know if you want to rebase. Thanks

rebase is not necessary as we use merge queues.
LGTM, sending to merge queue, thanks!

Patch:
diff --git a/api/v1beta2/bgppeer_types.go b/api/v1beta2/bgppeer_types.go
index cff1e225..56e93eb6 100644
--- a/api/v1beta2/bgppeer_types.go
+++ b/api/v1beta2/bgppeer_types.go
@@ -85,6 +85,14 @@ type BGPPeerSpec struct {
 	// +optional
 	BFDProfile string `json:"bfdProfile,omitempty"`
 
+	// EnableGracefulRestart allows BGP peer to continue to forward data packets along
+	// known routes while the routing protocol information is being restored.
+	// This field is immutable because it requires restart of the BGP session
+	// Supported for FRR mode only.
+	// +optional
+	// +kubebuilder:validation:XValidation:rule="self == oldSelf",message="EnableGracefulRestart cannot be changed after creation"
+	EnableGracefulRestart bool `json:"enableGracefulRestart,omitempty"`
+
 	// To set if the BGPPeer is multi-hops away. Needed for FRR mode only.
 	// +optional
 	EBGPMultiHop bool `json:"ebgpMultiHop,omitempty"`
diff --git a/charts/metallb/charts/crds/templates/crds.yaml b/charts/metallb/charts/crds/templates/crds.yaml
index 79497f12..b39c52de 100644
--- a/charts/metallb/charts/crds/templates/crds.yaml
+++ b/charts/metallb/charts/crds/templates/crds.yaml
@@ -525,6 +525,16 @@ spec:
                 ebgpMultiHop:
                   description: To set if the BGPPeer is multi-hops away. Needed for FRR mode only.
                   type: boolean
+                enableGracefulRestart:
+                  description: |-
+                    EnableGracefulRestart allows BGP peer to continue to forward data packets along
+                    known routes while the routing protocol information is being restored.
+                    This field is immutable because it requires restart of the BGP session
+                    Supported for FRR mode only.
+                  type: boolean
+                  x-kubernetes-validations:
+                    - message: EnableGracefulRestart cannot be changed after creation
+                      rule: self == oldSelf
                 holdTime:
                   description: Requested BGP hold time, per RFC4271.
                   type: string
diff --git a/config/crd/bases/metallb.io_bgppeers.yaml b/config/crd/bases/metallb.io_bgppeers.yaml
index eda91d2c..4bc888cc 100644
--- a/config/crd/bases/metallb.io_bgppeers.yaml
+++ b/config/crd/bases/metallb.io_bgppeers.yaml
@@ -198,6 +198,16 @@ spec:
                 description: To set if the BGPPeer is multi-hops away. Needed for
                   FRR mode only.
                 type: boolean
+              enableGracefulRestart:
+                description: |-
+                  EnableGracefulRestart allows BGP peer to continue to forward data packets along
+                  known routes while the routing protocol information is being restored.
+                  This field is immutable because it requires restart of the BGP session
+                  Supported for FRR mode only.
+                type: boolean
+                x-kubernetes-validations:
+                - message: EnableGracefulRestart cannot be changed after creation
+                  rule: self == oldSelf
               holdTime:
                 description: Requested BGP hold time, per RFC4271.
                 type: string
diff --git a/config/manifests/metallb-frr-k8s-prometheus.yaml b/config/manifests/metallb-frr-k8s-prometheus.yaml
index 25d01aff..76b5e7d7 100644
--- a/config/manifests/metallb-frr-k8s-prometheus.yaml
+++ b/config/manifests/metallb-frr-k8s-prometheus.yaml
@@ -552,6 +552,16 @@ spec:
                 description: To set if the BGPPeer is multi-hops away. Needed for
                   FRR mode only.
                 type: boolean
+              enableGracefulRestart:
+                description: |-
+                  EnableGracefulRestart allows BGP peer to continue to forward data packets along
+                  known routes while the routing protocol information is being restored.
+                  This field is immutable because it requires restart of the BGP session
+                  Supported for FRR mode only.
+                type: boolean
+                x-kubernetes-validations:
+                - message: EnableGracefulRestart cannot be changed after creation
+                  rule: self == oldSelf
               holdTime:
                 description: Requested BGP hold time, per RFC4271.
                 type: string
diff --git a/config/manifests/metallb-frr-k8s.yaml b/config/manifests/metallb-frr-k8s.yaml
index f63c6b08..3f49b355 100644
--- a/config/manifests/metallb-frr-k8s.yaml
+++ b/config/manifests/metallb-frr-k8s.yaml
@@ -552,6 +552,16 @@ spec:
                 description: To set if the BGPPeer is multi-hops away. Needed for
                   FRR mode only.
                 type: boolean
+              enableGracefulRestart:
+                description: |-
+                  EnableGracefulRestart allows BGP peer to continue to forward data packets along
+                  known routes while the routing protocol information is being restored.
+                  This field is immutable because it requires restart of the BGP session
+                  Supported for FRR mode only.
+                type: boolean
+                x-kubernetes-validations:
+                - message: EnableGracefulRestart cannot be changed after creation
+                  rule: self == oldSelf
               holdTime:
                 description: Requested BGP hold time, per RFC4271.
                 type: string
diff --git a/config/manifests/metallb-frr-prometheus.yaml b/config/manifests/metallb-frr-prometheus.yaml
index 80ebdac3..a0a4577e 100644
--- a/config/manifests/metallb-frr-prometheus.yaml
+++ b/config/manifests/metallb-frr-prometheus.yaml
@@ -552,6 +552,16 @@ spec:
                 description: To set if the BGPPeer is multi-hops away. Needed for
                   FRR mode only.
                 type: boolean
+              enableGracefulRestart:
+                description: |-
+                  EnableGracefulRestart allows BGP peer to continue to forward data packets along
+                  known routes while the routing protocol information is being restored.
+                  This field is immutable because it requires restart of the BGP session
+                  Supported for FRR mode only.
+                type: boolean
+                x-kubernetes-validations:
+                - message: EnableGracefulRestart cannot be changed after creation
+                  rule: self == oldSelf
               holdTime:
                 description: Requested BGP hold time, per RFC4271.
                 type: string
diff --git a/config/manifests/metallb-frr.yaml b/config/manifests/metallb-frr.yaml
index 2fb272f3..8be0fad9 100644
--- a/config/manifests/metallb-frr.yaml
+++ b/config/manifests/metallb-frr.yaml
@@ -552,6 +552,16 @@ spec:
                 description: To set if the BGPPeer is multi-hops away. Needed for
                   FRR mode only.
                 type: boolean
+              enableGracefulRestart:
+                description: |-
+                  EnableGracefulRestart allows BGP peer to continue to forward data packets along
+                  known routes while the routing protocol information is being restored.
+                  This field is immutable because it requires restart of the BGP session
+                  Supported for FRR mode only.
+                type: boolean
+                x-kubernetes-validations:
+                - message: EnableGracefulRestart cannot be changed after creation
+                  rule: self == oldSelf
               holdTime:
                 description: Requested BGP hold time, per RFC4271.
                 type: string
diff --git a/config/manifests/metallb-native-prometheus.yaml b/config/manifests/metallb-native-prometheus.yaml
index 38396840..62955204 100644
--- a/config/manifests/metallb-native-prometheus.yaml
+++ b/config/manifests/metallb-native-prometheus.yaml
@@ -552,6 +552,16 @@ spec:
                 description: To set if the BGPPeer is multi-hops away. Needed for
                   FRR mode only.
                 type: boolean
+              enableGracefulRestart:
+                description: |-
+                  EnableGracefulRestart allows BGP peer to continue to forward data packets along
+                  known routes while the routing protocol information is being restored.
+                  This field is immutable because it requires restart of the BGP session
+                  Supported for FRR mode only.
+                type: boolean
+                x-kubernetes-validations:
+                - message: EnableGracefulRestart cannot be changed after creation
+                  rule: self == oldSelf
               holdTime:
                 description: Requested BGP hold time, per RFC4271.
                 type: string
diff --git a/config/manifests/metallb-native.yaml b/config/manifests/metallb-native.yaml
index 455501a8..0e0ee5bd 100644
--- a/config/manifests/metallb-native.yaml
+++ b/config/manifests/metallb-native.yaml
@@ -552,6 +552,16 @@ spec:
                 description: To set if the BGPPeer is multi-hops away. Needed for
                   FRR mode only.
                 type: boolean
+              enableGracefulRestart:
+                description: |-
+                  EnableGracefulRestart allows BGP peer to continue to forward data packets along
+                  known routes while the routing protocol information is being restored.
+                  This field is immutable because it requires restart of the BGP session
+                  Supported for FRR mode only.
+                type: boolean
+                x-kubernetes-validations:
+                - message: EnableGracefulRestart cannot be changed after creation
+                  rule: self == oldSelf
               holdTime:
                 description: Requested BGP hold time, per RFC4271.
                 type: string
diff --git a/internal/bgp/bgp.go b/internal/bgp/bgp.go
index 2dd65acd..86983553 100644
--- a/internal/bgp/bgp.go
+++ b/internal/bgp/bgp.go
@@ -63,22 +63,23 @@ type Session interface {
 }
 
 type SessionParameters struct {
-	PeerAddress   string
-	SourceAddress net.IP
-	MyASN         uint32
-	RouterID      net.IP
-	PeerASN       uint32
-	HoldTime      time.Duration
-	KeepAliveTime time.Duration
-	ConnectTime   *time.Duration
-	Password      string
-	PasswordRef   v1.SecretReference
-	CurrentNode   string
-	BFDProfile    string
-	EBGPMultiHop  bool
-	VRFName       string
-	SessionName   string
-	DisableMP     bool
+	PeerAddress     string
+	SourceAddress   net.IP
+	MyASN           uint32
+	RouterID        net.IP
+	PeerASN         uint32
+	HoldTime        time.Duration
+	KeepAliveTime   time.Duration
+	ConnectTime     *time.Duration
+	Password        string
+	PasswordRef     v1.SecretReference
+	CurrentNode     string
+	BFDProfile      string
+	GracefulRestart bool
+	EBGPMultiHop    bool
+	VRFName         string
+	SessionName     string
+	DisableMP       bool
 }
 type SessionManager interface {
 	NewSession(logger log.Logger, args SessionParameters) (Session, error)
diff --git a/internal/bgp/frr/config.go b/internal/bgp/frr/config.go
index 3697d2e2..e21fd899 100644
--- a/internal/bgp/frr/config.go
+++ b/internal/bgp/frr/config.go
@@ -76,6 +76,7 @@ type neighborConfig struct {
 	Password            string
 	Advertisements      []*advertisementConfig
 	BFDProfile          string
+	GracefulRestart     bool
 	EBGPMultiHop        bool
 	VRFName             string
 	HasV4Advertisements bool
diff --git a/internal/bgp/frr/frr.go b/internal/bgp/frr/frr.go
index f348d688..9934c502 100644
--- a/internal/bgp/frr/frr.go
+++ b/internal/bgp/frr/frr.go
@@ -272,20 +272,21 @@ func (sm *sessionManager) createConfig() (*frrConfig, error) {
 			}
 
 			neighbor = &neighborConfig{
-				Name:           neighborName,
-				IPFamily:       family,
-				ASN:            s.PeerASN,
-				Addr:           host,
-				Port:           uint16(portUint),
-				HoldTime:       uint64(s.HoldTime / time.Second),
-				KeepaliveTime:  uint64(s.KeepAliveTime / time.Second),
-				ConnectTime:    connectTime,
-				Password:       s.Password,
-				Advertisements: make([]*advertisementConfig, 0),
-				BFDProfile:     s.BFDProfile,
-				EBGPMultiHop:   s.EBGPMultiHop,
-				VRFName:        s.VRFName,
-				DisableMP:      s.DisableMP,
+				Name:            neighborName,
+				IPFamily:        family,
+				ASN:             s.PeerASN,
+				Addr:            host,
+				Port:            uint16(portUint),
+				HoldTime:        uint64(s.HoldTime / time.Second),
+				KeepaliveTime:   uint64(s.KeepAliveTime / time.Second),
+				ConnectTime:     connectTime,
+				Password:        s.Password,
+				Advertisements:  make([]*advertisementConfig, 0),
+				BFDProfile:      s.BFDProfile,
+				GracefulRestart: s.GracefulRestart,
+				EBGPMultiHop:    s.EBGPMultiHop,
+				VRFName:         s.VRFName,
+				DisableMP:       s.DisableMP,
 			}
 			if s.SourceAddress != nil {
 				neighbor.SrcAddr = s.SourceAddress.String()
diff --git a/internal/bgp/frr/templates/frr.tmpl b/internal/bgp/frr/templates/frr.tmpl
index f732de91..d8f67121 100644
--- a/internal/bgp/frr/templates/frr.tmpl
+++ b/internal/bgp/frr/templates/frr.tmpl
@@ -30,6 +30,7 @@ router bgp {{$r.MyASN}}{{ if $r.VRF }} vrf {{$r.VRF}}{{end}}
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 {{ if $r.RouterID }}
   bgp router-id {{$r.RouterID}}
 {{- end }}
diff --git a/internal/bgp/frr/templates/neighborsession.tmpl b/internal/bgp/frr/templates/neighborsession.tmpl
index 715964fa..da876526 100644
--- a/internal/bgp/frr/templates/neighborsession.tmpl
+++ b/internal/bgp/frr/templates/neighborsession.tmpl
@@ -16,10 +16,13 @@
   {{ if .neighbor.SrcAddr -}}
   neighbor {{.neighbor.Addr}} update-source {{.neighbor.SrcAddr}}
   {{- end }}
-{{- if ne .neighbor.BFDProfile ""}}
+  {{- if .neighbor.GracefulRestart }}
+  neighbor {{.neighbor.Addr}} graceful-restart
+  {{- end }}
+  {{- if ne .neighbor.BFDProfile "" }}
   neighbor {{.neighbor.Addr}} bfd profile {{.neighbor.BFDProfile}}
-{{- end }}
-{{- if  mustDisableConnectedCheck .neighbor.IPFamily .routerASN .neighbor.ASN .neighbor.EBGPMultiHop }}
+  {{- end }}
+  {{- if  mustDisableConnectedCheck .neighbor.IPFamily .routerASN .neighbor.ASN .neighbor.EBGPMultiHop }}
   neighbor {{.neighbor.Addr}} disable-connected-check
-{{- end }}
+  {{- end }}
 {{- end -}}
diff --git a/internal/bgp/frrk8s/frrk8s.go b/internal/bgp/frrk8s/frrk8s.go
index 0339f65a..c47960f2 100644
--- a/internal/bgp/frrk8s/frrk8s.go
+++ b/internal/bgp/frrk8s/frrk8s.go
@@ -258,7 +258,8 @@ func (sm *sessionManager) updateConfig() error {
 				KeepaliveTime: &metav1.Duration{Duration: s.KeepAliveTime},
 				ConnectTime:   connectTime,
 				BFDProfile:    s.BFDProfile,
-				EBGPMultiHop:  s.EBGPMultiHop,
+				//Graceful:      s.GracefulRestart,
+				EBGPMultiHop: s.EBGPMultiHop,
 				ToAdvertise: frrv1beta1.Advertise{
 					Allowed: frrv1beta1.AllowedOutPrefixes{
 						Prefixes: make([]string, 0),
diff --git a/internal/config/config.go b/internal/config/config.go
index 66420a45..43b143be 100644
--- a/internal/config/config.go
+++ b/internal/config/config.go
@@ -119,6 +119,8 @@ type Peer struct {
 	PasswordRef corev1.SecretReference
 	// The optional BFD profile to be used for this BGP session
 	BFDProfile string
+	// Optional EnableGracefulRestart enable BGP graceful restart functionality at the peer level.
+	EnableGracefulRestart bool
 	// Optional ebgp peer is multi-hops away.
 	EBGPMultiHop bool
 	// Optional name of the vrf to establish the session from
@@ -451,23 +453,24 @@ func peerFromCR(p metallbv1beta2.BGPPeer, passwordSecrets map[string]corev1.Secr
 	}
 
 	return &Peer{
-		Name:          p.Name,
-		MyASN:         p.Spec.MyASN,
-		ASN:           p.Spec.ASN,
-		Addr:          ip,
-		SrcAddr:       src,
-		Port:          p.Spec.Port,
-		HoldTime:      holdTime,
-		KeepaliveTime: keepaliveTime,
-		ConnectTime:   connectTime,
-		RouterID:      routerID,
-		NodeSelectors: nodeSels,
-		Password:      password,
-		PasswordRef:   p.Spec.PasswordSecret,
-		BFDProfile:    p.Spec.BFDProfile,
-		EBGPMultiHop:  p.Spec.EBGPMultiHop,
-		VRF:           p.Spec.VRFName,
-		DisableMP:     p.Spec.DisableMP,
+		Name:                  p.Name,
+		MyASN:                 p.Spec.MyASN,
+		ASN:                   p.Spec.ASN,
+		Addr:                  ip,
+		SrcAddr:               src,
+		Port:                  p.Spec.Port,
+		HoldTime:              holdTime,
+		KeepaliveTime:         keepaliveTime,
+		ConnectTime:           connectTime,
+		RouterID:              routerID,
+		NodeSelectors:         nodeSels,
+		Password:              password,
+		PasswordRef:           p.Spec.PasswordSecret,
+		BFDProfile:            p.Spec.BFDProfile,
+		EnableGracefulRestart: p.Spec.EnableGracefulRestart,
+		EBGPMultiHop:          p.Spec.EBGPMultiHop,
+		VRF:                   p.Spec.VRFName,
+		DisableMP:             p.Spec.DisableMP,
 	}, nil
 }
 
diff --git a/internal/config/validation.go b/internal/config/validation.go
index 8b0b085c..92621b7f 100644
--- a/internal/config/validation.go
+++ b/internal/config/validation.go
@@ -42,6 +42,9 @@ func DiscardFRROnly(c ClusterResources) error {
 		if p.Spec.ConnectTime != nil {
 			return fmt.Errorf("peer %s has connect time set on native bgp mode", p.Spec.Address)
 		}
+		if p.Spec.EnableGracefulRestart {
+			return fmt.Errorf("peer %s has EnableGracefulRestart flag set on native bgp mode", p.Spec.Address)
+		}
 		if p.Spec.DisableMP {
 			return fmt.Errorf("peer %s has disable MP flag set on native bgp mode", p.Spec.Address)
 		}
diff --git a/speaker/bgp_controller.go b/speaker/bgp_controller.go
index cfe933b0..2fa5d0fa 100644
--- a/speaker/bgp_controller.go
+++ b/speaker/bgp_controller.go
@@ -223,22 +223,23 @@ func (c *bgpController) syncPeers(l log.Logger) error {
 			}
 			s, err := c.sessionManager.NewSession(c.logger,
 				bgp.SessionParameters{
-					PeerAddress:   net.JoinHostPort(p.cfg.Addr.String(), strconv.Itoa(int(p.cfg.Port))),
-					SourceAddress: p.cfg.SrcAddr,
-					MyASN:         p.cfg.MyASN,
-					RouterID:      routerID,
-					PeerASN:       p.cfg.ASN,
-					HoldTime:      p.cfg.HoldTime,
-					KeepAliveTime: p.cfg.KeepaliveTime,
-					ConnectTime:   p.cfg.ConnectTime,
-					Password:      p.cfg.Password,
-					PasswordRef:   p.cfg.PasswordRef,
-					CurrentNode:   c.myNode,
-					BFDProfile:    p.cfg.BFDProfile,
-					EBGPMultiHop:  p.cfg.EBGPMultiHop,
-					SessionName:   p.cfg.Name,
-					VRFName:       p.cfg.VRF,
-					DisableMP:     p.cfg.DisableMP,
+					PeerAddress:     net.JoinHostPort(p.cfg.Addr.String(), strconv.Itoa(int(p.cfg.Port))),
+					SourceAddress:   p.cfg.SrcAddr,
+					MyASN:           p.cfg.MyASN,
+					RouterID:        routerID,
+					PeerASN:         p.cfg.ASN,
+					HoldTime:        p.cfg.HoldTime,
+					KeepAliveTime:   p.cfg.KeepaliveTime,
+					ConnectTime:     p.cfg.ConnectTime,
+					Password:        p.cfg.Password,
+					PasswordRef:     p.cfg.PasswordRef,
+					CurrentNode:     c.myNode,
+					BFDProfile:      p.cfg.BFDProfile,
+					GracefulRestart: p.cfg.EnableGracefulRestart,
+					EBGPMultiHop:    p.cfg.EBGPMultiHop,
+					SessionName:     p.cfg.Name,
+					VRFName:         p.cfg.VRF,
+					DisableMP:       p.cfg.DisableMP,
 				},
 			)
 
diff --git a/website/content/apis/_index.md b/website/content/apis/_index.md
index 83b15d9d..a8648fdf 100644
--- a/website/content/apis/_index.md
+++ b/website/content/apis/_index.md
@@ -289,6 +289,7 @@ _Appears in:_
 | `password` _string_ | Authentication password for routers enforcing TCP MD5 authenticated sessions |
 | `passwordSecret` _[SecretReference](https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#secretreference-v1-core)_ | passwordSecret is name of the authentication secret for BGP Peer. the secret must be of type "kubernetes.io/basic-auth", and created in the same namespace as the MetalLB deployment. The password is stored in the secret as the key "password". |
 | `bfdProfile` _string_ | The name of the BFD Profile to be used for the BFD session associated to the BGP session. If not set, the BFD session won't be set up. |
+| `enableGracefulRestart` _boolean_ | EnableGracefulRestart allows BGP peer to continue to forward data packets along known routes while the routing protocol information is being restored. This field is immutable because it requires restart of the BGP session Supported for FRR mode only. |
 | `ebgpMultiHop` _boolean_ | To set if the BGPPeer is multi-hops away. Needed for FRR mode only. |
 | `vrf` _string_ | To set if we want to peer with the BGPPeer using an interface belonging to a host vrf |
 | `disableMP` _boolean_ | To set if we want to disable MP BGP that will separate IPv4 and IPv6 route exchanges into distinct BGP sessions. |


Test patch:
diff --git a/e2etest/bgptests/bgp.go b/e2etest/bgptests/bgp.go
index fb458266..9895d457 100644
--- a/e2etest/bgptests/bgp.go
+++ b/e2etest/bgptests/bgp.go
@@ -20,6 +20,7 @@ package bgptests
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"strings"
 	"time"
@@ -58,6 +59,9 @@ const (
 	CommunityNoAdv        = "65535:65282" // 0xFFFFFF02: NO_ADVERTISE
 	CommunityGracefulShut = "65535:0"     // GRACEFUL_SHUTDOWN
 	SpeakerContainerName  = "speaker"
+
+	GracefulRestartEnabled  = true
+	GracefulRestartDisabled = false
 )
 
 var (
@@ -129,7 +133,6 @@ var _ = ginkgo.Describe("BGP", func() {
 			validateService(svc, allNodes.Items, c)
 		}
 	},
-		ginkgo.Entry("IPV4", ipfamily.IPv4, []string{v4PoolAddresses}, func(_ *corev1.Service) {}),
 		ginkgo.Entry("IPV4", ipfamily.IPv4, []string{v4PoolAddresses}, func(_ *corev1.Service) {}),
 		ginkgo.Entry("IPV6", ipfamily.IPv6, []string{v6PoolAddresses}, func(_ *corev1.Service) {}),
 		ginkgo.Entry("DUALSTACK", ipfamily.DualStack, []string{v4PoolAddresses, v6PoolAddresses},
@@ -147,6 +150,117 @@ var _ = ginkgo.Describe("BGP", func() {
 			}),
 	)
 
+	ginkgo.Describe("GracefulRestart, when speakers restart", func() {
+
+		ginkgo.AfterEach(func() {
+			for _, c := range FRRContainers {
+				c.NeighborConfig.GracefulRestart = false
+			}
+		})
+
+		assertDuringSpeakerRestart := func(gracefulRestart bool, pairingIPFamily ipfamily.Family, poolAddresses []string, tweak testservice.Tweak) {
+			_, svc := setupBGPService(cs, testNamespace, pairingIPFamily, poolAddresses,
+				FRRContainers, func(svc *corev1.Service) {
+					testservice.TrafficPolicyCluster(svc)
+					tweak(svc)
+				})
+			defer testservice.Delete(cs, svc)
+
+			allNodes, err := cs.CoreV1().Nodes().List(context.Background(), metav1.ListOptions{})
+			Expect(err).NotTo(HaveOccurred())
+			testservice.ValidateDesiredLB(svc)
+
+			for _, c := range FRRContainers {
+				validateService(svc, allNodes.Items, c)
+			}
+			err = metallb.RestartSpeakerPods(cs)
+			Expect(err).NotTo(HaveOccurred())
+
+			if gracefulRestart == GracefulRestartDisabled {
+				Eventually(func() error {
+					for _, c := range FRRContainers {
+						err := validateServiceNoWait(svc, allNodes.Items, c)
+						if errors.Is(err, ErrStaleRoute) {
+							Expect(err).NotTo(HaveOccurred(),
+								"a stale route cannot be observed if GR disabled")
+						}
+						if err != nil {
+							return err
+						}
+					}
+					return nil
+				}, 2*time.Minute, time.Second).Should(HaveOccurred(), "a downtime should be observed")
+				return
+			}
+
+			Eventually(func() error {
+				for _, c := range FRRContainers {
+					err := validateServiceNoWait(svc, allNodes.Items, c)
+					if errors.Is(err, ErrStaleRoute) {
+						continue // when GR, is normal to observe stale routes
+					}
+					Expect(err).NotTo(HaveOccurred(), "downtime was observed")
+				}
+
+				pods, err := metallb.SpeakerPods(cs)
+				if err != nil {
+					return err
+				}
+
+				for _, p := range pods {
+					if !k8s.PodIsReady(p) {
+						return fmt.Errorf("speaker pods are not ready")
+					}
+				}
+
+				return nil
+			}, 2*time.Minute, time.Second).ShouldNot(HaveOccurred(), "no downtime until speakers are ready")
+
+			for _, c := range FRRContainers {
+				validateService(svc, allNodes.Items, c)
+			}
+		}
+
+		ginkgo.Context("and when GR enabled", func() {
+
+			assertDuringSpeakerRestartWithGR := func(pairingIPFamily ipfamily.Family, poolAddresses []string, tweak testservice.Tweak) {
+				assertDuringSpeakerRestart(GracefulRestartEnabled, pairingIPFamily, poolAddresses, tweak)
+			}
+
+			ginkgo.BeforeEach(func() {
+				for _, c := range FRRContainers {
+					c.NeighborConfig.GracefulRestart = true
+				}
+			})
+
+			ginkgo.DescribeTable("dataplane should keep working", assertDuringSpeakerRestartWithGR,
+				ginkgo.Entry("FRR-MODE IPV4", ipfamily.IPv4, []string{v4PoolAddresses}, func(_ *corev1.Service) {}),
+				ginkgo.Entry("FRR-MODE IPV6", ipfamily.IPv6, []string{v6PoolAddresses}, func(_ *corev1.Service) {}),
+				ginkgo.Entry("FRR-MODE DUALSTACK", ipfamily.DualStack, []string{v4PoolAddresses, v6PoolAddresses},
+					func(svc *corev1.Service) { testservice.DualStack(svc) }),
+			)
+		})
+
+		ginkgo.Context("when GR disabled", func() {
+			assertDuringSpeakerRestartWithoutGR := func(pairingIPFamily ipfamily.Family, poolAddresses []string, tweak testservice.Tweak) {
+				assertDuringSpeakerRestart(GracefulRestartDisabled, pairingIPFamily, poolAddresses, tweak)
+			}
+
+			ginkgo.BeforeEach(func() {
+				for _, c := range FRRContainers {
+					c.NeighborConfig.GracefulRestart = false
+				}
+			})
+
+			ginkgo.DescribeTable("dataplane should have a downtime", assertDuringSpeakerRestartWithoutGR,
+				ginkgo.Entry("FRR-MODE IPV4", ipfamily.IPv4, []string{v4PoolAddresses}, func(_ *corev1.Service) {}),
+				ginkgo.Entry("FRR-MODE IPV6", ipfamily.IPv6, []string{v6PoolAddresses}, func(_ *corev1.Service) {}),
+				ginkgo.Entry("FRR-MODE DUALSTACK", ipfamily.DualStack, []string{v4PoolAddresses, v6PoolAddresses},
+					func(svc *corev1.Service) { testservice.DualStack(svc) }),
+			)
+		})
+	})
+
 	ginkgo.Describe("Service with ETP=cluster", func() {
 		ginkgo.It("IPV4 - should not be announced from a node with a NetworkUnavailable condition", func() {
 			allNodes, err := cs.CoreV1().Nodes().List(context.Background(), metav1.ListOptions{})
@@ -427,6 +541,33 @@ var _ = ginkgo.Describe("BGP", func() {
 		ginkgo.Entry("IPV4", ipfamily.IPv4),
 		ginkgo.Entry("IPV6", ipfamily.IPv6))
 
+	ginkgo.DescribeTable("FRR-MODE configure peers with GracefulRestart and validate external containers are paired with nodes", func(ipFamily ipfamily.Family) {
+		ginkgo.By("configure peer")
+
+		resources := config.Resources{
+			Peers: metallb.WithGracefulRestart(metallb.PeersForContainers(FRRContainers, ipFamily)),
+		}
+
+		err := ConfigUpdater.Update(resources)
+		Expect(err).NotTo(HaveOccurred())
+
+		for _, c := range FRRContainers {
+			err = frrcontainer.PairWithNodes(cs, c, ipFamily)
+			Expect(err).NotTo(HaveOccurred())
+		}
+
+		for _, c := range FRRContainers {
+			validateFRRPeeredWithAllNodes(cs, c, ipFamily)
+			neighbors, err := frr.NeighborsInfo(c)
+			Expect(err).NotTo(HaveOccurred())
+			for _, n := range neighbors {
+				Expect(n.GRInfo.RemoteGrMode).To(Equal("Restart"))
+			}
+		}
+	},
+		ginkgo.Entry("IPV4", ipfamily.IPv4),
+		ginkgo.Entry("IPV6", ipfamily.IPv6))
+
 	ginkgo.DescribeTable("validate external containers are paired with nodes", func(ipFamily ipfamily.Family) {
 		ginkgo.By("configure peer")
 
diff --git a/e2etest/bgptests/validate.go b/e2etest/bgptests/validate.go
index d75623a8..85628fed 100644
--- a/e2etest/bgptests/validate.go
+++ b/e2etest/bgptests/validate.go
@@ -4,6 +4,7 @@ package bgptests
 
 import (
 	"context"
+	"errors"
 	"fmt"
 	"net"
 	"strconv"
@@ -26,6 +27,8 @@ import (
 	clientset "k8s.io/client-go/kubernetes"
 )
 
+var ErrStaleRoute = errors.New("stale route")
+
 func validateFRRPeeredWithAllNodes(cs clientset.Interface, c *frrcontainer.FRR, ipFamily ipfamily.Family) {
 	allNodes, err := cs.CoreV1().Nodes().List(context.Background(), metav1.ListOptions{})
 	Expect(err).NotTo(HaveOccurred())
@@ -118,6 +121,21 @@ func validateServiceNoWait(svc *corev1.Service, nodes []corev1.Node, c *frrconta
 				return err
 			}
 		}
+
+		var serr error
+		for k, v := range frrRoutesV4 {
+			if v.Stale {
+				serr = errors.Join(serr, errors.New(fmt.Sprintf("%s -%v", k, v)))
+			}
+		}
+		for k, v := range frrRoutesV6 {
+			if v.Stale {
+				serr = errors.Join(serr, errors.New(fmt.Sprintf("%s -%v", k, v)))
+			}
+		}
+		if serr != nil {
+			return errors.Join(ErrStaleRoute, serr)
+		}
 	}
 	return nil
 }
diff --git a/e2etest/pkg/frr/config/config.go b/e2etest/pkg/frr/config/config.go
index 075b4b93..df11432f 100644
--- a/e2etest/pkg/frr/config/config.go
+++ b/e2etest/pkg/frr/config/config.go
@@ -95,13 +95,14 @@ type RouterConfig struct {
 }
 
 type NeighborConfig struct {
-	ASN           uint32
-	Addr          string
-	Password      string
-	BFDEnabled    bool
-	ToAdvertiseV4 []string
-	ToAdvertiseV6 []string
-	MultiHop      bool
+	ASN             uint32
+	Addr            string
+	Password        string
+	BFDEnabled      bool
+	ToAdvertiseV4   []string
+	ToAdvertiseV6   []string
+	MultiHop        bool
+	GracefulRestart bool
 }
 
 type MultiProtocol bool
diff --git a/e2etest/pkg/frr/parse.go b/e2etest/pkg/frr/parse.go
index f7ef6572..9aeea022 100644
--- a/e2etest/pkg/frr/parse.go
+++ b/e2etest/pkg/frr/parse.go
@@ -21,6 +21,7 @@ type Neighbor struct {
 	PrefixSent              int
 	Port                    int
 	RemoteRouterID          string
+	GRInfo                  GracefulRestartInfo
 	MsgStats                MessageStats
 	ConfiguredHoldTime      int
 	ConfiguredKeepAliveTime int
@@ -33,27 +34,57 @@ type Route struct {
 	NextHops    []net.IP
 	LocalPref   uint32
 	Origin      string
+	Stale       bool
 }
 
 const bgpConnected = "Established"
 
 type FRRNeighbor struct {
-	RemoteAs                     int          `json:"remoteAs"`
-	LocalAs                      int          `json:"localAs"`
-	RemoteRouterID               string       `json:"remoteRouterId"`
-	BgpVersion                   int          `json:"bgpVersion"`
-	BgpState                     string       `json:"bgpState"`
-	PortForeign                  int          `json:"portForeign"`
-	MsgStats                     MessageStats `json:"messageStats"`
-	VRFName                      string       `json:"vrf"`
-	ConfiguredHoldTimeMSecs      int          `json:"bgpTimerConfiguredHoldTimeMsecs"`
-	ConfiguredKeepAliveTimeMSecs int          `json:"bgpTimerConfiguredKeepAliveIntervalMsecs"`
-	ConnectRetryTimer            int          `json:"connectRetryTimer"`
+	RemoteAs                     int                 `json:"remoteAs"`
+	LocalAs                      int                 `json:"localAs"`
+	RemoteRouterID               string              `json:"remoteRouterId"`
+	BgpVersion                   int                 `json:"bgpVersion"`
+	BgpState                     string              `json:"bgpState"`
+	PortForeign                  int                 `json:"portForeign"`
+	MsgStats                     MessageStats        `json:"messageStats"`
+	GRInfo                       GracefulRestartInfo `json:"gracefulRestartInfo"`
+	VRFName                      string              `json:"vrf"`
+	ConfiguredHoldTimeMSecs      int                 `json:"bgpTimerConfiguredHoldTimeMsecs"`
+	ConfiguredKeepAliveTimeMSecs int                 `json:"bgpTimerConfiguredKeepAliveIntervalMsecs"`
+	ConnectRetryTimer            int                 `json:"connectRetryTimer"`
 	AddressFamilyInfo            map[string]struct {
 		SentPrefixCounter int `json:"sentPrefixCounter"`
 	} `json:"addressFamilyInfo"`
 }
 
+type GracefulRestartInfo struct {
+	EndOfRibSend struct {
+		Ipv4Unicast bool `json:"ipv4Unicast"`
+	} `json:"endOfRibSend"`
+	EndOfRibRecv struct {
+		Ipv4Unicast bool `json:"ipv4Unicast"`
+	} `json:"endOfRibRecv"`
+	LocalGrMode  string `json:"localGrMode"`
+	RemoteGrMode string `json:"remoteGrMode"`
+	RBit         bool   `json:"rBit"`
+	NBit         bool   `json:"nBit"`
+	Timers       struct {
+		ConfiguredRestartTimer int `json:"configuredRestartTimer"`
+		ReceivedRestartTimer   int `json:"receivedRestartTimer"`
+	} `json:"timers"`
+	Ipv4Unicast struct {
+		FBit           bool `json:"fBit"`
+		EndOfRibStatus struct {
+			EndOfRibSend            bool `json:"endOfRibSend"`
+			EndOfRibSentAfterUpdate bool `json:"endOfRibSentAfterUpdate"`
+			EndOfRibRecv            bool `json:"endOfRibRecv"`
+		} `json:"endOfRibStatus"`
+		Timers struct {
+			StalePathTimer int `json:"stalePathTimer"`
+		} `json:"timers"`
+	} `json:"ipv4Unicast"`
+}
+
 type MessageStats struct {
 	OpensSent          int `json:"opensSent"`
 	OpensReceived      int `json:"opensRecv"`
@@ -72,6 +103,7 @@ type IPInfo struct {
 }
 
 type FRRRoute struct {
+	Stale     bool   `json:"stale"`
 	Valid     bool   `json:"valid"`
 	PeerID    string `json:"peerId"`
 	LocalPref uint32 `json:"locPrf"`
@@ -185,6 +217,7 @@ func ParseNeighbours(vtyshRes string) ([]*Neighbor, error) {
 			Port:                    n.PortForeign,
 			RemoteRouterID:          n.RemoteRouterID,
 			MsgStats:                n.MsgStats,
+			GRInfo:                  n.GRInfo,
 			ConfiguredKeepAliveTime: n.ConfiguredKeepAliveTimeMSecs,
 			ConfiguredHoldTime:      n.ConfiguredHoldTimeMSecs,
 			ConfiguredConnectTime:   n.ConnectRetryTimer,
@@ -217,6 +250,7 @@ func ParseRoutes(vtyshRes string) (map[string]Route, error) {
 		for _, n := range frrRoutes {
 			r.LocalPref = n.LocalPref
 			r.Origin = n.Origin
+			r.Stale = n.Stale
 		out:
 			for _, h := range n.Nexthops {
 				ip := net.ParseIP(h.IP)
diff --git a/e2etest/pkg/frr/provider/provider.go b/e2etest/pkg/frr/provider/provider.go
index 42dbbd3d..8f1bd5b4 100644
--- a/e2etest/pkg/frr/provider/provider.go
+++ b/e2etest/pkg/frr/provider/provider.go
@@ -22,7 +22,7 @@ type Provider interface {
 }
 
 type frrModeProvider struct {
-	speakers map[string]*corev1.Pod
+	cl *clientset.Clientset
 }
 
 // NewFRRMode returns a provider for a deployment using "frr" as its BGP mode.
@@ -33,7 +33,11 @@ func NewFRRMode(r *rest.Config) (Provider, error) {
 		return nil, err
 	}
 
-	speakerPods, err := metallb.SpeakerPods(cl)
+	return frrModeProvider{cl: cl}, nil
+}
+
+func (f frrModeProvider) FRRExecutorFor(ns, name string) (executor.Executor, error) {
+	speakerPods, err := metallb.SpeakerPods(f.cl)
 	if err != nil {
 		return nil, err
 	}
@@ -43,22 +47,28 @@ func NewFRRMode(r *rest.Config) (Provider, error) {
 		speakers[p.Name] = p
 	}
 
-	return frrModeProvider{speakers: speakers}, nil
-}
-
-func (f frrModeProvider) FRRExecutorFor(ns, name string) (executor.Executor, error) {
-	_, ok := f.speakers[name]
+	_, ok := speakers[name]
 	if !ok {
-		return nil, fmt.Errorf("speakers %s/%s not found in known speakers %v", ns, name, f.speakers)
+		return nil, fmt.Errorf("speakers %s/%s not found in known speakers %s", ns, name, speakers)
 	}
 
 	return executor.ForPod(ns, name, "frr"), nil
 }
 
 func (f frrModeProvider) BGPMetricsPodFor(ns, name string) (*corev1.Pod, string, error) {
-	p, ok := f.speakers[name]
+	speakerPods, err := metallb.SpeakerPods(f.cl)
+	if err != nil {
+		return nil, "", err
+	}
+
+	speakers := map[string]*corev1.Pod{}
+	for _, p := range speakerPods {
+		speakers[p.Name] = p
+	}
+
+	p, ok := speakers[name]
 	if !ok {
-		return nil, "", fmt.Errorf("speakers %s/%s not found in in known speakers %v", ns, name, f.speakers)
+		return nil, "", fmt.Errorf("speakers %s/%s not found in in known speakers %v", ns, name, speakers)
 	}
 
 	return p, "metallb", nil
diff --git a/e2etest/pkg/jigservice/jig.go b/e2etest/pkg/jigservice/jig.go
index 98a1c8ef..84f14a17 100644
--- a/e2etest/pkg/jigservice/jig.go
+++ b/e2etest/pkg/jigservice/jig.go
@@ -415,7 +415,7 @@ func (j *TestJig) UpdateService(ctx context.Context, update func(*v1.Service)) (
 
 // WaitForLoadBalancer waits the given service to have a LoadBalancer, or returns an error after the given timeout
 func (j *TestJig) WaitForLoadBalancer(ctx context.Context, timeout time.Duration) (*v1.Service, error) {
-	ginkgo.GinkgoWriter.Printf("Waiting up to %v for service %q to have a LoadBalancer", timeout, j.Name)
+	ginkgo.GinkgoWriter.Printf("\tWaiting up to %v for service %q to have a LoadBalancer\n", timeout, j.Name)
 	service, err := j.waitForCondition(ctx, timeout, "have a load balancer", func(svc *v1.Service) bool {
 		return len(svc.Status.LoadBalancer.Ingress) > 0
 	})
@@ -550,7 +550,7 @@ func (j *TestJig) waitForPodsCreated(ctx context.Context, replicas int) ([]strin
 	timeout := 2 * time.Minute
 	// List the pods, making sure we observe all the replicas.
 	label := labels.SelectorFromSet(labels.Set(j.Labels))
-	ginkgo.GinkgoWriter.Printf("Waiting up to %v for %d pods to be created", timeout, replicas)
+	ginkgo.GinkgoWriter.Printf("\tWaiting up to %v for %d pods to be created\n", timeout, replicas)
 	for start := time.Now(); time.Since(start) < timeout && ctx.Err() == nil; time.Sleep(2 * time.Second) {
 		options := metav1.ListOptions{LabelSelector: label.String()}
 		pods, err := j.Client.CoreV1().Pods(j.Namespace).List(ctx, options)
@@ -566,10 +566,10 @@ func (j *TestJig) waitForPodsCreated(ctx context.Context, replicas int) ([]strin
 			found = append(found, pod.Name)
 		}
 		if len(found) == replicas {
-			ginkgo.GinkgoWriter.Printf("Found all %d pods", replicas)
+			ginkgo.GinkgoWriter.Printf("\t\tFound all %d pods\n", replicas)
 			return found, nil
 		}
-		ginkgo.GinkgoWriter.Printf("Found %d/%d pods - will retry", len(found), replicas)
+		ginkgo.GinkgoWriter.Printf("\t\tFound %d/%d pods - will retry\n", len(found), replicas)
 	}
 	return nil, fmt.Errorf("timeout waiting for %d pods to be created", replicas)
 }
diff --git a/e2etest/pkg/k8s/pods.go b/e2etest/pkg/k8s/pods.go
index 4006b233..49edd519 100644
--- a/e2etest/pkg/k8s/pods.go
+++ b/e2etest/pkg/k8s/pods.go
@@ -38,9 +38,9 @@ func PodLogs(cs clientset.Interface, pod *corev1.Pod, podLogOpts corev1.PodLogOp
 	return str, nil
 }
 
-// PodIsReady returns the given pod's PodReady condition.
+// PodIsReady returns the given pod's PodReady and ContainersReady condition.
 func PodIsReady(p *corev1.Pod) bool {
-	return podConditionStatus(p, corev1.PodReady) == corev1.ConditionTrue
+	return podConditionStatus(p, corev1.PodReady) == corev1.ConditionTrue && podConditionStatus(p, corev1.ContainersReady) == corev1.ConditionTrue
 }
 
 // podConditionStatus returns the status of the condition for a given pod.
diff --git a/e2etest/pkg/metallb/config.go b/e2etest/pkg/metallb/config.go
index 81e691ac..917141e7 100644
--- a/e2etest/pkg/metallb/config.go
+++ b/e2etest/pkg/metallb/config.go
@@ -36,6 +36,10 @@ func PeersForContainers(containers []*frrcontainer.FRR, ipFamily ipfamily.Family
 		if i > 0 {
 			holdTime = time.Duration(i) * 180 * time.Second
 		}
+		gracefulRestart := false
+		if c.NeighborConfig.GracefulRestart {
+			gracefulRestart = true
+		}
 		ebgpMultihop := false
 		if c.NeighborConfig.MultiHop && c.NeighborConfig.ASN != c.RouterConfig.ASN {
 			ebgpMultihop = true
@@ -46,14 +50,15 @@ func PeersForContainers(containers []*frrcontainer.FRR, ipFamily ipfamily.Family
 					Name: c.Name + fmt.Sprint(i), // Otherwise the peers will override
 				},
 				Spec: metallbv1beta2.BGPPeerSpec{
-					Address:      address,
-					ASN:          c.RouterConfig.ASN,
-					MyASN:        c.NeighborConfig.ASN,
-					Port:         c.RouterConfig.BGPPort,
-					Password:     c.RouterConfig.Password,
-					HoldTime:     metav1.Duration{Duration: holdTime},
-					EBGPMultiHop: ebgpMultihop,
-					VRFName:      c.RouterConfig.VRF,
+					Address:               address,
+					ASN:                   c.RouterConfig.ASN,
+					MyASN:                 c.NeighborConfig.ASN,
+					Port:                  c.RouterConfig.BGPPort,
+					Password:              c.RouterConfig.Password,
+					HoldTime:              metav1.Duration{Duration: holdTime},
+					EnableGracefulRestart: gracefulRestart,
+					EBGPMultiHop:          ebgpMultihop,
+					VRFName:               c.RouterConfig.VRF,
 				}}
 			for _, f := range tweak {
 				f(&peer)
@@ -80,6 +85,14 @@ func WithRouterID(peers []metallbv1beta2.BGPPeer, routerID string) []metallbv1be
 	return peers
 }
 
+// WithGracefulRestart sets the GR to true to the peers.
+func WithGracefulRestart(peers []metallbv1beta2.BGPPeer) []metallbv1beta2.BGPPeer {
+	for i := range peers {
+		peers[i].Spec.EnableGracefulRestart = true
+	}
+	return peers
+}
+
 func BGPPeerSecretReferences(containers []*frrcontainer.FRR) map[string]corev1.Secret {
 	secretMap := make(map[string]corev1.Secret)
 	for _, c := range containers {
diff --git a/e2etest/pkg/metallb/metallb.go b/e2etest/pkg/metallb/metallb.go
index 878c3a36..9f917670 100644
--- a/e2etest/pkg/metallb/metallb.go
+++ b/e2etest/pkg/metallb/metallb.go
@@ -6,6 +6,7 @@ import (
 	"context"
 	"fmt"
 	"os"
+	"slices"
 	"time"
 
 	"errors"
@@ -33,6 +34,34 @@ func init() {
 	}
 }
 
+func RestartSpeakerPods(cs clientset.Interface) error {
+	pods, err := SpeakerPods(cs)
+	if err != nil {
+		return err
+	}
+	oldNames := []string{}
+	for _, p := range pods {
+		oldNames = append(oldNames, p.Name)
+		err := cs.CoreV1().Pods(Namespace).Delete(context.Background(), p.Name, metav1.DeleteOptions{})
+		if err != nil {
+			return err
+		}
+	}
+
+	return wait.PollUntilContextTimeout(context.Background(), 100*time.Millisecond, 10*time.Second, false, func(context.Context) (bool, error) {
+		npods, err := SpeakerPods(cs)
+		if err != nil {
+			return false, err
+		}
+		for _, p := range npods {
+			if slices.Contains(oldNames, p.Name) {
+				return false, nil
+			}
+		}
+		return true, nil
+	})
+}
+
 // SpeakerPods returns the set of pods running the speakers.
 func SpeakerPods(cs clientset.Interface) ([]*corev1.Pod, error) {
 	speakers, err := cs.CoreV1().Pods(Namespace).List(context.Background(), metav1.ListOptions{
diff --git a/e2etest/pkg/wget/wget.go b/e2etest/pkg/wget/wget.go
index d8fc1bd2..35da1e1a 100644
--- a/e2etest/pkg/wget/wget.go
+++ b/e2etest/pkg/wget/wget.go
@@ -25,7 +25,7 @@ func Do(address string, exc executor.Executor) error {
 
 	// Retry loop to handle wget NetworkFailure errors
 	for {
-		out, err = exc.Exec("wget", "-O-", "-q", address, "-T", "60")
+		out, err = exc.Exec("wget", "-O-", "-q", address, "-T", "5")
 		if exitErr, ok := err.(*exec.ExitError); err != nil && ok {
 			code = exitErr.ExitCode()
 		} else {
diff --git a/internal/bgp/frr/frr_test.go b/internal/bgp/frr/frr_test.go
index 649fd7a5..fff97b79 100644
--- a/internal/bgp/frr/frr_test.go
+++ b/internal/bgp/frr/frr_test.go
@@ -278,6 +278,31 @@ func TestSingleSessionClose(t *testing.T) {
 	session.Close()
 	testCheckConfigFile(t)
 }
+
+func TestSingleSessionWithGracefulRestart(t *testing.T) {
+	testSetup(t)
+
+	l := log.NewNopLogger()
+	sessionManager := mockNewSessionManager(l, logging.LevelInfo)
+	defer close(sessionManager.reloadConfig)
+	session, err := sessionManager.NewSession(l,
+		bgp.SessionParameters{
+			PeerAddress:     "10.2.2.254:179",
+			SourceAddress:   net.ParseIP("10.1.1.254"),
+			MyASN:           102,
+			RouterID:        net.ParseIP("10.1.1.254"),
+			PeerASN:         100,
+			GracefulRestart: true,
+			SessionName:     "test-peer"})
+
+	if err != nil {
+		t.Fatalf("Could not create session: %s", err)
+	}
+	defer session.Close()
+
+	testCheckConfigFile(t)
+}
+
 func TestTwoSessions(t *testing.T) {
 	testSetup(t)
 
diff --git a/internal/bgp/frr/testdata/TestBFDWithSession.golden b/internal/bgp/frr/testdata/TestBFDWithSession.golden
index 7e534f1c..da0ba0dc 100644
--- a/internal/bgp/frr/testdata/TestBFDWithSession.golden
+++ b/internal/bgp/frr/testdata/TestBFDWithSession.golden
@@ -20,6 +20,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestIPv4AndIPv6SessionsDisableMP.golden b/internal/bgp/frr/testdata/TestIPv4AndIPv6SessionsDisableMP.golden
index b9190a9f..e1b56328 100644
--- a/internal/bgp/frr/testdata/TestIPv4AndIPv6SessionsDisableMP.golden
+++ b/internal/bgp/frr/testdata/TestIPv4AndIPv6SessionsDisableMP.golden
@@ -32,6 +32,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10:2:2::254 remote-as 200
@@ -50,6 +51,7 @@ router bgp 300
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.3.3.254
   neighbor 10.4.4.255 remote-as 400
diff --git a/internal/bgp/frr/testdata/TestLargeCommunities.golden b/internal/bgp/frr/testdata/TestLargeCommunities.golden
index e145ba4c..89dd7b2e 100644
--- a/internal/bgp/frr/testdata/TestLargeCommunities.golden
+++ b/internal/bgp/frr/testdata/TestLargeCommunities.golden
@@ -43,6 +43,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestManyAdvertisementsSameCommunity.golden b/internal/bgp/frr/testdata/TestManyAdvertisementsSameCommunity.golden
index 93a8f5be..c0b132e3 100644
--- a/internal/bgp/frr/testdata/TestManyAdvertisementsSameCommunity.golden
+++ b/internal/bgp/frr/testdata/TestManyAdvertisementsSameCommunity.golden
@@ -73,6 +73,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestSingleAdvertisement.golden b/internal/bgp/frr/testdata/TestSingleAdvertisement.golden
index a4142cdb..6a70caaf 100644
--- a/internal/bgp/frr/testdata/TestSingleAdvertisement.golden
+++ b/internal/bgp/frr/testdata/TestSingleAdvertisement.golden
@@ -38,6 +38,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestSingleAdvertisementChange.golden b/internal/bgp/frr/testdata/TestSingleAdvertisementChange.golden
index f1559c98..f4868fa7 100644
--- a/internal/bgp/frr/testdata/TestSingleAdvertisementChange.golden
+++ b/internal/bgp/frr/testdata/TestSingleAdvertisementChange.golden
@@ -23,6 +23,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestSingleAdvertisementChangeVRF.golden b/internal/bgp/frr/testdata/TestSingleAdvertisementChangeVRF.golden
index b8d3f87d..cc00cc0f 100644
--- a/internal/bgp/frr/testdata/TestSingleAdvertisementChangeVRF.golden
+++ b/internal/bgp/frr/testdata/TestSingleAdvertisementChangeVRF.golden
@@ -23,6 +23,7 @@ router bgp 100 vrf red
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestSingleAdvertisementNoRouterID.golden b/internal/bgp/frr/testdata/TestSingleAdvertisementNoRouterID.golden
index f73ff571..42f5a429 100644
--- a/internal/bgp/frr/testdata/TestSingleAdvertisementNoRouterID.golden
+++ b/internal/bgp/frr/testdata/TestSingleAdvertisementNoRouterID.golden
@@ -23,6 +23,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   neighbor 10.2.2.254 remote-as 200
   neighbor 10.2.2.254 ebgp-multihop
diff --git a/internal/bgp/frr/testdata/TestSingleAdvertisementStop.golden b/internal/bgp/frr/testdata/TestSingleAdvertisementStop.golden
index 4c19a659..b752ef2f 100644
--- a/internal/bgp/frr/testdata/TestSingleAdvertisementStop.golden
+++ b/internal/bgp/frr/testdata/TestSingleAdvertisementStop.golden
@@ -20,6 +20,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestSingleAdvertisementVRF.golden b/internal/bgp/frr/testdata/TestSingleAdvertisementVRF.golden
index a0872797..33b0cdcb 100644
--- a/internal/bgp/frr/testdata/TestSingleAdvertisementVRF.golden
+++ b/internal/bgp/frr/testdata/TestSingleAdvertisementVRF.golden
@@ -38,6 +38,7 @@ router bgp 100 vrf red
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestSingleEBGPSessionMultiHop.golden b/internal/bgp/frr/testdata/TestSingleEBGPSessionMultiHop.golden
index 4c19a659..b752ef2f 100644
--- a/internal/bgp/frr/testdata/TestSingleEBGPSessionMultiHop.golden
+++ b/internal/bgp/frr/testdata/TestSingleEBGPSessionMultiHop.golden
@@ -20,6 +20,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestSingleEBGPSessionOneHop.golden b/internal/bgp/frr/testdata/TestSingleEBGPSessionOneHop.golden
index 61b39cb5..632381e8 100644
--- a/internal/bgp/frr/testdata/TestSingleEBGPSessionOneHop.golden
+++ b/internal/bgp/frr/testdata/TestSingleEBGPSessionOneHop.golden
@@ -20,6 +20,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 127.0.0.2 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestSingleIBGPSession.golden b/internal/bgp/frr/testdata/TestSingleIBGPSession.golden
index 98b23ed3..2c740f2e 100644
--- a/internal/bgp/frr/testdata/TestSingleIBGPSession.golden
+++ b/internal/bgp/frr/testdata/TestSingleIBGPSession.golden
@@ -20,6 +20,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 100
diff --git a/internal/bgp/frr/testdata/TestSingleIPv6EBGPSessionOneHop.golden b/internal/bgp/frr/testdata/TestSingleIPv6EBGPSessionOneHop.golden
index 75df6054..a56cb428 100644
--- a/internal/bgp/frr/testdata/TestSingleIPv6EBGPSessionOneHop.golden
+++ b/internal/bgp/frr/testdata/TestSingleIPv6EBGPSessionOneHop.golden
@@ -20,6 +20,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 127:0:0::2 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestSingleIPv6IBGPSession.golden b/internal/bgp/frr/testdata/TestSingleIPv6IBGPSession.golden
index 6f863306..4ba27170 100644
--- a/internal/bgp/frr/testdata/TestSingleIPv6IBGPSession.golden
+++ b/internal/bgp/frr/testdata/TestSingleIPv6IBGPSession.golden
@@ -20,6 +20,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10:2:2::254 remote-as 100
diff --git a/internal/bgp/frr/testdata/TestSingleSessionExtras.golden b/internal/bgp/frr/testdata/TestSingleSessionExtras.golden
index 0f03cfc7..976a521b 100644
--- a/internal/bgp/frr/testdata/TestSingleSessionExtras.golden
+++ b/internal/bgp/frr/testdata/TestSingleSessionExtras.golden
@@ -20,6 +20,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 127.0.0.2 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestSingleSessionWithGracefulRestart.golden b/internal/bgp/frr/testdata/TestSingleSessionWithGracefulRestart.golden
new file mode 100644
index 00000000..5c80852c
--- /dev/null
+++ b/internal/bgp/frr/testdata/TestSingleSessionWithGracefulRestart.golden
@@ -0,0 +1,43 @@
+log file /etc/frr/frr.log informational
+log timestamp precision 3
+hostname dummyhostname
+ip nht resolve-via-default
+ipv6 nht resolve-via-default
+route-map 10.2.2.254-in deny 20
+
+
+
+
+ip prefix-list 10.2.2.254-pl-ipv4 seq 1 deny any
+ipv6 prefix-list 10.2.2.254-pl-ipv4 seq 2 deny any
+
+route-map 10.2.2.254-out permit 1
+  match ip address prefix-list 10.2.2.254-pl-ipv4
+route-map 10.2.2.254-out permit 2
+  match ipv6 address prefix-list 10.2.2.254-pl-ipv4
+
+router bgp 102
+  no bgp ebgp-requires-policy
+  no bgp network import-check
+  no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
+
+  bgp router-id 10.1.1.254
+  neighbor 10.2.2.254 remote-as 100
+  neighbor 10.2.2.254 port 179
+  neighbor 10.2.2.254 timers 0 0
+  
+  neighbor 10.2.2.254 update-source 10.1.1.254
+  neighbor 10.2.2.254 graceful-restart
+
+  address-family ipv4 unicast
+    neighbor 10.2.2.254 activate
+    neighbor 10.2.2.254 route-map 10.2.2.254-in in
+    neighbor 10.2.2.254 route-map 10.2.2.254-out out
+  exit-address-family
+  address-family ipv6 unicast
+    neighbor 10.2.2.254 activate
+    neighbor 10.2.2.254 route-map 10.2.2.254-in in
+    neighbor 10.2.2.254 route-map 10.2.2.254-out out
+  exit-address-family
+
diff --git a/internal/bgp/frr/testdata/TestSingleVRFIBGPSession.golden b/internal/bgp/frr/testdata/TestSingleVRFIBGPSession.golden
index d0d9736f..55fee71f 100644
--- a/internal/bgp/frr/testdata/TestSingleVRFIBGPSession.golden
+++ b/internal/bgp/frr/testdata/TestSingleVRFIBGPSession.golden
@@ -20,6 +20,7 @@ router bgp 100 vrf red
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 100
diff --git a/internal/bgp/frr/testdata/TestTwoAdvertisements.golden b/internal/bgp/frr/testdata/TestTwoAdvertisements.golden
index 63e6baba..d2493f3c 100644
--- a/internal/bgp/frr/testdata/TestTwoAdvertisements.golden
+++ b/internal/bgp/frr/testdata/TestTwoAdvertisements.golden
@@ -32,6 +32,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestTwoAdvertisementsDuplicate.golden b/internal/bgp/frr/testdata/TestTwoAdvertisementsDuplicate.golden
index cc17ec86..6ebca453 100644
--- a/internal/bgp/frr/testdata/TestTwoAdvertisementsDuplicate.golden
+++ b/internal/bgp/frr/testdata/TestTwoAdvertisementsDuplicate.golden
@@ -23,6 +23,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestTwoAdvertisementsTwoSessions.golden b/internal/bgp/frr/testdata/TestTwoAdvertisementsTwoSessions.golden
index d18ef7f6..396f7bf8 100644
--- a/internal/bgp/frr/testdata/TestTwoAdvertisementsTwoSessions.golden
+++ b/internal/bgp/frr/testdata/TestTwoAdvertisementsTwoSessions.golden
@@ -68,6 +68,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestTwoAdvertisementsTwoSessionsOneVRF.golden b/internal/bgp/frr/testdata/TestTwoAdvertisementsTwoSessionsOneVRF.golden
index f1d6d931..f02f714b 100644
--- a/internal/bgp/frr/testdata/TestTwoAdvertisementsTwoSessionsOneVRF.golden
+++ b/internal/bgp/frr/testdata/TestTwoAdvertisementsTwoSessionsOneVRF.golden
@@ -68,6 +68,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
@@ -95,6 +96,7 @@ router bgp 100 vrf red
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.255 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestTwoAdvertisementsVRF.golden b/internal/bgp/frr/testdata/TestTwoAdvertisementsVRF.golden
index 323e0932..9e6a9ea2 100644
--- a/internal/bgp/frr/testdata/TestTwoAdvertisementsVRF.golden
+++ b/internal/bgp/frr/testdata/TestTwoAdvertisementsVRF.golden
@@ -32,6 +32,7 @@ router bgp 100 vrf red
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestTwoIPv6Sessions.golden b/internal/bgp/frr/testdata/TestTwoIPv6Sessions.golden
index 605e4645..befba677 100644
--- a/internal/bgp/frr/testdata/TestTwoIPv6Sessions.golden
+++ b/internal/bgp/frr/testdata/TestTwoIPv6Sessions.golden
@@ -32,6 +32,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10:2:2::254 remote-as 200
@@ -55,6 +56,7 @@ router bgp 300
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.3.3.254
   neighbor 10:4:4::255 remote-as 400
diff --git a/internal/bgp/frr/testdata/TestTwoSessions.golden b/internal/bgp/frr/testdata/TestTwoSessions.golden
index 19462a6d..05192f8c 100644
--- a/internal/bgp/frr/testdata/TestTwoSessions.golden
+++ b/internal/bgp/frr/testdata/TestTwoSessions.golden
@@ -32,6 +32,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
@@ -56,6 +57,7 @@ router bgp 300
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.3.3.254
   neighbor 10.4.4.255 remote-as 400
diff --git a/internal/bgp/frr/testdata/TestTwoSessionsDuplicate.golden b/internal/bgp/frr/testdata/TestTwoSessionsDuplicate.golden
index 4c19a659..b752ef2f 100644
--- a/internal/bgp/frr/testdata/TestTwoSessionsDuplicate.golden
+++ b/internal/bgp/frr/testdata/TestTwoSessionsDuplicate.golden
@@ -20,6 +20,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestTwoSessionsDuplicateRouter.golden b/internal/bgp/frr/testdata/TestTwoSessionsDuplicateRouter.golden
index 6df4c184..8a80a177 100644
--- a/internal/bgp/frr/testdata/TestTwoSessionsDuplicateRouter.golden
+++ b/internal/bgp/frr/testdata/TestTwoSessionsDuplicateRouter.golden
@@ -32,6 +32,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/bgp/frr/testdata/TestTwoSessionsOneVRF.golden b/internal/bgp/frr/testdata/TestTwoSessionsOneVRF.golden
index b151246b..6947c458 100644
--- a/internal/bgp/frr/testdata/TestTwoSessionsOneVRF.golden
+++ b/internal/bgp/frr/testdata/TestTwoSessionsOneVRF.golden
@@ -32,6 +32,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
@@ -55,6 +56,7 @@ router bgp 300 vrf red
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.3.3.254
   neighbor 10.4.4.255 remote-as 400
diff --git a/internal/bgp/frr/testdata/TestTwoSessionsSameIPRouterIDASNVRF.golden b/internal/bgp/frr/testdata/TestTwoSessionsSameIPRouterIDASNVRF.golden
index 9a25d276..9530ead0 100644
--- a/internal/bgp/frr/testdata/TestTwoSessionsSameIPRouterIDASNVRF.golden
+++ b/internal/bgp/frr/testdata/TestTwoSessionsSameIPRouterIDASNVRF.golden
@@ -32,6 +32,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   neighbor 10.2.2.254 remote-as 200
   neighbor 10.2.2.254 ebgp-multihop
@@ -54,6 +55,7 @@ router bgp 100 vrf red
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   neighbor 10.2.2.254 remote-as 200
   neighbor 10.2.2.254 ebgp-multihop
diff --git a/internal/bgp/frr/testdata/TestTwoSessionsSameIPVRF.golden b/internal/bgp/frr/testdata/TestTwoSessionsSameIPVRF.golden
index b93421ea..0d0bb743 100644
--- a/internal/bgp/frr/testdata/TestTwoSessionsSameIPVRF.golden
+++ b/internal/bgp/frr/testdata/TestTwoSessionsSameIPVRF.golden
@@ -32,6 +32,7 @@ router bgp 100
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
@@ -55,6 +56,7 @@ router bgp 300 vrf red
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.3.3.254
   neighbor 10.2.2.254 remote-as 400
diff --git a/internal/bgp/frr/testdata/TestVRFSingleEBGPSessionMultiHop.golden b/internal/bgp/frr/testdata/TestVRFSingleEBGPSessionMultiHop.golden
index b96d6816..fa853433 100644
--- a/internal/bgp/frr/testdata/TestVRFSingleEBGPSessionMultiHop.golden
+++ b/internal/bgp/frr/testdata/TestVRFSingleEBGPSessionMultiHop.golden
@@ -20,6 +20,7 @@ router bgp 100 vrf red
   no bgp ebgp-requires-policy
   no bgp network import-check
   no bgp default ipv4-unicast
+  bgp graceful-restart preserve-fw-state
 
   bgp router-id 10.1.1.254
   neighbor 10.2.2.254 remote-as 200
diff --git a/internal/config/config_test.go b/internal/config/config_test.go
index 7d109a00..39731620 100644
--- a/internal/config/config_test.go
+++ b/internal/config/config_test.go
@@ -69,16 +69,17 @@ func TestParse(t *testing.T) {
 							Name: "peer1",
 						},
 						Spec: v1beta2.BGPPeerSpec{
-							MyASN:        42,
-							ASN:          142,
-							Address:      "1.2.3.4",
-							Port:         1179,
-							HoldTime:     metav1.Duration{Duration: 180 * time.Second},
-							ConnectTime:  ptr.To(metav1.Duration{Duration: time.Second}),
-							RouterID:     "10.20.30.40",
-							SrcAddress:   "10.20.30.40",
-							EBGPMultiHop: true,
-							VRFName:      "foo",
+							MyASN:                 42,
+							ASN:                   142,
+							Address:               "1.2.3.4",
+							Port:                  1179,
+							HoldTime:              metav1.Duration{Duration: 180 * time.Second},
+							ConnectTime:           ptr.To(metav1.Duration{Duration: time.Second}),
+							RouterID:              "10.20.30.40",
+							SrcAddress:            "10.20.30.40",
+							EnableGracefulRestart: true,
+							EBGPMultiHop:          true,
+							VRFName:               "foo",
 						},
 					},
 					{
@@ -86,11 +87,12 @@ func TestParse(t *testing.T) {
 							Name: "peer2",
 						},
 						Spec: v1beta2.BGPPeerSpec{
-							MyASN:        100,
-							ASN:          200,
-							Address:      "2.3.4.5",
-							EBGPMultiHop: false,
-							ConnectTime:  ptr.To(metav1.Duration{Duration: time.Second}),
+							MyASN:                 100,
+							ASN:                   200,
+							Address:               "2.3.4.5",
+							EnableGracefulRestart: false,
+							EBGPMultiHop:          false,
+							ConnectTime:           ptr.To(metav1.Duration{Duration: time.Second}),
 							NodeSelectors: []metav1.LabelSelector{
 								{
 									MatchLabels: map[string]string{
@@ -222,30 +224,32 @@ func TestParse(t *testing.T) {
 			want: &Config{
 				Peers: map[string]*Peer{
 					"peer1": {
-						Name:          "peer1",
-						MyASN:         42,
-						ASN:           142,
-						Addr:          net.ParseIP("1.2.3.4"),
-						SrcAddr:       net.ParseIP("10.20.30.40"),
-						Port:          1179,
-						HoldTime:      180 * time.Second,
-						KeepaliveTime: 60 * time.Second,
-						ConnectTime:   ptr.To(time.Second),
-						RouterID:      net.ParseIP("10.20.30.40"),
-						NodeSelectors: []labels.Selector{labels.Everything()},
-						EBGPMultiHop:  true,
-						VRF:           "foo",
+						Name:                  "peer1",
+						MyASN:                 42,
+						ASN:                   142,
+						Addr:                  net.ParseIP("1.2.3.4"),
+						SrcAddr:               net.ParseIP("10.20.30.40"),
+						Port:                  1179,
+						HoldTime:              180 * time.Second,
+						KeepaliveTime:         60 * time.Second,
+						ConnectTime:           ptr.To(time.Second),
+						RouterID:              net.ParseIP("10.20.30.40"),
+						NodeSelectors:         []labels.Selector{labels.Everything()},
+						EnableGracefulRestart: true,
+						EBGPMultiHop:          true,
+						VRF:                   "foo",
 					},
 					"peer2": {
-						Name:          "peer2",
-						MyASN:         100,
-						ASN:           200,
-						Addr:          net.ParseIP("2.3.4.5"),
-						HoldTime:      90 * time.Second,
-						KeepaliveTime: 30 * time.Second,
-						ConnectTime:   ptr.To(time.Second),
-						NodeSelectors: []labels.Selector{selector("bar in (quux),foo=bar")},
-						EBGPMultiHop:  false,
+						Name:                  "peer2",
+						MyASN:                 100,
+						ASN:                   200,
+						Addr:                  net.ParseIP("2.3.4.5"),
+						HoldTime:              90 * time.Second,
+						KeepaliveTime:         30 * time.Second,
+						ConnectTime:           ptr.To(time.Second),
+						NodeSelectors:         []labels.Selector{selector("bar in (quux),foo=bar")},
+						EnableGracefulRestart: false,
+						EBGPMultiHop:          false,
 					},
 				},
 				Pools: &Pools{ByName: map[string]*Pool{
diff --git a/internal/config/validation_test.go b/internal/config/validation_test.go
index 3b7d873d..c527a327 100644
--- a/internal/config/validation_test.go
+++ b/internal/config/validation_test.go
@@ -70,6 +70,19 @@ func TestValidate(t *testing.T) {
 			},
 			mustFail: true,
 		},
+		{
+			desc: "enable BGP GracefulRestart",
+			config: ClusterResources{
+				Peers: []v1beta2.BGPPeer{
+					{
+						Spec: v1beta2.BGPPeerSpec{
+							EnableGracefulRestart: true,
+						},
+					},
+				},
+			},
+			mustFail: true,
+		},
 		{
 			desc: "disable BGP MP",
 			config: ClusterResources{