Repository: nginx/nginx-gateway-fabric

Issue:
NGINX Plus worker process takes extended time to shut down.
In the snippets filter functional test, occasionally an nginx plus worker process will take around 60 seconds to shut down when reloading nginx while it usually should take less than ~1 second. This isn't an issue in regards to our eventually consistent working environment, however is an issue when working with NGINX Agent because there is a section in their code which waits for NGINX to fully reload (all old worker processes to be shut down, 30s max timeout) before sending back a configuration has been updated message to NGF. This also means, when using nginx plus, there could be 30 seconds of downtime because of how we update upstreams using the plus api. 

This issue is to investigate whats causing the one nginx plus worker process to take an extended time to shut down.

Discussion:
@bjee19: The one nginx worker process ended up being related to NGINX Agent and a call to the NGINX Plus API when Agent was collecting some metrics. Workaround was decided to support these long-lived connections and reduce a refresh timeout on Agent configuration. 

Solution patch:
diff --git a/internal/controller/provisioner/templates.go b/internal/controller/provisioner/templates.go
index 78945d53..d52700ad 100644
--- a/internal/controller/provisioner/templates.go
+++ b/internal/controller/provisioner/templates.go
@@ -89,4 +89,13 @@ collector:
             "ngf":
                 receivers: ["host_metrics", "nginx_metrics"]
                 exporters: ["prometheus"]
-{{- end }}`
+{{- end }}
+data_plane_config:
+    nginx:
+        reload_backoff:
+            initial_interval: .5s
+            max_interval: 1.5s
+            max_elapsed_time: 3s
+            randomization_factor: 0.5
+            multiplier: 1.5
+`



Test patch:
diff --git a/tests/framework/timeout.go b/tests/framework/timeout.go
index 72e51ef1..394f9e35 100644
--- a/tests/framework/timeout.go
+++ b/tests/framework/timeout.go
@@ -49,7 +49,7 @@ func DefaultTimeoutConfig() TimeoutConfig {
 		DeleteNamespaceTimeout:  150 * time.Second,
 		GetTimeout:              10 * time.Second,
 		ManifestFetchTimeout:    10 * time.Second,
-		RequestTimeout:          30 * time.Second,
+		RequestTimeout:          10 * time.Second,
 		ContainerRestartTimeout: 10 * time.Second,
 		GetLeaderLeaseTimeout:   60 * time.Second,
 		GetStatusTimeout:        60 * time.Second,