Repository: static-web-server/static-web-server (language: rust)

Issue:
Configurable grace period on SIGTERM
## Problem

When hosting static-web-server in Kubernetes, there is a subtle race that exists during rollouts that causes upstream endpoint-based traffic steering (such as nginx-ingress controller) to fail requests.  With nginx upstream, for example, this results in a 502 Bad Gateway.

Suppose static-web-server is run within a Kubernetes Deployment. The race is as follows:
1. Update the deployment resource, or execute `kubectl rollout restart`, which causes new replacement pods to spin up.
2. The new pods come up and pass readiness checks. Traffic starts being directed to them.
3. Kubernetes begins terminating old pods. static-web-server dutifully responds to SIGTERM by shutting down immediately.
4. The Service or Ingress takes a moment to notice the updates to the available Endpoints.  During this time, traffic is still being directed to the dying or dead pods.
5. The Service or Ingress updates to reflect the change in endpoints and life returns to normal.

There is a discussion of this race over at https://github.com/kubernetes/ingress-gce/issues/769

One could argue Kubernetes should be providing the appropriate constructs to solve this problem (i.e. by withdrawing the to-be-terminated pods from the available endpoints to prevent traffic from going to pods *before* it kills them).  But the Kubernetes philosophy here appears to be to delegate this responsibility to the underlying workloads, for better or worse.

## Feature Request

To address the above issue, I propose that static-web-server provides a configurable grace period in seconds, whereupon receiving SIGTERM it continues to respond to requests during the grace period, but issues `Connection: close` for all requests, even if `Connection: keep-alive` is requested.  Once the configurable grace period has elapsed, then exit as usual.

This is basically the suggestion in [this comment](https://github.com/kubernetes/ingress-gce/issues/769#issuecomment-515218403) by Bowei Du on the Kubernetes team.



Discussion / hints:
Few questions from my side.

**If you are using v1**

> * Kubernetes begins terminating old pods. static-web-server dutifully responds to SIGTERM by shutting down immediately.

Did you try latest v2? Because v2 can shut down gracefully. v1 can not unfortunately. 
See sws v2 Linux logs of PR #62

--

**If you are using v2**

> ## Feature Request
> To address the above issue, I propose that static-web-server provides a configurable grace period in seconds, whereupon receiving SIGTERM it continues to respond to requests during the grace period, but issues `Connection: close` for all requests, even if `Connection: keep-alive` is requested. Once the configurable grace period has elapsed, then exit as usual.
> 
> This is basically the suggestion in [this comment](https://github.com/kubernetes/ingress-gce/issues/769#issuecomment-515218403) by Bowei Du on the Kubernetes team.

As I understood. It sounds like the current graceful showdown feature in v2 is not working for your use case.
And what do you would like essentially is to have the current graceful showdown + an extra configurable grace period in seconds on top.
I hope I understood it correctly. Or maybe not?
@joseluisq  apologies, I neglected to mention in my OP: yes, I'm using 2.4.0.

Reading through #62 (and I did somehow miss it initially, which I'll have to blame on a late whisky-fueled evening), these are definitely related inasmuch they aim to improve robustness around Kubernetes pod lifecycle management, but look to be subtly different. 

Whereas #62 ensures that in-flight requests are completed before shutting down, this FR is about providing a tolerance (grace period) to give Kubernetes sufficient time to withdraw terminating pods from receiving traffic.  I see them as different sides of the same coin.  Perhaps #62 was lower hanging fruit, and now we're scratching around narrower margins.

Personally I feel Kubernetes should be removing to-be-terminated pods from the endpoint list to give upstream proxies like nginx-ingress-controller the opportunity to a) notice the endpoint list change and b) update their configs and HUP the proxy, but that seems to be deliberately not done (for reasons that evade me).

So ultimately this FR is to address the race between Kubernetes sending SIGTERM to the container and the ingress controller updating its configuration to remove the pod from the list of available endpoints.  It seems like 3-5 seconds is enough for this, but for larger environments this convergence could take longer, hence my feeling that it should be configurable.

Hope that makes sense.
Don't worry.
Now I got what _**an extra configurable (additional) grace period in seconds**_ means.  And I see it happens _a posteriori_.
Sorry for my lack of Kubernetes taste.

So coming back to sws.
What I can think of doing at first glance is something like:
Leaving `SIGTERM` and then "the graceful showdown" happen (mostly intact). Which means after that the server will be essentially off.
Then right after, proceed with the timeout or grace period (if defined of course) and finally just exit the process.

Does this make sense for you? if so, then it should be not a big deal to implement. 

--

BTW cheers! I hope it was an scotch (my favorite).
> Leaving SIGTERM and then "the graceful showdown" happen (mostly intact). Which means after that the server will be essentially off. Then right after, proceed with the timeout or grace period (if defined of course) and finally just exit the process.

I'm not sure you mean by "Leaving SIGTERM" here? 

The important part is that sws actually continues to respond to requests even after SIGTERM, because nginx-ingress-controller will still be directing traffic its way for that small period of time until it updates its pool membership list.

Being more precise, my proposal is:

1. sws starts and runs normally
2. sws receives SIGTERM. The grace period begins.
3. New requests are still processed successfully, but whereas in normal operation `Connection: keep-alive` is respected, now all requests are include `Connection: close` in the header and the connection is terminated.
4. The grace period elapses, and sws's current shutdown behavior now begins, where new connections are refused, in-flight transactions are completed, and finally sws exits.

> BTW cheers! I hope it was an scotch (my favorite).

Glenfiddich 15. :)
FWIW, the `Connection: close` part of step 3 probably isn't crucial.  This was part of Bowei Du's recommendation, and it seems reasonable to encourage clients to move off the pod, but one way or the other their connections are going to get dumped -- either immediately, or later in step 4.  So if that's cumbersome to implement, I'd consider it optional, even though it may be more technically correct.
> I'm not sure you mean by "Leaving SIGTERM" here?

1. I thought that the grace period should start after the graceful shutdown. But it is right after the `SIGTERM` which means before the graceful shutdown.
So based on that premise, It should even more "easy" to implement. At least from my first glance on the code.

> The important part is that sws actually continues to respond to requests even after SIGTERM, because nginx-ingress-controller will still be directing traffic its way for that small period of time until it updates its pool membership list.
> 
> Being more precise, my proposal is:
> 
> 1. sws starts and runs normally
> 2. sws receives SIGTERM. The grace period begins.
> 3. New requests are still processed successfully, but whereas in normal operation `Connection: keep-alive` is respected, now all requests are include `Connection: close` in the header and the connection is terminated.
> 4. The grace period elapses, and sws's current shutdown behavior now begins, where new connections are refused, in-flight transactions are completed, and finally sws exits.

2. I see, so I now understand more clearly.
Just a generic highlight below that maybe could tell us what is happening right now for  `keep-alive` and  `close`.

`connection: close`

```log
2022-01-29T22:52:45.690231Z TRACE mio::poll: registering event source with poller: token=Token(4), interests=READABLE | WRITABLE    
2022-01-29T22:52:45.690996Z TRACE hyper::proto::h1::conn: Conn::read_head
2022-01-29T22:52:45.691324Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: Init, writing: Init, keep_alive: Busy }
2022-01-29T22:52:45.691674Z TRACE mio::poll: registering event source with poller: token=Token(5), interests=READABLE | WRITABLE    
2022-01-29T22:52:45.691734Z TRACE hyper::proto::h1::conn: Conn::read_head
2022-01-29T22:52:45.691747Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: Init, writing: Init, keep_alive: Busy }
2022-01-29T22:52:45.691817Z TRACE hyper::proto::h1::conn: Conn::read_head
2022-01-29T22:52:45.691840Z TRACE hyper::proto::h1::io: received 0 bytes
2022-01-29T22:52:45.691847Z TRACE hyper::proto::h1::io: parse eof
2022-01-29T22:52:45.691853Z TRACE hyper::proto::h1::conn: State::close_read()
2022-01-29T22:52:45.691867Z DEBUG hyper::proto::h1::conn: read eof
2022-01-29T22:52:45.691872Z TRACE hyper::proto::h1::conn: State::close_write()
2022-01-29T22:52:45.691878Z TRACE hyper::proto::h1::conn: State::close_read()
2022-01-29T22:52:45.691883Z TRACE hyper::proto::h1::conn: State::close_write()
2022-01-29T22:52:45.691888Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: Closed, writing: Closed, keep_alive: Disabled }
2022-01-29T22:52:45.691963Z TRACE hyper::proto::h1::conn: shut down IO complete
2022-01-29T22:52:45.691971Z TRACE mio::poll: deregistering event source from poller    
2022-01-29T22:52:45.692246Z TRACE hyper::proto::h1::conn: Conn::read_head
2022-01-29T22:52:45.692272Z TRACE hyper::proto::h1::io: received 221 bytes
2022-01-29T22:52:45.692318Z TRACE parse_headers: hyper::proto::h1::role: Request.parse bytes=221
2022-01-29T22:52:45.692335Z TRACE parse_headers: hyper::proto::h1::role: Request.parse Complete(221)
2022-01-29T22:52:45.692362Z TRACE parse_headers: hyper::proto::h1::role: close time.busy=46.2µs time.idle=1.54µs
2022-01-29T22:52:45.692377Z DEBUG hyper::proto::h1::io: parsed 7 headers
2022-01-29T22:52:45.692384Z DEBUG hyper::proto::h1::conn: incoming body is empty
2022-01-29T22:52:45.692390Z TRACE hyper::proto::h1::conn: remote disabling keep-alive
2022-01-29T22:52:45.692883Z TRACE static_web_server::static_files: dir? base="docker/", route="/"
2022-01-29T22:52:45.693400Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Disabled }
2022-01-29T22:52:45.693515Z DEBUG static_web_server::static_files: dir: appending index.html to directory path
2022-01-29T22:52:45.693522Z TRACE static_web_server::static_files: dir: "docker/index.html"
2022-01-29T22:52:45.693553Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Disabled }
2022-01-29T22:52:45.693603Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Disabled }
2022-01-29T22:52:45.693641Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Disabled }
2022-01-29T22:52:45.693694Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Disabled }
2022-01-29T22:52:45.693710Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Disabled }
2022-01-29T22:52:45.693730Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Disabled }
2022-01-29T22:52:45.693746Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Disabled }
2022-01-29T22:52:45.693767Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Disabled }
2022-01-29T22:52:45.693782Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Disabled }
2022-01-29T22:52:45.693802Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Disabled }
2022-01-29T22:52:45.693899Z TRACE encode_headers: hyper::proto::h1::role: Server::encode status=200, body=Some(Known(1801)), req_method=Some(GET)
2022-01-29T22:52:45.693936Z TRACE encode_headers: hyper::proto::h1::role: close time.busy=40.2µs time.idle=747ns
2022-01-29T22:52:45.693951Z TRACE hyper::proto::h1::io: buffer.queue self.len=156 buf.len=1801
2022-01-29T22:52:45.694029Z DEBUG hyper::proto::h1::io: flushed 1957 bytes
2022-01-29T22:52:45.694038Z TRACE hyper::proto::h1::conn: State::close()
2022-01-29T22:52:45.694044Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: Closed, writing: Closed, keep_alive: Disabled }
2022-01-29T22:52:45.694082Z TRACE hyper::proto::h1::conn: shut down IO complete
2022-01-29T22:52:45.694091Z TRACE mio::poll: deregistering event source from poller    
```

`connection: keep-alive`

```log
2022-01-29T22:54:23.055661Z TRACE mio::poll: registering event source with poller: token=Token(16777221), interests=READABLE | WRITABLE    
2022-01-29T22:54:23.055753Z TRACE hyper::proto::h1::conn: Conn::read_head
2022-01-29T22:54:23.055779Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: Init, writing: Init, keep_alive: Busy }
2022-01-29T22:54:23.057286Z TRACE mio::poll: registering event source with poller: token=Token(16777220), interests=READABLE | WRITABLE    
2022-01-29T22:54:23.057422Z TRACE hyper::proto::h1::conn: Conn::read_head
2022-01-29T22:54:23.057441Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: Init, writing: Init, keep_alive: Busy }
2022-01-29T22:54:23.057542Z TRACE hyper::proto::h1::conn: Conn::read_head
2022-01-29T22:54:23.057558Z TRACE hyper::proto::h1::io: received 0 bytes
2022-01-29T22:54:23.057565Z TRACE hyper::proto::h1::io: parse eof
2022-01-29T22:54:23.057573Z TRACE hyper::proto::h1::conn: State::close_read()
2022-01-29T22:54:23.057580Z DEBUG hyper::proto::h1::conn: read eof
2022-01-29T22:54:23.057586Z TRACE hyper::proto::h1::conn: State::close_write()
2022-01-29T22:54:23.057593Z TRACE hyper::proto::h1::conn: State::close_read()
2022-01-29T22:54:23.057599Z TRACE hyper::proto::h1::conn: State::close_write()
2022-01-29T22:54:23.057605Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: Closed, writing: Closed, keep_alive: Disabled }
2022-01-29T22:54:23.057680Z TRACE hyper::proto::h1::conn: shut down IO complete
2022-01-29T22:54:23.057690Z TRACE mio::poll: deregistering event source from poller    
2022-01-29T22:54:23.058053Z TRACE hyper::proto::h1::conn: Conn::read_head
2022-01-29T22:54:23.058080Z TRACE hyper::proto::h1::io: received 226 bytes
2022-01-29T22:54:23.058093Z TRACE parse_headers: hyper::proto::h1::role: Request.parse bytes=226
2022-01-29T22:54:23.058107Z TRACE parse_headers: hyper::proto::h1::role: Request.parse Complete(226)
2022-01-29T22:54:23.058128Z TRACE parse_headers: hyper::proto::h1::role: close time.busy=34.5µs time.idle=1.59µs
2022-01-29T22:54:23.058142Z DEBUG hyper::proto::h1::io: parsed 7 headers
2022-01-29T22:54:23.058150Z DEBUG hyper::proto::h1::conn: incoming body is empty
2022-01-29T22:54:23.058163Z TRACE static_web_server::static_files: dir? base="docker/", route="/"
2022-01-29T22:54:23.058256Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Busy }
2022-01-29T22:54:23.058331Z DEBUG static_web_server::static_files: dir: appending index.html to directory path
2022-01-29T22:54:23.058338Z TRACE static_web_server::static_files: dir: "docker/index.html"
2022-01-29T22:54:23.058354Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Busy }
2022-01-29T22:54:23.058397Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Busy }
2022-01-29T22:54:23.058428Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Busy }
2022-01-29T22:54:23.058462Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Busy }
2022-01-29T22:54:23.058476Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Busy }
2022-01-29T22:54:23.058494Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Busy }
2022-01-29T22:54:23.058509Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Busy }
2022-01-29T22:54:23.058525Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Busy }
2022-01-29T22:54:23.058540Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Busy }
2022-01-29T22:54:23.058558Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: KeepAlive, writing: Init, keep_alive: Busy }
2022-01-29T22:54:23.058618Z TRACE encode_headers: hyper::proto::h1::role: Server::encode status=200, body=Some(Known(1801)), req_method=Some(GET)
2022-01-29T22:54:23.058635Z TRACE encode_headers: hyper::proto::h1::role: close time.busy=16.4µs time.idle=781ns
2022-01-29T22:54:23.058645Z TRACE hyper::proto::h1::io: buffer.queue self.len=156 buf.len=1801
2022-01-29T22:54:23.058694Z DEBUG hyper::proto::h1::io: flushed 1957 bytes
2022-01-29T22:54:23.058702Z TRACE hyper::proto::h1::conn: flushed({role=server}): State { reading: Init, writing: Init, keep_alive: Idle }
```

> Glenfiddich 15. :)

Damn, you're 2yrs up on tastesness than my not so elegant JW Black. I feel too young on this long run.. :D 
>  So based on that premise, It should even more "easy" to implement. At least from my first glance on the code.

Wonderful!

>  Just a generic highlight below that maybe could tell us what is happening right now for keep-alive and close.

I'm a bit fuzzy on the nuances of the intermediate log lines, but the obvious thing that jumps out is "State::close()" and "shut down IO complete" in the `Connection: close` case but not in the `Connection: keep-alive` case, which looks right to me.

The trick now would be, while in the grace period, to enforce `Connection: close` behavior always regardless of the request headers.  Again I don't think this detail is super critical -- the more important thing is that the requests are still processed during the grace period following SIGTERM. And of course that the behavior from #62 to finish off in-flight requests is preserved in the final shutdown sequence.

> Damn, you're 2yrs up on tastesness than my not so elegant JW Black. I feel too young on this long run.. :D

No single malt snobbery here.  I've killed many a bottle of JW Black in my day.  What's not to love about campfire in a bottle. :)  Though I *am* an unapologetic sucker for sherry cask scotches like the Glenfiddich 15.
> I'm a bit fuzzy on the nuances of the intermediate log lines, but the obvious thing that jumps out is "State::close()" and "shut down IO complete" in the `Connection: close` case but not in the `Connection: keep-alive` case, which looks right to me.

Yes, you got it! I just wanted to share insights for `close` and `keep-alive` for the doubts above.

> The trick now would be, while in the grace period, to enforce `Connection: close` behavior always regardless of the request headers. Again I don't think this detail is super critical

You're right about strictness after `SIGTERM` because it implies an upcoming termination. However AFAICT it could require additional effort which is out of my radar right now. But I could try to dispel my doubts as long as the feature goes (if the case).

> the more important thing is that the requests are still processed during the grace period following SIGTERM. And of course that the behavior from #62 to finish off in-flight requests is preserved in the final shutdown sequence.

For the grace period following SIGTERM, Yes, in case of a SIGTERM the signal gets caught and as long as we don't notify the server for the shutdown process, it should be able to handle requests as shown on the logs above (normal requests).

And regarding the graceful shutdown itself, it basically remains intact.

> No single malt snobbery here. I've killed many a bottle of JW Black in my day. What's not to love about campfire in a bottle. :) Though I _am_ an unapologetic sucker for sherry cask scotches like the Glenfiddich 15.

I'm still enjoying this single malt on the rocks at least until the campfire starts to dissipate or if some of my friends convince my to do the jump xD. Or maybe I need to fan the fire a bit with a Double Black or similar haha.
But regardless, I have heard several times Glenfiddich so why not a taste.
> However AFAICT it could require additional effort which is out of my radar right now.

Absolutely, sounds good. If the grace period part is easy enough to implement, I can put it through its paces and determine how important the connection close behavior is in practice.  Happy to test dev builds.

> But regardless, I have heard several times Glenfiddich so why not a taste.

Like every distillery, there's quite a wide variety.  If you like sweet and fruity, I strongly recommend the 15. If you like oaky, the 14. But in any case, the price is a bit more than JW Black.  For a cheaper sherry cask single malt, it's hard to go wrong the Aberlour 12.  Tonight's sipper meanwhile is the Glenlivet Caribbean Reserve. Cheers! :)
> Absolutely, sounds good. If the grace period part is easy enough to implement, I can put it through its paces and determine how important the connection close behavior is in practice. Happy to test dev builds.

Definitely I will need help regarding dev testing on your Kubernetes environment if possible. I will let you know.

> Like every distillery, there's quite a wide variety. If you like sweet and fruity, I strongly recommend the 15. If you like oaky, the 14. But in any case, the price is a bit more than JW Black. For a cheaper sherry cask single malt, it's hard to go wrong the Aberlour 12. Tonight's sipper meanwhile is the Glenlivet Caribbean Reserve. Cheers! :)

Thanks for the advice! Yes, I think I'll give the Glenfiddich 14 a taste. Cheers!

There is a PR going on so I suggest to continue the discussion there.

Fix patch:
diff --git a/src/config.rs b/src/config.rs
index 7d297405..a6bd12ce 100644
--- a/src/config.rs
+++ b/src/config.rs
@@ -154,4 +154,8 @@ pub struct Config {
     /// It provides The "Basic" HTTP Authentication scheme using credentials as "user-id:password" pairs. Password must be encoded using the "BCrypt" password-hashing function.
     #[structopt(long, default_value = "", env = "SERVER_BASIC_AUTH")]
     pub basic_auth: String,
+
+    #[structopt(long, short = "q", default_value = "0", env = "SERVER_GRACE_PERIOD")]
+    /// Defines a grace period in seconds after a `SIGTERM` signal is caught which will delay the server before to shut it down gracefully. The maximum value is 255 seconds.
+    pub grace_period: u8,
 }
diff --git a/src/server.rs b/src/server.rs
index 8769033a..6e826ae7 100644
--- a/src/server.rs
+++ b/src/server.rs
@@ -121,6 +121,10 @@ impl Server {
         );
         let basic_auth = Arc::from(basic_auth);
 
+        // Grace period option
+        let grace_period = opts.grace_period;
+        tracing::info!("grace period before graceful shutdown: {}s", grace_period);
+
         // Create a service router for Hyper
         let router_service = RouterService::new(RequestHandler {
             opts: RequestHandlerOpts {
@@ -170,9 +174,10 @@ impl Server {
                 HyperServer::builder(TlsAcceptor::new(tls, incoming)).serve(router_service);
 
             #[cfg(not(windows))]
-            let server = server.with_graceful_shutdown(signals::wait_for_signals(signals));
+            let server =
+                server.with_graceful_shutdown(signals::wait_for_signals(signals, grace_period));
             #[cfg(windows)]
-            let server = server.with_graceful_shutdown(signals::wait_for_ctrl_c());
+            let server = server.with_graceful_shutdown(signals::wait_for_ctrl_c(grace_period));
 
             tracing::info!(
                 parent: tracing::info_span!("Server::start_server", ?addr_str, ?threads),
@@ -200,9 +205,10 @@ impl Server {
                 .serve(router_service);
 
             #[cfg(not(windows))]
-            let server = server.with_graceful_shutdown(signals::wait_for_signals(signals));
+            let server =
+                server.with_graceful_shutdown(signals::wait_for_signals(signals, grace_period));
             #[cfg(windows)]
-            let server = server.with_graceful_shutdown(signals::wait_for_ctrl_c());
+            let server = server.with_graceful_shutdown(signals::wait_for_ctrl_c(grace_period));
 
             tracing::info!(
                 parent: tracing::info_span!("Server::start_server", ?addr_str, ?threads),
diff --git a/src/signals.rs b/src/signals.rs
index 438ebcf8..6fc576d5 100644
--- a/src/signals.rs
+++ b/src/signals.rs
@@ -4,6 +4,8 @@ use {
     signal_hook_tokio::Signals,
 };
 
+use tokio::time::{sleep, Duration};
+
 #[cfg(not(windows))]
 /// It creates a common list of signals stream for `SIGTERM`, `SIGINT` and `SIGQUIT` to be observed.
 pub fn create_signals() -> Result<Signals> {
@@ -12,27 +14,43 @@ pub fn create_signals() -> Result<Signals> {
 
 #[cfg(not(windows))]
 /// It waits for a specific type of incoming signals included `ctrl+c`.
-pub async fn wait_for_signals(signals: Signals) {
+pub async fn wait_for_signals(signals: Signals, grace_period_secs: u8) {
     let mut signals = signals.fuse();
     while let Some(signal) = signals.next().await {
         match signal {
             SIGHUP => {
-                // Note: for now we don't do something for SIGHUPs
+                // NOTE: for now we don't do something for SIGHUPs
                 tracing::debug!("SIGHUP caught, nothing to do about")
             }
             SIGTERM | SIGINT | SIGQUIT => {
-                tracing::debug!("SIGTERM, SIGINT or SIGQUIT signal received, delegating graceful shutdown to the server");
+                tracing::info!("SIGTERM, SIGINT or SIGQUIT signal caught");
                 break;
             }
             _ => unreachable!(),
         }
     }
+    // NOTE: once loop above is done then an upstream graceful shutdown should come next.
+    delay_graceful_shutdown(grace_period_secs).await;
+    tracing::info!("delegating server's graceful shutdown");
+}
+
+/// Function intended to delay the server's graceful shutdown providing a grace period in seconds.
+async fn delay_graceful_shutdown(grace_period_secs: u8) {
+    if grace_period_secs > 0 {
+        tracing::info!(
+            "grace period of {}s after the SIGTERM started",
+            grace_period_secs
+        );
+        sleep(Duration::from_secs(grace_period_secs.into())).await;
+        tracing::info!("grace period has elapsed");
+    }
 }
 
 #[cfg(windows)]
 /// It waits for an incoming `ctrl+c` signal on Windows.
-pub async fn wait_for_ctrl_c() {
+pub async fn wait_for_ctrl_c(grace_period_secs: u8) {
     tokio::signal::ctrl_c()
         .await
         .expect("failed to install ctrl+c signal handler");
+    delay_graceful_shutdown(grace_period_secs).await;
 }


Test patch: