Make shutdown more graceful by delaying trying to shutdown

* Instead of trying to shut down immediately, wait for a bit before
trying to complete shutdown, in order to deal with in-flight
operations better.