Repository: quarkusio/quarkus
Issue: Unpredictable "Shutdown in progress" exception for code in dev mode

### Describe the bug

We found, that sometimes our test runs fail with "java.lang.IllegalStateException: Shutdown in progress" exceptions. It appears randomly([1],[2],[3]) but it looks like more demanding tasks tend to provoke this behaviour more often.

[1] https://github.com/quarkus-qe/quarkus-startstop/pull/102/checks?check_run_id=3220458632
[2] https://github.com/quarkus-qe/quarkus-startstop/runs/3297343795?check_suite_focus=true
[3] https://github.com/quarkus-qe/quarkus-startstop/runs/3206345439?check_suite_focus=true



### Expected behavior

Code finishes successfully of fails predictably.

### Actual behavior

There is an exception, with stacktraces like that:
```
2021-08-02 12:13:05,272 INFO  [io.quarkus] (Quarkus Main Thread) code-with-quarkus stopped in 0.007s
Press [space] to restart, [e] to edit command line args (currently ''), [r] to resume testing, [h] for more options>
2021-08-02 12:13:05,298 ERROR [io.qua.boo.cla.QuarkusClassLoader] (Quarkus Shutdown Thread) Failed to run close task: java.lang.IllegalStateException: Shutdown in progress
	at java.base/java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
	at java.base/java.lang.Runtime.removeShutdownHook(Runtime.java:244)
	at io.quarkus.datasource.deployment.devservices.DevServicesDatasourceProcessor$2.run(DevServicesDatasourceProcessor.java:169)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.close(QuarkusClassLoader.java:551)
	at io.quarkus.bootstrap.app.CuratedApplication.close(CuratedApplication.java:362)
	at io.quarkus.deployment.dev.IsolatedDevModeMain.close(IsolatedDevModeMain.java:327)
	at io.quarkus.deployment.dev.IsolatedDevModeMain$5.run(IsolatedDevModeMain.java:441)
	at java.base/java.lang.Thread.run(Thread.java:829)

Press [space] to restart, [e] to edit command line args (currently ''), [h] for more options>
2021-08-02 12:13:05,299 ERROR [io.qua.boo.cla.QuarkusClassLoader] (Quarkus Shutdown Thread) Failed to run close task: java.lang.IllegalStateException: Shutdown in progress
	at java.base/java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82)
	at java.base/java.lang.Runtime.removeShutdownHook(Runtime.java:244)
	at io.quarkus.mongodb.deployment.DevServicesMongoProcessor$2.run(DevServicesMongoProcessor.java:132)
	at io.quarkus.bootstrap.classloading.QuarkusClassLoader.close(QuarkusClassLoader.java:551)
	at io.quarkus.bootstrap.app.CuratedApplication.close(CuratedApplication.java:362)
	at io.quarkus.deployment.dev.IsolatedDevModeMain.close(IsolatedDevModeMain.java:327)
	at io.quarkus.deployment.dev.IsolatedDevModeMain$5.run(IsolatedDevModeMain.java:441)
	at java.base/java.lang.Thread.run(Thread.java:829)
```

### How to Reproduce?

1) Download examples from code.quarkus.io(the more extensions included, the better)
2) Run mvn quarkus:dev on this code
3) Repeat a lot(this bug cannot be reproduced robustly, probably due to data race)

### Output of `uname -a` or `ver`

ubuntu-20.04

### Output of `java -version`

11.0.11

### GraalVM version (if different from Java)

_No response_

### Quarkus version or git rev

2.1.0.Final, 2.1.1.Final

### Build tool (ie. output of `mvnw --version` or `gradlew --version`)

Maven 3.8.1 (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d)

### Additional information

A look at the offending code gives us some "peculiarities":
1) All shutdown hooks can be registered twice due to race condition.
2) The shutdown task is being added both:
2.1 As a closing task to the parent classloader
2.2 As a shutdown hook to the VM
3) Parent classloader removes the shutdown hook from the VM as a part of it's closing.

According to documentation of the Runtime class, "Once the shutdown sequence has begun it is impossible to register a new shutdown hook or de-register a previously-registered hook. Attempting either of these operations will cause an IllegalStateException to be thrown". Since we observe an IllegalStateException, I assume the situation goes like that:
1) Code registers the shutdown hook for the VM and for the classloader
2) VM's shutdown and close() method of parent classloader happen at the roughly the same time.
3) Classloader stops all databases and tries to remove a shutdown hook from the VM(not exactly in this order, because of peculiarity 1)
4) Since the VM is in the middle of the shutdown process, an attempt to remove the shutdown hook fails.

This code can be found in several files, it looks like it was first introduced into DevServicesDatasourceProcessor and then copied all over the devservices codebase. Affected files:
DevServicesDatasourceProcessor
DevServicesApicurioRegistryProcessor
DevServicesMongoProcessor
KeycloakDevServicesProcessor
DevServicesVaultProcessor

Registration of close task:
 - https://github.com/quarkusio/quarkus/blame/main/extensions/datasource/deployment/src/main/java/io/quarkus/datasource/deployment/devservices/DevServicesDatasourceProcessor.java#L163
 - https://github.com/quarkusio/quarkus/blame/main/extensions/datasource/deployment/src/main/java/io/quarkus/datasource/deployment/devservices/DevServicesDatasourceProcessor.java#L165

CC @stuartwdouglas 

Changed files:
extensions/oidc/deployment/src/main/java/io/quarkus/oidc/deployment/devservices/keycloak/KeycloakDevServicesProcessor.java
extensions/redis-client/deployment/src/main/java/io/quarkus/redis/client/deployment/DevServicesRedisProcessor.java
extensions/kafka-client/deployment/src/main/java/io/quarkus/kafka/client/deployment/DevServicesKafkaProcessor.java
test-framework/junit5/src/main/java/io/quarkus/test/junit/QuarkusTestExtension.java
extensions/mongodb-client/deployment/src/main/java/io/quarkus/mongodb/deployment/DevServicesMongoProcessor.java
extensions/smallrye-reactive-messaging-amqp/deployment/src/main/java/io/quarkus/smallrye/reactivemessaging/amqp/deployment/DevServicesAmqpProcessor.java
extensions/vault/deployment/src/main/java/io/quarkus/vault/deployment/DevServicesVaultProcessor.java
extensions/apicurio-registry-avro/deployment/src/main/java/io/quarkus/apicurio/registry/avro/DevServicesApicurioRegistryProcessor.java
extensions/datasource/deployment/src/main/java/io/quarkus/datasource/deployment/devservices/DevServicesDatasourceProcessor.java

diff --git a/extensions/apicurio-registry-avro/deployment/src/main/java/io/quarkus/apicurio/registry/avro/DevServicesApicurioRegistryProcessor.java b/extensions/apicurio-registry-avro/deployment/src/main/java/io/quarkus/apicurio/registry/avro/DevServicesApicurioRegistryProcessor.java
index 5fa11139635..1147572ecc9 100644
--- a/extensions/apicurio-registry-avro/deployment/src/main/java/io/quarkus/apicurio/registry/avro/DevServicesApicurioRegistryProcessor.java
+++ b/extensions/apicurio-registry-avro/deployment/src/main/java/io/quarkus/apicurio/registry/avro/DevServicesApicurioRegistryProcessor.java
@@ -102,14 +102,6 @@ public void run() {
             };
             QuarkusClassLoader cl = (QuarkusClassLoader) Thread.currentThread().getContextClassLoader();
             ((QuarkusClassLoader) cl.parent()).addCloseTask(closeTask);
-            Thread closeHookThread = new Thread(closeTask, "Apicurio Registry container shutdown thread");
-            Runtime.getRuntime().addShutdownHook(closeHookThread);
-            ((QuarkusClassLoader) cl.parent()).addCloseTask(new Runnable() {
-                @Override
-                public void run() {
-                    Runtime.getRuntime().removeShutdownHook(closeHookThread);
-                }
-            });
         }
     }
 
diff --git a/extensions/datasource/deployment/src/main/java/io/quarkus/datasource/deployment/devservices/DevServicesDatasourceProcessor.java b/extensions/datasource/deployment/src/main/java/io/quarkus/datasource/deployment/devservices/DevServicesDatasourceProcessor.java
index ca64741b09e..0881c793693 100644
--- a/extensions/datasource/deployment/src/main/java/io/quarkus/datasource/deployment/devservices/DevServicesDatasourceProcessor.java
+++ b/extensions/datasource/deployment/src/main/java/io/quarkus/datasource/deployment/devservices/DevServicesDatasourceProcessor.java
@@ -161,14 +161,6 @@ public void run() {
             };
             QuarkusClassLoader cl = (QuarkusClassLoader) Thread.currentThread().getContextClassLoader();
             ((QuarkusClassLoader) cl.parent()).addCloseTask(closeTask);
-            Thread closeHookThread = new Thread(closeTask, "Database shutdown thread");
-            Runtime.getRuntime().addShutdownHook(closeHookThread);
-            ((QuarkusClassLoader) cl.parent()).addCloseTask(new Runnable() {
-                @Override
-                public void run() {
-                    Runtime.getRuntime().removeShutdownHook(closeHookThread);
-                }
-            });
         }
         databases = closeableList;
         cachedProperties = propertiesMap;
diff --git a/extensions/kafka-client/deployment/src/main/java/io/quarkus/kafka/client/deployment/DevServicesKafkaProcessor.java b/extensions/kafka-client/deployment/src/main/java/io/quarkus/kafka/client/deployment/DevServicesKafkaProcessor.java
index 30e46fcd314..34415afe512 100644
--- a/extensions/kafka-client/deployment/src/main/java/io/quarkus/kafka/client/deployment/DevServicesKafkaProcessor.java
+++ b/extensions/kafka-client/deployment/src/main/java/io/quarkus/kafka/client/deployment/DevServicesKafkaProcessor.java
@@ -111,9 +111,6 @@ public DevServicesKafkaBrokerBuildItem startKafkaDevService(
             };
             QuarkusClassLoader cl = (QuarkusClassLoader) Thread.currentThread().getContextClassLoader();
             ((QuarkusClassLoader) cl.parent()).addCloseTask(closeTask);
-            Thread closeHookThread = new Thread(closeTask, "Kafka container shutdown thread");
-            Runtime.getRuntime().addShutdownHook(closeHookThread);
-            ((QuarkusClassLoader) cl.parent()).addCloseTask(() -> Runtime.getRuntime().removeShutdownHook(closeHookThread));
         }
         cfg = configuration;
 
diff --git a/extensions/mongodb-client/deployment/src/main/java/io/quarkus/mongodb/deployment/DevServicesMongoProcessor.java b/extensions/mongodb-client/deployment/src/main/java/io/quarkus/mongodb/deployment/DevServicesMongoProcessor.java
index 24b8183673f..1564f2fbbf3 100644
--- a/extensions/mongodb-client/deployment/src/main/java/io/quarkus/mongodb/deployment/DevServicesMongoProcessor.java
+++ b/extensions/mongodb-client/deployment/src/main/java/io/quarkus/mongodb/deployment/DevServicesMongoProcessor.java
@@ -117,14 +117,6 @@ public void run() {
             };
             QuarkusClassLoader cl = (QuarkusClassLoader) Thread.currentThread().getContextClassLoader();
             ((QuarkusClassLoader) cl.parent()).addCloseTask(closeTask);
-            Thread closeHookThread = new Thread(closeTask, "Mongo container shutdown thread");
-            Runtime.getRuntime().addShutdownHook(closeHookThread);
-            ((QuarkusClassLoader) cl.parent()).addCloseTask(new Runnable() {
-                @Override
-                public void run() {
-                    Runtime.getRuntime().removeShutdownHook(closeHookThread);
-                }
-            });
         }
         closeables = currentCloseables;
         capturedProperties = currentCapturedProperties;
diff --git a/extensions/oidc/deployment/src/main/java/io/quarkus/oidc/deployment/devservices/keycloak/KeycloakDevServicesProcessor.java b/extensions/oidc/deployment/src/main/java/io/quarkus/oidc/deployment/devservices/keycloak/KeycloakDevServicesProcessor.java
index 3e053fb9290..74791296bc7 100644
--- a/extensions/oidc/deployment/src/main/java/io/quarkus/oidc/deployment/devservices/keycloak/KeycloakDevServicesProcessor.java
+++ b/extensions/oidc/deployment/src/main/java/io/quarkus/oidc/deployment/devservices/keycloak/KeycloakDevServicesProcessor.java
@@ -160,14 +160,6 @@ public void run() {
             };
             QuarkusClassLoader cl = (QuarkusClassLoader) Thread.currentThread().getContextClassLoader();
             ((QuarkusClassLoader) cl.parent()).addCloseTask(closeTask);
-            Thread closeHookThread = new Thread(closeTask, "Keycloak container shutdown thread");
-            Runtime.getRuntime().addShutdownHook(closeHookThread);
-            ((QuarkusClassLoader) cl.parent()).addCloseTask(new Runnable() {
-                @Override
-                public void run() {
-                    Runtime.getRuntime().removeShutdownHook(closeHookThread);
-                }
-            });
         }
 
         capturedKeycloakUrl = startResult.url + "/auth";
diff --git a/extensions/redis-client/deployment/src/main/java/io/quarkus/redis/client/deployment/DevServicesRedisProcessor.java b/extensions/redis-client/deployment/src/main/java/io/quarkus/redis/client/deployment/DevServicesRedisProcessor.java
index 8d55ad11c6e..52a73e09abf 100644
--- a/extensions/redis-client/deployment/src/main/java/io/quarkus/redis/client/deployment/DevServicesRedisProcessor.java
+++ b/extensions/redis-client/deployment/src/main/java/io/quarkus/redis/client/deployment/DevServicesRedisProcessor.java
@@ -117,9 +117,6 @@ public void startRedisContainers(LaunchModeBuildItem launchMode,
             };
             QuarkusClassLoader cl = (QuarkusClassLoader) Thread.currentThread().getContextClassLoader();
             ((QuarkusClassLoader) cl.parent()).addCloseTask(closeTask);
-            Thread closeHookThread = new Thread(closeTask, "Redis container shutdown thread");
-            Runtime.getRuntime().addShutdownHook(closeHookThread);
-            ((QuarkusClassLoader) cl.parent()).addCloseTask(() -> Runtime.getRuntime().removeShutdownHook(closeHookThread));
         }
     }
 
diff --git a/extensions/smallrye-reactive-messaging-amqp/deployment/src/main/java/io/quarkus/smallrye/reactivemessaging/amqp/deployment/DevServicesAmqpProcessor.java b/extensions/smallrye-reactive-messaging-amqp/deployment/src/main/java/io/quarkus/smallrye/reactivemessaging/amqp/deployment/DevServicesAmqpProcessor.java
index 4dd81decd3c..5031d9c77c8 100644
--- a/extensions/smallrye-reactive-messaging-amqp/deployment/src/main/java/io/quarkus/smallrye/reactivemessaging/amqp/deployment/DevServicesAmqpProcessor.java
+++ b/extensions/smallrye-reactive-messaging-amqp/deployment/src/main/java/io/quarkus/smallrye/reactivemessaging/amqp/deployment/DevServicesAmqpProcessor.java
@@ -107,9 +107,6 @@ public DevServicesAmqpBrokerBuildItem startAmqpDevService(
             };
             QuarkusClassLoader cl = (QuarkusClassLoader) Thread.currentThread().getContextClassLoader();
             ((QuarkusClassLoader) cl.parent()).addCloseTask(closeTask);
-            Thread closeHookThread = new Thread(closeTask, "AMQP container shutdown thread");
-            Runtime.getRuntime().addShutdownHook(closeHookThread);
-            ((QuarkusClassLoader) cl.parent()).addCloseTask(() -> Runtime.getRuntime().removeShutdownHook(closeHookThread));
         }
         cfg = configuration;
         return artemis;
diff --git a/extensions/vault/deployment/src/main/java/io/quarkus/vault/deployment/DevServicesVaultProcessor.java b/extensions/vault/deployment/src/main/java/io/quarkus/vault/deployment/DevServicesVaultProcessor.java
index a76c25835fe..0d897b7f310 100644
--- a/extensions/vault/deployment/src/main/java/io/quarkus/vault/deployment/DevServicesVaultProcessor.java
+++ b/extensions/vault/deployment/src/main/java/io/quarkus/vault/deployment/DevServicesVaultProcessor.java
@@ -93,14 +93,6 @@ public void run() {
             };
             QuarkusClassLoader cl = (QuarkusClassLoader) Thread.currentThread().getContextClassLoader();
             ((QuarkusClassLoader) cl.parent()).addCloseTask(closeTask);
-            Thread closeHookThread = new Thread(closeTask, "Vault container shutdown thread");
-            Runtime.getRuntime().addShutdownHook(closeHookThread);
-            ((QuarkusClassLoader) cl.parent()).addCloseTask(new Runnable() {
-                @Override
-                public void run() {
-                    Runtime.getRuntime().removeShutdownHook(closeHookThread);
-                }
-            });
         }
         for (Map.Entry<String, String> entry : connectionProperties.entrySet()) {
             devConfig.produce(new DevServicesConfigResultBuildItem(entry.getKey(), entry.getValue()));
diff --git a/test-framework/junit5/src/main/java/io/quarkus/test/junit/QuarkusTestExtension.java b/test-framework/junit5/src/main/java/io/quarkus/test/junit/QuarkusTestExtension.java
index fba678a1b19..7daaf031e8a 100644
--- a/test-framework/junit5/src/main/java/io/quarkus/test/junit/QuarkusTestExtension.java
+++ b/test-framework/junit5/src/main/java/io/quarkus/test/junit/QuarkusTestExtension.java
@@ -338,6 +338,7 @@ public Thread newThread(Runnable r) {
                         .setTest(true)
                         .build()
                         .bootstrap();
+                shutdownTasks.add(curatedApplication::close);
             }
 
             if (curatedApplication.getAppModel().getUserDependencies().isEmpty()) {
@@ -1260,7 +1261,11 @@ public void close() {
                         ConfigProviderResolver.setInstance(null);
                     }
                 }
-                Runtime.getRuntime().removeShutdownHook(shutdownHook);
+                try {
+                    Runtime.getRuntime().removeShutdownHook(shutdownHook);
+                } catch (Throwable t) {
+                    //won't work if we are already shutting down
+                }
             }
         }
     }