Repository: kotlin/kotlinx.coroutines
Issue: Closing newFixedThreadPoolContext on K/N may lead to an application crash

We have a non-trivial bug on the edge of `MultiWorkerDispatcher` shutdown sequence and our channels' linearizability.

1) All tasks are dispatched within `newFixedThreadPoolContext` using the channel as a substitute for blocking queue
2) Shutdown sequence if dispatcher first closes the channel, then invokes `Worker.requestTermination` for each existing worker in the pool 
3) `Channels.close` is linearizable via helping technique: when any channel's operation detects `close` operation in progress, it starts helping it -- it processes a list of existing enqueued waiters about the close and `resume`s them.
3.1) Helping is "non-atomic", meaning that first helper removes the enqueued waiter from the internal list and then it resumes it.


Now consider the following scenario:

[T1] `close` starts closing the channel
[T2] one of the workers start helping, removes single waiter from the queue, gets preempted 
[T1] `close` finishes -- there are no more waiters, channel is in its final state
[T1] All the workers are closed via `requestTermination`. There is no work for workers, so they are terminated
[T2] finally gets scheduled back and invokes `resume` on the waiter, leading to `Worker.executeAfter` on an already destroyed worker


Relevant stacktrace:
```
Uncaught Kotlin exception: kotlin.IllegalStateException: Worker is already terminated
Invalid connection: com.apple.coresymbolicationd
    at 0   workerWithNewMM.kexe                0x1040bae7d        ThrowWorkerAlreadyTerminated + 157 (/opt/buildAgent/work/5f69639f351c4725/kotlin/kotlin-native/runtime/src/main/kotlin/kotlin/native/concurrent/Internal.kt:68:15)
    at 1   workerWithNewMM.kexe                0x1040bdd4f        kfun:kotlin.native.concurrent.Worker#executeAfter(kotlin.Long;kotlin.Function0<kotlin.Unit>){} + 639 (/opt/buildAgent/work/5f69639f351c4725/kotlin/kotlin-native/runtime/src/main/kotlin/kotlin/native/concurrent/Worker.kt:125:9)
    at 2   workerWithNewMM.kexe                0x10416097c        kfun:kotlinx.coroutines.EventLoopImplBase#enqueue(kotlinx.coroutines.Runnable){} + 140 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/EventLoop.common.kt:291:13)
Child process terminated with signal 6: Abort trap
    at 3   workerWithNewMM.kexe                0x104156c10        kfun:kotlinx.coroutines.CancellableContinuationImpl.dispatchResume#internal + 688 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/CancellableContinuationImpl.kt:<unknown>)
    at 4   workerWithNewMM.kexe                0x104157474        kfun:kotlinx.coroutines.CancellableContinuationImpl.resumeImpl#internal + 564 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/CancellableContinuationImpl.kt:451:21)
    at 5   workerWithNewMM.kexe                0x10415758d        kfun:kotlinx.coroutines.CancellableContinuationImpl#resumeImpl$default(kotlin.Any?;kotlin.Int;kotlin.Function1<kotlin.Throwable,kotlin.Unit>?;kotlin.Int){} + 141 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/CancellableContinuationImpl.kt:440:13)
    at 6   workerWithNewMM.kexe                0x1041561e7        kfun:kotlinx.coroutines.CancellableContinuationImpl#resumeWith(kotlin.Result<1:0>){} + 471 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/CancellableContinuationImpl.kt:348:9)
    at 7   workerWithNewMM.kexe                0x1041871b1        kfun:kotlinx.coroutines.channels.BufferedChannel.closeWaiter#internal + 1441 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/channels/BufferedChannel.kt:1753:37)
    at 8   workerWithNewMM.kexe                0x104185fb4        kfun:kotlinx.coroutines.channels.BufferedChannel.completeClose#internal + 2500 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/channels/BufferedChannel.kt:1739:42)
    at 9   workerWithNewMM.kexe                0x104187802        kfun:kotlinx.coroutines.channels.BufferedChannel.isClosed#internal + 146 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/channels/BufferedChannel.kt:<unknown>)
    at 10  workerWithNewMM.kexe                0x104183b92        kfun:kotlinx.coroutines.channels.BufferedChannel.BufferedChannelIterator.$hasNextCOROUTINE$2784#invokeSuspend(kotlin.Result<kotlin.Any?>){}kotlin.Any? + 418 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/channels/BufferedChannel.kt:<unknown>)
    at 11  workerWithNewMM.kexe                0x104184846        kfun:kotlinx.coroutines.channels.BufferedChannel.BufferedChannelIterator#hasNext(){}kotlin.Boolean + 198 (/opt/buildAgent/work/44ec6e850d5c63f0/kotlinx-coroutines-core/common/src/channels/BufferedChannel.kt:1395:26)
```

Changed files:
kotlinx-coroutines-core/native/src/EventLoop.kt
kotlinx-coroutines-core/native/src/MultithreadedDispatchers.kt

diff --git a/kotlinx-coroutines-core/native/src/EventLoop.kt b/kotlinx-coroutines-core/native/src/EventLoop.kt
index 25c3c12b7..149c6fe0b 100644
--- a/kotlinx-coroutines-core/native/src/EventLoop.kt
+++ b/kotlinx-coroutines-core/native/src/EventLoop.kt
@@ -13,7 +13,14 @@ internal actual abstract class EventLoopImplPlatform : EventLoop() {
     private val current = Worker.current
 
     protected actual fun unpark() {
-        current.executeAfter(0L, {})// send an empty task to unpark the waiting event loop
+        try {
+            current.executeAfter(0L, {}) // send an empty task to unpark the waiting event loop
+        } catch (e: IllegalStateException) {
+            // We deliberately ignore ISE here as they are expected
+            // due to peculiarities of Workers API.
+            // Unfortunately, race-free termination of workers is unachievable by its current state,
+            // see https://github.com/Kotlin/kotlinx.coroutines/issues/3578
+        }
     }
 
     protected actual fun reschedule(now: Long, delayedTask: EventLoopImplBase.DelayedTask) {
diff --git a/kotlinx-coroutines-core/native/src/MultithreadedDispatchers.kt b/kotlinx-coroutines-core/native/src/MultithreadedDispatchers.kt
index bf91e7003..1f495a4b6 100644
--- a/kotlinx-coroutines-core/native/src/MultithreadedDispatchers.kt
+++ b/kotlinx-coroutines-core/native/src/MultithreadedDispatchers.kt
@@ -79,14 +79,23 @@ private class MultiWorkerDispatcher(
         }
     }
 
-    private fun workerRunLoop() = runBlocking {
-        // NB: we leverage tail-call optimization in this loop, do not replace it with
-        // .receive() without proper evaluation
-        for (task in tasksQueue) {
-            /**
-             * Any unhandled exception here will pass through worker's boundary and will be properly reported.
-             */
-            task.run()
+    private fun workerRunLoop() {
+        try {
+            runBlocking {
+                // NB: we leverage tail-call optimization in this loop, do not replace it with
+                // .receive() without proper evaluation
+                for (task in tasksQueue) {
+                    /**
+                     * Any unhandled exception here will pass through worker's boundary and will be properly reported.
+                     */
+                    task.run()
+                }
+            }
+        } catch (e: IllegalStateException) {
+            // We deliberately ignore ISE here as they are expected
+            // due to peculiarities of Workers API.
+            // Unfortunately, race-free termination is unachievable by its current state,
+            // see https://github.com/Kotlin/kotlinx.coroutines/issues/3578
         }
     }