Repository: wordpress-mobile/WordPress-Android (language: kotlin)

Issue:
CompletionHandlerException: Exception in resume onCancellation handler for CancellableContinuation(DispatchedContinuation[Dis...
Sentry Issue: [JETPACK-ANDROID-92S](https://a8c.sentry.io/issues/4295575735/?referrer=github_integration)

```
IllegalStateException: This mutex is not locked
    at kotlinx.coroutines.sync.MutexImpl.unlock(Mutex.kt:207)
    at kotlinx.coroutines.sync.MutexImpl$CancellableContinuationWithOwner$tryResume$token$1.invoke(Mutex.kt:255)
    at kotlinx.coroutines.sync.MutexImpl$CancellableContinuationWithOwner$tryResume$token$1.invoke(Mutex.kt:252)
    at kotlinx.coroutines.CancellableContinuationImpl.callOnCancellation(CancellableContinuationImpl.kt:259)
    at kotlinx.coroutines.CompletedContinuation.invokeHandlers(CancellableContinuationImpl.kt:659)
...
(8 additional frame(s) were not displayed)

CompletionHandlerException: Exception in resume onCancellation handler for CancellableContinuation(DispatchedContinuation[Dispatchers.IO, Continuation at org.wordpress.android.ui.uploads.UploadStarter$upload$2.invokeSuspend(UploadStarter.kt:211)@df58903]){Completed}@c9c7080
    at kotlinx.coroutines.CancellableContinuationImpl.callOnCancellation(CancellableContinuationImpl.kt:264)
    at kotlinx.coroutines.CompletedContinuation.invokeHandlers(CancellableContinuationImpl.kt:659)
    at kotlinx.coroutines.CancellableContinuationImpl.cancelCompletedResult$kotlinx_coroutines_core(CancellableContinuationImpl.kt:180)
    at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)
    at kotlinx.coroutines.internal.LimitedDispatcher$Worker.run(LimitedDispatcher.kt:115)
...
(5 additional frame(s) were not displayed)
```

This is not a duplicate of #17463 since this is now happening on coroutines 1.7.1 which fixed the underlying root cause of that other ticket. This has a similar stacktrace, but it looks like we might be misusing the Mutex in our `UploadStarter` implementation.


Discussion hints:
**TLDR:** I think our code is silently making the crash look like it’s an internal Mutex problem but in reality, we might be calling `unlock` unnecessarily when the Mutex internals are not expecting it. We should remove [this line](https://github.com/wordpress-mobile/WordPress-Android/blob/1b5e1f3fc97e13d0068ea803d5df7777f421180b/WordPress/src/main/java/org/wordpress/android/ui/uploads/UploadStarter.kt#L186) and I think it might solve our issues on coroutines 1.7.1.

Based on the [breadcrumbs](https://a8c.sentry.io/issues/4295575735/?project=5731682&referrer=issue-stream&sort=freq&statsPeriod=14d&stream_index=1#breadcrumbs) I believe the actual issue is that we tried to [start the post upload service](https://github.com/wordpress-mobile/WordPress-Android/blob/1b5e1f3fc97e13d0068ea803d5df7777f421180b/WordPress/src/main/java/org/wordpress/android/ui/uploads/UploadStarter.kt#L175-L179) while the app was in the background and the system didn’t let us, which caused all `upload` jobs (created [here](https://github.com/wordpress-mobile/WordPress-Android/blob/1b5e1f3fc97e13d0068ea803d5df7777f421180b/WordPress/src/main/java/org/wordpress/android/ui/uploads/UploadStarter.kt#L138-L142)) to be canceled since the [Parent Job](https://github.com/wordpress-mobile/WordPress-Android/blob/1b5e1f3fc97e13d0068ea803d5df7777f421180b/WordPress/src/main/java/org/wordpress/android/ui/uploads/UploadStarter.kt#L68) is not a `SupervisorJob`.

This made all jobs (except the first one which failed because of an `IllegalStateException`) [fall here](https://github.com/wordpress-mobile/WordPress-Android/blob/1b5e1f3fc97e13d0068ea803d5df7777f421180b/WordPress/src/main/java/org/wordpress/android/ui/uploads/UploadStarter.kt#L183-L187) (as can be seen in the breadcrumb logs).

**What I first thought could be happening is:**

We use the `bgDispatcher` ([Dispatchers.Default](https://github.com/wordpress-mobile/WordPress-Android/blob/e03d208247c7487a7497f873d91f6b0e52a3622e/WordPress/src/main/java/org/wordpress/android/modules/ThreadModule.kt#L40-L42)) which has several threads in the pool and we use the same `mutex` for all upload jobs. Because of that, multiple coroutines (running in different threads) are suspended awaiting the to [acquire](https://github.com/Kotlin/kotlinx.coroutines/blob/02382479398518ea1c4507170aa92692595b6839/kotlinx-coroutines-core/common/src/sync/Semaphore.kt#L196-L201C6) the semaphore (which [creates a queue of segments](https://github.com/Kotlin/kotlinx.coroutines/blob/02382479398518ea1c4507170aa92692595b6839/kotlinx-coroutines-core/common/src/sync/Semaphore.kt#L285-L315)).

Since multiple jobs are being canceled at once, I think that the cancellation was not able to be properly propagated and the `segment` state for one of them was not marked as `CANCELLED` making the `Semaphore#tryResumeNextFromQueue` ([which is called on](https://github.com/Kotlin/kotlinx.coroutines/blob/1.7.1/kotlinx-coroutines-core/common/src/sync/Semaphore.kt#L265) `Semaphore#release` [via the](https://github.com/Kotlin/kotlinx.coroutines/blob/02382479398518ea1c4507170aa92692595b6839/kotlinx-coroutines-core/common/src/sync/Mutex.kt#L216) `Mutex#unlock`) [call the](https://github.com/Kotlin/kotlinx.coroutines/blob/02382479398518ea1c4507170aa92692595b6839/kotlinx-coroutines-core/common/src/sync/Semaphore.kt#L329-L341) `tryResumeAcquire`. Which in turn [calls](https://github.com/Kotlin/kotlinx.coroutines/blob/02382479398518ea1c4507170aa92692595b6839/kotlinx-coroutines-core/common/src/sync/Semaphore.kt#L347C23-L347C23) `Mutex#tryResume` but since that is canceled it runs the `onCancellation` [callback](https://github.com/Kotlin/kotlinx.coroutines/blob/02382479398518ea1c4507170aa92692595b6839/kotlinx-coroutines-core/common/src/sync/Mutex.kt#L253-L255) when trying to resume the underlying continuation, which is [where the crash is happening](https://github.com/Kotlin/kotlinx.coroutines/blob/02382479398518ea1c4507170aa92692595b6839/kotlinx-coroutines-core/common/src/sync/Mutex.kt#L250-L262).

But I am not 100% sure about that, as I feel like this scenario would be pretty common for a synchronization mechanism using coroutines (like `Mutex` and `Semaphore`), which is why I believe that `onCancellation` callback also unlocks the `Mutex`, in case of that late propagation of the cancellation to the mutex `Segment`.

**So, considering that the code makes sense and works (hopefully) what I think might be the issue is:**

We also have a [call to](https://github.com/wordpress-mobile/WordPress-Android/blob/1b5e1f3fc97e13d0068ea803d5df7777f421180b/WordPress/src/main/java/org/wordpress/android/ui/uploads/UploadStarter.kt#L186) `unlock` on our side and from the breadcrumbs we know that the code is reaching that line. So it might be that this `unlock` is actually unlocking the mutex **at the same time as** the whole cancellation flow I mentioned above is happening (_which would be possible because of the multiple threads in the Dispatcher pool_), so when the cancelation reaches the `onCancellation` [callback here](https://github.com/Kotlin/kotlinx.coroutines/blob/02382479398518ea1c4507170aa92692595b6839/kotlinx-coroutines-core/common/src/sync/Mutex.kt#L253-L255) our code already unlocked it, causing the crash to happen in the `Mutex` internals.

This is my best guess, but the Mutex/Semaphore code is a bit complex and I didn’t fully understand it yet (and probably never will 😆).

### Suggestion
Remove that `unlock` [call ASAP](https://github.com/wordpress-mobile/WordPress-Android/blob/1b5e1f3fc97e13d0068ea803d5df7777f421180b/WordPress/src/main/java/org/wordpress/android/ui/uploads/UploadStarter.kt#L186) as we really don’t need it since we use `withLock` which is better in 2 ways:
1. It makes sure to `unlock` when exceptions happen ([try/finally](https://github.com/Kotlin/kotlinx.coroutines/blob/02382479398518ea1c4507170aa92692595b6839/kotlinx-coroutines-core/common/src/sync/Mutex.kt#L120-L131))
2. It only calls `unlock` if it was successfully locked before running the `action`, so if my hypothesis above is correct, the internal `lockSuspend` code will cause a cancellation before reaching that `try/finally` block


Pull request: [Post Upload] Crash fix: remove call to mutex#unlock from UploadStarter
Fixes #18958

The `mutex.unlock` call is redundant (because we already use `withLock`, which unlocks after the block runs) and can be the source of crashes since the mutex cannot be unlocked if it's not locked and the internal `Mutex` source code tries to unlock it in certain Cancellation situations.

More info in the issue comment: https://github.com/wordpress-mobile/WordPress-Android/issues/18958#issuecomment-1675239908

To test:
We could not find a way of reproducing this issue as it might be related to some race conditions, multiple threads, and likely, how the device / system handles them.

## Regression Notes
1. Potential unintended areas of impact
N/A

2. What I did to test those areas of impact (or what existing automated tests I relied on)
N/A

3. What automated tests I added (or what prevented me from doing so)
WIP

PR submission checklist:

- [x] I have completed the Regression Notes.
- [x] I have considered adding accessibility improvements for my changes.
- [x] I have considered if this change warrants user-facing release notes and have added them to `RELEASE-NOTES.txt` if necessary.

UI Changes testing checklist:
None

Test command: ./gradlew :WordPress:testWordpressVanillaDebugUnitTest --tests "org.wordpress.android.ui.uploads.UploadStarterMutexTest"

Fix patch:
diff --git a/WordPress/src/main/java/org/wordpress/android/ui/uploads/UploadStarter.kt b/WordPress/src/main/java/org/wordpress/android/ui/uploads/UploadStarter.kt
index 66bff7046d62..ceca1428fc62 100644
--- a/WordPress/src/main/java/org/wordpress/android/ui/uploads/UploadStarter.kt
+++ b/WordPress/src/main/java/org/wordpress/android/ui/uploads/UploadStarter.kt
@@ -136,7 +136,7 @@ class UploadStarter @Inject constructor(
         }
 
         sites.forEach {
-            launch(ioDispatcher) {
+            launch {
                 upload(site = it)
             }
         }
@@ -148,8 +148,8 @@ class UploadStarter @Inject constructor(
     private suspend fun upload(site: SiteModel) = coroutineScope {
         try {
             mutex.withLock {
-                val posts = async { postStore.getPostsWithLocalChanges(site) }
-                val pages = async { pageStore.getPagesWithLocalChanges(site) }
+                val posts = async(ioDispatcher) { postStore.getPostsWithLocalChanges(site) }
+                val pages = async(ioDispatcher) { pageStore.getPagesWithLocalChanges(site) }
                 val list = posts.await() + pages.await()
 
                 list.asSequence()
@@ -181,9 +181,7 @@ class UploadStarter @Inject constructor(
             }
         } catch (e: CancellationException) {
             AppLog.e(T.MEDIA, e)
-            // Do any needed actions while we are still holding the mutex lock, then release it and rethrow the
-            // exception so it can be handled upstream
-            mutex.unlock()
+            // rethrow the exception so it can be handled upstream
             throw e
         }
     }


Test patch:
diff --git a/WordPress/src/test/java/org/wordpress/android/ui/uploads/UploadStarterMutexTest.kt b/WordPress/src/test/java/org/wordpress/android/ui/uploads/UploadStarterMutexTest.kt
new file mode 100644
index 000000000000..611e018567a2
--- /dev/null
+++ b/WordPress/src/test/java/org/wordpress/android/ui/uploads/UploadStarterMutexTest.kt
@@ -0,0 +1,142 @@
+package org.wordpress.android.ui.uploads
+
+import android.content.Context
+import androidx.lifecycle.MutableLiveData
+import kotlinx.coroutines.ExperimentalCoroutinesApi
+import kotlinx.coroutines.asCoroutineDispatcher
+import kotlinx.coroutines.delay
+import org.junit.Before
+import org.junit.Test
+import org.mockito.Mock
+import org.mockito.kotlin.any
+import org.mockito.kotlin.doAnswer
+import org.mockito.kotlin.doSuspendableAnswer
+import org.mockito.kotlin.mock
+import org.mockito.kotlin.whenever
+import org.wordpress.android.BaseUnitTest
+import org.wordpress.android.fluxc.Dispatcher
+import org.wordpress.android.fluxc.model.PostModel
+import org.wordpress.android.fluxc.store.PageStore
+import org.wordpress.android.fluxc.store.PostStore
+import org.wordpress.android.fluxc.store.SiteStore
+import org.wordpress.android.util.NetworkUtilsWrapper
+import org.wordpress.android.util.analytics.AnalyticsTrackerWrapper
+import org.wordpress.android.viewmodel.helpers.ConnectionStatus
+import java.util.concurrent.Executors
+
+/**
+ * This class tries to reproduce the situation of trying to start the upload service when the device is in the
+ * background. Such situation causes the system to throw a IllegalStateException, as it's not allowed to create
+ * services from the background, which in turns causes a failure in the coroutine jobs, which should be cancelled
+ * without crashing the application.
+ *
+ * This was created because the situation above was causing a crash in our app:
+ * https://a8c.sentry.io/issues/4295575735/?project=573168
+ */
+@OptIn(ExperimentalCoroutinesApi::class)
+class UploadStarterMutexTest : BaseUnitTest() {
+    private val bgDispatcher = Executors.newFixedThreadPool(DEFAULT_THREADS).asCoroutineDispatcher()
+    private val ioDispatcher = Executors.newFixedThreadPool(IO_THREADS).asCoroutineDispatcher()
+
+    @Mock
+    lateinit var context: Context
+
+    @Mock
+    lateinit var dispatcher: Dispatcher
+
+    @Mock
+    lateinit var postStore: PostStore
+
+    @Mock
+    lateinit var pageStore: PageStore
+
+    @Mock
+    lateinit var siteStore: SiteStore
+
+    @Mock
+    lateinit var uploadActionUseCase: UploadActionUseCase
+
+    @Mock
+    lateinit var tracker: AnalyticsTrackerWrapper
+
+    @Mock
+    lateinit var uploadServiceFacade: UploadServiceFacade
+
+    @Mock
+    lateinit var networkUtilsWrapper: NetworkUtilsWrapper
+
+    private var connectionStatus: MutableLiveData<ConnectionStatus> = MutableLiveData()
+
+    private lateinit var uploadStarter: UploadStarter
+
+    @Before
+    fun setUp() {
+        uploadStarter = UploadStarter(
+            context = context,
+            dispatcher = dispatcher,
+            postStore = postStore,
+            pageStore = pageStore,
+            siteStore = siteStore,
+            uploadActionUseCase = uploadActionUseCase,
+            tracker = tracker,
+            bgDispatcher = bgDispatcher,
+            ioDispatcher = ioDispatcher,
+            uploadServiceFacade = uploadServiceFacade,
+            networkUtilsWrapper = networkUtilsWrapper,
+            connectionStatus = connectionStatus
+        )
+    }
+
+    @Test
+    fun `should not crash if uploadPost throws an exception`() = test {
+        // network is available
+        whenever(networkUtilsWrapper.isNetworkAvailable()).thenReturn(true)
+        // return 10 sites for queuing
+        whenever(siteStore.sites).thenReturn(List(10) { mock() })
+        // each site has 10 posts with local changes
+        whenever(postStore.getPostsWithLocalChanges(any())).doAnswer {
+            Thread.sleep(10)
+            List(10) { fakePostModel(it, false) }
+        }
+        // each site has 5 pages with local changes
+        whenever(pageStore.getPagesWithLocalChanges(any())).doSuspendableAnswer {
+            delay(10)
+            List(5) {
+                fakePostModel(
+                    100 + it,
+                    true
+                )
+            }
+        }
+
+        whenever(uploadActionUseCase.getAutoUploadAction(any(), any()))
+            .thenReturn(UploadActionUseCase.UploadAction.UPLOAD)
+
+        // throw IllegalStateException when uploading post (emulate trying to start background service)
+        whenever(uploadServiceFacade.uploadPost(any(), any<PostModel>(), any()))
+            .thenThrow(IllegalStateException("FAKE: Not allowed to start service intent"))
+
+        // ACT
+        uploadStarter.queueUploadFromAllSites().join()
+    }
+
+    companion object {
+        // # of threads in Dispatchers.Default is equal to the # of CPU cores, 8 is a common # of cores in modern phones
+        // https://kotlinlang.org/api/kotlinx.coroutines/kotlinx-coroutines-core/kotlinx.coroutines/-dispatchers/-default.html
+        private const val DEFAULT_THREADS = 8
+
+        // default # of threads in Dispatchers.IO is 64 but it can be even larger, let's use 64, which is enough
+        // https://kotlinlang.org/api/kotlinx.coroutines/kotlinx-coroutines-core/kotlinx.coroutines/-dispatchers/-i-o.html
+        private const val IO_THREADS = 64
+
+        private fun fakePostModel(
+            id: Int,
+            isPage: Boolean,
+        ) = PostModel().apply {
+            setId(id)
+            setIsPage(isPage)
+            setTitle("Title $id")
+            setStatus("draft")
+        }
+    }
+}