Repository: mockito/mockito
Issue: Set up Android Github Action pipeline

Coming from https://github.com/mockito/mockito/issues/2890#issuecomment-1409200309:

> we started an effort to set up some basic Android tests for us: https://github.com/mockito/mockito/tree/main/subprojects/androidTest However, this stalled as we couldn't get it to work on CI. If anybody affected here would like to dedicate some time to prevent these kind of issues from happening, please take a look at the existing test suite and infrastructure and submit a PR to get it passing on CI.

The presence of such Github Action would have helped to prevent regressions like https://github.com/mockito/mockito/issues/2890. 

@TimvdLippe I will try to pull it off (I don't work with Android sadly), it seems like Github Actions have support for it.

Issue comments:
I am seeing some failures on CI with the new API 33 level job:

```
> Task :androidTest:connectedDebugAndroidTest
Exception thrown during onBeforeAll invocation of plugin com.google.testing.platform.plugin.android.AndroidDevicePlugin.
Failed to install APK(s): /Users/runner/work/mockito/mockito/subprojects/androidTest/build/outputs/apk/androidTest/debug/androidTest-debug-androidTest.apk
Unknown failure: cmd: Can't find service: package
com.android.ddmlib.InstallException: Unknown failure: cmd: Can't find service: package
	at com.android.ddmlib.internal.DeviceImpl.installRemotePackage(DeviceImpl.java:1315)
	at com.android.ddmlib.internal.DeviceImpl.installPackage(DeviceImpl.java:1141)
	at com.android.tools.utp.plugins.deviceprovider.ddmlib.DdmlibAndroidDevice.installPackage(DdmlibAndroidDevice.kt)
	at com.android.tools.utp.plugins.deviceprovider.ddmlib.DdmlibAndroidDeviceController$executeAsync$deferred$1.invokeSuspend(DdmlibAndroidDeviceController.kt:173)
	at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
	at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:106)
	at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:571)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:750)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:678)
	at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:665)
```

From https://github.com/mockito/mockito/actions/runs/4106509220/jobs/7084834115

@TWiStErRob do you know what is going on here?
I have an idea, will raise a PR a bit later. You can rerun, and it should very likely work the second time.
Created #2903, hopefully it will help. I've got many builds as you can see on the commit history, none of them failed for any weird reason, so I'm hopeful.
Unfortunately we got another failure, this time on 26: https://github.com/mockito/mockito/actions/runs/4150981851/jobs/7180984789
🙄 I'm starting to think it's related to the time of day...
> 🙄 I'm starting to think it's related to the time of day...

Should we try to reduce just to one Android version for now? It won't solve the issue but we could have less often CI failures while we are looking for solution(s) to make it stable?
I would say let's leave it for a week or two, and see actually how bad it is. For now we see it failed once... out of 2 executions. That could be 1% or 50%, depending on how many you measure with.

If it's too bad, I guess we could move to be part of the release process to reduce the burden, but keep the safety the tests provide.

Or we could add retry the same way you have retry for normal tests, which was surprising to see.
Unfortunately another failure on `main`: https://github.com/mockito/mockito/actions/runs/4168588270/jobs/7215565389#step:4:706

```
Failed to install APK(s): /Users/runner/work/mockito/mockito/subprojects/androidTest/build/outputs/apk/androidTest/debug/androidTest-debug-androidTest.apk
Unknown failure: cmd: Can't find service: package
com.android.ddmlib.InstallException: Unknown failure: cmd: Can't find service: package
```
@TWiStErRob @TimvdLippe so what should we do? I would still try to reduce Android matrix to one version, to minimize the number of possible build failures ...
I would try to add retry first, but up to you. We could reduce it to 1 API version, which would still fail sometimes, or move the whole thing to release time, and run it once at the end of the cycle; or somewhere in the middle, running it scheduled in the background on `main`, but then who/when will look at the results?

@chrisbanes as the main tamer of Android UI libs how do you think we can reduce flakiness on this trivial Android test project? The problem is that the emulator doesn't start up quite often, the actual tests are fine so far.

CI: https://github.com/mockito/mockito/blob/main/.github/workflows/ci.yml#L89

Project: https://github.com/mockito/mockito/tree/main/subprojects/androidTest

Example failures: #2903's OP ("Investigation" section, first 3)
To be honest, I've had very little issues with GitHub Actions (other than starting the emulator is slow).

I'd recommend sticking to the `default` images, and `google_apis` if necessary.

You might also want to explore Gradle Managed Devices. https://developer.android.com/studio/test/gradle-managed-devices

It simplifies the setup greatly, but you need to use recent AGP versions for recent fixes.
The Android 33 job fails the most (2 instances tonight, with the latest being https://github.com/mockito/mockito/actions/runs/4231340587/jobs/7361785524). Therefore, I am inclined to remove Android 33 for now, as Android 26 appears stable for now.
I found [a way to consistently trigger a quick boot on the emulator from a clean state (WIP)](https://github.com/TWiStErRob/github-workflows/pull/7), this might solve issues 1-3 listed in https://github.com/mockito/mockito/pull/2903 and potentially could help with 4 as well, since the emulator always starts from a consistent state. The tradeoff: ~2GB of cache for the emulator Disk and RAM.

Example cache:
![image](https://github.com/mockito/mockito/assets/2906988/2e87bec1-c0f9-40c2-96ed-473289cd8fe9)

Example quick boot:
![image](https://github.com/mockito/mockito/assets/2906988/74e0cc45-dc51-41e2-9dec-24e1e2d08bc7)

<details><summary>Compared to a "slow boot"</summary>

which might or might not time out.

![image](https://github.com/mockito/mockito/assets/2906988/8f8be731-2007-4708-bed4-8d7f0662133d)

</details>

Note: the slow boot still needs to be performed probably once a week to fill the cache, but the chance of failure **should** go way down.

I'll raise a PR to change the CI once I'm done with my workflow change and used it a bit "in production".
This is great, thanks a lot for the update @TWiStErRob !

Pull request review comments:
You probably don't even need these two steps, because Android Gradle Plugin will download everything automatically (what is needed) and GitHub Actions hosted runners are set up with Android SDK already.

`3.` is only necessary if you use self-hosted runners.
`4.` is not necessary if you use Gradle.
Could you please add a step (in the same job) for running `gradlew test` too so that unit tests with mocks are also tested. These **should** be just normal JVM tests, but we never know what can break :)
maths :D
I have a feeling matrix won't be necessary like this, because mockito-android has its own mockmaker, right?
ah never mind, that would be covered by normal CI, nice 👏

(iff all the conditions in settings.gradle are met in normal CI)
👍 thanks! still figuring out how to make it work :)
👍 
:D yeah, adding and removing steps, will fix that 😄 
It might be the case indeed, I will check it shortly 
FYI https://github.com/mockito/mockito/pull/2894
(I don't think it's necessary for this pipeline PR, but was inspired :) )
This test seemed to be not valid: the exception was thrown but no validations / verifications or what not
Do you want to keep a 1-element matrix?
It could be fully inlined to the env var values?
Unnecessary comment is unnecessary ;)
All actions are GitHub org/repo@tag (or hash) format. So this mapping is the default resolution.
Agree, it was probably testing failure, because a CI job is as good as how you can diagnose failures.
Upload resulting report as artifact?
Actually might want to introduce more API levels in the future, because they have different JVMs. See my comment on the issue, API 21 fails, while API 31 passes.
Yeah, I think we could remove the mock-maker part
That's a good question for @TimvdLippe , I don't see test reports uploaded (but only coverage reports).
I think we settled on level 26 as the minimum: https://github.com/mockito/mockito/blob/main/build.gradle#L102
For reference the [official docs](https://developer.android.com/about/versions/oreo/android-8.0#art) only list some new APIs, so even Android 8 doesn't support Java 11. However, as I understand with [desugaring](https://developer.android.com/studio/write/java11-default-support-table) Java 11+ APIs can be used as low as API 1. This is of course not practical, I think an API 21 minimum would be [enough](https://apilevels.com/). That said [official full Java 11 support is only in 33](https://developer.android.com/about/versions/13/features#core-libraries).

Anyway, based on the info, starting with `android-api: [26,27,28,29,30,31,33]` would work.
([32 has no Java changes](https://developer.android.com/sdk/api_diff/32/changes), it was a "feature drop", not given a major version, if it works, it could be included though)
😲 so how do you know what failed?
https://github.com/mockito/mockito/actions/runs/1865554496/jobs/2625339758
Just to clarify, the matrix should only change the emulator version, the compile/target/minSdk should be hardcoded to highest/lowest values as [Google recommends it (TL;DR: we have ~1 year to use the latest target, otherwise cannot publish)](https://support.google.com/googleplay/android-developer/answer/11926878).
You look into Github action output, like https://github.com/mockito/mockito/pull/2893#issuecomment-1410906236
Hmm... that leaves so many questions open:
```
java.lang.NoClassDefFoundError: org.mockito.internal.debugging.LocationImpl
at org.mockito.internal.debugging.LocationFactory.create(LocationFactory.java:17)
```
is a very minimal output, where is it called from? which code path got here? what's the root "caused by"? What got output to stdout/stderr? How long did it take? etc. All of these are recorded in the report. As an example open any `IntgTest` from [this workflow's "Test Results HTML" artifacts](https://github.com/TWiStErRob/net.twisterrob.gradle/actions/runs/4052565235). Anyway up to you if you (and your contributors) like reading logs :), it would drive me crazy.
:D @TWiStErRob I am totally in for test report publishing (don't get me wrong), I have just included the most interesting part but here is the complete log [1], this is how the test failures are troubleshot at the moment. 

[1] https://github.com/reta/mockito/actions/runs/4057324609/jobs/6982933109
@TWiStErRob so I have to remove 30-31 API levels (for now), the emulator GA seems to incorrectly deduct the arch:

```
/bin/sh -c \sdkmanager --install 'system-images;android-31;default;x86' --channel=0 > /dev/null
  Warning: Failed to find package 'system-images;android-31;default;x86'
```

The result is that emulator is not installed and the test run fails. I think we could hack the arch parameters or just get to that later on when it is fixed. I think it looks good now. Thanks!
It just doesn't exist: `sdkmanager --list | grep android- | grep default`:
```
  system-images;android-26;default;arm64-v8a                                               | 1            | ARM 64 v8a System Image
  system-images;android-26;default;x86                                                     | 1            | Intel x86 Atom System Image
  system-images;android-26;default;x86_64                                                  | 1            | Intel x86 Atom_64 System Image
  system-images;android-27;default;arm64-v8a                                               | 1            | ARM 64 v8a System Image
  system-images;android-27;default;x86                                                     | 1            | Intel x86 Atom System Image
  system-images;android-27;default;x86_64                                                  | 1            | Intel x86 Atom_64 System Image
  system-images;android-28;default;arm64-v8a                                               | 1            | ARM 64 v8a System Image
  system-images;android-28;default;x86                                                     | 4            | Intel x86 Atom System Image
  system-images;android-28;default;x86_64                                                  | 4            | Intel x86 Atom_64 System Image
  system-images;android-29;default;arm64-v8a                                               | 8            | ARM 64 v8a System Image
  system-images;android-29;default;x86                                                     | 8            | Intel x86 Atom System Image
  system-images;android-29;default;x86_64                                                  | 8            | Intel x86 Atom_64 System Image
  system-images;android-30;default;arm64-v8a                                               | 1            | ARM 64 v8a System Image
  system-images;android-30;default;x86_64                                                  | 10           | Intel x86 Atom_64 System Image
  system-images;android-31;default;arm64-v8a                                               | 4            | ARM 64 v8a System Image
  system-images;android-31;default;x86_64                                                  | 4            | Intel x86 Atom_64 System Image

```
I would be very surprised if the "latest" MacOS, that GitHub just released recently would be x86.

The action doesn't deduce the architecture, it's hardcoded to x86. Can you try adding the numbers and forcing the `arch`?

https://github.com/ReactiveCircus/android-emulator-runner/blob/e1248ae048e31ec2e171adc06be585d06c1f936e/action.yml#L14-L16

```diff
      uses: reactivecircus/android-emulator-runner@v2
      with:
        api-level: ${{ matrix.android-api }}
+       arch: x86_64 # or arm64-v8a (don't know if these Macs are M1/M2 or not)
```
`x86_64` does not work for sure, I tried that before, let me try `arm`
What does "does not work for sure" means, because in their examples they also use that:
https://github.com/ReactiveCircus/android-emulator-runner#usage--examples
`PANIC: Avd's CPU Architecture 'arm64' is not supported by the QEMU2 emulator on x86_64 host.` sounds good.
It seems like going up to `31` is fine, updated the matrix, after 31 - failures :(
I think 33 would work, told you 32 is "special": https://github.com/mockito/mockito/pull/2893#discussion_r1093002196

Notice that in the above table there's no 32 and 33, but there are emulators:
```
  system-images;android-32;android-desktop;arm64-v8a                                       | 5            | Desktop ARM 64 v8a System Image
  system-images;android-32;android-desktop;x86_64                                          | 5            | Desktop Intel x86 Atom_64 System Image
  system-images;android-32;google_apis;arm64-v8a                                           | 3            | Google APIs ARM 64 v8a System Image
  system-images;android-32;google_apis;x86_64                                              | 3            | Google APIs Intel x86 Atom_64 System Image
  system-images;android-32;google_apis_playstore;x86_64                                    | 3            | Google Play Intel x86 Atom_64 System Image

  system-images;android-33;google_apis;arm64-v8a                                           | 8            | Google APIs ARM 64 v8a System Image
  system-images;android-33;google_apis;x86_64                                              | 8            | Google APIs Intel x86 Atom_64 System Image
  system-images;android-33;google_apis_playstore;x86_64                                    | 7            | Google Play Intel x86 Atom_64 System Image
```

Let's merge it as is, this is a huge improvement over "nothing". We can figure out 32/33 later.
Oh sorry, I missed that the issue was only specific to 32
Let's see why https://issuetracker.google.com/issues/267458959
We don't have test reporting setup. For that, we rely on console output solely and locally reproducing the error. With CI that is sometimes tricky, but that's the setup we got atm. If you have any suggestions, feel free to create an issue and we can discuss further there.
I don't think we want to test so many Android releases, since we don't have a lot of CPU capacity on GitHub actions. Instead, I think only having 26 (which is the oldest version that we still support) is sufficient. If we notice gaps in Android support, we can always extend the matrix, but for now I think just 1 Android version is good enough.
Good point, very likely if it works on the oldest, it'll work elsewhere too.
That said, I would follow this up with a PR for running 33, because that has full on Java 11 support, which can mess things up. But this PR, even with 1 version is good enough.
Kept only `26`

Patch that resolved the issue:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index 0c57e59086..6ea135eac2 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -85,6 +85,38 @@ jobs:
         chmod +x codecov
         ./codecov
 
+  #
+  # Android build job
+  #
+  android:
+    runs-on: macos-latest
+    if: "! contains(toJSON(github.event.commits.*.message), '[skip ci]')"
+
+    # Definition of the build matrix
+    strategy:
+      matrix:
+        java: [ 11 ]
+        android-api: [ 26 ]
+
+    # All build steps
+    steps:
+    - name: 1. Check out code
+      uses: actions/checkout@v3
+      with:
+        fetch-depth: '0' # https://github.com/shipkit/shipkit-changelog#fetch-depth-on-ci
+
+    - name: 2. Set up Java ${{ matrix.java }}
+      uses: actions/setup-java@v3
+      with:
+        distribution: 'zulu'
+        java-version: ${{ matrix.java }}
+
+    - name: 3. Run Android tests on Java ${{ matrix.java }} with Android API level ${{ matrix.android-api }}
+      uses: reactivecircus/android-emulator-runner@v2
+      with:
+        arch: x86_64
+        api-level: ${{ matrix.android-api }}
+        script: ./gradlew :androidTest:connectedCheck --no-daemon --no-build-cache
 
   #
   # Release job, only for pushes to the main development branch


Test patch:
diff --git a/subprojects/androidTest/src/androidTest/java/org/mockitousage/androidtest/BasicInstrumentedTests.kt b/subprojects/androidTest/src/androidTest/java/org/mockitousage/androidtest/BasicInstrumentedTests.kt
index 2fa04c8bd6..07378171fc 100644
--- a/subprojects/androidTest/src/androidTest/java/org/mockitousage/androidtest/BasicInstrumentedTests.kt
+++ b/subprojects/androidTest/src/androidTest/java/org/mockitousage/androidtest/BasicInstrumentedTests.kt
@@ -35,7 +35,6 @@ class BasicInstrumentedTests {
     fun mockAndUseBasicClassUsingAnnotatedMock() {
         val basicClass = BasicOpenClassReceiver(mockedViaAnnotationBasicOpenClass)
         basicClass.callDependencyMethod()
-        throw Exception("java")
     }
 
     @Test