Dexファイルにinvoke-polymorphicとinvoke-customが現れることを確認する

Dexファイルフォーマットのバージョン`038`で、[`invoke-polymorphic`と`invoke-custom`が追加](https://source.android.com/devices/tech/dalvik/dex-format.html#definitions)されました。

> Note: Support for version 038 of the format was added in the Android 8.0 release. Version 038 added new bytecodes (invoke-polymorphic and invoke-custom) and data for method handles.

これらがDexファイルに現れることを確認します。

以下に、コンパイル対象のコードを示します。

```java:app/src/main/java/com/example/myapplication/Main.java
package com.example.myapplication;

import java.lang.invoke.MethodHandles;
import java.lang.invoke.MethodType;
import java.util.function.BiFunction;

public class Main {

    public static void main(String[] args) throws Throwable {
        System.out.println(MethodHandles.lookup()
                                   .findStatic(Main.class,
                                               "add",
                                               MethodType.methodType(int.class, int.class, int.class))
                                   .invoke(2, 3));
    }

    static int add(int a, int b) {
        BiFunction<Integer, Integer, Integer> f = (i0, i1) -> i0 + i1;
        return f.apply(a, b);
    }

}
```

Dexファイルがどうなるかを確認するためだけのコードです。
Android上で動作させることは考えていません。

build.gradleはそれぞれ以下のとおりです。

```groovy:build.gradle
buildscript {
    repositories {
        jcenter()
    }
    dependencies {
        classpath 'com.android.tools.build:gradle:2.4.0-alpha3'
    }
}

allprojects {
    repositories {
        jcenter()
    }
}

task clean(type: Delete) {
    delete rootProject.buildDir
}
```

```groovy:app/build.gradle
apply plugin: 'com.android.application'

android {
    compileSdkVersion 'android-O'
    buildToolsVersion '26.0.0-rc1'
    defaultConfig {
        jackOptions {
            enabled = true
        }
        applicationId "com.example.myapplication"
        minSdkVersion 'O'
        versionCode 1
        versionName '1.0'
    }
    compileOptions {
        targetCompatibility JavaVersion.VERSION_1_8
        sourceCompatibility JavaVersion.VERSION_1_8
    }
}
```

`compileSdkVersion`を`android-O`に、`minSdkVersion`を`O`にしています。
また、ラムダ式を使用しているので、Jackを有効にしています。

ビルドします。

```
$ ./gradlew clean assembleDebug
:clean
:app:clean
:app:preBuild UP-TO-DATE
:app:resolveDebugDependencies
:app:preDebugBuild
:app:compileDebugNdk NO-SOURCE
:app:checkDebugManifest
:app:prepareDebugDependencies
:app:compileDebugAidl
:app:compileDebugRenderscript
:app:generateDebugBuildConfig
:app:generateDebugResValues
:app:handleDebugMicroApk
:app:generateDebugResources
:app:mergeDebugResources
:app:processDebugManifest
:app:processDebugResources
:app:generateDebugSources
:app:transformClassesWithPreJackPackagedLibrariesForDebug
:app:transformClassesWithPreJackRuntimeLibrariesForDebug
:app:processDebugJavaRes NO-SOURCE
:app:transformResourcesWithMergeJavaResForDebug
:app:transformJackAndJavaSourcesWithJackCompileForDebug
/home/tmura/AndroidStudioProjects/MyApplication2/app/src/main/java/com/example/myapplication/Main.java:10:28-14:48: Call to a method with a polymorphic signature requires at least an android provisional level o-b1
:app:transformJackAndJavaSourcesWithJackCompileForDebug FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:transformJackAndJavaSourcesWithJackCompileForDebug'.
> com.android.build.api.transform.TransformException: com.android.builder.core.JackToolchain$ToolchainException: Jack compilation exception

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED

Total time: 3.683 secs
```

`Call to a method with a polymorphic signature requires at least an android provisional level o-b1`とのことで、レベルを設定する必要がありそうです。

`jack.jar`のオプションを確認します。

```
$ java -jar $ANDROID_HOME/build-tools/26.0.0-rc1/jack.jar --help-properties

...

jack.android.min-api-level:
     Minimum Android API level compatibility (default is '1')
     a released API level as an integer belonging to [1 .. 9223372036854775807] or a provisional API level as {o-b1,o-b2} (case insensitive)
          o-b1: Dex file 0x37 with invoke-poly
          o-b2: Dex file 0x38 with invoke-{poly,custom}

...
```

`jack.android.min-api-level`というプロパティに`o-b1`または`o-b2`を指定すれば良さそうです。

`app/build.gradle`にて、上記オプションを設定します。

```groovy:app/build.gradle
android {
    defaultConfig {
        jackOptions {
            enabled = true
            additionalParameters 'jack.android.min-api-level': 'o-b1'
        }
    }
}
```

まずは、`o-b1`を試します。

ビルドします。

```
$ ./gradlew clean assembleDebug
:clean
:app:clean
:app:preBuild UP-TO-DATE
:app:resolveDebugDependencies
:app:preDebugBuild
:app:compileDebugNdk NO-SOURCE
:app:checkDebugManifest
:app:prepareDebugDependencies
:app:compileDebugAidl
:app:compileDebugRenderscript
:app:generateDebugBuildConfig
:app:generateDebugResValues
:app:handleDebugMicroApk
:app:generateDebugResources
:app:mergeDebugResources
:app:processDebugManifest
:app:processDebugResources
:app:generateDebugSources
:app:transformClassesWithPreJackPackagedLibrariesForDebug
:app:transformClassesWithPreJackRuntimeLibrariesForDebug
:app:processDebugJavaRes NO-SOURCE
:app:transformResourcesWithMergeJavaResForDebug
:app:transformJackAndJavaSourcesWithJackCompileForDebug
:app:compileDebugSources
:app:mergeDebugShaders
:app:compileDebugShaders
:app:generateDebugAssets
:app:mergeDebugAssets
:app:transformJackWithJackDexerForDebug
:app:mergeDebugJniLibFolders
:app:transformNativeLibsWithMergeJniLibsForDebug
:app:validateSigningDebug
:app:packageDebug
:app:assembleDebug

BUILD SUCCESSFUL

Total time: 4.636 secs
```

`dexdump`します。

```
$ $ANDROID_HOME/build-tools/26.0.0-rc1/dexdump -d -h app/build/intermediates/transforms/jackDexer/debug/folders/1000/1f/main/classes.dex
Processing 'app/build/intermediates/transforms/jackDexer/debug/folders/1000/1f/main/classes.dex'...
Opened 'app/build/intermediates/transforms/jackDexer/debug/folders/1000/1f/main/classes.dex', DEX version '037'

...

000634:                                        |[000634] com.example.myapplication.Main.main:([Ljava/lang/String;)V
000644: 6200 0a00                              |0000: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream; // field@000a
000648: 7100 1400 0000                         |0002: invoke-static {}, Ljava/lang/invoke/MethodHandles;.lookup:()Ljava/lang/invoke/MethodHandles$Lookup; // method@0014
00064e: 0c01                                   |0005: move-result-object v1
000650: 1c02 0300                              |0006: const-class v2, Lcom/example/myapplication/Main; // type@0003
000654: 1b03 3900 0000                         |0008: const-string/jumbo v3, "add" // string@00000039
00065a: 6204 0900                              |000b: sget-object v4, Ljava/lang/Integer;.TYPE:Ljava/lang/Class; // field@0009
00065e: 6205 0900                              |000d: sget-object v5, Ljava/lang/Integer;.TYPE:Ljava/lang/Class; // field@0009
000662: 1216                                   |000f: const/4 v6, #int 1 // #1
000664: 2366 1b00                              |0010: new-array v6, v6, [Ljava/lang/Class; // type@001b
000668: 6207 0900                              |0012: sget-object v7, Ljava/lang/Integer;.TYPE:Ljava/lang/Class; // field@0009
00066c: 1208                                   |0014: const/4 v8, #int 0 // #0
00066e: 4d07 0608                              |0015: aput-object v7, v6, v8
000672: 7130 1500 5406                         |0017: invoke-static {v4, v5, v6}, Ljava/lang/invoke/MethodType;.methodType:(Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;)Ljava/lang/invoke/MethodType; // method@0015
000678: 0c04                                   |001a: move-result-object v4
00067a: 6e40 1300 2143                         |001b: invoke-virtual {v1, v2, v3, v4}, Ljava/lang/invoke/MethodHandles$Lookup;.findStatic:(Ljava/lang/Class;Ljava/lang/String;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/MethodHandle; // method@0013
000680: 0c01                                   |001e: move-result-object v1
000682: 1222                                   |001f: const/4 v2, #int 2 // #2
000684: 1233                                   |0020: const/4 v3, #int 3 // #3
000686: fa30 1200 2103 0400                    |0021: invoke-polymorphic  {v1, v2, v3}, Ljava/lang/invoke/MethodHandle;.invoke:([Ljava/lang/Object;)Ljava/lang/Object;, (II)Ljava/lang/Object; // method@0012, proto@0004
00068e: 0c01                                   |0025: move-result-object v1
000690: 6e20 0d00 1000                         |0026: invoke-virtual {v0, v1}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V // method@000d
000696: 0e00                                   |0029: return-void

...
```

Dexファイルのバージョンは`037`です。
`MethodHandler#invoke()`の箇所(下から4行目)で`invoke-polymorphic`が出現しています。
上記で確認したオプションの説明通りです。

次に、`o-b2`を試します。

```groovy:app/build.gradle
android {
    defaultConfig {
        jackOptions {
            enabled = true
            additionalParameters 'jack.android.min-api-level': 'o-b2'
        }
    }
}
```

ビルドして、確認してみます。

```
$ ./gradlew clean assembleDebug --quiet
$ $ANDROID_HOME/build-tools/26.0.0-rc1/dexdump -d -h app/build/intermediates/transforms/jackDexer/debug/folders/1000/1f/main/classes.dex
Processing 'app/build/intermediates/transforms/jackDexer/debug/folders/1000/1f/main/classes.dex'...
Opened 'app/build/intermediates/transforms/jackDexer/debug/folders/1000/1f/main/classes.dex', DEX version '038'

...

000634:                                        |[000634] com.example.myapplication.Main.main:([Ljava/lang/String;)V
000644: 6200 0a00                              |0000: sget-object v0, Ljava/lang/System;.out:Ljava/io/PrintStream; // field@000a
000648: 7100 1400 0000                         |0002: invoke-static {}, Ljava/lang/invoke/MethodHandles;.lookup:()Ljava/lang/invoke/MethodHandles$Lookup; // method@0014
00064e: 0c01                                   |0005: move-result-object v1
000650: 1c02 0300                              |0006: const-class v2, Lcom/example/myapplication/Main; // type@0003
000654: 1b03 3900 0000                         |0008: const-string/jumbo v3, "add" // string@00000039
00065a: 6204 0900                              |000b: sget-object v4, Ljava/lang/Integer;.TYPE:Ljava/lang/Class; // field@0009
00065e: 6205 0900                              |000d: sget-object v5, Ljava/lang/Integer;.TYPE:Ljava/lang/Class; // field@0009
000662: 1216                                   |000f: const/4 v6, #int 1 // #1
000664: 2366 1b00                              |0010: new-array v6, v6, [Ljava/lang/Class; // type@001b
000668: 6207 0900                              |0012: sget-object v7, Ljava/lang/Integer;.TYPE:Ljava/lang/Class; // field@0009
00066c: 1208                                   |0014: const/4 v8, #int 0 // #0
00066e: 4d07 0608                              |0015: aput-object v7, v6, v8
000672: 7130 1500 5406                         |0017: invoke-static {v4, v5, v6}, Ljava/lang/invoke/MethodType;.methodType:(Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;)Ljava/lang/invoke/MethodType; // method@0015
000678: 0c04                                   |001a: move-result-object v4
00067a: 6e40 1300 2143                         |001b: invoke-virtual {v1, v2, v3, v4}, Ljava/lang/invoke/MethodHandles$Lookup;.findStatic:(Ljava/lang/Class;Ljava/lang/String;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/MethodHandle; // method@0013
000680: 0c01                                   |001e: move-result-object v1
000682: 1222                                   |001f: const/4 v2, #int 2 // #2
000684: 1233                                   |0020: const/4 v3, #int 3 // #3
000686: fa30 1200 2103 0400                    |0021: invoke-polymorphic  {v1, v2, v3}, Ljava/lang/invoke/MethodHandle;.invoke:([Ljava/lang/Object;)Ljava/lang/Object;, (II)Ljava/lang/Object; // method@0012, proto@0004
00068e: 0c01                                   |0025: move-result-object v1
000690: 6e20 0d00 1000                         |0026: invoke-virtual {v0, v1}, Ljava/io/PrintStream;.println:(Ljava/lang/Object;)V // method@000d
000696: 0e00                                   |0029: return-void

...
```

Dexファイルのバージョンは`038`です。
下から4行目に`invoke-polymorphic`が出現しているのはオプションの説明通りですが、一方、`invoke-custom`は見つかりませんでした。

```
$ $ANDROID_HOME/build-tools/26.0.0-rc1/dexdump -d -h app/build/intermediates/transforms/jackDexer/debug/folders/1000/1f/main/classes.dex \
> | grep -i invoke-custom | wc -l
0
```

Javaのクラスファイルでは`CallSite`が現れているので、`invoke-custom`が出現するかと思いましたが、そうはなっていませんでした。

```
$ ./gradlew clean compileDebugJavaWithJavac --quiet
$ javap -v -private app/build/intermediates/classes/debug/com/example/myapplication/Main.class

...

  static int add(int, int);
    descriptor: (II)I
    flags: ACC_STATIC
    Code:
      stack=3, locals=3, args_size=2
         0: invokedynamic #12,  0             // InvokeDynamic #0:apply:()Ljava/util/function/BiFunction;
         5: astore_2
         6: aload_2
         7: iload_0
         8: invokestatic  #13                 // Method java/lang/Integer.valueOf:(I)Ljava/lang/Integer;
        11: iload_1
        12: invokestatic  #13                 // Method java/lang/Integer.valueOf:(I)Ljava/lang/Integer;
        15: invokeinterface #14,  3           // InterfaceMethod java/util/function/BiFunction.apply:(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
        20: checkcast     #15                 // class java/lang/Integer
        23: invokevirtual #16                 // Method java/lang/Integer.intValue:()I
        26: ireturn
      LineNumberTable:
        line 18: 0
        line 19: 6
      LocalVariableTable:
        Start  Length  Slot  Name   Signature
            0      27     0     a   I
            0      27     1     b   I
            6      21     2     f   Ljava/util/function/BiFunction;
      LocalVariableTypeTable:
        Start  Length  Slot  Name   Signature
            6      21     2     f   Ljava/util/function/BiFunction<Ljava/lang/Integer;Ljava/lang/Integer;Ljava/lang/Integer;>;

...

SourceFile: "Main.java"
InnerClasses:
     public static final #81= #57 of #50; //Lookup=class java/lang/invoke/MethodHandles$Lookup of class java/lang/invoke/MethodHandles
BootstrapMethods:
  0: #64 invokestatic java/lang/invoke/LambdaMetafactory.metafactory:(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodType;Ljava/lang/invoke/MethodHandle;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/CallSite;
    Method arguments:
      #65 (Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
      #66 invokestatic com/example/myapplication/Main.lambda$add$0:(Ljava/lang/Integer;Ljava/lang/Integer;)Ljava/lang/Integer;
      #67 (Ljava/lang/Integer;Ljava/lang/Integer;)Ljava/lang/Integer;
```

Jackの[`RopBuilderVisitor.java`](https://android.googlesource.com/toolchain/jack/+/ub-jack/jack/src/com/android/jack/backend/dex/rop/RopBuilderVisitor.java#1213)を見ると、メソッド呼び出しが`invoke-custom`かどうかを`InvokeCustomHelper#isInvokeCustom(JAbstractMethodCall)`で判断しています。
[`InvokeCustomHelper.java`](https://android.googlesource.com/toolchain/jack/+/ub-jack/jack/src/com/android/jack/backend/dex/invokecustom/InvokeCustomHelper.java)を見ると、メソッドに`CalledByInvokeCustom`アノテーションが付与されていれば、`invoke-custom`と判断しているようです。

```java:jack/src/com/android/jack/backend/dex/invokecustom/InvokeCustomHelper.java
public static boolean isInvokeCustom(@Nonnull JAbstractMethodCall call) {
  Collection<JMethod> methods = call.getMethodIdWide().getMethods();
  Iterator<JMethod> methodsIt = methods.iterator();
  boolean isInvokeCustom = false;
  while (methodsIt.hasNext()) {
    JMethod method = methodsIt.next();
    if (getInvokeCustomCallsite(method) != null) {
      isInvokeCustom = true;
      break;
    }
  }

  assert !isInvokeCustom || methods.size() == 1;

  return isInvokeCustom;
}

@CheckForNull
public static JAnnotation getInvokeCustomCallsite(@Nonnull JMethod method) {
  for (JAnnotation annotation : method.getAnnotations()) {
    if (annotation.getType().getName().equals("CalledByInvokeCustom")) {
      return annotation;
    }
  }
  return null;
}
```

`CalledByInvokeCustom`は[`jack-tests`ディレクトリ配下](https://android.googlesource.com/toolchain/jack/+/ub-jack/jack-tests/src/com/android/jack/annotations/)にあるので、これを使ってみます。

```
$ git clone https://android.googlesource.com/toolchain/jack
$ (cd jack; git checkout -b ub-jack origin/ub-jack)
$ (cd jack/jack-tests/src; cp -p --parent com/android/jack/annotations/*.java ../../../app/src/main/java/)
```

さらに、`Main.java`を以下のように書きかえます。

```java:app/src/main/java/com/example/myapplication/Main.java
package com.example.myapplication;

import com.android.jack.annotations.CalledByInvokeCustom;
import com.android.jack.annotations.LinkerMethodHandle;
import com.android.jack.annotations.MethodHandleKind;

import java.lang.invoke.CallSite;
import java.lang.invoke.ConstantCallSite;
import java.lang.invoke.MethodHandles;
import java.lang.invoke.MethodType;
import java.util.function.BiFunction;

public class Main {

    public static void main(String[] args) throws Throwable {
        System.out.println(MethodHandles.lookup()
                                   .findStatic(Main.class,
                                               "add",
                                               MethodType.methodType(int.class, int.class, int.class))
                                   .invoke(2, 3));
    }

    static int add(int a, int b) {
        BiFunction<Integer, Integer, Integer> f = new BiFunction<Integer, Integer, Integer>() {
            @Override
            public Integer apply(Integer integer, Integer integer2) {
                return add0(a, b);
            }
        };
        return f.apply(a, b);
    }

    @CalledByInvokeCustom(
            invokeMethodHandle = @LinkerMethodHandle(kind = MethodHandleKind.INVOKE_STATIC,
                                                     enclosingType = Main.class,
                                                     name = "linkerMethod",
                                                     argumentTypes = {MethodHandles.Lookup.class, String.class, MethodType.class}),
            name = "add0",
            returnType = Integer.class,
            argumentTypes = {Integer.class, Integer.class})
    static Integer add0(Integer a, Integer b) {
        return a + b;
    }

    private static CallSite linkerMethod(MethodHandles.Lookup caller, String name, MethodType methodType)
            throws NoSuchMethodException, IllegalAccessException {
        return new ConstantCallSite(caller.findStatic(caller.lookupClass(), name, methodType));
    }

}
```

ビルドして、確認してみます。

```
$ ./gradlew clean assembleDebug --quiet
$ $ANDROID_HOME/build-tools/26.0.0-rc1/dexdump -d -h app/build/intermediates/transforms/jackDexer/debug/folders/1000/1f/main/classes.dex
Processing 'app/build/intermediates/transforms/jackDexer/debug/folders/1000/1f/main/classes.dex'...
Opened 'app/build/intermediates/transforms/jackDexer/debug/folders/1000/1f/main/classes.dex', DEX version '038'

...

000c18:                                        |[000c18] com.example.myapplication.Main.1.apply:(Ljava/lang/Integer;Ljava/lang/Integer;)Ljava/lang/Integer;
000c28: 5220 0e00                              |0000: iget v0, v2, Lcom/example/myapplication/Main$1;.val$a:I // field@000e
000c2c: 7110 3000 0000                         |0002: invoke-static {v0}, Ljava/lang/Integer;.valueOf:(I)Ljava/lang/Integer; // method@0030
000c32: 0c00                                   |0005: move-result-object v0
000c34: 5221 0f00                              |0006: iget v1, v2, Lcom/example/myapplication/Main$1;.val$b:I // field@000f
000c38: 7110 3000 0100                         |0008: invoke-static {v1}, Ljava/lang/Integer;.valueOf:(I)Ljava/lang/Integer; // method@0030
000c3e: 0c01                                   |000b: move-result-object v1
000c40: fc20 0000 1000                         |000c: invoke-custom {v0, v1}, call_site@0000
000c46: 0c00                                   |000f: move-result-object v0
000c48: 1100                                   |0010: return-object v0

...
```

`invoke-custom`が現れました(下から3行目)。

将来的にはラムダ式の呼び出しは`invoke-custom`になるのでしょうか？
[Jackは非推奨になる](https://android-developers.googleblog.com/2017/03/future-of-java-8-language-feature.html)らしいので、dx側で`invokedynamic`を`invoke-custom`に変換するようになるのかもしれません。

ちなみに、`jack.jar`には`jack.lambda.anonymous`というプロパティがあるので

```
jack.lambda.anonymous:
     Enable lambda support with an anonymous class (default is 'true')
     {true,yes,on,1,false,no,off,0} (case insensitive)
```

これを`false`にしてみたのですが、残念ながらエラーになってしまいました。

```groovy:app/build.gradle
android {
    defaultConfig {
        jackOptions {
            enabled = true
            additionalParameters 'jack.android.min-api-level': 'o-b2'
            additionalParameters 'jack.lambda.anonymous': 'false'
        }
    }
}
```

```
$ ./gradlew clean assembleDebug --stacktrace

...

:app:transformJackAndJavaSourcesWithJackCompileForDebug FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:transformJackAndJavaSourcesWithJackCompileForDebug'.
> com.android.sched.scheduler.RunnerProcessException: Error during 'CodeItemBuilder' runner on 'static int com.example.myapplication.Main.add(int a, int b) (/home/tmura/AndroidStudioProjects/MyApplication2/app/src/main/java/com/example/myapplication/Main.java:17.5-20.5)': Unexpected error during visit: com.android.jack.ir.ast.JLambda at "/home/tmura/AndroidStudioProjects/MyApplication2/app/src/main/java/com/example/myapplication/Main.java:18.51-18.69"

* Try:
Run with --info or --debug option to get more log output.

* Exception is:
org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':app:transformJackAndJavaSourcesWithJackCompileForDebug'.

...

Caused by: java.lang.AssertionError
        at com.android.jack.backend.dex.rop.RopBuilderVisitor$AssignBuilderVisitor.visit(RopBuilderVisitor.java:224)
        at com.android.jack.ir.ast.JLambda.traverse(JLambda.java:108)
        at com.android.jack.ir.ast.JVisitor.accept(JVisitor.java:51)
        ... 10 more
```

[エラー元](https://android.googlesource.com/toolchain/jack/+/ub-jack/jack/src/com/android/jack/backend/dex/rop/RopBuilderVisitor.java#224)を見ると、無条件で`throw new AssertionError()`しているので、現状このオプションを`false`にすることはできないようです。

以上です。

### 参考
Dalvik bytecode
https://source.android.com/devices/tech/dalvik/dalvik-bytecode.html