Repository: deepjavalibrary/djl
Pull request #1844 (2022-07-29 04:12:49)

[WIP] add DJL Android ONNXRuntime support
## Description ##

Brief description of what this PR is about
- Add DJL Android support for the ONNXRuntime engine to enable it to use onnx format models.- 
- But now it is [WIP].

codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1844?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
> Merging [#1844](https://codecov.io/gh/deepjavalibrary/djl/pull/1844?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (8ba0f0e) into [master](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (bb5073f) will **decrease** coverage by `2.50%`.
> The diff coverage is `65.20%`.

```diff
@@             Coverage Diff              @@
##             master    #1844      +/-   ##
============================================
- Coverage     72.08%   69.58%   -2.51%     
- Complexity     5126     5511     +385     
============================================
  Files           473      526      +53     
  Lines         21970    24445    +2475     
  Branches       2351     2665     +314     
============================================
+ Hits          15838    17010    +1172     
- Misses         4925     6124    +1199     
- Partials       1207     1311     +104     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/1844?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L011bHRpQm94UHJpb3IuamF2YQ==) | `76.00% <ø> (ø)` | |
| [...rc/main/java/ai/djl/modality/cv/output/Joints.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9Kb2ludHMuamF2YQ==) | `71.42% <ø> (ø)` | |
| [.../main/java/ai/djl/modality/cv/output/Landmark.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9MYW5kbWFyay5qYXZh) | `100.00% <ø> (ø)` | |
| [...main/java/ai/djl/modality/cv/output/Rectangle.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9SZWN0YW5nbGUuamF2YQ==) | `72.41% <0.00%> (ø)` | |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <0.00%> (-5.24%)` | :arrow_down: |
| [...odality/cv/translator/BigGANTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `33.33% <0.00%> (+8.33%)` | :arrow_up: |
| [...nslator/InstanceSegmentationTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW5zdGFuY2VTZWdtZW50YXRpb25UcmFuc2xhdG9yRmFjdG9yeS5qYXZh) | `14.28% <0.00%> (-3.90%)` | :arrow_down: |
| [.../cv/translator/SemanticSegmentationTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU2VtYW50aWNTZWdtZW50YXRpb25UcmFuc2xhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../cv/translator/StyleTransferTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU3R5bGVUcmFuc2ZlclRyYW5zbGF0b3JGYWN0b3J5LmphdmE=) | `40.00% <ø> (ø)` | |
| ... and [423 more](https://codecov.io/gh/deepjavalibrary/djl/pull/1844/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1844?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1844?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Last update [587763a...8ba0f0e](https://codecov.io/gh/deepjavalibrary/djl/pull/1844?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).

lanking520: As discussed offline. Let's just rollback to the previous trasient dependency version

Patch:
diff --git a/android/README.md b/android/README.md
index 9da89e238..0786d5ce7 100644
--- a/android/README.md
+++ b/android/README.md
@@ -5,6 +5,7 @@ It has the following two modules:
 
 - Core package: contains some Image processing toolkit for Android user using DJL
 - PyTorch Native: contains DJL PyTorch Android native package
+- ONNXRuntime: contains DJL ONNXRuntime Android package
 
 ## Installation
 You need to have Android SDK and Android NDK installed on your machine.
@@ -15,9 +16,10 @@ In gradle, you can include the snapshot repository and add the 4 modules in your
 
 ```
 dependencies {
-    implementation "ai.djl:api:0.18.0-SNAPSHOT"
-    implementation "ai.djl.android:core:0.18.0-SNAPSHOT"
-    androidRuntimeOnly "ai.djl.pytorch:pytorch-engine:0.18.0-SNAPSHOT"
-    androidRuntimeOnly "ai.djl.android:pytorch-native:0.18.0-SNAPSHOT"
+    implementation "ai.djl:api:0.19.0-SNAPSHOT"
+    implementation "ai.djl.android:core:0.19.0-SNAPSHOT"
+    androidRuntimeOnly "ai.djl.pytorch:pytorch-engine:0.19.0-SNAPSHOT"
+    androidRuntimeOnly "ai.djl.android:pytorch-native:0.19.0-SNAPSHOT"
+    androidRuntimeOnly "ai.djl.android:onnxruntime:0.19.0-SNAPSHOT"
 }
 ```
diff --git a/android/core/build.gradle b/android/core/build.gradle
index 4ab1e3783..0a31876cf 100644
--- a/android/core/build.gradle
+++ b/android/core/build.gradle
@@ -102,6 +102,7 @@ dependencies {
     testImplementation 'junit:junit:4.13.2'
     androidTestRuntimeOnly "ai.djl.pytorch:pytorch-engine:${version}"
     androidTestRuntimeOnly "ai.djl.android:pytorch-native:${version}"
+    androidTestImplementation "ai.djl.android:onnxruntime:${version}"
     androidTestImplementation 'androidx.test.ext:junit:1.1.3'
     androidTestImplementation 'androidx.test.espresso:espresso-core:3.4.0'
 }
diff --git a/android/gradlew.bat b/android/gradlew.bat
deleted file mode 120000
index 284013288..000000000
--- a/android/gradlew.bat
+++ /dev/null
@@ -1,1 +0,0 @@
-../gradlew.bat
\ No newline at end of file
diff --git a/android/gradlew.bat b/android/gradlew.bat
new file mode 100644
index 000000000..b6d31956c
--- /dev/null
+++ b/android/gradlew.bat
@@ -0,0 +1,1 @@
+call ..\gradlew.bat
\ No newline at end of file
diff --git a/android/settings.gradle b/android/settings.gradle
index 1406c75cc..543320081 100644
--- a/android/settings.gradle
+++ b/android/settings.gradle
@@ -1,3 +1,3 @@
 rootProject.name='android'
-include ':pytorch-native'
 include ':core'
+include ':pytorch-native'


Test patch:
diff --git a/android/core/src/androidTest/java/ai/djl/android/core/ModelLoadingTest.java b/android/core/src/androidTest/java/ai/djl/android/core/ModelLoadingTest.java
index a15b13a53..1a0f3387d 100644
--- a/android/core/src/androidTest/java/ai/djl/android/core/ModelLoadingTest.java
+++ b/android/core/src/androidTest/java/ai/djl/android/core/ModelLoadingTest.java
@@ -20,6 +20,8 @@ import ai.djl.modality.cv.ImageFactory;
 import ai.djl.modality.cv.transform.Resize;
 import ai.djl.modality.cv.transform.ToTensor;
 import ai.djl.modality.cv.translator.ImageClassificationTranslator;
+import ai.djl.onnxruntime.zoo.tabular.softmax_regression.IrisClassificationTranslatorFactory;
+import ai.djl.onnxruntime.zoo.tabular.softmax_regression.IrisFlower;
 import ai.djl.repository.zoo.Criteria;
 import ai.djl.repository.zoo.ModelZoo;
 import ai.djl.repository.zoo.ZooModel;
@@ -35,30 +37,46 @@ public class ModelLoadingTest {
 
     @Test
     public void testModelLoading() throws IOException, ModelException, TranslateException {
-
         String modelUrl = "https://resources.djl.ai/demo/pytorch/traced_resnet18.zip";
-        ImageClassificationTranslator.builder()
-                .addTransform(new Resize(224, 224))
-                .addTransform(new ToTensor())
-                .optApplySoftmax(true)
-                .build();
+        ImageClassificationTranslator translator =
+                ImageClassificationTranslator.builder()
+                        .addTransform(new Resize(224, 224))
+                        .addTransform(new ToTensor())
+                        .optApplySoftmax(true)
+                        .build();
         Criteria<Image, Classifications> criteria =
                 Criteria.builder()
                         .setTypes(Image.class, Classifications.class)
                         .optModelUrls(modelUrl)
-                        .optTranslator(
-                                ImageClassificationTranslator.builder()
-                                        .addTransform(new Resize(224, 224))
-                                        .addTransform(new ToTensor())
-                                        .optApplySoftmax(true)
-                                        .build())
+                        .optTranslator(translator)
                         .build();
-        Image image =
-                ImageFactory.getInstance().fromUrl("https://resources.djl.ai/images/kitten.jpg");
+
+        ImageFactory factory = ImageFactory.getInstance();
+        Image image = factory.fromUrl("https://resources.djl.ai/images/kitten.jpg");
         try (ZooModel<Image, Classifications> model = ModelZoo.loadModel(criteria);
                 Predictor<Image, Classifications> predictor = model.newPredictor()) {
             Classifications result = predictor.predict(image);
             Assert.assertEquals("n02124075 Egyptian cat", result.best().getClassName());
         }
     }
+
+    @Test
+    public void testONNXRuntimeModelLoading()
+            throws IOException, ModelException, TranslateException {
+        String modelUrl =
+                "https://mlrepo.djl.ai/model/tabular/softmax_regression/ai/djl/onnxruntime/iris_flowers/0.0.1/iris_flowers.zip";
+        Criteria<IrisFlower, Classifications> criteria =
+                Criteria.builder()
+                        .setTypes(IrisFlower.class, Classifications.class)
+                        .optModelUrls(modelUrl)
+                        .optTranslatorFactory(new IrisClassificationTranslatorFactory())
+                        .optEngine("OnnxRuntime") // use OnnxRuntime engine by default
+                        .build();
+
+        ZooModel<IrisFlower, Classifications> model = criteria.loadModel();
+        Predictor<IrisFlower, Classifications> predictor = model.newPredictor();
+        IrisFlower info = new IrisFlower(1.0f, 2.0f, 3.0f, 4.0f);
+        Classifications result = predictor.predict(info);
+        Assert.assertEquals("virginica", result.best().getClassName());
+    }
 }