Repository: deepjavalibrary/djl
Pull request #1402 (2021-12-04 18:11:13)

Adds load model from InputStream for OnnxRuntime and TFLite
Change-Id: I5485c87f9dedf88acf5fa2e5f74c66e799c28319

## Description ##

Brief description of what this PR is about

- If this change is a backward incompatible change, why must this change be made?
- Interesting edge cases to note here


codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1402?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
> Merging [#1402](https://codecov.io/gh/deepjavalibrary/djl/pull/1402?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (5e60b5c) into [master](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (bb5073f) will **decrease** coverage by `1.51%`.
> The diff coverage is `51.08%`.

[![Impacted file tree graph](https://codecov.io/gh/deepjavalibrary/djl/pull/1402/graphs/tree.svg?width=650&height=150&src=pr&token=mDKVNwFXRV&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)](https://codecov.io/gh/deepjavalibrary/djl/pull/1402?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)

```diff
@@             Coverage Diff              @@
##             master    #1402      +/-   ##
============================================
- Coverage     72.08%   70.56%   -1.52%     
- Complexity     5126     5247     +121     
============================================
  Files           473      487      +14     
  Lines         21970    23078    +1108     
  Branches       2351     2518     +167     
============================================
+ Hits          15838    16286     +448     
- Misses         4925     5524     +599     
- Partials       1207     1268      +61     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/1402?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1402/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1402/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <ø> (-5.24%)` | :arrow_down: |
| [...odality/cv/translator/BigGANTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1402/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `33.33% <0.00%> (+8.33%)` | :arrow_up: |
| [...nslator/InstanceSegmentationTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1402/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW5zdGFuY2VTZWdtZW50YXRpb25UcmFuc2xhdG9yRmFjdG9yeS5qYXZh) | `14.28% <0.00%> (-3.90%)` | :arrow_down: |
| [.../modality/cv/translator/YoloTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1402/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1RyYW5zbGF0b3JGYWN0b3J5LmphdmE=) | `8.33% <0.00%> (-1.67%)` | :arrow_down: |
| [...i/djl/modality/cv/translator/YoloV5Translator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1402/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1Y1VHJhbnNsYXRvci5qYXZh) | `5.69% <0.00%> (ø)` | |
| [...odality/cv/translator/YoloV5TranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1402/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1Y1VHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `8.33% <0.00%> (-1.67%)` | :arrow_down: |
| [...modality/nlp/embedding/TrainableWordEmbedding.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1402/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L25scC9lbWJlZGRpbmcvVHJhaW5hYmxlV29yZEVtYmVkZGluZy5qYXZh) | `45.00% <0.00%> (-3.34%)` | :arrow_down: |
| [...pi/src/main/java/ai/djl/ndarray/BytesSupplier.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1402/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL25kYXJyYXkvQnl0ZXNTdXBwbGllci5qYXZh) | `54.54% <0.00%> (-12.13%)` | :arrow_down: |
| [...i/src/main/java/ai/djl/ndarray/NDArrayAdapter.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1402/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL25kYXJyYXkvTkRBcnJheUFkYXB0ZXIuamF2YQ==) | `15.23% <0.00%> (-0.83%)` | :arrow_down: |
| ... and [143 more](https://codecov.io/gh/deepjavalibrary/djl/pull/1402/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1402?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1402?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Last update [b0e2ea3...5e60b5c](https://codecov.io/gh/deepjavalibrary/djl/pull/1402?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/engine/OrtModel.java b/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/engine/OrtModel.java
index d2cf9dc51..55641e7c8 100644
--- a/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/engine/OrtModel.java
+++ b/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/engine/OrtModel.java
@@ -18,11 +18,13 @@ import ai.djl.MalformedModelException;
 import ai.djl.Model;
 import ai.djl.ndarray.NDManager;
 import ai.djl.ndarray.types.DataType;
+import ai.djl.util.Utils;
 import ai.onnxruntime.OrtEnvironment;
 import ai.onnxruntime.OrtException;
 import ai.onnxruntime.OrtSession;
 import java.io.FileNotFoundException;
 import java.io.IOException;
+import java.io.InputStream;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.util.Map;
@@ -84,6 +86,32 @@ public class OrtModel extends BaseModel {
         }
     }
 
+    /** {@inheritDoc} */
+    @Override
+    public void load(InputStream is, Map<String, ?> options)
+            throws IOException, MalformedModelException {
+        if (block != null) {
+            throw new UnsupportedOperationException("ONNX Runtime does not support dynamic blocks");
+        }
+        modelDir = Files.createTempDirectory("ort-model");
+        modelDir.toFile().deleteOnExit();
+        try {
+            byte[] buf = Utils.toByteArray(is);
+            Device device = manager.getDevice();
+            OrtSession session;
+            if (device.isGpu()) {
+                OrtSession.SessionOptions sessionOptions = new OrtSession.SessionOptions();
+                sessionOptions.addCUDA(manager.getDevice().getDeviceId());
+                session = env.createSession(buf, sessionOptions);
+            } else {
+                session = env.createSession(buf);
+            }
+            block = new OrtSymbolBlock(session, (OrtNDManager) manager);
+        } catch (OrtException e) {
+            throw new MalformedModelException("ONNX Model cannot be loaded", e);
+        }
+    }
+
     private Path findModelFile(String prefix) {
         if (Files.isRegularFile(modelDir)) {
             Path file = modelDir;
diff --git a/engines/tflite/tflite-engine/src/main/java/ai/djl/tflite/engine/TfLiteModel.java b/engines/tflite/tflite-engine/src/main/java/ai/djl/tflite/engine/TfLiteModel.java
index 7ae8d9c44..5e0cdc570 100644
--- a/engines/tflite/tflite-engine/src/main/java/ai/djl/tflite/engine/TfLiteModel.java
+++ b/engines/tflite/tflite-engine/src/main/java/ai/djl/tflite/engine/TfLiteModel.java
@@ -14,10 +14,14 @@ package ai.djl.tflite.engine;
 
 import ai.djl.BaseModel;
 import ai.djl.Model;
+import ai.djl.engine.Engine;
 import ai.djl.ndarray.NDManager;
 import ai.djl.ndarray.types.DataType;
+import ai.djl.util.Utils;
 import java.io.FileNotFoundException;
 import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.util.Map;
@@ -62,6 +66,22 @@ public class TfLiteModel extends BaseModel {
         setBlock(new TfLiteSymbolBlock(interpreter, getNDManager()));
     }
 
+    /** {@inheritDoc} */
+    @Override
+    public void load(InputStream is, Map<String, ?> options) throws IOException {
+        if (block != null) {
+            throw new UnsupportedOperationException("TFLite does not support dynamic blocks");
+        }
+        modelDir = Files.createTempDirectory("tflite-model");
+        modelDir.toFile().deleteOnExit();
+        byte[] buf = Utils.toByteArray(is);
+        Engine engine = Engine.getEngine(TfLiteEngine.ENGINE_NAME);
+        ByteBuffer bb = engine.newBaseManager().allocateDirect(buf.length);
+        bb.put(buf);
+        Interpreter interpreter = new Interpreter(bb);
+        setBlock(new TfLiteSymbolBlock(interpreter, getNDManager()));
+    }
+
     /** {@inheritDoc} */
     @Override
     public TfLiteNDManager getNDManager() {


Test patch:
diff --git a/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java b/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java
index cc421bb33..29e949ed9 100644
--- a/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java
+++ b/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java
@@ -27,6 +27,8 @@ import ai.djl.repository.zoo.ModelNotFoundException;
 import ai.djl.repository.zoo.ZooModel;
 import ai.djl.translate.TranslateException;
 import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
 import java.nio.file.Path;
 import org.testng.Assert;
 import org.testng.SkipException;
@@ -58,6 +60,13 @@ public class OrtTest {
                 m.load(modelFile);
 
                 m.close();
+
+                // Test load model from stream
+                Model stream = Model.newInstance("model", "OnnxRuntime");
+                try (InputStream is = Files.newInputStream(modelFile)) {
+                    stream.load(is);
+                }
+                stream.close();
             }
         } catch (UnsatisfiedLinkError e) {
             /*
diff --git a/engines/tflite/tflite-engine/src/test/java/ai/djl/tflite/engine/TfLiteTest.java b/engines/tflite/tflite-engine/src/test/java/ai/djl/tflite/engine/TfLiteTest.java
index 8414be2b9..1a3650f87 100644
--- a/engines/tflite/tflite-engine/src/test/java/ai/djl/tflite/engine/TfLiteTest.java
+++ b/engines/tflite/tflite-engine/src/test/java/ai/djl/tflite/engine/TfLiteTest.java
@@ -13,6 +13,7 @@
 package ai.djl.tflite.engine;
 
 import ai.djl.MalformedModelException;
+import ai.djl.Model;
 import ai.djl.inference.Predictor;
 import ai.djl.modality.Classifications;
 import ai.djl.modality.cv.Image;
@@ -23,6 +24,9 @@ import ai.djl.repository.zoo.ZooModel;
 import ai.djl.testing.TestRequirements;
 import ai.djl.translate.TranslateException;
 import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
 import org.testng.Assert;
 import org.testng.annotations.Test;
 
@@ -47,6 +51,14 @@ public class TfLiteTest {
                             .fromUrl("https://resources.djl.ai/images/sachertorte.jpg");
             Classifications prediction = predictor.predict(image);
             Assert.assertEquals(prediction.best().getClassName(), "Sachertorte");
+
+            Path modelPath = model.getModelPath();
+            Path modelFile = modelPath.resolve("aiyDish.tflite");
+            Model m = Model.newInstance("aiyDish", "TFLite");
+            try (InputStream is = Files.newInputStream(modelFile)) {
+                m.load(is);
+            }
+            m.close();
         }
     }
 }