Repository: deepjavalibrary/djl
Pull request #1479 (2022-01-22 19:13:59)

[onnx] Add SessionOptions support for OnnxRuntime
Change-Id: I99890422056a3a68f7419088cf5cf849e75d4da9

## Description ##

Brief description of what this PR is about

- If this change is a backward incompatible change, why must this change be made?
- Interesting edge cases to note here


codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1479?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
> Merging [#1479](https://codecov.io/gh/deepjavalibrary/djl/pull/1479?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (ec0b016) into [master](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (bb5073f) will **decrease** coverage by `1.37%`.
> The diff coverage is `49.21%`.

[![Impacted file tree graph](https://codecov.io/gh/deepjavalibrary/djl/pull/1479/graphs/tree.svg?width=650&height=150&src=pr&token=mDKVNwFXRV&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)](https://codecov.io/gh/deepjavalibrary/djl/pull/1479?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)

```diff
@@             Coverage Diff              @@
##             master    #1479      +/-   ##
============================================
- Coverage     72.08%   70.71%   -1.38%     
- Complexity     5126     5310     +184     
============================================
  Files           473      493      +20     
  Lines         21970    23263    +1293     
  Branches       2351     2541     +190     
============================================
+ Hits          15838    16451     +613     
- Misses         4925     5535     +610     
- Partials       1207     1277      +70     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/1479?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...ava/ai/djl/modality/cv/output/DetectedObjects.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9EZXRlY3RlZE9iamVjdHMuamF2YQ==) | `95.23% <ø> (+0.23%)` | :arrow_up: |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <ø> (-5.24%)` | :arrow_down: |
| [...odality/cv/translator/BigGANTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `33.33% <0.00%> (+8.33%)` | :arrow_up: |
| [...nslator/InstanceSegmentationTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW5zdGFuY2VTZWdtZW50YXRpb25UcmFuc2xhdG9yRmFjdG9yeS5qYXZh) | `14.28% <0.00%> (-3.90%)` | :arrow_down: |
| [.../modality/cv/translator/YoloTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1RyYW5zbGF0b3JGYWN0b3J5LmphdmE=) | `8.33% <0.00%> (-1.67%)` | :arrow_down: |
| [...i/djl/modality/cv/translator/YoloV5Translator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1Y1VHJhbnNsYXRvci5qYXZh) | `5.69% <0.00%> (ø)` | |
| [...odality/cv/translator/YoloV5TranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1Y1VHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `8.33% <0.00%> (-1.67%)` | :arrow_down: |
| [...pi/src/main/java/ai/djl/ndarray/BytesSupplier.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL25kYXJyYXkvQnl0ZXNTdXBwbGllci5qYXZh) | `54.54% <0.00%> (-12.13%)` | :arrow_down: |
| [...i/src/main/java/ai/djl/ndarray/NDArrayAdapter.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1479/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL25kYXJyYXkvTkRBcnJheUFkYXB0ZXIuamF2YQ==) | `15.23% <0.00%> (-0.83%)` | :arrow_down: |
| ... and [165 more](https://codecov.io/gh/deepjavalibrary/djl/pull/1479/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1479?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1479?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Last update [3a67b8e...ec0b016](https://codecov.io/gh/deepjavalibrary/djl/pull/1479?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/engine/OrtModel.java b/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/engine/OrtModel.java
index 55641e7c8..c3d856a83 100644
--- a/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/engine/OrtModel.java
+++ b/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/engine/OrtModel.java
@@ -22,6 +22,9 @@ import ai.djl.util.Utils;
 import ai.onnxruntime.OrtEnvironment;
 import ai.onnxruntime.OrtException;
 import ai.onnxruntime.OrtSession;
+import ai.onnxruntime.OrtSession.SessionOptions;
+import ai.onnxruntime.OrtSession.SessionOptions.ExecutionMode;
+import ai.onnxruntime.OrtSession.SessionOptions.OptLevel;
 import java.io.FileNotFoundException;
 import java.io.IOException;
 import java.io.InputStream;
@@ -38,6 +41,7 @@ import java.util.Map;
 public class OrtModel extends BaseModel {
 
     private OrtEnvironment env;
+    private SessionOptions sessionOptions;
 
     /**
      * Constructs a new Model on a given device.
@@ -52,6 +56,7 @@ public class OrtModel extends BaseModel {
         this.manager.setName("ortModel");
         this.env = env;
         dataType = DataType.FLOAT32;
+        sessionOptions = new SessionOptions();
     }
 
     /** {@inheritDoc} */
@@ -71,15 +76,12 @@ public class OrtModel extends BaseModel {
         }
 
         try {
+            SessionOptions ortOptions = getSessionOptions(options);
             Device device = manager.getDevice();
-            OrtSession session;
             if (device.isGpu()) {
-                OrtSession.SessionOptions sessionOptions = new OrtSession.SessionOptions();
-                sessionOptions.addCUDA(manager.getDevice().getDeviceId());
-                session = env.createSession(modelFile.toString(), sessionOptions);
-            } else {
-                session = env.createSession(modelFile.toString());
+                ortOptions.addCUDA(manager.getDevice().getDeviceId());
             }
+            OrtSession session = env.createSession(modelFile.toString(), ortOptions);
             block = new OrtSymbolBlock(session, (OrtNDManager) manager);
         } catch (OrtException e) {
             throw new MalformedModelException("ONNX Model cannot be loaded", e);
@@ -97,15 +99,12 @@ public class OrtModel extends BaseModel {
         modelDir.toFile().deleteOnExit();
         try {
             byte[] buf = Utils.toByteArray(is);
+            SessionOptions ortOptions = getSessionOptions(options);
             Device device = manager.getDevice();
-            OrtSession session;
             if (device.isGpu()) {
-                OrtSession.SessionOptions sessionOptions = new OrtSession.SessionOptions();
-                sessionOptions.addCUDA(manager.getDevice().getDeviceId());
-                session = env.createSession(buf, sessionOptions);
-            } else {
-                session = env.createSession(buf);
+                ortOptions.addCUDA(manager.getDevice().getDeviceId());
             }
+            OrtSession session = env.createSession(buf, ortOptions);
             block = new OrtSymbolBlock(session, (OrtNDManager) manager);
         } catch (OrtException e) {
             throw new MalformedModelException("ONNX Model cannot be loaded", e);
@@ -139,4 +138,48 @@ public class OrtModel extends BaseModel {
         }
         return modelFile;
     }
+
+    /** {@inheritDoc} */
+    @Override
+    public void close() {
+        super.close();
+        try {
+            sessionOptions.close();
+        } catch (IllegalArgumentException ignore) {
+            // ignore
+        }
+    }
+
+    private SessionOptions getSessionOptions(Map<String, ?> options) throws OrtException {
+        if (options == null) {
+            return sessionOptions;
+        }
+
+        SessionOptions ortSession = sessionOptions;
+        if (options.containsKey("sessionOptions")) {
+            ortSession = (SessionOptions) options.get("sessionOptions");
+        }
+
+        String interOpNumThreads = (String) options.get("interOpNumThreads");
+        if (interOpNumThreads != null) {
+            ortSession.setInterOpNumThreads(Integer.parseInt(interOpNumThreads));
+        }
+        String intraOpNumThreads = (String) options.get("intraOpNumThreads");
+        if (interOpNumThreads != null) {
+            ortSession.setIntraOpNumThreads(Integer.parseInt(intraOpNumThreads));
+        }
+        String executionMode = (String) options.get("executionMode");
+        if (executionMode != null) {
+            ortSession.setExecutionMode(ExecutionMode.valueOf(executionMode));
+        }
+        String optLevel = (String) options.get("optLevel");
+        if (optLevel != null) {
+            ortSession.setOptimizationLevel(OptLevel.valueOf(optLevel));
+        }
+        String memoryOptimization = (String) options.get("memoryPatternOptimization");
+        if (Boolean.parseBoolean(memoryOptimization)) {
+            ortSession.setMemoryPatternOptimization(true);
+        }
+        return ortSession;
+    }
 }


Test patch:
diff --git a/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java b/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java
index 29e949ed9..94fab66b0 100644
--- a/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java
+++ b/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java
@@ -43,6 +43,11 @@ public class OrtTest {
                     Criteria.builder()
                             .setTypes(IrisFlower.class, Classifications.class)
                             .optEngine("OnnxRuntime") // use OnnxRuntime engine
+                            .optOption("interOpNumThreads", "1")
+                            .optOption("intraOpNumThreads", "1")
+                            .optOption("executionMode", "SEQUENTIAL")
+                            .optOption("optLevel", "NO_OPT")
+                            .optOption("memoryPatternOptimization", "true")
                             .build();
 
             IrisFlower virginica = new IrisFlower(1.0f, 2.0f, 3.0f, 4.0f);