Repository: deepjavalibrary/djl
Pull request #1808 (2022-07-19 01:57:45)

Compatibility changes for Semantic Segmentation app
## Description ##

Adds compatibility for the DJL-Demo Semantic Segmentation app

- Transposes NDArray if dimensions matchup in BitmapImageFactory
- DataType for NDArray would change from the app due to using BitmapImageFactory and not BufferImageFactory, so added line to change DataType back to `UINT8` before returning image.


codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1808?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
> Merging [#1808](https://codecov.io/gh/deepjavalibrary/djl/pull/1808?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (b203b70) into [master](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (bb5073f) will **decrease** coverage by `2.52%`.
> The diff coverage is `64.98%`.

```diff
@@             Coverage Diff              @@
##             master    #1808      +/-   ##
============================================
- Coverage     72.08%   69.56%   -2.53%     
- Complexity     5126     5495     +369     
============================================
  Files           473      524      +51     
  Lines         21970    24339    +2369     
  Branches       2351     2652     +301     
============================================
+ Hits          15838    16931    +1093     
- Misses         4925     6093    +1168     
- Partials       1207     1315     +108     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/1808?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1808/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1808/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L011bHRpQm94UHJpb3IuamF2YQ==) | `76.00% <ø> (ø)` | |
| [...rc/main/java/ai/djl/modality/cv/output/Joints.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1808/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9Kb2ludHMuamF2YQ==) | `71.42% <ø> (ø)` | |
| [.../main/java/ai/djl/modality/cv/output/Landmark.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1808/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9MYW5kbWFyay5qYXZh) | `100.00% <ø> (ø)` | |
| [...main/java/ai/djl/modality/cv/output/Rectangle.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1808/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9SZWN0YW5nbGUuamF2YQ==) | `72.41% <0.00%> (ø)` | |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1808/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <ø> (-5.24%)` | :arrow_down: |
| [...odality/cv/translator/BigGANTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1808/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `33.33% <0.00%> (+8.33%)` | :arrow_up: |
| [...nslator/InstanceSegmentationTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1808/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW5zdGFuY2VTZWdtZW50YXRpb25UcmFuc2xhdG9yRmFjdG9yeS5qYXZh) | `14.28% <0.00%> (-3.90%)` | :arrow_down: |
| [.../cv/translator/SemanticSegmentationTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1808/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU2VtYW50aWNTZWdtZW50YXRpb25UcmFuc2xhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../cv/translator/StyleTransferTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1808/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU3R5bGVUcmFuc2ZlclRyYW5zbGF0b3JGYWN0b3J5LmphdmE=) | `40.00% <ø> (ø)` | |
| ... and [415 more](https://codecov.io/gh/deepjavalibrary/djl/pull/1808/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1808?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1808?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Last update [13453b4...b203b70](https://codecov.io/gh/deepjavalibrary/djl/pull/1808?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/android/core/src/main/java/ai/djl/android/core/BitmapImageFactory.java b/android/core/src/main/java/ai/djl/android/core/BitmapImageFactory.java
index f670a4590..9d78fa787 100644
--- a/android/core/src/main/java/ai/djl/android/core/BitmapImageFactory.java
+++ b/android/core/src/main/java/ai/djl/android/core/BitmapImageFactory.java
@@ -93,8 +93,13 @@ public class BitmapImageFactory extends ImageFactory {
         }
         if (shape.get(0) == 1) {
             throw new UnsupportedOperationException("Grayscale image is not supported");
-        } else if (shape.get(0) != 3){
-            throw new IllegalArgumentException("First dimension should be number of channel with value 1 or 3");
+        } else if (shape.get(0) != 3) {
+            if (shape.get(2) == 3) {
+                array = array.transpose(2, 0, 1);
+                shape = array.getShape();
+            } else {
+                throw new IllegalArgumentException("First or last dimension should be number of channel with value 1 or 3");
+            }
         }
         int height = (int) shape.get(1);
         int width = (int) shape.get(2);
diff --git a/api/src/main/java/ai/djl/modality/cv/translator/SemanticSegmentationTranslator.java b/api/src/main/java/ai/djl/modality/cv/translator/SemanticSegmentationTranslator.java
index b9c4b382b..0cda70c01 100644
--- a/api/src/main/java/ai/djl/modality/cv/translator/SemanticSegmentationTranslator.java
+++ b/api/src/main/java/ai/djl/modality/cv/translator/SemanticSegmentationTranslator.java
@@ -110,10 +110,12 @@ public class SemanticSegmentationTranslator extends BaseImageTranslator<Image> {
             }
             int originW = (int) ctx.getAttachment("originalWidth");
             int originH = (int) ctx.getAttachment("originalHeight");
-            NDArray intRet = manager.create(bb, new Shape(height, width, CHANNEL), DataType.UINT8);
-            NDArray resized = NDImageUtils.resize(intRet, originW, originH);
+            NDArray fullImage =
+                    manager.create(bb, new Shape(height, width, CHANNEL), DataType.UINT8);
+            NDArray resized = NDImageUtils.resize(fullImage, originW, originH);
+            NDArray ret = resized.toType(DataType.UINT8, false);
 
-            return ImageFactory.getInstance().fromNDArray(resized);
+            return ImageFactory.getInstance().fromNDArray(ret);
         }
     }
 
diff --git a/api/src/main/java/ai/djl/ndarray/NDArrayAdapter.java b/api/src/main/java/ai/djl/ndarray/NDArrayAdapter.java
index 60d1d02a9..541e93d75 100644
--- a/api/src/main/java/ai/djl/ndarray/NDArrayAdapter.java
+++ b/api/src/main/java/ai/djl/ndarray/NDArrayAdapter.java
@@ -74,6 +74,7 @@ public abstract class NDArrayAdapter implements NDArray {
         manager.attachInternal(getUid(), this);
         alternativeManager = ((BaseNDManager) manager).getAlternativeManager();
         if (alternativeManager == null) {
+            // to prevent hybrid engine memory leak
             alternativeManager = manager;
         }
     }
@@ -195,6 +196,12 @@ public abstract class NDArrayAdapter implements NDArray {
         throw new UnsupportedOperationException("Not implemented yet.");
     }
 
+    /** {@inheritDoc} */
+    @Override
+    public NDArray get(NDIndex index) {
+        return get(alternativeManager, index);
+    }
+
     /** {@inheritDoc} */
     @Override
     public void set(Buffer data) {
@@ -1075,7 +1082,11 @@ public abstract class NDArrayAdapter implements NDArray {
     /** {@inheritDoc} */
     @Override
     public NDArrayEx getNDArrayInternal() {
-        return getAlternativeArray().getNDArrayInternal();
+        NDArray array = getAlternativeArray();
+        if (array instanceof NDArrayAdapter) {
+            throw new UnsupportedOperationException("Operation not supported.");
+        }
+        return array.getNDArrayInternal();
     }
 
     /** {@inheritDoc} */
@@ -1083,11 +1094,11 @@ public abstract class NDArrayAdapter implements NDArray {
     public void close() {
         if (!isClosed) {
             manager.detachInternal(getUid());
+            isClosed = true;
             if (alternativeArray != null) {
                 alternativeArray.close();
                 alternativeArray = null;
             }
-            isClosed = true;
         }
     }
 
diff --git a/engines/onnxruntime/onnxruntime-engine/build.gradle b/engines/onnxruntime/onnxruntime-engine/build.gradle
index c961d5691..162663344 100644
--- a/engines/onnxruntime/onnxruntime-engine/build.gradle
+++ b/engines/onnxruntime/onnxruntime-engine/build.gradle
@@ -7,7 +7,8 @@ dependencies {
     testImplementation("org.testng:testng:${testng_version}") {
         exclude group: "junit", module: "junit"
     }
-    testImplementation(project(":testing"))
+    testImplementation project(":testing")
+    testImplementation project(":engines:pytorch:pytorch-engine")
 
     testRuntimeOnly "org.slf4j:slf4j-simple:${slf4j_version}"
 }
diff --git a/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/engine/OrtNDArray.java b/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/engine/OrtNDArray.java
index 0a1f3758e..07088cdd2 100644
--- a/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/engine/OrtNDArray.java
+++ b/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/engine/OrtNDArray.java
@@ -25,11 +25,12 @@ import java.nio.ByteBuffer;
 import java.nio.ByteOrder;
 import java.nio.charset.Charset;
 import java.util.UUID;
+import java.util.concurrent.atomic.AtomicReference;
 
 /** {@code OrtNDArray} is the ONNX Runtime implementation of {@link NDArray}. */
 public class OrtNDArray extends NDArrayAdapter {
 
-    private OnnxTensor tensor;
+    private AtomicReference<OnnxTensor> tensor;
 
     /**
      * Constructs an ONNX Runtime NDArray from a {@link OnnxTensor} (internal. Use {@link NDManager}
@@ -41,19 +42,24 @@ public class OrtNDArray extends NDArrayAdapter {
      */
     OrtNDArray(OrtNDManager manager, NDManager alternativeManager, OnnxTensor tensor) {
         super(manager, alternativeManager, null, null, UUID.randomUUID().toString());
-        this.tensor = tensor;
+        this.tensor = new AtomicReference<>(tensor);
         manager.attachInternal(uid, this);
     }
 
-    OnnxTensor getTensor() {
-        return tensor;
+    /**
+     * Returns the {@code OnnxTensor} representation of this OrtNDArray.
+     *
+     * @return the {@code OnnxTensor} representation of this OrtNDArray
+     */
+    public OnnxTensor getTensor() {
+        return tensor.get();
     }
 
     /** {@inheritDoc} */
     @Override
     public DataType getDataType() {
         if (dataType == null) {
-            dataType = OrtUtils.toDataType(tensor.getInfo().type);
+            dataType = OrtUtils.toDataType(tensor.get().getInfo().type);
         }
         return dataType;
     }
@@ -62,7 +68,7 @@ public class OrtNDArray extends NDArrayAdapter {
     @Override
     public Shape getShape() {
         if (shape == null) {
-            shape = new Shape(tensor.getInfo().getShape());
+            shape = new Shape(tensor.get().getInfo().getShape());
         }
         return shape;
     }
@@ -70,8 +76,7 @@ public class OrtNDArray extends NDArrayAdapter {
     /** {@inheritDoc} */
     @Override
     public void intern(NDArray replaced) {
-        tensor.close();
-        tensor = ((OrtNDArray) replaced).tensor;
+        tensor.getAndSet(((OrtNDArray) replaced).getTensor()).close();
     }
 
     /** {@inheritDoc} */
@@ -85,7 +90,7 @@ public class OrtNDArray extends NDArrayAdapter {
     @Override
     public String[] toStringArray(Charset charset) {
         try {
-            Object obj = tensor.getValue();
+            Object obj = tensor.get().getValue();
             if (obj instanceof String) {
                 // Scalar type;
                 return new String[] {(String) obj};
@@ -102,15 +107,16 @@ public class OrtNDArray extends NDArrayAdapter {
         if (getDataType() == DataType.STRING) {
             throw new IllegalArgumentException("Please use toStringArray() for String NDArray.");
         }
-        return tensor.getByteBuffer().order(ByteOrder.nativeOrder());
+        return tensor.get().getByteBuffer().order(ByteOrder.nativeOrder());
     }
 
     /** {@inheritDoc} */
     @Override
     public void close() {
-        if (!isClosed) {
-            tensor.close();
-            super.close();
+        super.close();
+        OnnxTensor ortTensor = tensor.getAndSet(null);
+        if (ortTensor != null) {
+            ortTensor.close();
         }
     }
 }


Test patch:
diff --git a/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java b/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java
index c5cba2a56..a07f59672 100644
--- a/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java
+++ b/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java
@@ -14,6 +14,7 @@ package ai.djl.onnxruntime.engine;
 
 import ai.djl.Model;
 import ai.djl.ModelException;
+import ai.djl.engine.Engine;
 import ai.djl.inference.Predictor;
 import ai.djl.modality.Classifications;
 import ai.djl.ndarray.NDArray;
@@ -25,6 +26,7 @@ import ai.djl.repository.zoo.Criteria;
 import ai.djl.repository.zoo.ZooModel;
 import ai.djl.testing.TestRequirements;
 import ai.djl.translate.TranslateException;
+import ai.onnxruntime.OrtException;
 
 import org.testng.Assert;
 import org.testng.SkipException;
@@ -32,6 +34,7 @@ import org.testng.annotations.Test;
 
 import java.io.IOException;
 import java.io.InputStream;
+import java.lang.reflect.Field;
 import java.nio.file.Files;
 import java.nio.file.Path;
 
@@ -88,7 +91,7 @@ public class OrtTest {
     }
 
     @Test
-    public void testNDArray() {
+    public void testNDArray() throws OrtException {
         TestRequirements.notArm();
 
         try (NDManager manager = OrtNDManager.getSystemManager().newSubManager()) {
@@ -111,6 +114,9 @@ public class OrtTest {
 
             array = manager.create(new String[] {"string1", "string2"});
             Assert.assertEquals(array.toStringArray()[1], "string2");
+
+            float[][] value = (float[][]) ((OrtNDArray) ones).getTensor().getValue();
+            Assert.assertEquals(value[0], new float[] {1, 1});
         }
     }
 
@@ -118,7 +124,7 @@ public class OrtTest {
     public void testStringTensor() throws ModelException, IOException, TranslateException {
         TestRequirements.notArm();
 
-        System.setProperty("ai.djl.onnx.disable_alternative", "true");
+        setAlternativeEngineDisabled(true);
         Criteria<NDList, NDList> criteria =
                 Criteria.builder()
                         .setTypes(NDList.class, NDList.class)
@@ -137,6 +143,38 @@ public class OrtTest {
             Assert.assertEquals(result.size(), 2);
             Assert.assertEquals(result.get(0).toLongArray(), new long[] {1});
         }
-        System.clearProperty("ai.djl.onnx.disable_alternative");
+        setAlternativeEngineDisabled(false);
+    }
+
+    @Test
+    public void testAlternativeArray() {
+        TestRequirements.notArm();
+
+        try (NDManager manager = OrtNDManager.getSystemManager().newSubManager()) {
+            NDArray array = manager.zeros(new Shape(1, 2));
+            Assert.assertEquals(array.get(0).toFloatArray(), new float[] {0, 0});
+        }
+
+        setAlternativeEngineDisabled(true);
+        try (NDManager manager = OrtNDManager.getSystemManager().newSubManager()) {
+            NDArray array = manager.zeros(new Shape(1, 2));
+            Assert.expectThrows(UnsupportedOperationException.class, () -> array.get(0));
+        }
+        setAlternativeEngineDisabled(false);
+    }
+
+    private void setAlternativeEngineDisabled(boolean enable) {
+        System.setProperty("ai.djl.onnx.disable_alternative", String.valueOf(enable));
+        Engine engine = Engine.getEngine(OrtEngine.ENGINE_NAME);
+        try {
+            Field field = OrtEngine.class.getDeclaredField("initialized");
+            field.setAccessible(true);
+            field.setBoolean(engine, false);
+            field = OrtEngine.class.getDeclaredField("alternativeEngine");
+            field.setAccessible(true);
+            field.set(engine, null);
+        } catch (ReflectiveOperationException ignore) {
+            // ignore
+        }
     }
 }