Repository: deepjavalibrary/djl
Pull request #1833 (2022-07-26 01:26:56)

Updated image factories for HWC and CHW
## Description ##

Both image factories now handle HWC and CHW formats

- Transpose is not available for all engines so we must handle the different formats without transposing to a single format type
- Test on the apps and within DJL repo


codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1833?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
> Merging [#1833](https://codecov.io/gh/deepjavalibrary/djl/pull/1833?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (0aa9e46) into [master](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (bb5073f) will **decrease** coverage by `2.39%`.
> The diff coverage is `63.93%`.

```diff
@@             Coverage Diff              @@
##             master    #1833      +/-   ##
============================================
- Coverage     72.08%   69.69%   -2.40%     
- Complexity     5126     5522     +396     
============================================
  Files           473      526      +53     
  Lines         21970    24441    +2471     
  Branches       2351     2662     +311     
============================================
+ Hits          15838    17035    +1197     
- Misses         4925     6094    +1169     
- Partials       1207     1312     +105     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/1833?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1833/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1833/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L011bHRpQm94UHJpb3IuamF2YQ==) | `76.00% <ø> (ø)` | |
| [...rc/main/java/ai/djl/modality/cv/output/Joints.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1833/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9Kb2ludHMuamF2YQ==) | `71.42% <ø> (ø)` | |
| [.../main/java/ai/djl/modality/cv/output/Landmark.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1833/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9MYW5kbWFyay5qYXZh) | `100.00% <ø> (ø)` | |
| [...main/java/ai/djl/modality/cv/output/Rectangle.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1833/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9SZWN0YW5nbGUuamF2YQ==) | `72.41% <0.00%> (ø)` | |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1833/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <0.00%> (-5.24%)` | :arrow_down: |
| [...odality/cv/translator/BigGANTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1833/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `33.33% <0.00%> (+8.33%)` | :arrow_up: |
| [...nslator/InstanceSegmentationTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1833/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW5zdGFuY2VTZWdtZW50YXRpb25UcmFuc2xhdG9yRmFjdG9yeS5qYXZh) | `14.28% <0.00%> (-3.90%)` | :arrow_down: |
| [.../cv/translator/SemanticSegmentationTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1833/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU2VtYW50aWNTZWdtZW50YXRpb25UcmFuc2xhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../cv/translator/StyleTransferTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1833/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU3R5bGVUcmFuc2ZlclRyYW5zbGF0b3JGYWN0b3J5LmphdmE=) | `40.00% <ø> (ø)` | |
| ... and [420 more](https://codecov.io/gh/deepjavalibrary/djl/pull/1833/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1833?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1833?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Last update [404c733...0aa9e46](https://codecov.io/gh/deepjavalibrary/djl/pull/1833?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/android/core/src/main/java/ai/djl/android/core/BitmapImageFactory.java b/android/core/src/main/java/ai/djl/android/core/BitmapImageFactory.java
index f5e29a6db..aeca1d997 100644
--- a/android/core/src/main/java/ai/djl/android/core/BitmapImageFactory.java
+++ b/android/core/src/main/java/ai/djl/android/core/BitmapImageFactory.java
@@ -90,15 +90,26 @@ public class BitmapImageFactory extends ImageFactory {
         } else if (shape.get(0) == 1 || shape.get(2) == 1) {
             throw new UnsupportedOperationException("Grayscale image is not supported");
         }
+        int[] raw = array.toType(DataType.UINT8, false).toUint8Array();
         if (NDImageUtils.isCHW(shape)) {
-            array = array.transpose(1, 2, 0);
-            shape = array.getShape();
+            int height = (int) shape.get(1);
+            int width = (int) shape.get(2);
+            Bitmap bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
+            int imageArea = width * height;
+            for (int h = 0; h < height; ++h) {
+                for (int w = 0; w < width; ++w) {
+                    int index = h * width + w;
+                    int red = raw[index] & 0xFF;
+                    int green = raw[imageArea + index] & 0xFF;
+                    int blue = raw[imageArea * 2 + index] & 0xFF;
+                    bitmap.setPixel(w, h, Color.argb(255, red, green, blue));
+                }
+            }
+            return new BitMapWrapper(bitmap);
         }
-
         int height = (int) shape.get(0);
         int width = (int) shape.get(1);
         Bitmap bitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
-        int[] raw = array.toType(DataType.UINT8, false).toUint8Array();
         for (int h = 0; h < height; ++h) {
             for (int w = 0; w < width; ++w) {
                 int pos = (h * width + w) * 3;
diff --git a/api/src/main/java/ai/djl/modality/cv/BufferedImageFactory.java b/api/src/main/java/ai/djl/modality/cv/BufferedImageFactory.java
index c02c36464..a66050657 100644
--- a/api/src/main/java/ai/djl/modality/cv/BufferedImageFactory.java
+++ b/api/src/main/java/ai/djl/modality/cv/BufferedImageFactory.java
@@ -90,14 +90,28 @@ public class BufferedImageFactory extends ImageFactory {
         } else if (shape.get(0) == 1 || shape.get(2) == 1) {
             throw new UnsupportedOperationException("Grayscale image is not supported");
         }
+        int[] raw = array.toType(DataType.UINT8, false).toUint8Array();
         if (NDImageUtils.isCHW(shape)) {
-            array = array.transpose(1, 2, 0);
-            shape = array.getShape();
+            int height = (int) shape.get(1);
+            int width = (int) shape.get(2);
+            BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_INT_RGB);
+            int[] pixels = new int[width * height];
+            int imageArea = height * width;
+            for (int h = 0; h < height; ++h) {
+                for (int w = 0; w < width; ++w) {
+                    int index = h * width + w;
+                    int red = raw[index];
+                    int green = raw[imageArea + index];
+                    int blue = raw[imageArea * 2 + index];
+                    pixels[index] = (red << 16) | (green << 8) | blue;
+                }
+            }
+            image.setRGB(0, 0, width, height, pixels, 0, width);
+            return new BufferedImageWrapper(image);
         }
         int height = (int) shape.get(0);
         int width = (int) shape.get(1);
         BufferedImage image = new BufferedImage(width, height, BufferedImage.TYPE_INT_RGB);
-        int[] raw = array.toType(DataType.UINT8, false).toUint8Array();
         int[] pixels = new int[width * height];
         for (int h = 0; h < height; ++h) {
             for (int w = 0; w < width; ++w) {
diff --git a/engines/pytorch/pytorch-engine/README.md b/engines/pytorch/pytorch-engine/README.md
index 6f6bea49b..302ee497d 100644
--- a/engines/pytorch/pytorch-engine/README.md
+++ b/engines/pytorch/pytorch-engine/README.md
@@ -234,7 +234,6 @@ For the Windows platform, you can choose between CPU and GPU.
 
 - ai.djl.pytorch:pytorch-jni:1.11.0-0.18.0
 - ai.djl.pytorch:pytorch-native-cu113:1.11.0:win-x86_64
-- ai.djl.pytorch:pytorch-native-cu102:1.11.0:win-x86_64
 
 ```xml
 <dependency>
@@ -252,22 +251,6 @@ For the Windows platform, you can choose between CPU and GPU.
 </dependency>
 ```
 
-```xml
-<dependency>
-    <groupId>ai.djl.pytorch</groupId>
-    <artifactId>pytorch-native-cu102</artifactId>
-    <classifier>win-x86_64</classifier>
-    <version>1.11.0</version>
-    <scope>runtime</scope>
-</dependency>
-<dependency>
-    <groupId>ai.djl.pytorch</groupId>
-    <artifactId>pytorch-jni</artifactId>
-    <version>1.11.0-0.18.0</version>
-    <scope>runtime</scope>
-</dependency>
-```
-
 ### Windows CPU
 
 - ai.djl.pytorch:pytorch-jni:1.11.0-0.18.0


Test patch:
diff --git a/android/core/src/androidTest/java/ai/djl/android/core/BitmapWrapperTest.java b/android/core/src/androidTest/java/ai/djl/android/core/BitmapWrapperTest.java
index 20d42e038..d54f8dcd9 100644
--- a/android/core/src/androidTest/java/ai/djl/android/core/BitmapWrapperTest.java
+++ b/android/core/src/androidTest/java/ai/djl/android/core/BitmapWrapperTest.java
@@ -90,6 +90,12 @@ public class BitmapWrapperTest {
                     .transpose(2, 0, 1)
                     .toType(DataType.FLOAT32, true);
             assertEquals(array, converted);
+
+            array = manager.arange(0, 12, 1, DataType.UINT8).reshape(3, 2, 2);
+            array = array.transpose(1, 2, 0);
+            image = factory.fromNDArray(array);
+            converted = image.toNDArray(manager);
+            assertEquals(array, converted);
         }
     }
 }
diff --git a/integration/src/main/java/ai/djl/integration/tests/modality/cv/BufferedImageFactoryTest.java b/integration/src/main/java/ai/djl/integration/tests/modality/cv/BufferedImageFactoryTest.java
index a01db157f..f7420c750 100644
--- a/integration/src/main/java/ai/djl/integration/tests/modality/cv/BufferedImageFactoryTest.java
+++ b/integration/src/main/java/ai/djl/integration/tests/modality/cv/BufferedImageFactoryTest.java
@@ -47,6 +47,11 @@ public class BufferedImageFactoryTest {
             NDArray converted =
                     image.toNDArray(manager).transpose(2, 0, 1).toType(DataType.FLOAT32, true);
             Assertions.assertAlmostEquals(array, converted);
+
+            array = array.transpose(1, 2, 0);
+            image = factory.fromNDArray(array);
+            converted = image.toNDArray(manager).toType(DataType.FLOAT32, false);
+            Assertions.assertAlmostEquals(array, converted);
         }
     }
 }