Repository: deepjavalibrary/djl
Pull request #1654 (2022-05-14 15:59:09)

[api] Improves ServingTranslator output handling
The output content-type will match the input content-type.

Change-Id: I06d864449b89e8e416371f2e2b32b91406ee4883

## Description ##

Brief description of what this PR is about

- If this change is a backward incompatible change, why must this change be made?
- Interesting edge cases to note here


codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1654?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
> Merging [#1654](https://codecov.io/gh/deepjavalibrary/djl/pull/1654?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (a4d7e67) into [master](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (bb5073f) will **decrease** coverage by `1.26%`.
> The diff coverage is `61.67%`.

```diff
@@             Coverage Diff              @@
##             master    #1654      +/-   ##
============================================
- Coverage     72.08%   70.82%   -1.27%     
- Complexity     5126     5438     +312     
============================================
  Files           473      507      +34     
  Lines         21970    23792    +1822     
  Branches       2351     2590     +239     
============================================
+ Hits          15838    16850    +1012     
- Misses         4925     5647     +722     
- Partials       1207     1295      +88     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/1654?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1654/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...rc/main/java/ai/djl/modality/cv/output/Joints.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1654/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9Kb2ludHMuamF2YQ==) | `71.42% <ø> (ø)` | |
| [.../main/java/ai/djl/modality/cv/output/Landmark.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1654/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9MYW5kbWFyay5qYXZh) | `100.00% <ø> (ø)` | |
| [...main/java/ai/djl/modality/cv/output/Rectangle.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1654/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9SZWN0YW5nbGUuamF2YQ==) | `72.41% <ø> (ø)` | |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1654/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <ø> (-5.24%)` | :arrow_down: |
| [...odality/cv/translator/BigGANTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1654/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `33.33% <0.00%> (+8.33%)` | :arrow_up: |
| [...nslator/InstanceSegmentationTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1654/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW5zdGFuY2VTZWdtZW50YXRpb25UcmFuc2xhdG9yRmFjdG9yeS5qYXZh) | `14.28% <0.00%> (-3.90%)` | :arrow_down: |
| [.../modality/cv/translator/YoloTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1654/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1RyYW5zbGF0b3JGYWN0b3J5LmphdmE=) | `8.33% <0.00%> (-1.67%)` | :arrow_down: |
| [...i/djl/modality/cv/translator/YoloV5Translator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1654/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1Y1VHJhbnNsYXRvci5qYXZh) | `5.69% <0.00%> (ø)` | |
| [...odality/cv/translator/YoloV5TranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1654/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1Y1VHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `8.33% <0.00%> (-1.67%)` | :arrow_down: |
| ... and [248 more](https://codecov.io/gh/deepjavalibrary/djl/pull/1654/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1654?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1654?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Last update [9d12c04...a4d7e67](https://codecov.io/gh/deepjavalibrary/djl/pull/1654?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/api/src/main/java/ai/djl/ndarray/NDList.java b/api/src/main/java/ai/djl/ndarray/NDList.java
index 8852501c1..0a580300f 100644
--- a/api/src/main/java/ai/djl/ndarray/NDList.java
+++ b/api/src/main/java/ai/djl/ndarray/NDList.java
@@ -292,8 +292,18 @@ public class NDList extends ArrayList<NDArray> implements NDResource, BytesSuppl
      * @return the byte array
      */
     public byte[] encode() {
+        return encode(false);
+    }
+
+    /**
+     * Encodes the NDList to byte array.
+     *
+     * @param numpy encode in npz format if true
+     * @return the byte array
+     */
+    public byte[] encode(boolean numpy) {
         try (ByteArrayOutputStream baos = new ByteArrayOutputStream()) {
-            encode(baos);
+            encode(baos, numpy);
             return baos.toByteArray();
         } catch (IOException e) {
             throw new AssertionError("NDList is not writable", e);
diff --git a/api/src/main/java/ai/djl/translate/ServingTranslatorFactory.java b/api/src/main/java/ai/djl/translate/ServingTranslatorFactory.java
index 11353777b..54db7afe9 100644
--- a/api/src/main/java/ai/djl/translate/ServingTranslatorFactory.java
+++ b/api/src/main/java/ai/djl/translate/ServingTranslatorFactory.java
@@ -251,6 +251,7 @@ public class ServingTranslatorFactory implements TranslatorFactory {
         public NDList processInput(TranslatorContext ctx, Input input) throws TranslateException {
             NDManager manager = ctx.getNDManager();
             try {
+                ctx.setAttachment("properties", input.getProperties());
                 return input.getDataAsNDList(manager);
             } catch (IllegalArgumentException e) {
                 throw new TranslateException("Input is not a NDList data type", e);
@@ -259,11 +260,19 @@ public class ServingTranslatorFactory implements TranslatorFactory {
 
         /** {@inheritDoc} */
         @Override
+        @SuppressWarnings("unchecked")
         public Output processOutput(TranslatorContext ctx, NDList list) {
+            Map<String, String> prop = (Map<String, String>) ctx.getAttachment("properties");
+            String contentType = prop.get("Content-Type");
+
             Output output = new Output();
-            // TODO: find a way to pass NDList out
-            output.add(list.getAsBytes());
-            output.addProperty("Content-Type", "tensor/ndlist");
+            if ("tensor/npz".equalsIgnoreCase(contentType)) {
+                output.add(list.encode(true));
+                output.addProperty("Content-Type", "tensor/npz");
+            } else {
+                output.add(list.encode(false));
+                output.addProperty("Content-Type", "tensor/ndlist");
+            }
             return output;
         }
     }
diff --git a/gradle.properties b/gradle.properties
index ef8fc8506..0ff6242d6 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -26,7 +26,7 @@ commons_compress_version=1.21
 commons_csv_version=1.9.0
 commons_logging_version=1.2
 gson_version=2.9.0
-jna_version=5.10.0
+jna_version=5.11.0
 slf4j_version=1.7.36
 log4j_slf4j_version=2.17.2
 awssdk_version=2.17.151


Test patch:
diff --git a/api/src/test/java/ai/djl/translate/BatchifierTest.java b/api/src/test/java/ai/djl/translate/BatchifierTest.java
new file mode 100644
index 000000000..c7ef460e9
--- /dev/null
+++ b/api/src/test/java/ai/djl/translate/BatchifierTest.java
@@ -0,0 +1,27 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.translate;
+
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+public class BatchifierTest {
+
+    @Test
+    public void testBatchifier() {
+        Batchifier batchifier = Batchifier.fromString("stack");
+        Assert.assertEquals(batchifier, Batchifier.STACK);
+
+        Assert.assertThrows(() -> Batchifier.fromString("invalid"));
+    }
+}
diff --git a/api/src/test/java/ai/djl/translate/ServingTranslatorTest.java b/api/src/test/java/ai/djl/translate/ServingTranslatorTest.java
new file mode 100644
index 000000000..6ac8da183
--- /dev/null
+++ b/api/src/test/java/ai/djl/translate/ServingTranslatorTest.java
@@ -0,0 +1,84 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.translate;
+
+import ai.djl.Model;
+import ai.djl.ModelException;
+import ai.djl.inference.Predictor;
+import ai.djl.modality.Input;
+import ai.djl.modality.Output;
+import ai.djl.ndarray.NDList;
+import ai.djl.ndarray.NDManager;
+import ai.djl.ndarray.types.DataType;
+import ai.djl.ndarray.types.Shape;
+import ai.djl.nn.Block;
+import ai.djl.nn.Blocks;
+import ai.djl.repository.zoo.Criteria;
+import ai.djl.repository.zoo.ZooModel;
+import ai.djl.util.Utils;
+import java.io.IOException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.Test;
+
+public class ServingTranslatorTest {
+
+    @AfterClass
+    public void tierDown() {
+        Utils.deleteQuietly(Paths.get("build/model"));
+    }
+
+    @Test
+    public void testNumpy() throws IOException, TranslateException, ModelException {
+        Path path = Paths.get("build/model");
+        Files.createDirectories(path);
+        Input input = new Input();
+
+        try (NDManager manager = NDManager.newBaseManager()) {
+            Block block = Blocks.identityBlock();
+            block.initialize(manager, DataType.FLOAT32, new Shape(1));
+            Model model = Model.newInstance("identity");
+            model.setBlock(block);
+            model.save(path, null);
+            model.close();
+            NDList list = new NDList();
+            list.add(manager.create(10f));
+            input.add(list.encode(true));
+            input.add("Content-Type", "tensor/npz");
+        }
+
+        Criteria<Input, Output> criteria =
+                Criteria.builder()
+                        .setTypes(Input.class, Output.class)
+                        .optModelPath(path)
+                        .optModelName("identity")
+                        .optBlock(Blocks.identityBlock())
+                        .build();
+
+        try (ZooModel<Input, Output> model = criteria.loadModel();
+                Predictor<Input, Output> predictor = model.newPredictor()) {
+            Output output = predictor.predict(input);
+            try (NDManager manager = NDManager.newBaseManager()) {
+                NDList list = output.getDataAsNDList(manager);
+                Assert.assertEquals(list.size(), 1);
+                Assert.assertEquals(list.get(0).toFloatArray()[0], 10f);
+            }
+            Input invalid = new Input();
+            invalid.add("String");
+            Assert.assertThrows(TranslateException.class, () -> predictor.predict(invalid));
+        }
+    }
+}
diff --git a/api/src/test/java/ai/djl/translate/package-info.java b/api/src/test/java/ai/djl/translate/package-info.java
new file mode 100644
index 000000000..31c8c2e84
--- /dev/null
+++ b/api/src/test/java/ai/djl/translate/package-info.java
@@ -0,0 +1,15 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+
+/** Contains tests for {@link ai.djl.translate}. */
+package ai.djl.translate;