Repository: deepjavalibrary/djl
Pull request #3494 (2024-10-04 00:31:37)

[api] Visualize sam2 output for Sam2ServingTranslator
- **[api] Adds sam2 model to onnxruntime model zoo**
- **[api] Standardizes CV output format**
- **[api] Visualize sam2 output for Sam2ServingTranslator**

## Description ##

Brief description of what this PR is about

- If this change is a backward incompatible change, why must this change be made?
- Interesting edge cases to note here


codecov-commenter: :warning: Please install the !['codecov app svg image'](https://github.com/codecov/engineering-team/assets/152432831/e90313f4-9d3a-4b63-8b54-cfe14e7ec20d) to ensure uploads and comments are reliably processed by Codecov.

## [Codecov](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
Attention: Patch coverage is `63.97849%` with `67 lines` in your changes missing coverage. Please review.
> Project coverage is 72.39%. Comparing base [(`b6656ed`)](https://app.codecov.io/gh/deepjavalibrary/djl/commit/b6656ed60d823ee835bd69fb4e42a8244038b807?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) to head [(`7020502`)](https://app.codecov.io/gh/deepjavalibrary/djl/commit/702050217402c3e620960d0188a553d81f21341f?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).
> Report is 1 commits behind head on master.

| [Files with missing lines](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Patch % | Lines |
|---|---|---|
| [.../modality/cv/translator/Sam2ServingTranslator.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?src=pr&el=tree&filepath=api%2Fsrc%2Fmain%2Fjava%2Fai%2Fdjl%2Fmodality%2Fcv%2Ftranslator%2FSam2ServingTranslator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU2FtMlNlcnZpbmdUcmFuc2xhdG9yLmphdmE=) | 8.33% | [33 Missing :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [.../ai/djl/modality/cv/translator/Sam2Translator.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?src=pr&el=tree&filepath=api%2Fsrc%2Fmain%2Fjava%2Fai%2Fdjl%2Fmodality%2Fcv%2Ftranslator%2FSam2Translator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU2FtMlRyYW5zbGF0b3IuamF2YQ==) | 73.91% | [14 Missing and 4 partials :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [...pencv/src/main/java/ai/djl/opencv/OpenCVImage.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?src=pr&el=tree&filepath=extensions%2Fopencv%2Fsrc%2Fmain%2Fjava%2Fai%2Fdjl%2Fopencv%2FOpenCVImage.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-ZXh0ZW5zaW9ucy9vcGVuY3Yvc3JjL21haW4vamF2YS9haS9kamwvb3BlbmN2L09wZW5DVkltYWdlLmphdmE=) | 0.00% | [11 Missing :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [...main/java/ai/djl/onnxruntime/engine/OrtEngine.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?src=pr&el=tree&filepath=engines%2Fonnxruntime%2Fonnxruntime-engine%2Fsrc%2Fmain%2Fjava%2Fai%2Fdjl%2Fonnxruntime%2Fengine%2FOrtEngine.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-ZW5naW5lcy9vbm54cnVudGltZS9vbm54cnVudGltZS1lbmdpbmUvc3JjL21haW4vamF2YS9haS9kamwvb25ueHJ1bnRpbWUvZW5naW5lL09ydEVuZ2luZS5qYXZh) | 33.33% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [...ava/ai/djl/modality/cv/output/DetectedObjects.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?src=pr&el=tree&filepath=api%2Fsrc%2Fmain%2Fjava%2Fai%2Fdjl%2Fmodality%2Fcv%2Foutput%2FDetectedObjects.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9EZXRlY3RlZE9iamVjdHMuamF2YQ==) | 66.66% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [.../modality/cv/translator/Sam2TranslatorFactory.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?src=pr&el=tree&filepath=api%2Fsrc%2Fmain%2Fjava%2Fai%2Fdjl%2Fmodality%2Fcv%2Ftranslator%2FSam2TranslatorFactory.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU2FtMlRyYW5zbGF0b3JGYWN0b3J5LmphdmE=) | 75.00% | [0 Missing and 1 partial :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [...pi/src/main/java/ai/djl/util/JsonSerializable.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?src=pr&el=tree&filepath=api%2Fsrc%2Fmain%2Fjava%2Fai%2Fdjl%2Futil%2FJsonSerializable.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL3V0aWwvSnNvblNlcmlhbGl6YWJsZS5qYXZh) | 80.00% | [1 Missing :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |

:exclamation: Your organization needs to install the [Codecov GitHub app](https://github.com/apps/codecov/installations/select_target) to enable full functionality.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff              @@
##             master    #3494      +/-   ##
============================================
- Coverage     72.52%   72.39%   -0.13%     
- Complexity     7394     7410      +16     
============================================
  Files           672      674       +2     
  Lines         32909    33027     +118     
  Branches       3501     3516      +15     
============================================
+ Hits          23868    23911      +43     
- Misses         7400     7483      +83     
+ Partials       1641     1633       -8     
```

| [Flag](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | `?` | |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3494?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/api/src/main/java/ai/djl/modality/cv/translator/Sam2ServingTranslator.java b/api/src/main/java/ai/djl/modality/cv/translator/Sam2ServingTranslator.java
index 909c3d01f..97432dd03 100644
--- a/api/src/main/java/ai/djl/modality/cv/translator/Sam2ServingTranslator.java
+++ b/api/src/main/java/ai/djl/modality/cv/translator/Sam2ServingTranslator.java
@@ -14,7 +14,9 @@ package ai.djl.modality.cv.translator;
 
 import ai.djl.modality.Input;
 import ai.djl.modality.Output;
+import ai.djl.modality.cv.Image;
 import ai.djl.modality.cv.output.DetectedObjects;
+import ai.djl.modality.cv.output.Rectangle;
 import ai.djl.modality.cv.translator.Sam2Translator.Sam2Input;
 import ai.djl.ndarray.BytesSupplier;
 import ai.djl.ndarray.NDList;
@@ -23,7 +25,13 @@ import ai.djl.translate.TranslateException;
 import ai.djl.translate.Translator;
 import ai.djl.translate.TranslatorContext;
 
+import org.apache.commons.codec.binary.Base64OutputStream;
+
+import java.io.ByteArrayOutputStream;
 import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.LinkedHashMap;
+import java.util.Map;
 
 /** A {@link Translator} that can serve SAM2 model. */
 public class Sam2ServingTranslator implements Translator<Input, Output> {
@@ -47,11 +55,29 @@ public class Sam2ServingTranslator implements Translator<Input, Output> {
 
     /** {@inheritDoc} */
     @Override
-    public Output processOutput(TranslatorContext ctx, NDList list) throws Exception {
+    public Output processOutput(TranslatorContext ctx, NDList list) throws IOException {
         Output output = new Output();
+        Sam2Input sam2 = (Sam2Input) ctx.getAttachment("input");
         output.addProperty("Content-Type", "application/json");
-        DetectedObjects obj = translator.processOutput(ctx, list);
-        output.add(BytesSupplier.wrapAsJson(obj));
+        DetectedObjects detection = translator.processOutput(ctx, list);
+        Map<String, Object> ret = new LinkedHashMap<>(); // NOPMD
+        ret.put("result", detection);
+        if (sam2.isVisualize()) {
+            Image img = sam2.getImage();
+            img.drawBoundingBoxes(detection, 0.8f);
+            img.drawMarks(sam2.getPoints());
+            for (Rectangle rect : sam2.getBoxes()) {
+                img.drawRectangle(rect, 0xff0000, 6);
+            }
+            ByteArrayOutputStream os = new ByteArrayOutputStream();
+            os.write("data:image/png;base64,".getBytes(StandardCharsets.UTF_8));
+            Base64OutputStream bos = new Base64OutputStream(os, true, 0, null);
+            img.save(bos, "png");
+            bos.close();
+            os.close();
+            ret.put("image", os.toString(StandardCharsets.UTF_8.name()));
+        }
+        output.add(BytesSupplier.wrapAsJson(ret));
         return output;
     }
 
@@ -64,6 +90,7 @@ public class Sam2ServingTranslator implements Translator<Input, Output> {
                 throw new TranslateException("Input data is empty.");
             }
             Sam2Input sam2 = Sam2Input.fromJson(data.getAsString());
+            ctx.setAttachment("input", sam2);
             return translator.processInput(ctx, sam2);
         } catch (IOException e) {
             throw new TranslateException("Input is not an Image data type", e);
diff --git a/api/src/main/java/ai/djl/modality/cv/translator/Sam2Translator.java b/api/src/main/java/ai/djl/modality/cv/translator/Sam2Translator.java
index 0c05c569f..f1b75dd95 100644
--- a/api/src/main/java/ai/djl/modality/cv/translator/Sam2Translator.java
+++ b/api/src/main/java/ai/djl/modality/cv/translator/Sam2Translator.java
@@ -205,6 +205,7 @@ public class Sam2Translator implements NoBatchifyTranslator<Sam2Input, DetectedO
         private Image image;
         private Point[] points;
         private int[] labels;
+        private boolean visualize;
 
         /**
          * Constructs a {@code Sam2Input} instance.
@@ -214,9 +215,22 @@ public class Sam2Translator implements NoBatchifyTranslator<Sam2Input, DetectedO
          * @param labels the labels for the locations (0: background, 1: foreground)
          */
         public Sam2Input(Image image, Point[] points, int[] labels) {
+            this(image, points, labels, false);
+        }
+
+        /**
+         * Constructs a {@code Sam2Input} instance.
+         *
+         * @param image the image
+         * @param points the locations on the image
+         * @param labels the labels for the locations (0: background, 1: foreground)
+         * @param visualize true if output visualized image
+         */
+        public Sam2Input(Image image, Point[] points, int[] labels, boolean visualize) {
             this.image = image;
             this.points = points;
             this.labels = labels;
+            this.visualize = visualize;
         }
 
         /**
@@ -228,6 +242,15 @@ public class Sam2Translator implements NoBatchifyTranslator<Sam2Input, DetectedO
             return image;
         }
 
+        /**
+         * Returns {@code true} if output visualized image.
+         *
+         * @return {@code true} if output visualized image
+         */
+        public boolean isVisualize() {
+            return visualize;
+        }
+
         /**
          * Returns the locations.
          *
@@ -288,13 +311,16 @@ public class Sam2Translator implements NoBatchifyTranslator<Sam2Input, DetectedO
         public static Sam2Input fromJson(String input) throws IOException {
             Prompt prompt = JsonUtils.GSON.fromJson(input, Prompt.class);
             if (prompt.image == null) {
-                throw new IllegalArgumentException("Missing url value");
+                throw new IllegalArgumentException("Missing image value");
             }
             if (prompt.prompt == null || prompt.prompt.length == 0) {
                 throw new IllegalArgumentException("Missing prompt value");
             }
             Image image = ImageFactory.getInstance().fromUrl(prompt.image);
             Builder builder = builder(image);
+            if (prompt.visualize) {
+                builder.visualize();
+            }
             for (Location location : prompt.prompt) {
                 int[] data = location.data;
                 if ("point".equals(location.type)) {
@@ -322,6 +348,7 @@ public class Sam2Translator implements NoBatchifyTranslator<Sam2Input, DetectedO
             private Image image;
             private List<Point> points;
             private List<Integer> labels;
+            private boolean visualize;
 
             Builder(Image image) {
                 this.image = image;
@@ -380,6 +407,16 @@ public class Sam2Translator implements NoBatchifyTranslator<Sam2Input, DetectedO
                 return this;
             }
 
+            /**
+             * Sets the visualize for the {@code Sam2Input}.
+             *
+             * @return the builder
+             */
+            public Builder visualize() {
+                visualize = true;
+                return this;
+            }
+
             /**
              * Builds the {@code Sam2Input}.
              *
@@ -388,7 +425,7 @@ public class Sam2Translator implements NoBatchifyTranslator<Sam2Input, DetectedO
             public Sam2Input build() {
                 Point[] location = points.toArray(new Point[0]);
                 int[] array = labels.stream().mapToInt(Integer::intValue).toArray();
-                return new Sam2Input(image, location, array);
+                return new Sam2Input(image, location, array, visualize);
             }
         }
 
@@ -413,6 +450,7 @@ public class Sam2Translator implements NoBatchifyTranslator<Sam2Input, DetectedO
         private static final class Prompt {
             String image;
             Location[] prompt;
+            boolean visualize;
 
             public void setImage(String image) {
                 this.image = image;
@@ -421,6 +459,10 @@ public class Sam2Translator implements NoBatchifyTranslator<Sam2Input, DetectedO
             public void setPrompt(Location[] prompt) {
                 this.prompt = prompt;
             }
+
+            public void setVisualize(boolean visualize) {
+                this.visualize = visualize;
+            }
         }
     }
 }
diff --git a/api/src/main/java/ai/djl/util/JsonSerializable.java b/api/src/main/java/ai/djl/util/JsonSerializable.java
index b998e94e5..e321ec21e 100644
--- a/api/src/main/java/ai/djl/util/JsonSerializable.java
+++ b/api/src/main/java/ai/djl/util/JsonSerializable.java
@@ -49,11 +49,7 @@ public interface JsonSerializable extends Serializable, BytesSupplier {
         return ByteBuffer.wrap(toJson().getBytes(StandardCharsets.UTF_8));
     }
 
-    /**
-     * Serializes the object to the {@code JsonElement}.
-     *
-     * @return the {@code JsonElement}
-     */
+    /** {@inheritDoc} */
     JsonElement serialize();
 
     /** A customized Gson serializer to serialize the {@code Segmentation} object. */


Test patch:
diff --git a/api/src/test/java/ai/djl/modality/cv/translator/Sam2InputTest.java b/api/src/test/java/ai/djl/modality/cv/translator/Sam2InputTest.java
index 9f9811bda..a742e92d4 100644
--- a/api/src/test/java/ai/djl/modality/cv/translator/Sam2InputTest.java
+++ b/api/src/test/java/ai/djl/modality/cv/translator/Sam2InputTest.java
@@ -33,15 +33,17 @@ public class Sam2InputTest {
                 "{\"image\": \""
                         + file.toUri().toURL()
                         + "\",\n"
+                        + "\"visualize\": true,\n"
                         + "\"prompt\": [\n"
                         + "    {\"type\": \"point\", \"data\": [575, 750], \"label\": 0},\n"
                         + "    {\"type\": \"rectangle\", \"data\": [425, 600, 700, 875]}\n"
                         + "]}";
         Sam2Input input = Sam2Input.fromJson(json);
+        Assert.assertTrue(input.isVisualize());
         Assert.assertEquals(input.getPoints().size(), 1);
         Assert.assertEquals(input.getBoxes().size(), 1);
 
-        input = Sam2Input.builder(img).addPoint(0, 1).addBox(0, 0, 1, 1).build();
+        input = Sam2Input.builder(img).visualize().addPoint(0, 1).addBox(0, 0, 1, 1).build();
         Assert.assertEquals(input.getPoints().size(), 1);
         Assert.assertEquals(input.getBoxes().size(), 1);
     }