Repository: deepjavalibrary/djl
Pull request #3500 (2024-10-06 20:05:17)

[api] Fixes QaServingTranslator output format
## Description ##

Brief description of what this PR is about

- If this change is a backward incompatible change, why must this change be made?
- Interesting edge cases to note here


codecov-commenter: :warning: Please install the !['codecov app svg image'](https://github.com/codecov/engineering-team/assets/152432831/e90313f4-9d3a-4b63-8b54-cfe14e7ec20d) to ensure uploads and comments are reliably processed by Codecov.

## [Codecov](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3500?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
Attention: Patch coverage is `68.00000%` with `8 lines` in your changes missing coverage. Please review.
> Project coverage is 58.91%. Comparing base [(`b6656ed`)](https://app.codecov.io/gh/deepjavalibrary/djl/commit/b6656ed60d823ee835bd69fb4e42a8244038b807?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) to head [(`ee2ecb3`)](https://app.codecov.io/gh/deepjavalibrary/djl/commit/ee2ecb3a5e8c2af2c4f721df9029e79e720dd188?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).
> Report is 9 commits behind head on master.

| [Files with missing lines](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3500?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Patch % | Lines |
|---|---|---|
| [...l/modality/nlp/translator/QaServingTranslator.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3500?src=pr&el=tree&filepath=api%2Fsrc%2Fmain%2Fjava%2Fai%2Fdjl%2Fmodality%2Fnlp%2Ftranslator%2FQaServingTranslator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L25scC90cmFuc2xhdG9yL1FhU2VydmluZ1RyYW5zbGF0b3IuamF2YQ==) | 40.00% | [6 Missing :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [...anslator/TokenClassificationServingTranslator.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3500?src=pr&el=tree&filepath=api%2Fsrc%2Fmain%2Fjava%2Fai%2Fdjl%2Fmodality%2Fnlp%2Ftranslator%2FTokenClassificationServingTranslator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L25scC90cmFuc2xhdG9yL1Rva2VuQ2xhc3NpZmljYXRpb25TZXJ2aW5nVHJhbnNsYXRvci5qYXZh) | 0.00% | [1 Missing :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [...rc/main/java/ai/djl/ndarray/BytesSupplierImpl.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3500?src=pr&el=tree&filepath=api%2Fsrc%2Fmain%2Fjava%2Fai%2Fdjl%2Fndarray%2FBytesSupplierImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL25kYXJyYXkvQnl0ZXNTdXBwbGllckltcGwuamF2YQ==) | 50.00% | [1 Missing :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3500?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |

:exclamation: Your organization needs to install the [Codecov GitHub app](https://github.com/apps/codecov/installations/select_target) to enable full functionality.

<details><summary>Additional details and impacted files</summary>


```diff
@@              Coverage Diff              @@
##             master    #3500       +/-   ##
=============================================
- Coverage     72.52%   58.91%   -13.62%     
+ Complexity     7394     5856     -1538     
=============================================
  Files           672      674        +2     
  Lines         32909    33042      +133     
  Branches       3501     3523       +22     
=============================================
- Hits          23868    19467     -4401     
- Misses         7400    12093     +4693     
+ Partials       1641     1482      -159     
```

| [Flag](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3500/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3500/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | `?` | |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#carryforward-flags-in-the-pull-request-comment) to find out more.

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/deepjavalibrary/djl/pull/3500?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/.github/workflows/serving_publish.yml b/.github/workflows/serving_publish.yml
deleted file mode 100644
index 8ba5e4b54..000000000
--- a/.github/workflows/serving_publish.yml
+++ /dev/null
@@ -1,110 +0,0 @@
-name: Serving publish
-
-on:
-  schedule:
-    - cron: '0 12 * * *'
-  workflow_dispatch:
-    inputs:
-      mode:
-        description: 'staging/snapshot, default is snapshot'
-        required: true
-        default: 'snapshot'
-      serving-branch:
-        description: 'djl-serving branch, default is master'
-        required: true
-        default: 'master'
-      repo-id:
-        description: 'staging repository id for djl api packages'
-        required: false
-        default: ''
-
-permissions:
-  id-token: write
-  contents: read
-
-jobs:
-  publish:
-    if: github.repository == 'deepjavalibrary/djl'
-    runs-on: ubuntu-latest
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          repository: deepjavalibrary/djl-serving
-          ref: ${{ github.event.inputs.serving-branch }}
-      - name: Set up JDK 17
-        uses: actions/setup-java@v4
-        with:
-          distribution: 'corretto'
-          java-version: 17
-      # Enable gradle cache: https://github.com/actions/cache/blob/master/examples.md#java---gradle
-      - uses: actions/cache@v4
-        with:
-          path: ~/.gradle/caches
-          key: ${{ runner.os }}-gradle-${{ hashFiles('*/build.gradle.kts', 'engines/**/build.gradle.kts', 'extensions/**/build.gradle.kts') }}
-          restore-keys: |
-            ${{ runner.os }}-gradle-
-      - name: Configure AWS Credentials
-        uses: aws-actions/configure-aws-credentials@v4
-        with:
-          role-to-assume: arn:aws:iam::425969335547:role/djl-ci-publish-role
-          aws-region: us-east-2
-      - name: Copy serving snapshot artifacts to S3
-        if: ${{ github.event.inputs.mode == '' || github.event.inputs.mode == 'snapshot' }}
-        run: |
-          ./gradlew :serving:createDeb -Psnapshot
-          DJL_VERSION=$(awk -F '=' '/djl / {gsub(/ ?"/, "", $2); print $2}' gradle/libs.versions.toml)
-          aws s3 cp serving/build/distributions/*.deb s3://djl-ai/publish/djl-serving/
-          aws cloudfront create-invalidation --distribution-id E371VB8JQ6NRVY --paths "/djl-serving/djl-serving_${DJL_VERSION}*"
-      - name: Copy serving release artifacts to S3
-        if: ${{ github.event.inputs.mode == 'staging' }}
-        run: |
-          ./gradlew :serving:dZ :serving:createDeb -Pstaging
-          DJL_VERSION=$(awk -F '=' '/djl / {gsub(/ ?"/, "", $2); print $2}' gradle/libs.versions.toml)
-          aws s3 cp serving/build/distributions/*.deb s3://djl-ai/publish/djl-serving/
-          aws cloudfront create-invalidation --distribution-id E371VB8JQ6NRVY --paths "/djl-serving/djl-serving_${DJL_VERSION}*"
-          if [[ $(aws s3 ls s3://djl-ai/publish/djl-serving/serving-$DJL_VERSION.tar | wc -l) -eq 0 ]]; \
-            then aws s3 cp serving/build/distributions/*.tar s3://djl-ai/publish/djl-serving/; \
-            else echo serving tarball published already!; fi
-          aws s3 cp serving/build/distributions/*.zip s3://djl-ai/publish/djl-serving/
-          aws cloudfront create-invalidation --distribution-id E371VB8JQ6NRVY --paths "/djl-serving/serving-${DJL_VERSION}*"
-      - name: Copy benchmark release artifacts to S3
-        if: ${{ github.event.inputs.mode == 'staging' }}
-        run: |
-          ./gradlew :benchmark:dZ :benchmark:createDeb -Pstaging
-          DJL_VERSION=$(awk -F '=' '/djl / {gsub(/ ?"/, "", $2); print $2}' gradle/libs.versions.toml)
-          aws s3 cp benchmark/build/distributions/*.tar s3://djl-ai/publish/djl-bench/${DJL_VERSION}/
-          aws s3 cp benchmark/build/distributions/*.deb s3://djl-ai/publish/djl-bench/${DJL_VERSION}/
-          aws s3 cp benchmark/build/distributions/*.zip s3://djl-ai/publish/djl-bench/${DJL_VERSION}/
-          aws cloudfront create-invalidation --distribution-id E371VB8JQ6NRVY --paths "/djl-bench/${DJL_VERSION}/*"
-      - name: Copy awscurl snapshot artifacts to S3
-        if: ${{ github.event.inputs.mode == '' || github.event.inputs.mode == 'snapshot' }}
-        run: |
-          ./gradlew :awscurl:jar
-          aws s3 cp awscurl/build/awscurl s3://djl-ai/publish/awscurl/
-          aws cloudfront create-invalidation --distribution-id E371VB8JQ6NRVY --paths "/awscurl/awscurl"
-      - name: Copy awscurl staging artifacts to S3
-        if: ${{ github.event.inputs.mode == 'staging' }}
-        run: |
-          ./gradlew :awscurl:jar -Pstaging
-          DJL_VERSION=$(awk -F '=' '/djl / {gsub(/ ?"/, "", $2); print $2}' gradle/libs.versions.toml)
-          aws s3 cp awscurl/build/awscurl s3://djl-ai/publish/awscurl/${DJL_VERSION}/
-          aws cloudfront create-invalidation --distribution-id E371VB8JQ6NRVY --paths "/awscurl/${DJL_VERSION}/*"
-      - name: Publish to snapshot repository
-        if: ${{ github.event.inputs.mode == '' || github.event.inputs.mode == 'snapshot' }}
-        run: ./gradlew publish -Psnapshot --refresh-dependencies
-        env:
-          ORG_GRADLE_PROJECT_signingKey: ${{ secrets.ORG_GRADLE_PROJECT_signingKey }}
-          ORG_GRADLE_PROJECT_signingPassword: ${{ secrets.ORG_GRADLE_PROJECT_signingPassword }}
-          ORG_GRADLE_PROJECT_ossrhUsername: ${{ secrets.ORG_GRADLE_PROJECT_ossrhUsername }}
-          ORG_GRADLE_PROJECT_ossrhPassword: ${{ secrets.ORG_GRADLE_PROJECT_ossrhPassword }}
-          DJL_STAGING: ${{ github.event.inputs.repo-id }}
-      - name: Publish to staging repository
-        if: ${{ github.event.inputs.mode == 'staging' }}
-        run: ./gradlew publish -Pstaging --refresh-dependencies
-        env:
-          ORG_GRADLE_PROJECT_signingKey: ${{ secrets.ORG_GRADLE_PROJECT_signingKey }}
-          ORG_GRADLE_PROJECT_signingPassword: ${{ secrets.ORG_GRADLE_PROJECT_signingPassword }}
-          ORG_GRADLE_PROJECT_ossrhUsername: ${{ secrets.ORG_GRADLE_PROJECT_ossrhUsername }}
-          ORG_GRADLE_PROJECT_ossrhPassword: ${{ secrets.ORG_GRADLE_PROJECT_ossrhPassword }}
-          DJL_STAGING: ${{ github.event.inputs.repo-id }}
diff --git a/api/src/main/java/ai/djl/modality/nlp/translator/QaServingTranslator.java b/api/src/main/java/ai/djl/modality/nlp/translator/QaServingTranslator.java
index 45e3f59af..7224a6d40 100644
--- a/api/src/main/java/ai/djl/modality/nlp/translator/QaServingTranslator.java
+++ b/api/src/main/java/ai/djl/modality/nlp/translator/QaServingTranslator.java
@@ -30,14 +30,17 @@ import com.google.gson.JsonParseException;
 import com.google.gson.reflect.TypeToken;
 
 import java.lang.reflect.Type;
+import java.util.ArrayList;
+import java.util.Collections;
 import java.util.List;
+import java.util.Map;
 
 /**
  * A {@link Translator} that can handle generic question answering {@link Input} and {@link Output}.
  */
 public class QaServingTranslator implements NoBatchifyTranslator<Input, Output> {
 
-    private static final Type LIST_TYPE = new TypeToken<List<String>>() {}.getType();
+    private static final Type LIST_TYPE = new TypeToken<List<QAInput>>() {}.getType();
 
     private Translator<QAInput, String> translator;
 
@@ -116,13 +119,20 @@ public class QaServingTranslator implements NoBatchifyTranslator<Input, Output>
         Output output = new Output();
         output.addProperty("Content-Type", "application/json");
         if (ctx.getAttachment("batch") != null) {
-            output.add(BytesSupplier.wrapAsJson(translator.batchProcessOutput(ctx, list)));
+            List<String> answers = translator.batchProcessOutput(ctx, list);
+            List<Map<String, String>> ret = new ArrayList<>();
+            for (String answer : answers) {
+                ret.add(Collections.singletonMap("answer", answer));
+            }
+            output.add(BytesSupplier.wrapAsJson(ret));
         } else {
             Batchifier batchifier = translator.getBatchifier();
             if (batchifier != null) {
                 list = batchifier.unbatchify(list)[0];
             }
-            output.add(translator.processOutput(ctx, list));
+            String answer = translator.processOutput(ctx, list);
+            Map<String, String> ret = Collections.singletonMap("answer", answer);
+            output.add(BytesSupplier.wrapAsJson(ret));
         }
         return output;
     }
diff --git a/api/src/main/java/ai/djl/modality/nlp/translator/TokenClassificationServingTranslator.java b/api/src/main/java/ai/djl/modality/nlp/translator/TokenClassificationServingTranslator.java
index a6ab6e8af..f6b1fd386 100644
--- a/api/src/main/java/ai/djl/modality/nlp/translator/TokenClassificationServingTranslator.java
+++ b/api/src/main/java/ai/djl/modality/nlp/translator/TokenClassificationServingTranslator.java
@@ -56,6 +56,7 @@ public class TokenClassificationServingTranslator implements NoBatchifyTranslato
         TextPrompt prompt = TextPrompt.parseInput(input);
         if (prompt.isBatch()) {
             ctx.setAttachment("batch", Boolean.TRUE);
+            return translator.batchProcessInput(ctx, prompt.getBatch());
         }
 
         NDList ret = translator.processInput(ctx, prompt.getText());
diff --git a/api/src/main/java/ai/djl/ndarray/BytesSupplierImpl.java b/api/src/main/java/ai/djl/ndarray/BytesSupplierImpl.java
index d3f7198c1..e6af90ebb 100644
--- a/api/src/main/java/ai/djl/ndarray/BytesSupplierImpl.java
+++ b/api/src/main/java/ai/djl/ndarray/BytesSupplierImpl.java
@@ -40,7 +40,7 @@ class BytesSupplierImpl implements BytesSupplier {
     public byte[] getAsBytes() {
         if (buf == null) {
             if (value == null) {
-                value = JsonUtils.toJson(obj) + '\n';
+                value = JsonUtils.toJson(obj);
             }
             buf = value.getBytes(StandardCharsets.UTF_8);
         }
@@ -52,7 +52,7 @@ class BytesSupplierImpl implements BytesSupplier {
     public String getAsString() {
         if (value == null) {
             if (obj != null) {
-                value = JsonUtils.toJson(obj) + '\n';
+                value = JsonUtils.toJson(obj);
             } else {
                 value = new String(buf, StandardCharsets.UTF_8);
             }
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
index c10e7c3e6..cd6eaf289 100644
--- a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
@@ -36,6 +36,7 @@ import java.util.Arrays;
 import java.util.List;
 import java.util.Locale;
 import java.util.Map;
+import java.util.Objects;
 import java.util.concurrent.ConcurrentHashMap;
 
 /**
@@ -232,6 +233,9 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
      * @return the {@code Encoding} of the input sentence
      */
     public Encoding encode(String text, boolean addSpecialTokens, boolean withOverflowingTokens) {
+        if (text == null) {
+            throw new NullPointerException("text cannot be null");
+        }
         if (doLowerCase != null) {
             text = text.toLowerCase(doLowerCase);
         }
@@ -261,6 +265,10 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
      */
     public Encoding encode(
             String text, String textPair, boolean addSpecialTokens, boolean withOverflowingTokens) {
+        if (text == null || textPair == null) {
+            throw new NullPointerException("text/text_pair cannot be null");
+        }
+
         if (doLowerCase != null) {
             text = text.toLowerCase(doLowerCase);
             textPair = textPair.toLowerCase(doLowerCase);
@@ -322,6 +330,8 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
             for (int i = 0; i < inputs.length; ++i) {
                 inputs[i] = inputs[i].toLowerCase(doLowerCase);
             }
+        } else if (Arrays.stream(inputs).anyMatch(Objects::isNull)) {
+            throw new NullPointerException("input text cannot be null");
         }
         long encoding = TokenizersLibrary.LIB.encodeList(getHandle(), inputs, addSpecialTokens);
         return toEncoding(encoding, withOverflowingTokens);
@@ -377,6 +387,8 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
             for (int i = 0; i < inputs.length; ++i) {
                 inputs[i] = inputs[i].toLowerCase(doLowerCase);
             }
+        } else if (Arrays.stream(inputs).anyMatch(Objects::isNull)) {
+            throw new NullPointerException("input text cannot be null");
         }
         long[] encodings = TokenizersLibrary.LIB.batchEncode(getHandle(), inputs, addSpecialTokens);
         Encoding[] ret = new Encoding[encodings.length];
@@ -418,6 +430,13 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
             for (int i = 0; i < textPair.length; ++i) {
                 textPair[i] = textPair[i].toLowerCase(doLowerCase);
             }
+        } else {
+            if (inputs.keys().stream().anyMatch(Objects::isNull)) {
+                throw new NullPointerException("text pair key cannot be null");
+            }
+            if (inputs.values().stream().anyMatch(Objects::isNull)) {
+                throw new NullPointerException("text pair value cannot be null");
+            }
         }
         long[] encodings =
                 TokenizersLibrary.LIB.batchEncodePair(


Test patch:
diff --git a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
index 5c1e63873..27fb1056b 100644
--- a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
+++ b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
@@ -27,6 +27,7 @@ import java.io.IOException;
 import java.nio.file.Path;
 import java.nio.file.Paths;
 import java.util.Arrays;
+import java.util.Collections;
 import java.util.List;
 import java.util.Locale;
 import java.util.Map;
@@ -102,6 +103,17 @@ public class HuggingFaceTokenizerTest {
                 Assert.assertEquals(charSpansExpected[i].getStart(), charSpansResult[i].getStart());
                 Assert.assertEquals(charSpansExpected[i].getEnd(), charSpansResult[i].getEnd());
             }
+
+            Assert.assertThrows(() -> tokenizer.encode((String) null));
+            Assert.assertThrows(() -> tokenizer.encode(new String[] {null}));
+            Assert.assertThrows(() -> tokenizer.encode(null, null));
+            Assert.assertThrows(() -> tokenizer.encode("null", null));
+            Assert.assertThrows(() -> tokenizer.batchEncode(new String[] {null}));
+            List<String> empty = Collections.singletonList(null);
+            List<String> some = Collections.singletonList("null");
+
+            Assert.assertThrows(() -> tokenizer.batchEncode(new PairList<>(empty, some)));
+            Assert.assertThrows(() -> tokenizer.batchEncode(new PairList<>(some, empty)));
         }
 
         Map<String, String> options = new ConcurrentHashMap<>();
diff --git a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/QuestionAnsweringTranslatorTest.java b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/QuestionAnsweringTranslatorTest.java
index acb283734..b67d2a7bc 100644
--- a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/QuestionAnsweringTranslatorTest.java
+++ b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/QuestionAnsweringTranslatorTest.java
@@ -102,7 +102,7 @@ public class QuestionAnsweringTranslatorTest {
             input.add("question", question);
             input.add("paragraph", paragraph);
             Output res = predictor.predict(input);
-            Assert.assertEquals(res.getAsString(0), "December 2004");
+            Assert.assertEquals(res.getAsString(0), "{\"answer\":\"December 2004\"}");
 
             Assert.assertThrows(
                     "Input data is empty.",