Repository: deepjavalibrary/djl
Pull request #1855 (2022-08-02 16:49:49)

Allow overriding special token flags in encode and decode methods
## Description ##

The tokenizer python and rust apis allow users to pass boolean flags to encode and decode methods to override special token behavior set during initialization. 

This change adds apis for the user to override the behavior in java similar to python and rust.

I also updated the naming of this flag for the decoding method to match the tokenizers naming.


codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1855?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
> Merging [#1855](https://codecov.io/gh/deepjavalibrary/djl/pull/1855?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (fa8dc58) into [master](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (bb5073f) will **decrease** coverage by `2.34%`.
> The diff coverage is `64.54%`.

> :exclamation: Current head fa8dc58 differs from pull request most recent head 28a33be. Consider uploading reports for the commit 28a33be to get more accurate results

```diff
@@             Coverage Diff              @@
##             master    #1855      +/-   ##
============================================
- Coverage     72.08%   69.74%   -2.35%     
- Complexity     5126     5542     +416     
============================================
  Files           473      527      +54     
  Lines         21970    24471    +2501     
  Branches       2351     2669     +318     
============================================
+ Hits          15838    17067    +1229     
- Misses         4925     6093    +1168     
- Partials       1207     1311     +104     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/1855?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1855/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1855/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L011bHRpQm94UHJpb3IuamF2YQ==) | `76.00% <ø> (ø)` | |
| [...rc/main/java/ai/djl/modality/cv/output/Joints.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1855/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9Kb2ludHMuamF2YQ==) | `71.42% <ø> (ø)` | |
| [.../main/java/ai/djl/modality/cv/output/Landmark.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1855/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9MYW5kbWFyay5qYXZh) | `100.00% <ø> (ø)` | |
| [...main/java/ai/djl/modality/cv/output/Rectangle.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1855/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9SZWN0YW5nbGUuamF2YQ==) | `72.41% <0.00%> (ø)` | |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1855/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <0.00%> (-5.24%)` | :arrow_down: |
| [...odality/cv/translator/BigGANTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1855/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `33.33% <0.00%> (+8.33%)` | :arrow_up: |
| [...nslator/InstanceSegmentationTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1855/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW5zdGFuY2VTZWdtZW50YXRpb25UcmFuc2xhdG9yRmFjdG9yeS5qYXZh) | `14.28% <0.00%> (-3.90%)` | :arrow_down: |
| [.../cv/translator/SemanticSegmentationTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1855/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU2VtYW50aWNTZWdtZW50YXRpb25UcmFuc2xhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../cv/translator/StyleTransferTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1855/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU3R5bGVUcmFuc2ZlclRyYW5zbGF0b3JGYWN0b3J5LmphdmE=) | `40.00% <ø> (ø)` | |
| ... and [424 more](https://codecov.io/gh/deepjavalibrary/djl/pull/1855/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1855?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1855?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Last update [95d9519...28a33be](https://codecov.io/gh/deepjavalibrary/djl/pull/1855?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/extensions/tokenizers/rust/src/lib.rs b/extensions/tokenizers/rust/src/lib.rs
index 1b46b92af..3b0d52a7e 100644
--- a/extensions/tokenizers/rust/src/lib.rs
+++ b/extensions/tokenizers/rust/src/lib.rs
@@ -383,7 +383,7 @@ pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_
     _: JObject,
     handle: jlong,
     ids: jlongArray,
-    add_special_tokens: jboolean,
+    skip_special_tokens: jboolean,
 ) -> jstring {
     let tokenizer = cast_handle::<Tokenizer>(handle);
     let long_ids = env.get_long_array_elements(ids, ReleaseMode::NoCopyBack).unwrap();
@@ -396,7 +396,7 @@ pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_
             decode_ids.push(*val as u32);
         }
     }
-    let decoding: String = tokenizer.decode(decode_ids, add_special_tokens == JNI_TRUE).unwrap();
+    let decoding: String = tokenizer.decode(decode_ids, skip_special_tokens == JNI_TRUE).unwrap();
     let ret = env
         .new_string(decoding)
         .expect("Couldn't create java string!")
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
index 74ffefa35..123ba7c84 100644
--- a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
@@ -140,35 +140,62 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
      * Returns the {@code Encoding} of the input sentence.
      *
      * @param text the input sentence
+     * @param addSpecialTokens whether to encode the sequence with special tokens relative to their
+     *     model
      * @return the {@code Encoding} of the input sentence
      */
-    public Encoding encode(String text) {
+    public Encoding encode(String text, boolean addSpecialTokens) {
         long encoding = TokenizersLibrary.LIB.encode(getHandle(), text, addSpecialTokens);
         return toEncoding(encoding);
     }
 
+    /**
+     * Returns the {@code Encoding} of the input sentence.
+     *
+     * @param text the input sentence
+     * @return the {@code Encoding} of the input sentence
+     */
+    public Encoding encode(String text) {
+        return encode(text, addSpecialTokens);
+    }
+
     /**
      * Returns the {@code Encoding} of the input sentence.
      *
      * @param text the input sentence
      * @param textPair the second input sentence
+     * @param addSpecialTokens whether to encode the sequence with special tokens relative to their
+     *     model
      * @return the {@code Encoding} of the input sentence
      */
-    public Encoding encode(String text, String textPair) {
+    public Encoding encode(String text, String textPair, boolean addSpecialTokens) {
         long encoding =
                 TokenizersLibrary.LIB.encodeDual(getHandle(), text, textPair, addSpecialTokens);
         return toEncoding(encoding);
     }
 
+    /**
+     * Returns the {@code Encoding} of the input sentence.
+     *
+     * @param text the input sentence
+     * @param textPair the second input sentence
+     * @return the {@code Encoding} of the input sentence
+     */
+    public Encoding encode(String text, String textPair) {
+        return encode(text, textPair, addSpecialTokens);
+    }
+
     /**
      * Returns the {@code Encoding} of the input sentences.
      *
      * @param inputs the input sentences
+     * @param addSpecialTokens whether to encode the sequence with special tokens relative to their
+     *     model
      * @return the {@code Encoding} of the input sentences
      */
-    public Encoding encode(List<String> inputs) {
+    public Encoding encode(List<String> inputs, boolean addSpecialTokens) {
         String[] array = inputs.toArray(new String[0]);
-        return encode(array);
+        return encode(array, addSpecialTokens);
     }
 
     /**
@@ -177,20 +204,44 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
      * @param inputs the input sentences
      * @return the {@code Encoding} of the input sentences
      */
-    public Encoding encode(String[] inputs) {
+    public Encoding encode(List<String> inputs) {
+        return encode(inputs, addSpecialTokens);
+    }
+
+    /**
+     * Returns the {@code Encoding} of the input sentences.
+     *
+     * @param inputs the input sentences
+     * @param addSpecialTokens whether to encode the sequence with special tokens relative to their
+     *     model
+     * @return the {@code Encoding} of the input sentences
+     */
+    public Encoding encode(String[] inputs, boolean addSpecialTokens) {
         long encoding = TokenizersLibrary.LIB.encodeList(getHandle(), inputs, addSpecialTokens);
         return toEncoding(encoding);
     }
 
+    /**
+     * Returns the {@code Encoding} of the input sentences.
+     *
+     * @param inputs the input sentences
+     * @return the {@code Encoding} of the input sentences
+     */
+    public Encoding encode(String[] inputs) {
+        return encode(inputs, addSpecialTokens);
+    }
+
     /**
      * Returns the {@code Encoding} of the input sentence in batch.
      *
      * @param inputs the batch of input sentence
+     * @param addSpecialTokens whether to encode the sequence with special tokens relative to their
+     *     model
      * @return the {@code Encoding} of the input sentence in batch
      */
-    public Encoding[] batchEncode(List<String> inputs) {
+    public Encoding[] batchEncode(List<String> inputs, boolean addSpecialTokens) {
         String[] array = inputs.toArray(new String[0]);
-        return batchEncode(array);
+        return batchEncode(array, addSpecialTokens);
     }
 
     /**
@@ -199,7 +250,19 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
      * @param inputs the batch of input sentence
      * @return the {@code Encoding} of the input sentence in batch
      */
-    public Encoding[] batchEncode(String[] inputs) {
+    public Encoding[] batchEncode(List<String> inputs) {
+        return batchEncode(inputs, addSpecialTokens);
+    }
+
+    /**
+     * Returns the {@code Encoding} of the input sentence in batch.
+     *
+     * @param inputs the batch of input sentence
+     * @param addSpecialTokens whether to encode the sequence with special tokens relative to their
+     *     model
+     * @return the {@code Encoding} of the input sentence in batch
+     */
+    public Encoding[] batchEncode(String[] inputs, boolean addSpecialTokens) {
         long[] encodings = TokenizersLibrary.LIB.batchEncode(getHandle(), inputs, addSpecialTokens);
         Encoding[] ret = new Encoding[encodings.length];
         for (int i = 0; i < encodings.length; ++i) {
@@ -208,6 +271,27 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
         return ret;
     }
 
+    /**
+     * Returns the {@code Encoding} of the input sentence in batch.
+     *
+     * @param inputs the batch of input sentence
+     * @return the {@code Encoding} of the input sentence in batch
+     */
+    public Encoding[] batchEncode(String[] inputs) {
+        return batchEncode(inputs, addSpecialTokens);
+    }
+
+    /**
+     * Returns the decoded String from the input ids.
+     *
+     * @param ids the input ids
+     * @param skipSpecialTokens whether to remove special tokens in the decoding
+     * @return the decoded String from the input ids
+     */
+    public String decode(long[] ids, boolean skipSpecialTokens) {
+        return TokenizersLibrary.LIB.decode(getHandle(), ids, skipSpecialTokens);
+    }
+
     /**
      * Returns the decoded String from the input ids.
      *
@@ -215,7 +299,7 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
      * @return the decoded String from the input ids
      */
     public String decode(long[] ids) {
-        return TokenizersLibrary.LIB.decode(getHandle(), ids, addSpecialTokens);
+        return decode(ids, !addSpecialTokens);
     }
 
     private Encoding toEncoding(long encoding) {


Test patch:
diff --git a/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java b/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java
index a07f59672..6287bd526 100644
--- a/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java
+++ b/engines/onnxruntime/onnxruntime-engine/src/test/java/ai/djl/onnxruntime/engine/OrtTest.java
@@ -24,7 +24,6 @@ import ai.djl.ndarray.types.Shape;
 import ai.djl.onnxruntime.zoo.tabular.softmax_regression.IrisFlower;
 import ai.djl.repository.zoo.Criteria;
 import ai.djl.repository.zoo.ZooModel;
-import ai.djl.testing.TestRequirements;
 import ai.djl.translate.TranslateException;
 import ai.onnxruntime.OrtException;
 
@@ -42,8 +41,6 @@ public class OrtTest {
 
     @Test
     public void testOrt() throws TranslateException, ModelException, IOException {
-        TestRequirements.notArm();
-
         try {
             Criteria<IrisFlower, Classifications> criteria =
                     Criteria.builder()
@@ -92,8 +89,6 @@ public class OrtTest {
 
     @Test
     public void testNDArray() throws OrtException {
-        TestRequirements.notArm();
-
         try (NDManager manager = OrtNDManager.getSystemManager().newSubManager()) {
             NDArray zeros = manager.zeros(new Shape(1, 2));
             float[] data = zeros.toFloatArray();
@@ -122,8 +117,6 @@ public class OrtTest {
 
     @Test
     public void testStringTensor() throws ModelException, IOException, TranslateException {
-        TestRequirements.notArm();
-
         setAlternativeEngineDisabled(true);
         Criteria<NDList, NDList> criteria =
                 Criteria.builder()
@@ -148,8 +141,6 @@ public class OrtTest {
 
     @Test
     public void testAlternativeArray() {
-        TestRequirements.notArm();
-
         try (NDManager manager = OrtNDManager.getSystemManager().newSubManager()) {
             NDArray array = manager.zeros(new Shape(1, 2));
             Assert.assertEquals(array.get(0).toFloatArray(), new float[] {0, 0});
diff --git a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
index 0e2f5f209..8999517eb 100644
--- a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
+++ b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
@@ -122,13 +122,35 @@ public class HuggingFaceTokenizerTest {
                             101, 3570, 1110, 170, 21162, 1285, 119, 2750, 4250, 146, 112, 173, 1474,
                             102
                         });
-        List<String> expectedDecodings =
+        List<String> expectedDecodingsNoSpecialTokens =
                 Arrays.asList(
                         "Hello, y ' all! How are you?",
                         "Today is a sunny day. Good weather I ' d say");
+        List<String> expectedDecodingsWithSpecialTokens =
+                Arrays.asList(
+                        "[CLS] Hello, y ' all! How are you [UNK]? [SEP]",
+                        "[CLS] Today is a sunny day. Good weather I ' d say [SEP]");
         try (HuggingFaceTokenizer tokenizer = HuggingFaceTokenizer.newInstance("bert-base-cased")) {
             for (int i = 0; i < testIds.size(); ++i) {
-                Assert.assertEquals(tokenizer.decode(testIds.get(i)), expectedDecodings.get(i));
+                Assert.assertEquals(
+                        tokenizer.decode(testIds.get(i)),
+                        expectedDecodingsWithSpecialTokens.get(i));
+                Assert.assertEquals(
+                        tokenizer.decode(testIds.get(i), true),
+                        expectedDecodingsNoSpecialTokens.get(i));
+            }
+        }
+
+        Map<String, String> options = new ConcurrentHashMap<>();
+        options.put("addSpecialTokens", "false");
+        try (HuggingFaceTokenizer tokenizer =
+                HuggingFaceTokenizer.newInstance("bert-base-cased", options)) {
+            for (int i = 0; i < testIds.size(); ++i) {
+                Assert.assertEquals(
+                        tokenizer.decode(testIds.get(i)), expectedDecodingsNoSpecialTokens.get(i));
+                Assert.assertEquals(
+                        tokenizer.decode(testIds.get(i), false),
+                        expectedDecodingsWithSpecialTokens.get(i));
             }
         }
     }