Repository: deepjavalibrary/djl
Pull request #1843 (2022-07-28 23:56:22)

Tokenizers decode
## Description ##
Expose tokenizers decode method https://github.com/deepjavalibrary/djl/issues/1829


codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1843?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
> Merging [#1843](https://codecov.io/gh/deepjavalibrary/djl/pull/1843?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (bec8cbb) into [master](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (bb5073f) will **decrease** coverage by `2.36%`.
> The diff coverage is `64.61%`.

```diff
@@             Coverage Diff              @@
##             master    #1843      +/-   ##
============================================
- Coverage     72.08%   69.72%   -2.37%     
- Complexity     5126     5525     +399     
============================================
  Files           473      526      +53     
  Lines         21970    24446    +2476     
  Branches       2351     2665     +314     
============================================
+ Hits          15838    17044    +1206     
- Misses         4925     6090    +1165     
- Partials       1207     1312     +105     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/1843?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1843/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1843/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L011bHRpQm94UHJpb3IuamF2YQ==) | `76.00% <ø> (ø)` | |
| [...rc/main/java/ai/djl/modality/cv/output/Joints.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1843/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9Kb2ludHMuamF2YQ==) | `71.42% <ø> (ø)` | |
| [.../main/java/ai/djl/modality/cv/output/Landmark.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1843/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9MYW5kbWFyay5qYXZh) | `100.00% <ø> (ø)` | |
| [...main/java/ai/djl/modality/cv/output/Rectangle.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1843/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9SZWN0YW5nbGUuamF2YQ==) | `72.41% <0.00%> (ø)` | |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1843/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <0.00%> (-5.24%)` | :arrow_down: |
| [...odality/cv/translator/BigGANTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1843/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `33.33% <0.00%> (+8.33%)` | :arrow_up: |
| [...nslator/InstanceSegmentationTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1843/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW5zdGFuY2VTZWdtZW50YXRpb25UcmFuc2xhdG9yRmFjdG9yeS5qYXZh) | `14.28% <0.00%> (-3.90%)` | :arrow_down: |
| [.../cv/translator/SemanticSegmentationTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1843/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU2VtYW50aWNTZWdtZW50YXRpb25UcmFuc2xhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../cv/translator/StyleTransferTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1843/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU3R5bGVUcmFuc2ZlclRyYW5zbGF0b3JGYWN0b3J5LmphdmE=) | `40.00% <ø> (ø)` | |
| ... and [420 more](https://codecov.io/gh/deepjavalibrary/djl/pull/1843/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1843?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1843?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Last update [2f6e078...bec8cbb](https://codecov.io/gh/deepjavalibrary/djl/pull/1843?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/extensions/tokenizers/rust/src/lib.rs b/extensions/tokenizers/rust/src/lib.rs
index a02c906de..1b46b92af 100644
--- a/extensions/tokenizers/rust/src/lib.rs
+++ b/extensions/tokenizers/rust/src/lib.rs
@@ -19,9 +19,9 @@ use tk::tokenizer::{EncodeInput, Encoding};
 use tk::Tokenizer;
 use tk::{FromPretrainedParameters, Offsets};
 
-use jni::objects::{JClass, JMethodID, JObject, JString, JValue};
+use jni::objects::{JClass, JMethodID, JObject, JString, JValue, ReleaseMode};
 use jni::sys::{
-    jboolean, jint, jlong, jlongArray, jobjectArray, jsize, JNI_TRUE,
+    jboolean, jint, jlong, jlongArray, jobjectArray, jsize, jstring, JNI_TRUE,
 };
 use jni::JNIEnv;
 
@@ -377,6 +377,34 @@ pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_
     array
 }
 
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_decode(
+    env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+    ids: jlongArray,
+    add_special_tokens: jboolean,
+) -> jstring {
+    let tokenizer = cast_handle::<Tokenizer>(handle);
+    let long_ids = env.get_long_array_elements(ids, ReleaseMode::NoCopyBack).unwrap();
+    let long_ids_ptr = long_ids.as_ptr();
+    let len = long_ids.size().unwrap() as usize;
+    let mut decode_ids: Vec<u32> = Vec::new();
+    for i in 0..len {
+        unsafe {
+            let val = long_ids_ptr.add(i);
+            decode_ids.push(*val as u32);
+        }
+    }
+    let decoding: String = tokenizer.decode(decode_ids, add_special_tokens == JNI_TRUE).unwrap();
+    let ret = env
+        .new_string(decoding)
+        .expect("Couldn't create java string!")
+        .into_inner();
+
+    ret
+}
+
 fn to_handle<T: 'static>(val: T) -> jlong {
     let handle = Box::into_raw(Box::new(val)) as jlong;
     handle
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
index fac3c1f39..74ffefa35 100644
--- a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
@@ -208,6 +208,16 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
         return ret;
     }
 
+    /**
+     * Returns the decoded String from the input ids.
+     *
+     * @param ids the input ids
+     * @return the decoded String from the input ids
+     */
+    public String decode(long[] ids) {
+        return TokenizersLibrary.LIB.decode(getHandle(), ids, addSpecialTokens);
+    }
+
     private Encoding toEncoding(long encoding) {
         long[] ids = TokenizersLibrary.LIB.getTokenIds(encoding);
         long[] typeIds = TokenizersLibrary.LIB.getTypeIds(encoding);
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java
index b0471385e..b532886bf 100644
--- a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java
@@ -50,4 +50,6 @@ public final class TokenizersLibrary {
     public native long[] getSpecialTokenMask(long encoding);
 
     public native CharSpan[] getTokenCharSpans(long encoding);
+
+    public native String decode(long tokenizer, long[] ids, boolean addSpecialTokens);
 }
diff --git a/extensions/tokenizers/src/main/python/.gitignore b/extensions/tokenizers/src/main/python/.gitignore
new file mode 100644
index 000000000..bee8a64b7
--- /dev/null
+++ b/extensions/tokenizers/src/main/python/.gitignore
@@ -0,0 +1,1 @@
+__pycache__
diff --git a/extensions/tokenizers/src/main/python/arg_parser.py b/extensions/tokenizers/src/main/python/arg_parser.py
index c31348401..2d64e787f 100644
--- a/extensions/tokenizers/src/main/python/arg_parser.py
+++ b/extensions/tokenizers/src/main/python/arg_parser.py
@@ -23,7 +23,7 @@ def converter_args():
                         default=1,
                         help="Max amount of models to convert")
     parser.add_argument("-o", "--output-dir", help="Model output directory")
-    parser.add_argument("-t",
+    parser.add_argument("-c",
                         "--categories",
                         nargs="*",
                         help="Model categories to convert")
diff --git a/extensions/tokenizers/src/main/python/huggingface_converter.py b/extensions/tokenizers/src/main/python/huggingface_converter.py
index b822a4d8a..9b130c025 100644
--- a/extensions/tokenizers/src/main/python/huggingface_converter.py
+++ b/extensions/tokenizers/src/main/python/huggingface_converter.py
@@ -21,7 +21,7 @@ import shutil
 from shasum import sha1_sum
 
 import json
-from huggingface_hub import HfApi
+from huggingface_hub import HfApi, ModelSearchArguments
 from huggingface_hub import hf_hub_download
 from transformers import pipeline
 import torch
@@ -84,6 +84,7 @@ class HuggingfaceConverter:
                 self.processed_models = json.load(f)
 
     def convert_huggingface_models(self, category: str):
+        languages = ModelSearchArguments().language
         api = HfApi()
         models = api.list_models(filter=f"{category},pytorch",
                                  sort="downloads",
@@ -93,6 +94,15 @@ class HuggingfaceConverter:
         temp_dir = f"{self.output_dir}/tmp"
 
         for model_info in models:
+            is_english = True
+            for tag in model_info.tags:
+                if tag in languages and tag != 'en':
+                    is_english = False
+                    break
+
+            if not is_english:
+                continue
+
             if self.processed_models.get(model_info.modelId):
                 logging.info(f"Skip converted mode: {model_info.modelId}.")
                 continue


Test patch:
diff --git a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
index 566bf6186..0e2f5f209 100644
--- a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
+++ b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
@@ -110,4 +110,26 @@ public class HuggingFaceTokenizerTest {
             Assert.assertEquals(encoding.getIds(), ids);
         }
     }
+
+    @Test
+    public void testTokenizerDecoding() {
+        List<long[]> testIds =
+                Arrays.asList(
+                        new long[] {
+                            101, 8667, 117, 194, 112, 1155, 106, 1731, 1132, 1128, 100, 136, 102
+                        },
+                        new long[] {
+                            101, 3570, 1110, 170, 21162, 1285, 119, 2750, 4250, 146, 112, 173, 1474,
+                            102
+                        });
+        List<String> expectedDecodings =
+                Arrays.asList(
+                        "Hello, y ' all! How are you?",
+                        "Today is a sunny day. Good weather I ' d say");
+        try (HuggingFaceTokenizer tokenizer = HuggingFaceTokenizer.newInstance("bert-base-cased")) {
+            for (int i = 0; i < testIds.size(); ++i) {
+                Assert.assertEquals(tokenizer.decode(testIds.get(i)), expectedDecodings.get(i));
+            }
+        }
+    }
 }