Repository: deepjavalibrary/djl
Pull request #1926 (2022-08-18 17:25:11)

[tokenizers] load truncation and padding settings from options, updat…
[tokenizers] load truncation and padding settings from options, update apis

## Description ##
- loading truncation/padding settings from options when creating new tokenizer instance
- Updated APIs for setting truncation/padding post tokenizer creation. Of note:
  - maxLength parameter is shared by both truncation and padding, and settable by its own method
  - No support for stride in this implementation
 - Updated unit test coverage - used the supported combinations from https://huggingface.co/docs/transformers/v4.20.1/en/pad_truncation to guide test cases



codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1926?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
> Merging [#1926](https://codecov.io/gh/deepjavalibrary/djl/pull/1926?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (4e755dd) into [master](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (bb5073f) will **decrease** coverage by `2.32%`.
> The diff coverage is `67.01%`.

```diff
@@             Coverage Diff              @@
##             master    #1926      +/-   ##
============================================
- Coverage     72.08%   69.75%   -2.33%     
- Complexity     5126     5687     +561     
============================================
  Files           473      543      +70     
  Lines         21970    25119    +3149     
  Branches       2351     2715     +364     
============================================
+ Hits          15838    17523    +1685     
- Misses         4925     6266    +1341     
- Partials       1207     1330     +123     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/1926?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1926/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1926/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L011bHRpQm94UHJpb3IuamF2YQ==) | `76.00% <ø> (ø)` | |
| [...rc/main/java/ai/djl/modality/cv/output/Joints.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1926/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9Kb2ludHMuamF2YQ==) | `71.42% <ø> (ø)` | |
| [.../main/java/ai/djl/modality/cv/output/Landmark.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1926/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9MYW5kbWFyay5qYXZh) | `100.00% <ø> (ø)` | |
| [...main/java/ai/djl/modality/cv/output/Rectangle.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1926/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9SZWN0YW5nbGUuamF2YQ==) | `72.41% <0.00%> (ø)` | |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1926/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <0.00%> (-5.24%)` | :arrow_down: |
| [...odality/cv/translator/BigGANTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1926/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `33.33% <0.00%> (+8.33%)` | :arrow_up: |
| [...nslator/InstanceSegmentationTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1926/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW5zdGFuY2VTZWdtZW50YXRpb25UcmFuc2xhdG9yRmFjdG9yeS5qYXZh) | `14.28% <0.00%> (-3.90%)` | :arrow_down: |
| [.../cv/translator/SemanticSegmentationTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1926/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU2VtYW50aWNTZWdtZW50YXRpb25UcmFuc2xhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../cv/translator/StyleTransferTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1926/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU3R5bGVUcmFuc2ZlclRyYW5zbGF0b3JGYWN0b3J5LmphdmE=) | `40.00% <ø> (ø)` | |
| ... and [443 more](https://codecov.io/gh/deepjavalibrary/djl/pull/1926/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)


Patch:
diff --git a/.github/workflows/native_jni_s3_pytorch.yml b/.github/workflows/native_jni_s3_pytorch.yml
index 291e5688b..fea755e68 100644
--- a/.github/workflows/native_jni_s3_pytorch.yml
+++ b/.github/workflows/native_jni_s3_pytorch.yml
@@ -226,11 +226,9 @@ jobs:
           yum -y install patch git cmake3 python3-devel java-11-amazon-corretto
           ln -sf /usr/bin/cmake3 /usr/bin/cmake
           pip3 install awscli --upgrade
+      - uses: actions/checkout@v2
       - name: Release JNI prep
         run: |
-          BRANCH=$(sed -e s:refs/heads/::g -e s:/:-:g <<< ${{ github.event.ref }})
-          echo $BRANCH
-          rm -rf djl && git clone https://github.com/deepjavalibrary/djl -b $BRANCH && cd djl
           export JAVA_HOME=/usr/lib/jvm/java-11-amazon-corretto.aarch64
           export PATH=$PATH:$JAVA_HOME
           PYTORCH_VERSION=${{ github.event.inputs.pt_version }}
@@ -246,7 +244,6 @@ jobs:
           aws-region: us-east-2
       - name: Copy files to S3 with the AWS CLI
         run: |
-          cd djl
           PYTORCH_VERSION=${{ github.event.inputs.pt_version }}
           PYTORCH_VERSION=${PYTORCH_VERSION:-$(cat gradle.properties | awk -F '=' '/pytorch_version/ {print $2}')}
           aws s3 sync engines/pytorch/pytorch-native/jnilib s3://djl-ai/publish/pytorch/${PYTORCH_VERSION}/jnilib
diff --git a/api/src/main/java/ai/djl/translate/NoopServingTranslatorFactory.java b/api/src/main/java/ai/djl/translate/NoopServingTranslatorFactory.java
new file mode 100644
index 000000000..c23f0efea
--- /dev/null
+++ b/api/src/main/java/ai/djl/translate/NoopServingTranslatorFactory.java
@@ -0,0 +1,92 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.translate;
+
+import ai.djl.Model;
+import ai.djl.modality.Input;
+import ai.djl.modality.Output;
+import ai.djl.ndarray.NDList;
+import ai.djl.ndarray.NDManager;
+import ai.djl.util.Pair;
+
+import java.lang.reflect.Type;
+import java.util.Collections;
+import java.util.Map;
+import java.util.Set;
+
+/** A {@link TranslatorFactory} that creates a {@code RawTranslator} instance. */
+public class NoopServingTranslatorFactory implements TranslatorFactory {
+
+    /** {@inheritDoc} */
+    @Override
+    public Set<Pair<Type, Type>> getSupportedTypes() {
+        return Collections.singleton(new Pair<>(Input.class, Output.class));
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public Translator<?, ?> newInstance(
+            Class<?> input, Class<?> output, Model model, Map<String, ?> arguments)
+            throws TranslateException {
+        if (!isSupported(input, output)) {
+            throw new IllegalArgumentException("Unsupported input/output types.");
+        }
+        String batchifier = ArgumentsUtil.stringValue(arguments, "batchifier", "none");
+        return new NoopServingTranslator(Batchifier.fromString(batchifier));
+    }
+
+    static final class NoopServingTranslator implements Translator<Input, Output> {
+
+        private Batchifier batchifier;
+
+        NoopServingTranslator(Batchifier batchifier) {
+            this.batchifier = batchifier;
+        }
+
+        /** {@inheritDoc} */
+        @Override
+        public Batchifier getBatchifier() {
+            return batchifier;
+        }
+
+        /** {@inheritDoc} */
+        @Override
+        public NDList processInput(TranslatorContext ctx, Input input) throws TranslateException {
+            NDManager manager = ctx.getNDManager();
+            try {
+                ctx.setAttachment("properties", input.getProperties());
+                return input.getDataAsNDList(manager);
+            } catch (IllegalArgumentException e) {
+                throw new TranslateException("Input is not a NDList data type", e);
+            }
+        }
+
+        /** {@inheritDoc} */
+        @Override
+        @SuppressWarnings("unchecked")
+        public Output processOutput(TranslatorContext ctx, NDList list) {
+            Map<String, String> prop = (Map<String, String>) ctx.getAttachment("properties");
+            String contentType = prop.get("Content-Type");
+
+            Output output = new Output();
+            if ("tensor/npz".equalsIgnoreCase(contentType)) {
+                output.add(list.encode(true));
+                output.addProperty("Content-Type", "tensor/npz");
+            } else {
+                output.add(list.encode(false));
+                output.addProperty("Content-Type", "tensor/ndlist");
+            }
+            return output;
+        }
+    }
+}
diff --git a/engines/pytorch/pytorch-engine/src/main/java/ai/djl/pytorch/jni/LibUtils.java b/engines/pytorch/pytorch-engine/src/main/java/ai/djl/pytorch/jni/LibUtils.java
index addea42aa..dd983fa18 100644
--- a/engines/pytorch/pytorch-engine/src/main/java/ai/djl/pytorch/jni/LibUtils.java
+++ b/engines/pytorch/pytorch-engine/src/main/java/ai/djl/pytorch/jni/LibUtils.java
@@ -507,11 +507,11 @@ public final class LibUtils {
             this.apiVersion = platform.getApiVersion();
             this.classifier = platform.getClassifier();
             version = Utils.getEnvOrSystemProperty("PYTORCH_VERSION");
-            if (version == null) {
+            if (version == null || version.isEmpty()) {
                 version = platform.getVersion();
             }
             flavor = Utils.getEnvOrSystemProperty("PYTORCH_FLAVOR");
-            if (flavor == null) {
+            if (flavor == null || flavor.isEmpty()) {
                 if (CudaUtils.getGpuCount() > 0) {
                     flavor = "cu" + CudaUtils.getCudaVersionString() + "-precxx11";
                 } else {
diff --git a/extensions/tokenizers/rust/src/lib.rs b/extensions/tokenizers/rust/src/lib.rs
index 17bfa79b8..e2a62f95a 100644
--- a/extensions/tokenizers/rust/src/lib.rs
+++ b/extensions/tokenizers/rust/src/lib.rs
@@ -16,15 +16,13 @@ extern crate tokenizers as tk;
 
 use std::str::FromStr;
 use tk::tokenizer::{EncodeInput, Encoding};
+use tk::utils::padding::{PaddingParams, PaddingStrategy};
+use tk::utils::truncation::{TruncationParams, TruncationStrategy};
 use tk::Tokenizer;
 use tk::{FromPretrainedParameters, Offsets};
-use tk::utils::truncation::{TruncationParams, TruncationStrategy};
-use tk::utils::padding::{PaddingParams, PaddingStrategy};
 
 use jni::objects::{JClass, JMethodID, JObject, JString, JValue, ReleaseMode};
-use jni::sys::{
-    jboolean, jint, jlong, jlongArray, jobjectArray, jsize, jstring, JNI_TRUE,
-};
+use jni::sys::{jboolean, jint, jlong, jlongArray, jobjectArray, jsize, jstring, JNI_TRUE};
 use jni::JNIEnv;
 
 #[no_mangle]
@@ -388,7 +386,9 @@ pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_
     skip_special_tokens: jboolean,
 ) -> jstring {
     let tokenizer = cast_handle::<Tokenizer>(handle);
-    let long_ids = env.get_long_array_elements(ids, ReleaseMode::NoCopyBack).unwrap();
+    let long_ids = env
+        .get_long_array_elements(ids, ReleaseMode::NoCopyBack)
+        .unwrap();
     let long_ids_ptr = long_ids.as_ptr();
     let len = long_ids.size().unwrap() as usize;
     let mut decode_ids: Vec<u32> = Vec::new();
@@ -398,7 +398,9 @@ pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_
             decode_ids.push(*val as u32);
         }
     }
-    let decoding: String = tokenizer.decode(decode_ids, skip_special_tokens == JNI_TRUE).unwrap();
+    let decoding: String = tokenizer
+        .decode(decode_ids, skip_special_tokens == JNI_TRUE)
+        .unwrap();
     let ret = env
         .new_string(decoding)
         .expect("Couldn't create java string!")
@@ -407,14 +409,114 @@ pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_
     ret
 }
 
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_getTruncationStrategy(
+    env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+) -> jstring {
+    let tokenizer = cast_handle::<Tokenizer>(handle);
+    let truncation = tokenizer.get_truncation();
+    let strategy = match truncation {
+        Some(val) => val.strategy.as_ref(),
+        None => "DO_NOT_TRUNCATE",
+    };
+
+    let ret = env
+        .new_string(strategy.to_string())
+        .expect("Couldn't create java string!")
+        .into_inner();
+
+    ret
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_getPaddingStrategy(
+    env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+) -> jstring {
+    let tokenizer = cast_handle::<Tokenizer>(handle);
+    let padding = tokenizer.get_padding();
+    let strategy = match padding {
+        Some(val) => match val.strategy {
+            PaddingStrategy::BatchLongest => "LONGEST",
+            _ => "MAX_LENGTH",
+        },
+        None => "DO_NOT_PAD",
+    };
+
+    let ret = env
+        .new_string(strategy)
+        .expect("Couldn't create java string!")
+        .into_inner();
+
+    ret
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_getMaxLength(
+    _env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+) -> jint {
+    let tokenizer = cast_handle::<Tokenizer>(handle);
+    let truncation = tokenizer.get_truncation();
+    let mut max_length = match truncation {
+        Some(val) => val.max_length as jint,
+        None => -1,
+    };
+    if max_length == -1 {
+        let padding = tokenizer.get_padding();
+        max_length = match padding {
+            Some(param) => match param.strategy {
+                PaddingStrategy::Fixed(i) => i as jint,
+                _ => -1,
+            },
+            _ => -1,
+        };
+    }
+    max_length
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_getStride(
+    _env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+) -> jint {
+    let tokenizer = cast_handle::<Tokenizer>(handle);
+    let truncation = tokenizer.get_truncation();
+    let ret = match truncation {
+        Some(val) => val.stride,
+        None => 0,
+    };
+    ret as jint
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_getPadToMultipleOf(
+    _env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+) -> jint {
+    let tokenizer = cast_handle::<Tokenizer>(handle);
+    let padding = tokenizer.get_padding();
+    let ret = match padding {
+        Some(val) => val.pad_to_multiple_of.unwrap_or(0),
+        None => 0,
+    };
+    ret as jint
+}
+
 #[no_mangle]
 pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_setPadding(
     env: JNIEnv,
     _: JObject,
     handle: jlong,
-    max_length: jlong,
+    max_length: jint,
     padding_strategy: JString,
-    pad_to_multiple_of: jlong,
+    pad_to_multiple_of: jint,
 ) {
     let strategy: String = env
         .get_string(padding_strategy)
@@ -422,9 +524,9 @@ pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_
         .into();
     let len = max_length as usize;
     let res_strategy = match strategy.as_ref() {
-        "batch_longest" => Ok(PaddingStrategy::BatchLongest),
-        "fixed" => Ok(PaddingStrategy::Fixed(len)),
-        _ => Err("strategy must be one of [batch_longest, fixed]"),
+        "LONGEST" => Ok(PaddingStrategy::BatchLongest),
+        "MAX_LENGTH" => Ok(PaddingStrategy::Fixed(len)),
+        _ => Err("strategy must be one of [longest, max_length]"),
     };
 
     let mut params = PaddingParams::default();
@@ -449,19 +551,19 @@ pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_
     env: JNIEnv,
     _: JObject,
     handle: jlong,
-    truncation_max_length: jlong,
+    truncation_max_length: jint,
     truncation_strategy: JString,
-    truncation_stride: jlong,
+    truncation_stride: jint,
 ) {
     let strategy: String = env
         .get_string(truncation_strategy)
         .expect("Couldn't get java string!")
         .into();
     let res_strategy = match strategy.as_ref() {
-       "longest_first" => Ok(TruncationStrategy::LongestFirst),
-       "only_first" => Ok(TruncationStrategy::OnlyFirst),
-       "only_second" => Ok(TruncationStrategy::OnlySecond),
-       _ => Err("strategy must be one of [longest_first, only_first, only_second]"),
+        "LONGEST_FIRST" => Ok(TruncationStrategy::LongestFirst),
+        "ONLY_FIRST" => Ok(TruncationStrategy::OnlyFirst),
+        "ONLY_SECOND" => Ok(TruncationStrategy::OnlySecond),
+        _ => Err("strategy must be one of [longest_first, only_first, only_second]"),
     };
     let mut params = TruncationParams::default();
     params.max_length = truncation_max_length as usize;
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
index f80779960..8ff7edd2b 100644
--- a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
@@ -16,9 +16,13 @@ import ai.djl.huggingface.tokenizers.jni.CharSpan;
 import ai.djl.huggingface.tokenizers.jni.LibUtils;
 import ai.djl.huggingface.tokenizers.jni.TokenizersLibrary;
 import ai.djl.modality.nlp.preprocess.Tokenizer;
+import ai.djl.translate.ArgumentsUtil;
 import ai.djl.util.NativeResource;
 import ai.djl.util.Utils;
 
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
 import java.io.IOException;
 import java.io.InputStream;
 import java.nio.file.Files;
@@ -33,18 +37,49 @@ import java.util.Map;
  */
 public final class HuggingFaceTokenizer extends NativeResource<Long> implements Tokenizer {
 
+    private static final Logger logger = LoggerFactory.getLogger(HuggingFaceTokenizer.class);
+
     private boolean addSpecialTokens;
+    private TruncationStrategy truncation;
+    private PaddingStrategy padding;
+    private int maxLength;
+    private int stride;
+    private int padToMultipleOf;
+    private int modelMaxLength;
 
     private HuggingFaceTokenizer(long handle, Map<String, String> options) {
         super(handle);
-        this.addSpecialTokens =
-                options == null
-                        || !options.containsKey("addSpecialTokens")
-                        || Boolean.parseBoolean(options.get("addSpecialTokens"));
+        String val = TokenizersLibrary.LIB.getTruncationStrategy(handle);
+        truncation = TruncationStrategy.fromValue(val);
+        val = TokenizersLibrary.LIB.getPaddingStrategy(handle);
+        padding = PaddingStrategy.fromValue(val);
+        maxLength = TokenizersLibrary.LIB.getMaxLength(handle);
+        stride = TokenizersLibrary.LIB.getStride(handle);
+        padToMultipleOf = TokenizersLibrary.LIB.getPadToMultipleOf(handle);
+
+        if (options != null) {
+            val = options.getOrDefault("addSpecialTokens", "true");
+            addSpecialTokens = Boolean.parseBoolean(val);
+            modelMaxLength = ArgumentsUtil.intValue(options, "modelMaxLength", 512);
+            if (options.containsKey("truncation")) {
+                truncation = TruncationStrategy.fromValue(options.get("truncation"));
+            }
+            if (options.containsKey("padding")) {
+                padding = PaddingStrategy.fromValue(options.get("padding"));
+            }
+            maxLength = ArgumentsUtil.intValue(options, "maxLength", maxLength);
+            stride = ArgumentsUtil.intValue(options, "stride", stride);
+            padToMultipleOf = ArgumentsUtil.intValue(options, "padToMultipleOf", padToMultipleOf);
+        } else {
+            addSpecialTokens = true;
+            modelMaxLength = 512;
+        }
+
+        updateTruncationAndPadding();
     }
 
     /**
-     * Create a pre-trained {@code HuggingFaceTokenizer} instance from hugginface hub.
+     * Creates a pre-trained {@code HuggingFaceTokenizer} instance from huggingface hub.
      *
      * @param name the name of the huggingface tokenizer
      * @return a {@code HuggingFaceTokenizer} instance
@@ -54,7 +89,7 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
     }
 
     /**
-     * Create a pre-trained {@code HuggingFaceTokenizer} instance from hugginface hub.
+     * Create a pre-trained {@code HuggingFaceTokenizer} instance from huggingface hub.
      *
      * @param identifier the identifier of the huggingface tokenizer
      * @param options tokenizer options
@@ -303,86 +338,130 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
     }
 
     /**
-     * Sets the truncation and padding behavior for the tokenizer.
+     * Enables or Disables default truncation behavior for the tokenizer.
      *
-     * @param truncationStrategy the {@code TruncationStrategy} to use
-     * @param paddingStrategy the {@code PaddingStrategy} to use
-     * @param maxLength the maximum length to pad/truncate sequences to
+     * @param enabled whether to enable default truncation behavior
      */
-    public void setTruncationAndPadding(
-            TruncationStrategy truncationStrategy,
-            PaddingStrategy paddingStrategy,
-            long maxLength) {
-        setTruncationAndPadding(truncationStrategy, paddingStrategy, maxLength, 0, 0);
+    public void setTruncation(boolean enabled) {
+        TruncationStrategy strategy =
+                enabled ? TruncationStrategy.LONGEST_FIRST : TruncationStrategy.DO_NOT_TRUNCATE;
+        if (strategy != truncation) {
+            truncation = strategy;
+            updateTruncationAndPadding();
+        }
+    }
+
+    /** Enables truncation to only truncate the first item. */
+    public void setTruncateFirstOnly() {
+        if (truncation != TruncationStrategy.ONLY_FIRST) {
+            truncation = TruncationStrategy.ONLY_FIRST;
+            updateTruncationAndPadding();
+        }
+    }
+
+    /** Enables truncation to only truncate the second item. */
+    public void setTruncateSecondOnly() {
+        if (truncation != TruncationStrategy.ONLY_SECOND) {
+            truncation = TruncationStrategy.ONLY_SECOND;
+            updateTruncationAndPadding();
+        }
     }
 
     /**
-     * Sets the truncation and padding behavior for the tokenizer.
+     * Enables or Disables default padding behavior for the tokenizer.
      *
-     * @param truncationStrategy the {@code TruncationStrategy} to use
-     * @param paddingStrategy the {@code PaddingStrategy} to use
-     * @param maxLength the maximum length to pad/truncate sequences to
-     * @param stride value to use when handling overflow
+     * @param enabled whether to enable default padding behavior
      */
-    public void setTruncationAndPadding(
-            TruncationStrategy truncationStrategy,
-            PaddingStrategy paddingStrategy,
-            long maxLength,
-            long stride) {
-        setTruncationAndPadding(truncationStrategy, paddingStrategy, maxLength, stride, 0);
+    public void setPadding(boolean enabled) {
+        PaddingStrategy p = enabled ? PaddingStrategy.LONGEST : PaddingStrategy.DO_NOT_PAD;
+        if (padding != p) {
+            padding = p;
+            updateTruncationAndPadding();
+        }
     }
 
     /**
-     * Sets the truncation and padding behavior for the tokenizer.
-     *
-     * @param truncationStrategy the {@code TruncationStrategy} to use
-     * @param paddingStrategy the {@code PaddingStrategy} to use
-     * @param maxLength the maximum length to pad/truncate sequences to
-     * @param stride value to use when handling overflow
-     * @param padToMultipleOf pad sequence length to multiple of value
+     * Enables padding to pad sequences to previously specified maxLength, or modelMaxLength if not
+     * specified.
      */
-    public void setTruncationAndPadding(
-            TruncationStrategy truncationStrategy,
-            PaddingStrategy paddingStrategy,
-            long maxLength,
-            long stride,
-            long padToMultipleOf) {
-        setTruncation(truncationStrategy, maxLength, stride);
-        setPadding(paddingStrategy, maxLength, padToMultipleOf);
+    public void setPaddingMaxLength() {
+        if (padding != PaddingStrategy.MAX_LENGTH) {
+            padding = PaddingStrategy.MAX_LENGTH;
+            updateTruncationAndPadding();
+        }
     }
 
     /**
-     * Sets the truncation behavior for the tokenizer.
+     * Sets maxLength for padding and truncation.
      *
-     * @param truncationStrategy the {@code TruncationStrategy} to use
-     * @param maxLength the maximum length to truncate sequences to
-     * @param stride value to use when handling overflow
+     * @param maxLength the length to truncate and/or pad sequences to
      */
-    public void setTruncation(TruncationStrategy truncationStrategy, long maxLength, long stride) {
-        if (truncationStrategy == TruncationStrategy.DO_NOT_TRUNCATE) {
-            TokenizersLibrary.LIB.disableTruncation(getHandle());
-        } else {
-            TokenizersLibrary.LIB.setTruncation(
-                    getHandle(), maxLength, truncationStrategy.toString().toLowerCase(), stride);
+    public void setMaxLength(int maxLength) {
+        if (this.maxLength != maxLength) {
+            this.maxLength = maxLength;
+            updateTruncationAndPadding();
         }
     }
 
     /**
-     * Sets the padding behavior for the tokenizer.
+     * Sets padToMultipleOf for padding.
      *
-     * @param paddingStrategy the {@code PaddingStrategy} to use
-     * @param maxLength the maximum length to pad sequences to
-     * @param padToMultipleOf pad sequence length to multiple of value
+     * @param padToMultipleOf the multiple of sequences should be padded to
      */
-    public void setPadding(PaddingStrategy paddingStrategy, long maxLength, long padToMultipleOf) {
-        if (paddingStrategy == PaddingStrategy.DO_NOT_PAD) {
+    public void setPadToMultipleOf(int padToMultipleOf) {
+        if (this.padToMultipleOf != padToMultipleOf) {
+            this.padToMultipleOf = padToMultipleOf;
+            updateTruncationAndPadding();
+        }
+    }
+
+    /*
+     * See: https://huggingface.co/docs/transformers/pad_truncation
+     */
+    private void updateTruncationAndPadding() {
+        boolean isTruncate = truncation != TruncationStrategy.DO_NOT_TRUNCATE;
+        if (padding == PaddingStrategy.MAX_LENGTH || isTruncate) {
+            if (maxLength == -1) {
+                logger.warn(
+                        "maxLength is not explicitly specified, use modelMaxLength: "
+                                + modelMaxLength);
+                maxLength = modelMaxLength;
+            } else if (maxLength > modelMaxLength) {
+                logger.warn(
+                        "maxLength is greater then modelMaxLength, change to: " + modelMaxLength);
+                maxLength = modelMaxLength;
+            }
+
+            if (padding == PaddingStrategy.MAX_LENGTH && isTruncate && padToMultipleOf != 0) {
+                int remainder = maxLength % padToMultipleOf;
+                if (remainder != 0) {
+                    int newMaxLength = maxLength + padToMultipleOf - maxLength % padToMultipleOf;
+                    if (newMaxLength > modelMaxLength) {
+                        newMaxLength -= padToMultipleOf;
+                    }
+                    logger.warn(
+                            "maxLength ("
+                                    + maxLength
+                                    + ") is not a multiple of padToMultipleOf ("
+                                    + padToMultipleOf
+                                    + "), change to: "
+                                    + newMaxLength);
+                    maxLength = newMaxLength;
+                }
+            }
+        }
+
+        if (isTruncate) {
+            TokenizersLibrary.LIB.setTruncation(getHandle(), maxLength, truncation.name(), stride);
+        } else {
+            TokenizersLibrary.LIB.disableTruncation(getHandle());
+        }
+
+        if (padding == PaddingStrategy.DO_NOT_PAD) {
             TokenizersLibrary.LIB.disablePadding(getHandle());
         } else {
             TokenizersLibrary.LIB.setPadding(
-                    getHandle(),
-                    maxLength,
-                    paddingStrategy.toString().toLowerCase(),
-                    padToMultipleOf);
+                    getHandle(), maxLength, padding.name(), padToMultipleOf);
         }
     }
 
@@ -400,16 +479,60 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
                 ids, typeIds, tokens, wordIds, attentionMask, specialTokenMask, charSpans);
     }
 
-    enum TruncationStrategy {
-        DO_NOT_TRUNCATE,
+    /** An enum to represent the different available truncation strategies. */
+    private enum TruncationStrategy {
         LONGEST_FIRST,
         ONLY_FIRST,
-        ONLY_SECOND
+        ONLY_SECOND,
+        DO_NOT_TRUNCATE;
+
+        /**
+         * Converts the String to the matching TruncationStrategy type.
+         *
+         * @param value the String to convert
+         * @return the matching PaddingStrategy type
+         * @throws IllegalArgumentException if the value does not match any TruncationStrategy type
+         */
+        static TruncationStrategy fromValue(String value) {
+            if ("true".equals(value)) {
+                return TruncationStrategy.LONGEST_FIRST;
+            } else if ("false".equals(value)) {
+                return TruncationStrategy.DO_NOT_TRUNCATE;
+            }
+            for (TruncationStrategy strategy : TruncationStrategy.values()) {
+                if (strategy.name().equalsIgnoreCase(value)) {
+                    return strategy;
+                }
+            }
+            throw new IllegalArgumentException("Invalid TruncationStrategy: " + value);
+        }
     }
 
-    enum PaddingStrategy {
-        DO_NOT_PAD,
-        BATCH_LONGEST,
-        FIXED
+    /** An enum to represent the different available padding strategies. */
+    private enum PaddingStrategy {
+        LONGEST,
+        MAX_LENGTH,
+        DO_NOT_PAD;
+
+        /**
+         * Converts the String to the matching PaddingStrategy type.
+         *
+         * @param value the String to convert
+         * @return the matching PaddingStrategy type
+         * @throws IllegalArgumentException if the value does not match any PaddingStrategy type
+         */
+        static PaddingStrategy fromValue(String value) {
+            if ("true".equals(value)) {
+                return PaddingStrategy.LONGEST;
+            } else if ("false".equals(value)) {
+                return PaddingStrategy.DO_NOT_PAD;
+            }
+            for (PaddingStrategy strategy : PaddingStrategy.values()) {
+                if (strategy.name().equalsIgnoreCase(value)) {
+                    return strategy;
+                }
+            }
+            throw new IllegalArgumentException("Invalid PaddingStrategy: " + value);
+        }
     }
 }
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java
index b0e9095ac..caada24c6 100644
--- a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java
@@ -53,13 +53,23 @@ public final class TokenizersLibrary {
 
     public native String decode(long tokenizer, long[] ids, boolean addSpecialTokens);
 
+    public native String getTruncationStrategy(long tokenizer);
+
+    public native String getPaddingStrategy(long tokenizer);
+
+    public native int getMaxLength(long tokenizer);
+
+    public native int getStride(long tokenizer);
+
+    public native int getPadToMultipleOf(long tokenizer);
+
     public native void disablePadding(long tokenizer);
 
     public native void setPadding(
-            long tokenizer, long maxLength, String paddingStrategy, long padToMultipleOf);
+            long tokenizer, int maxLength, String paddingStrategy, int padToMultipleOf);
 
     public native void disableTruncation(long tokenizer);
 
     public native void setTruncation(
-            long tokenizer, long maxLength, String truncationStrategy, long stride);
+            long tokenizer, int maxLength, String truncationStrategy, int stride);
 }


Test patch:
diff --git a/api/src/test/java/ai/djl/translate/NoopServingTranslatorFactoryTest.java b/api/src/test/java/ai/djl/translate/NoopServingTranslatorFactoryTest.java
new file mode 100644
index 000000000..69c11b368
--- /dev/null
+++ b/api/src/test/java/ai/djl/translate/NoopServingTranslatorFactoryTest.java
@@ -0,0 +1,49 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.translate;
+
+import ai.djl.ModelException;
+import ai.djl.modality.Input;
+import ai.djl.modality.Output;
+import ai.djl.nn.Blocks;
+import ai.djl.repository.zoo.Criteria;
+import ai.djl.repository.zoo.ZooModel;
+
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+
+public class NoopServingTranslatorFactoryTest {
+
+    @Test
+    public void testRawTranslatorFactory() throws ModelException, IOException {
+        NoopServingTranslatorFactory factory = new NoopServingTranslatorFactory();
+        Assert.assertEquals(factory.getSupportedTypes().size(), 1);
+
+        Path modelPath = Paths.get("src/test/resources/identity");
+        Criteria<Input, Output> criteria =
+                Criteria.builder()
+                        .setTypes(Input.class, Output.class)
+                        .optModelPath(modelPath)
+                        .optBlock(Blocks.identityBlock())
+                        .optTranslatorFactory(factory)
+                        .build();
+
+        try (ZooModel<Input, Output> model = criteria.loadModel()) {
+            Assert.assertNotNull(model);
+        }
+    }
+}
diff --git a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
index 22cf0782d..bf5ffe74b 100644
--- a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
+++ b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
@@ -14,7 +14,6 @@
 package ai.djl.huggingface.tokenizers;
 
 import ai.djl.huggingface.tokenizers.jni.CharSpan;
-import ai.djl.testing.TestRequirements;
 import ai.djl.training.util.DownloadUtils;
 
 import org.testng.Assert;
@@ -32,8 +31,6 @@ public class HuggingFaceTokenizerTest {
 
     @Test
     public void testTokenizer() throws IOException {
-        TestRequirements.notArm();
-
         String input = "Hello, y'all! How are you 😁 ?";
         String[] inputs = {"Hello, y'all!", "How are you 😁 ?"};
 
@@ -113,7 +110,6 @@ public class HuggingFaceTokenizerTest {
 
     @Test
     public void testTokenizerDecoding() {
-        TestRequirements.notArm();
         List<long[]> testIds =
                 Arrays.asList(
                         new long[] {
@@ -157,74 +153,227 @@ public class HuggingFaceTokenizerTest {
     }
 
     @Test
-    public void testTruncationAndPadding() {
-        TestRequirements.notArm();
+    public void testMaxLengthTruncationAndAllPaddings() {
+        List<String> inputs =
+                Arrays.asList(
+                        "Hello, y'all! How are you?",
+                        "Today is a sunny day. Good weather I'd say",
+                        "I am happy");
 
-        try (HuggingFaceTokenizer tokenizer = HuggingFaceTokenizer.newInstance("bert-base-cased")) {
-            List<String> inputs =
-                    Arrays.asList(
-                            "Hello, y'all! How are you?",
-                            "Today is a sunny day. Good weather I'd say",
-                            "I am happy");
-            tokenizer.setTruncationAndPadding(
-                    HuggingFaceTokenizer.TruncationStrategy.LONGEST_FIRST,
-                    HuggingFaceTokenizer.PaddingStrategy.BATCH_LONGEST,
-                    10);
-            List<long[]> expectedIds =
+        Map<String, String> options = new ConcurrentHashMap<>();
+        options.put("truncation", "longest_first");
+        options.put("padding", "longest");
+        options.put("maxLength", "10");
+
+        try (HuggingFaceTokenizer tokenizer =
+                HuggingFaceTokenizer.newInstance("bert-base-cased", options)) {
+            List<long[]> expectedIdsWithTruncationAndPadding =
                     Arrays.asList(
                             new long[] {101, 8667, 117, 194, 112, 1155, 106, 1731, 1132, 102},
                             new long[] {101, 3570, 1110, 170, 21162, 1285, 119, 2750, 4250, 102},
                             new long[] {101, 146, 1821, 2816, 102, 0, 0, 0, 0, 0});
-            List<long[]> expectedAttentionMasks =
+            Encoding[] encodings = tokenizer.batchEncode(inputs);
+            for (int i = 0; i < encodings.length; ++i) {
+                Assert.assertEquals(
+                        encodings[i].getIds(), expectedIdsWithTruncationAndPadding.get(i));
+            }
+
+            List<long[]> expectedIdsWithTruncationAndPaddingToMultiple =
                     Arrays.asList(
-                            new long[] {1, 1, 1, 1, 1, 1, 1, 1, 1, 1},
-                            new long[] {1, 1, 1, 1, 1, 1, 1, 1, 1, 1},
-                            new long[] {1, 1, 1, 1, 1, 0, 0, 0, 0, 0});
+                            new long[] {101, 8667, 117, 194, 112, 1155, 106, 1731, 1132, 102, 0, 0},
+                            new long[] {
+                                101, 3570, 1110, 170, 21162, 1285, 119, 2750, 4250, 102, 0, 0
+                            },
+                            new long[] {101, 146, 1821, 2816, 102, 0, 0, 0, 0, 0, 0, 0});
+            tokenizer.setPadToMultipleOf(3);
+            encodings = tokenizer.batchEncode(inputs);
+            for (int i = 0; i < encodings.length; ++i) {
+                Assert.assertEquals(
+                        encodings[i].getIds(),
+                        expectedIdsWithTruncationAndPaddingToMultiple.get(i));
+            }
+
+            List<long[]> expectedIdsNoPadding =
+                    Arrays.asList(
+                            new long[] {101, 8667, 117, 194, 112, 1155, 106, 1731, 1132, 102},
+                            new long[] {101, 3570, 1110, 170, 21162, 1285, 119, 2750, 4250, 102},
+                            new long[] {101, 146, 1821, 2816, 102});
+            tokenizer.setPadding(false);
+            encodings = tokenizer.batchEncode(inputs);
+            for (int i = 0; i < encodings.length; ++i) {
+                Assert.assertEquals(encodings[i].getIds(), expectedIdsNoPadding.get(i));
+            }
+        }
+    }
+
+    @Test
+    public void testMaxModelLengthTruncationAndAllPaddings() {
+        String repeat = "hi,";
+        int numRepeats = 513;
+        int capacity = numRepeats * 3;
+        StringBuilder stringBuilder = new StringBuilder(capacity);
+        for (int i = 0; i < numRepeats; ++i) {
+            stringBuilder.append(repeat);
+        }
+        List<String> inputs = Arrays.asList(stringBuilder.toString(), "This is a short sentence");
+        try (HuggingFaceTokenizer tokenizer = HuggingFaceTokenizer.newInstance("bert-base-cased")) {
+            int[] expectedNumberOfIdsNoTruncationNoPadding = new int[] {numRepeats * 2 + 2, 7};
             Encoding[] encodings = tokenizer.batchEncode(inputs);
             for (int i = 0; i < encodings.length; ++i) {
-                Assert.assertEquals(encodings[i].getIds(), expectedIds.get(i));
-                Assert.assertEquals(encodings[i].getAttentionMask(), expectedAttentionMasks.get(i));
+                Assert.assertEquals(
+                        encodings[i].getIds().length, expectedNumberOfIdsNoTruncationNoPadding[i]);
             }
 
-            tokenizer.setTruncationAndPadding(
-                    HuggingFaceTokenizer.TruncationStrategy.LONGEST_FIRST,
-                    HuggingFaceTokenizer.PaddingStrategy.FIXED,
-                    12);
-            expectedIds =
+            tokenizer.setTruncation(true);
+            int[] expectedNumberOfIdsDefaultTruncationNoPadding = new int[] {512, 7};
+            encodings = tokenizer.batchEncode(inputs);
+            for (int i = 0; i < encodings.length; ++i) {
+                Assert.assertEquals(
+                        encodings[i].getIds().length,
+                        expectedNumberOfIdsDefaultTruncationNoPadding[i]);
+            }
+
+            tokenizer.setPadding(true);
+            encodings = tokenizer.batchEncode(inputs);
+            for (Encoding encoding : encodings) {
+                Assert.assertEquals(encoding.getIds().length, 512);
+            }
+        }
+    }
+
+    @Test
+    public void testNoTruncationAndAllPaddings() {
+        List<String> inputs =
+                Arrays.asList(
+                        "Hello, y'all! How are you?",
+                        "Today is a sunny day. Good weather I'd say",
+                        "I am happy");
+        try (HuggingFaceTokenizer tokenizer = HuggingFaceTokenizer.newInstance("bert-base-cased")) {
+            List<long[]> expectedIdsNoTruncationOrPadding =
                     Arrays.asList(
                             new long[] {
                                 101, 8667, 117, 194, 112, 1155, 106, 1731, 1132, 1128, 136, 102
                             },
                             new long[] {
-                                101, 3570, 1110, 170, 21162, 1285, 119, 2750, 4250, 146, 112, 102
+                                101, 3570, 1110, 170, 21162, 1285, 119, 2750, 4250, 146, 112, 173,
+                                1474, 102
                             },
-                            new long[] {101, 146, 1821, 2816, 102, 0, 0, 0, 0, 0, 0, 0});
-            expectedAttentionMasks =
+                            new long[] {101, 146, 1821, 2816, 102});
+
+            Encoding[] encodings = tokenizer.batchEncode(inputs);
+            for (int i = 0; i < encodings.length; ++i) {
+                Assert.assertEquals(encodings[i].getIds(), expectedIdsNoTruncationOrPadding.get(i));
+            }
+            tokenizer.setPadding(true);
+            List<long[]> expectedIdsNoTruncationDefaultPadding =
                     Arrays.asList(
-                            new long[] {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1},
-                            new long[] {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1},
-                            new long[] {1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0});
+                            new long[] {
+                                101, 8667, 117, 194, 112, 1155, 106, 1731, 1132, 1128, 136, 102, 0,
+                                0
+                            },
+                            new long[] {
+                                101, 3570, 1110, 170, 21162, 1285, 119, 2750, 4250, 146, 112, 173,
+                                1474, 102
+                            },
+                            new long[] {101, 146, 1821, 2816, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0});
             encodings = tokenizer.batchEncode(inputs);
             for (int i = 0; i < encodings.length; ++i) {
-                Assert.assertEquals(encodings[i].getIds(), expectedIds.get(i));
-                Assert.assertEquals(encodings[i].getAttentionMask(), expectedAttentionMasks.get(i));
+                Assert.assertEquals(
+                        encodings[i].getIds(), expectedIdsNoTruncationDefaultPadding.get(i));
             }
 
-            tokenizer.setTruncationAndPadding(
-                    HuggingFaceTokenizer.TruncationStrategy.ONLY_FIRST,
-                    HuggingFaceTokenizer.PaddingStrategy.BATCH_LONGEST,
-                    8);
-            Encoding encoding = tokenizer.encode("Hello there my friend", "How are you");
+            tokenizer.setPaddingMaxLength();
+            encodings = tokenizer.batchEncode(inputs);
+            for (int i = 0; i < encodings.length; ++i) {
+                Assert.assertEquals(encodings[i].getIds().length, 512);
+            }
+
+            tokenizer.setMaxLength(15);
+            List<long[]> expectedIdsNoTruncationMaxLengthPadding =
+                    Arrays.asList(
+                            new long[] {
+                                101, 8667, 117, 194, 112, 1155, 106, 1731, 1132, 1128, 136, 102, 0,
+                                0, 0
+                            },
+                            new long[] {
+                                101, 3570, 1110, 170, 21162, 1285, 119, 2750, 4250, 146, 112, 173,
+                                1474, 102, 0
+                            },
+                            new long[] {101, 146, 1821, 2816, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0});
+            encodings = tokenizer.batchEncode(inputs);
+            for (int i = 0; i < encodings.length; ++i) {
+                Assert.assertEquals(
+                        encodings[i].getIds(), expectedIdsNoTruncationMaxLengthPadding.get(i));
+            }
+
+            tokenizer.setPadToMultipleOf(8);
+            List<long[]> expectedIdsNoTruncationMaxLengthPadToMultiplePadding =
+                    Arrays.asList(
+                            new long[] {
+                                101, 8667, 117, 194, 112, 1155, 106, 1731, 1132, 1128, 136, 102, 0,
+                                0, 0, 0
+                            },
+                            new long[] {
+                                101, 3570, 1110, 170, 21162, 1285, 119, 2750, 4250, 146, 112, 173,
+                                1474, 102, 0, 0
+                            },
+                            new long[] {
+                                101, 146, 1821, 2816, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
+                            });
+            encodings = tokenizer.batchEncode(inputs);
+            for (int i = 0; i < encodings.length; ++i) {
+                Assert.assertEquals(
+                        encodings[i].getIds(),
+                        expectedIdsNoTruncationMaxLengthPadToMultiplePadding.get(i));
+            }
+        }
+    }
+
+    @Test
+    public void testTruncationAndPaddingForPairInputs() {
+        Map<String, String> options = new ConcurrentHashMap<>();
+        options.put("truncation", "only_first");
+        options.put("padding", "longest");
+        options.put("maxLength", "8");
+        String text = "Hello there my friend";
+        String textPair = "How are you";
+
+        try (HuggingFaceTokenizer tokenizer =
+                HuggingFaceTokenizer.newInstance("bert-base-cased", options)) {
+            Encoding encoding = tokenizer.encode(text, textPair);
             long[] expectedId = new long[] {101, 8667, 1175, 102, 1731, 1132, 1128, 102};
             Assert.assertEquals(encoding.getIds(), expectedId);
 
-            tokenizer.setTruncationAndPadding(
-                    HuggingFaceTokenizer.TruncationStrategy.ONLY_SECOND,
-                    HuggingFaceTokenizer.PaddingStrategy.DO_NOT_PAD,
-                    8);
+            tokenizer.setPadToMultipleOf(3);
+            encoding = tokenizer.encode(text, textPair);
+            long[] expectedIdPadToMultiple =
+                    new long[] {101, 8667, 1175, 102, 1731, 1132, 1128, 102, 0};
+            Assert.assertEquals(encoding.getIds(), expectedIdPadToMultiple);
+        }
+
+        options.put("truncation", "only_second");
+        options.remove("padding");
+        options.put("maxLength", "8");
+
+        try (HuggingFaceTokenizer tokenizer =
+                HuggingFaceTokenizer.newInstance("bert-base-cased", options)) {
+            Encoding encoding = tokenizer.encode(text, textPair);
+            long[] expectedId = new long[] {101, 8667, 1175, 1139, 1910, 102, 1731, 102};
+            Assert.assertEquals(encoding.getIds(), expectedId);
+
+            tokenizer.setPadToMultipleOf(3);
+            tokenizer.setPadding(true);
+            encoding = tokenizer.encode(text, textPair);
+            long[] expectedIdPadToMultiple =
+                    new long[] {101, 8667, 1175, 1139, 1910, 102, 1731, 102, 0};
+            Assert.assertEquals(encoding.getIds(), expectedIdPadToMultiple);
+        }
 
-            encoding = tokenizer.encode("Hello there my friend", "How are you");
-            expectedId = new long[] {101, 8667, 1175, 1139, 1910, 102, 1731, 102};
+        options.put("truncation", "longest_first");
+        try (HuggingFaceTokenizer tokenizer =
+                HuggingFaceTokenizer.newInstance("bert-base-cased", options)) {
+            Encoding encoding = tokenizer.encode(text, textPair);
+            long[] expectedId = new long[] {101, 8667, 1175, 1139, 102, 1731, 1132, 102};
             Assert.assertEquals(encoding.getIds(), expectedId);
         }
     }