Repository: deepjavalibrary/djl
Pull request #1406 (2021-12-08 22:29:06)

[huggingface] Adds huggingface tokenizer extension
Change-Id: I1af56de1e841dbcf00c947d2feefb7800699831d

## Description ##

Brief description of what this PR is about

- If this change is a backward incompatible change, why must this change be made?
- Interesting edge cases to note here


codecov-commenter: # [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1406?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
> Merging [#1406](https://codecov.io/gh/deepjavalibrary/djl/pull/1406?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (e1feea4) into [master](https://codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) (bb5073f) will **decrease** coverage by `1.47%`.
> The diff coverage is `51.08%`.

[![Impacted file tree graph](https://codecov.io/gh/deepjavalibrary/djl/pull/1406/graphs/tree.svg?width=650&height=150&src=pr&token=mDKVNwFXRV&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)](https://codecov.io/gh/deepjavalibrary/djl/pull/1406?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)

```diff
@@             Coverage Diff              @@
##             master    #1406      +/-   ##
============================================
- Coverage     72.08%   70.61%   -1.48%     
- Complexity     5126     5274     +148     
============================================
  Files           473      491      +18     
  Lines         21970    23178    +1208     
  Branches       2351     2528     +177     
============================================
+ Hits          15838    16367     +529     
- Misses         4925     5538     +613     
- Partials       1207     1273      +66     
```


| [Impacted Files](https://codecov.io/gh/deepjavalibrary/djl/pull/1406?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Coverage Δ | |
|---|---|---|
| [api/src/main/java/ai/djl/modality/cv/Image.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1406/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0ltYWdlLmphdmE=) | `69.23% <ø> (-4.11%)` | :arrow_down: |
| [...i/djl/modality/cv/translator/BigGANTranslator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1406/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvci5qYXZh) | `21.42% <ø> (-5.24%)` | :arrow_down: |
| [...odality/cv/translator/BigGANTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1406/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvQmlnR0FOVHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `33.33% <0.00%> (+8.33%)` | :arrow_up: |
| [...nslator/InstanceSegmentationTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1406/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvSW5zdGFuY2VTZWdtZW50YXRpb25UcmFuc2xhdG9yRmFjdG9yeS5qYXZh) | `14.28% <0.00%> (-3.90%)` | :arrow_down: |
| [.../modality/cv/translator/YoloTranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1406/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1RyYW5zbGF0b3JGYWN0b3J5LmphdmE=) | `8.33% <0.00%> (-1.67%)` | :arrow_down: |
| [...i/djl/modality/cv/translator/YoloV5Translator.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1406/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1Y1VHJhbnNsYXRvci5qYXZh) | `5.69% <0.00%> (ø)` | |
| [...odality/cv/translator/YoloV5TranslatorFactory.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1406/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvWW9sb1Y1VHJhbnNsYXRvckZhY3RvcnkuamF2YQ==) | `8.33% <0.00%> (-1.67%)` | :arrow_down: |
| [...modality/nlp/embedding/TrainableWordEmbedding.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1406/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L25scC9lbWJlZGRpbmcvVHJhaW5hYmxlV29yZEVtYmVkZGluZy5qYXZh) | `45.00% <0.00%> (-3.34%)` | :arrow_down: |
| [...pi/src/main/java/ai/djl/ndarray/BytesSupplier.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1406/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL25kYXJyYXkvQnl0ZXNTdXBwbGllci5qYXZh) | `54.54% <0.00%> (-12.13%)` | :arrow_down: |
| [...i/src/main/java/ai/djl/ndarray/NDArrayAdapter.java](https://codecov.io/gh/deepjavalibrary/djl/pull/1406/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL25kYXJyYXkvTkRBcnJheUFkYXB0ZXIuamF2YQ==) | `15.23% <0.00%> (-0.83%)` | :arrow_down: |
| ... and [145 more](https://codecov.io/gh/deepjavalibrary/djl/pull/1406/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1406?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/deepjavalibrary/djl/pull/1406?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Last update [209ed16...e1feea4](https://codecov.io/gh/deepjavalibrary/djl/pull/1406?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS
index 1f9d3db0a..4a2e34a10 100644
--- a/.github/CODEOWNERS
+++ b/.github/CODEOWNERS
@@ -1,2 +1,2 @@
 # Global Owners
-* @zachgk @frankfliu
\ No newline at end of file
+* @zachgk @frankfliu @DeepJavaLibrary/djl-admin
\ No newline at end of file
diff --git a/.github/workflows/continuous.yml b/.github/workflows/continuous.yml
index 1f4b381b3..a463d0bef 100644
--- a/.github/workflows/continuous.yml
+++ b/.github/workflows/continuous.yml
@@ -33,12 +33,14 @@ jobs:
             ${{ runner.os }}-gradle-
       - name: check disk space
         run: df -h
-      - name: Compile PyTorch JNI
-        run: ./gradlew :engines:pytorch:pytorch-native:compileJNI
       - name: Compile Sentencepiece JNI
-        run: ./gradlew :extension:sentencepiece:compileJNI
+        run: ./gradlew :extensions:sentencepiece:compileJNI
       - name: Compile fastText JNI
-        run: ./gradlew :extension:fasttext:compileJNI
+        run: ./gradlew :extensions:fasttext:compileJNI
+      - name: Compile tokenizers JNI
+        run: ./gradlew :extensions:tokenizers:compileJNI
+      - name: Compile PyTorch JNI
+        run: ./gradlew :engines:pytorch:pytorch-native:compileJNI
       - name: Compile PaddlePaddle JNI
         run: ./gradlew :engines:paddlepaddle:paddlepaddle-native:compileJNI
       - name: Build with Gradle
@@ -73,10 +75,14 @@ jobs:
           key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*') }}
           restore-keys: |
             ${{ runner.os }}-gradle-
+      - name: Compile Sentencepiece JNI
+        run: ./gradlew :extensions:sentencepiece:compileJNI
       - name: Compile PyTorch JNI
         run: ./gradlew :engines:pytorch:pytorch-native:compileJNI
       - name: Compile PaddlePaddle JNI
         run: ./gradlew :engines:paddlepaddle:paddlepaddle-native:compileJNI
+      - name: Compile tokenizers JNI
+        run: ./gradlew :extensions:tokenizers:compileJNI
       - name: Integration tests
         # MXNet integration test on Github Action fail due to memory limitation on Windows
         run: |
diff --git a/.github/workflows/native_s3_huggingface.yml b/.github/workflows/native_s3_huggingface.yml
new file mode 100644
index 000000000..e11523eac
--- /dev/null
+++ b/.github/workflows/native_s3_huggingface.yml
@@ -0,0 +1,116 @@
+name: Native S3 Huggingface tokenizers
+
+on:
+  workflow_dispatch:
+
+jobs:
+  build-tokenizers-jni-osx:
+    runs-on: macos-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up JDK 1.8
+        uses: actions/setup-java@v1
+        with:
+          java-version: 1.8
+      - uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+      - uses: actions/cache@v2
+        with:
+          path: ~/.gradle/caches
+          key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*') }}
+          restore-keys: |
+            ${{ runner.os }}-gradle-
+      - name: Release JNI prep
+        run: |
+          ./gradlew :extensions:tokenizers:compileJNI
+          ./gradlew -Pjni :extensions:tokenizers:test
+      - name: Configure AWS Credentials
+        uses: aws-actions/configure-aws-credentials@v1
+        with:
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          aws-region: us-east-2
+      - name: Copy files to S3 with the AWS CLI
+        run: |
+          TOKENIZERS_VERSION="$(cat gradle.properties | awk -F '=' '/tokenizers_version/ {print $2}')"
+          aws s3 sync extensions/tokenizers/jnilib s3://djl-ai/publish/tokenizers/${TOKENIZERS_VERSION}/jnilib/
+          aws cloudfront create-invalidation --distribution-id E371VB8JQ6NRVY --paths "/tokenizers/${TOKENIZERS_VERSION}/jnilib/*"
+
+  build-tokenizers-jni-linux:
+    runs-on: ubuntu-latest
+    container: amazonlinux:2
+    steps:
+      - name: Install Environment
+        run: |
+          yum -y update
+          yum -y groupinstall "Development Tools"
+          yum -y install python3-pip perl-IPC-Cmd
+          pip3 install awscli --upgrade
+      - uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+      - uses: actions/checkout@v2
+      - name: Set up JDK 1.8
+        uses: actions/setup-java@v1
+        with:
+          java-version: 1.8
+      - uses: actions/cache@v2
+        with:
+          path: ~/.gradle/caches
+          key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*') }}
+          restore-keys: |
+            ${{ runner.os }}-gradle-
+      - name: Release JNI prep
+        run: |
+          ./gradlew :extensions:tokenizers:compileJNI
+          ./gradlew -Pjni :extensions:tokenizers:test
+      - name: Configure AWS Credentials
+        uses: aws-actions/configure-aws-credentials@v1
+        with:
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          aws-region: us-east-2
+      - name: Copy files to S3 with the AWS CLI
+        run: |
+          TOKENIZERS_VERSION="$(cat gradle.properties | awk -F '=' '/tokenizers_version/ {print $2}')"
+          aws s3 sync extensions/tokenizers/jnilib s3://djl-ai/publish/tokenizers/${TOKENIZERS_VERSION}/jnilib/
+          aws cloudfront create-invalidation --distribution-id E371VB8JQ6NRVY --paths "/tokenizers/${TOKENIZERS_VERSION}/jnilib/*"
+
+  build-tokenizer-jni-windows:
+    runs-on: windows-latest
+    steps:
+      - name: Install Environment
+        run: |
+          choco install -y mingw
+      - uses: actions-rs/toolchain@v1
+        with:
+          toolchain: stable
+      - uses: actions/checkout@v2
+      - name: Set up JDK 1.8
+        uses: actions/setup-java@v1
+        with:
+          java-version: 1.8
+      - uses: actions/cache@v2
+        with:
+          path: ~/.gradle/caches
+          key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*') }}
+          restore-keys: |
+            ${{ runner.os }}-gradle-
+      - name: Release CPU JNI
+        shell: cmd
+        run: |
+          call "C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" amd64
+          gradlew :extensions:tokenizer:compileJNI
+      - name: Configure AWS Credentials
+        uses: aws-actions/configure-aws-credentials@v1
+        with:
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          aws-region: us-east-2
+      - name: Copy files to S3 with the AWS CLI
+        shell: bash
+        run: |
+          TOKENIZERS_VERSION="$(cat gradle.properties | awk -F '=' '/tokenizers_version/ {print $2}')"
+          aws s3 sync extensions/tokenizers/jnilib s3://djl-ai/publish/tokenizers/${TOKENIZERS_VERSION}/jnilib/
+          aws cloudfront create-invalidation --distribution-id E371VB8JQ6NRVY --paths "/tokenizers/${TOKENIZERS_VERSION}/jnilib/*"
diff --git a/.github/workflows/native_s3_sentencepiece.yml b/.github/workflows/native_s3_sentencepiece.yml
index 27557ca24..962b3ad58 100644
--- a/.github/workflows/native_s3_sentencepiece.yml
+++ b/.github/workflows/native_s3_sentencepiece.yml
@@ -71,3 +71,34 @@ jobs:
           SENTENCEPIECE_VERSION="$(cat gradle.properties | awk -F '=' '/sentencepiece_version/ {print $2}')"
           aws s3 sync extensions/sentencepiece/jnilib s3://djl-ai/publish/sentencepiece-${SENTENCEPIECE_VERSION}/jnilib/
           aws cloudfront create-invalidation --distribution-id E371VB8JQ6NRVY --paths "/sentencepiece-${SENTENCEPIECE_VERSION}/jnilib/*"
+
+  build-sentencepiece-jni-windows:
+    runs-on: windows-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Set up JDK 1.8
+        uses: actions/setup-java@v1
+        with:
+          java-version: 1.8
+      - uses: actions/cache@v2
+        with:
+          path: ~/.gradle/caches
+          key: ${{ runner.os }}-gradle-${{ hashFiles('**/*.gradle*') }}
+          restore-keys: |
+            ${{ runner.os }}-gradle-
+      - name: Release JNI prep
+        run: |
+          ./gradlew :extensions:sentencepiece:compileJNI
+          ./gradlew -Pjni :extensions:sentencepiece:test
+      - name: Configure AWS Credentials
+        uses: aws-actions/configure-aws-credentials@v1
+        with:
+          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          aws-region: us-east-2
+      - name: Copy files to S3 with the AWS CLI
+        shell: bash
+        run: |
+          SENTENCEPIECE_VERSION="$(cat gradle.properties | awk -F '=' '/sentencepiece_version/ {print $2}')"
+          aws s3 sync extensions/sentencepiece/jnilib s3://djl-ai/publish/sentencepiece-${SENTENCEPIECE_VERSION}/jnilib/
+          aws cloudfront create-invalidation --distribution-id E371VB8JQ6NRVY --paths "/sentencepiece-${SENTENCEPIECE_VERSION}/jnilib/*"
diff --git a/api/src/main/java/ai/djl/training/DefaultTrainingConfig.java b/api/src/main/java/ai/djl/training/DefaultTrainingConfig.java
index 35b25683a..1218404e9 100644
--- a/api/src/main/java/ai/djl/training/DefaultTrainingConfig.java
+++ b/api/src/main/java/ai/djl/training/DefaultTrainingConfig.java
@@ -24,6 +24,7 @@ import ai.djl.training.optimizer.Optimizer;
 import ai.djl.util.PairList;
 import java.util.ArrayList;
 import java.util.Arrays;
+import java.util.Collection;
 import java.util.List;
 import java.util.concurrent.ExecutorService;
 import java.util.concurrent.ForkJoinPool;
@@ -136,6 +137,18 @@ public class DefaultTrainingConfig implements TrainingConfig {
         return this;
     }
 
+    /**
+     * Adds multiple {@link Evaluator}s that needs to be computed during training.
+     *
+     * @param evaluators the evaluators to be added
+     * @param <T> the type of evaluator to be added
+     * @return this {@code DefaultTrainingConfig}
+     */
+    public <T extends Evaluator> DefaultTrainingConfig addEvaluators(Collection<T> evaluators) {
+        evaluators.forEach(this::addEvaluator);
+        return this;
+    }
+
     /**
      * Adds an {@link Evaluator} that needs to be computed during training.
      *
diff --git a/api/src/main/java/ai/djl/training/evaluator/AbstractAccuracy.java b/api/src/main/java/ai/djl/training/evaluator/AbstractAccuracy.java
index acfba1fff..b58dcb9c2 100644
--- a/api/src/main/java/ai/djl/training/evaluator/AbstractAccuracy.java
+++ b/api/src/main/java/ai/djl/training/evaluator/AbstractAccuracy.java
@@ -29,30 +29,26 @@ public abstract class AbstractAccuracy extends Evaluator {
 
     protected Map<String, Long> correctInstances;
     protected int axis;
-    protected int index;
 
     /**
-     * Creates an accuracy evaluator that computes accuracy across axis 1 along given index.
+     * Creates an accuracy evaluator that computes accuracy across axis 1.
      *
      * @param name the name of the evaluator, default is "Accuracy"
-     * @param index the index of the NDArray in labels to compute accuracy for
      */
-    public AbstractAccuracy(String name, int index) {
-        this(name, index, 1);
+    public AbstractAccuracy(String name) {
+        this(name, 1);
     }
 
     /**
      * Creates an accuracy evaluator.
      *
      * @param name the name of the evaluator, default is "Accuracy"
-     * @param index the index of the NDArray in labels to compute accuracy for
      * @param axis the axis that represent classes in prediction, default 1
      */
-    public AbstractAccuracy(String name, int index, int axis) {
+    public AbstractAccuracy(String name, int axis) {
         super(name);
         correctInstances = new ConcurrentHashMap<>();
         this.axis = axis;
-        this.index = index;
     }
 
     /**
diff --git a/api/src/main/java/ai/djl/training/evaluator/Accuracy.java b/api/src/main/java/ai/djl/training/evaluator/Accuracy.java
index 5c4fe5ed6..5b9695243 100644
--- a/api/src/main/java/ai/djl/training/evaluator/Accuracy.java
+++ b/api/src/main/java/ai/djl/training/evaluator/Accuracy.java
@@ -25,36 +25,34 @@ public class Accuracy extends AbstractAccuracy {
      * index.
      */
     public Accuracy() {
-        this("Accuracy", 0, 1);
+        this("Accuracy", 1);
     }
 
     /**
-     * Creates a multiclass accuracy evaluator that computes accuracy across axis 1 along given
+     * Creates a multiclass accuracy evaluator that computes accuracy across axis 1 along the 0th
      * index.
      *
      * @param name the name of the evaluator, default is "Accuracy"
-     * @param index the index of the NDArray in labels to compute accuracy for
      */
-    public Accuracy(String name, int index) {
-        super(name, index);
+    public Accuracy(String name) {
+        this(name, 1);
     }
 
     /**
      * Creates a multiclass accuracy evaluator.
      *
      * @param name the name of the evaluator, default is "Accuracy"
-     * @param index the index of the NDArray in labels to compute accuracy for
      * @param axis the axis that represent classes in prediction, default 1
      */
-    public Accuracy(String name, int index, int axis) {
-        super(name, index, axis);
+    public Accuracy(String name, int axis) {
+        super(name, axis);
     }
 
     /** {@inheritDoc} */
     @Override
     protected Pair<Long, NDArray> accuracyHelper(NDList labels, NDList predictions) {
-        NDArray label = labels.get(index);
-        NDArray prediction = predictions.get(index);
+        NDArray label = labels.head();
+        NDArray prediction = predictions.head();
         checkLabelShapes(label, prediction);
         NDArray predictionReduced;
         if (!label.getShape().equals(prediction.getShape())) {
diff --git a/api/src/main/java/ai/djl/training/evaluator/BinaryAccuracy.java b/api/src/main/java/ai/djl/training/evaluator/BinaryAccuracy.java
index 42bf45b0f..3468be2d4 100644
--- a/api/src/main/java/ai/djl/training/evaluator/BinaryAccuracy.java
+++ b/api/src/main/java/ai/djl/training/evaluator/BinaryAccuracy.java
@@ -35,11 +35,10 @@ public class BinaryAccuracy extends AbstractAccuracy {
      *
      * @param name the name of the evaluator, default is "Accuracy"
      * @param threshold the value differentiating the posive and negative classes (usually 0 or .5)
-     * @param index the index of the NDArray in labels to compute accuracy for
      * @param axis the axis that represent classes in prediction, default 1
      */
-    public BinaryAccuracy(String name, float threshold, int index, int axis) {
-        super(name, index, axis);
+    public BinaryAccuracy(String name, float threshold, int axis) {
+        super(name, axis);
         this.threshold = threshold;
     }
 
@@ -49,10 +48,9 @@ public class BinaryAccuracy extends AbstractAccuracy {
      *
      * @param name the name of the evaluator, default is "Accuracy"
      * @param threshold the value differentiating the posive and negative classes (usually 0 or .5)
-     * @param index the index of the NDArray in labels to compute accuracy for
      */
-    public BinaryAccuracy(String name, float threshold, int index) {
-        this(name, threshold, index, 1);
+    public BinaryAccuracy(String name, float threshold) {
+        this(name, threshold, 1);
     }
 
     /**
@@ -62,7 +60,7 @@ public class BinaryAccuracy extends AbstractAccuracy {
      * @param threshold the value differentiating the posive and negative classes (usually 0 or .5)
      */
     public BinaryAccuracy(float threshold) {
-        this("BinaryAccuracy", threshold, 0, 1);
+        this("BinaryAccuracy", threshold, 1);
     }
 
     /** Creates a binary (two class) accuracy evaluator with 0 threshold. */
@@ -76,8 +74,8 @@ public class BinaryAccuracy extends AbstractAccuracy {
         Preconditions.checkArgument(
                 labels.size() == predictions.size(),
                 "labels and prediction length does not match.");
-        NDArray label = labels.get(index);
-        NDArray prediction = predictions.get(index);
+        NDArray label = labels.head();
+        NDArray prediction = predictions.head();
         checkLabelShapes(label, prediction, false);
         NDArray predictionReduced = prediction.gte(threshold);
         // result of sum is int64 now
diff --git a/api/src/main/java/ai/djl/training/evaluator/IndexEvaluator.java b/api/src/main/java/ai/djl/training/evaluator/IndexEvaluator.java
new file mode 100644
index 000000000..a7fe08b61
--- /dev/null
+++ b/api/src/main/java/ai/djl/training/evaluator/IndexEvaluator.java
@@ -0,0 +1,95 @@
+/*
+ * Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.training.evaluator;
+
+import ai.djl.ndarray.NDArray;
+import ai.djl.ndarray.NDList;
+
+/**
+ * A wrapper for an {@link Evaluator} that evaluates on only a particular {@link NDArray} in the
+ * predictions and/or labels {@link NDList}s.
+ */
+public class IndexEvaluator extends Evaluator {
+
+    private Evaluator evaluator;
+    private Integer predictionsIndex;
+    private Integer labelsIndex;
+
+    /**
+     * Constructs an {@link IndexEvaluator} with the same index for both predictions and labels.
+     *
+     * @param evaluator the base evaluator
+     * @param index the index for both predictions and labels
+     */
+    public IndexEvaluator(Evaluator evaluator, int index) {
+        this(evaluator, index, index);
+    }
+
+    /**
+     * Constructs an {@link IndexEvaluator}.
+     *
+     * @param evaluator the base evaluator
+     * @param predictionsIndex the predictions index
+     * @param labelsIndex the labels index
+     */
+    public IndexEvaluator(Evaluator evaluator, Integer predictionsIndex, Integer labelsIndex) {
+        super(evaluator.getName());
+        this.evaluator = evaluator;
+        this.predictionsIndex = predictionsIndex;
+        this.labelsIndex = labelsIndex;
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public NDArray evaluate(NDList labels, NDList predictions) {
+        return evaluator.evaluate(getLabels(labels), getPredictions(predictions));
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public void addAccumulator(String key) {
+        evaluator.addAccumulator(key);
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public void updateAccumulator(String key, NDList labels, NDList predictions) {
+        evaluator.updateAccumulator(key, getLabels(labels), getPredictions(predictions));
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public void resetAccumulator(String key) {
+        evaluator.resetAccumulator(key);
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public float getAccumulator(String key) {
+        return evaluator.getAccumulator(key);
+    }
+
+    private NDList getPredictions(NDList predictions) {
+        if (predictionsIndex == null) {
+            return predictions;
+        }
+        return new NDList(predictions.get(predictionsIndex));
+    }
+
+    private NDList getLabels(NDList labels) {
+        if (labelsIndex == null) {
+            return labels;
+        }
+        return new NDList(labels.get(labelsIndex));
+    }
+}
diff --git a/api/src/main/java/ai/djl/training/evaluator/TopKAccuracy.java b/api/src/main/java/ai/djl/training/evaluator/TopKAccuracy.java
index f7b56acf1..4a4fa0210 100644
--- a/api/src/main/java/ai/djl/training/evaluator/TopKAccuracy.java
+++ b/api/src/main/java/ai/djl/training/evaluator/TopKAccuracy.java
@@ -34,11 +34,10 @@ public class TopKAccuracy extends AbstractAccuracy {
      * Creates a {@code TopKAccuracy} instance.
      *
      * @param name the accuracy name, default "Top_K_Accuracy"
-     * @param index the index of the {@link NDArray} in labels to compute topK accuracy for
      * @param topK the value of K
      */
-    public TopKAccuracy(String name, int index, int topK) {
-        super(name, index);
+    public TopKAccuracy(String name, int topK) {
+        super(name);
         if (topK > 1) {
             this.topK = topK;
         } else {
@@ -46,17 +45,6 @@ public class TopKAccuracy extends AbstractAccuracy {
         }
     }
 
-    /**
-     * Creates an instance of {@code TopKAccuracy} evaluator that computes topK accuracy across axis
-     * 1 along the given index.
-     *
-     * @param index the index of the {@link NDArray} in labels to compute topK accuracy for
-     * @param topK the value of K
-     */
-    public TopKAccuracy(int index, int topK) {
-        this("Top_" + topK + "_Accuracy", index, topK);
-    }
-
     /**
      * Creates an instance of {@code TopKAccuracy} evaluator that computes topK accuracy across axis
      * 1 along the 0th index.
@@ -64,14 +52,14 @@ public class TopKAccuracy extends AbstractAccuracy {
      * @param topK the value of K
      */
     public TopKAccuracy(int topK) {
-        this("Top_" + topK + "_Accuracy", 0, topK);
+        this("Top_" + topK + "_Accuracy", topK);
     }
 
     /** {@inheritDoc} */
     @Override
     protected Pair<Long, NDArray> accuracyHelper(NDList labels, NDList predictions) {
-        NDArray label = labels.get(index);
-        NDArray prediction = predictions.get(index);
+        NDArray label = labels.head();
+        NDArray prediction = predictions.head();
         // number of labels and predictions should be the same
         checkLabelShapes(label, prediction);
         // ascending by default
diff --git a/api/src/main/java/ai/djl/training/loss/IndexLoss.java b/api/src/main/java/ai/djl/training/loss/IndexLoss.java
new file mode 100644
index 000000000..4f05c8781
--- /dev/null
+++ b/api/src/main/java/ai/djl/training/loss/IndexLoss.java
@@ -0,0 +1,71 @@
+/*
+ * Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.training.loss;
+
+import ai.djl.ndarray.NDArray;
+import ai.djl.ndarray.NDList;
+
+/**
+ * A wrapper for a {@link Loss} that evaluates on only a particular {@link NDArray} in the
+ * predictions and/or labels {@link NDList}s.
+ */
+public class IndexLoss extends Loss {
+
+    private Loss loss;
+    private Integer predictionsIndex;
+    private Integer labelsIndex;
+
+    /**
+     * Constructs an {@link IndexLoss} with the same index for both predictions and labels.
+     *
+     * @param loss the base evaluator
+     * @param index the index for both predictions and labels
+     */
+    public IndexLoss(Loss loss, int index) {
+        this(loss, index, index);
+    }
+
+    /**
+     * Constructs an {@link IndexLoss}.
+     *
+     * @param loss the base evaluator
+     * @param predictionsIndex the predictions index
+     * @param labelsIndex the labels index
+     */
+    public IndexLoss(Loss loss, Integer predictionsIndex, Integer labelsIndex) {
+        super(loss.getName());
+        this.loss = loss;
+        this.predictionsIndex = predictionsIndex;
+        this.labelsIndex = labelsIndex;
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public NDArray evaluate(NDList labels, NDList predictions) {
+        return loss.evaluate(getLabels(labels), getPredictions(predictions));
+    }
+
+    private NDList getPredictions(NDList predictions) {
+        if (predictionsIndex == null) {
+            return predictions;
+        }
+        return new NDList(predictions.get(predictionsIndex));
+    }
+
+    private NDList getLabels(NDList labels) {
+        if (labelsIndex == null) {
+            return labels;
+        }
+        return new NDList(labels.get(labelsIndex));
+    }
+}
diff --git a/api/src/main/java/ai/djl/training/loss/SimpleCompositeLoss.java b/api/src/main/java/ai/djl/training/loss/SimpleCompositeLoss.java
index ccaa46d7c..2dc0519aa 100644
--- a/api/src/main/java/ai/djl/training/loss/SimpleCompositeLoss.java
+++ b/api/src/main/java/ai/djl/training/loss/SimpleCompositeLoss.java
@@ -15,15 +15,13 @@ package ai.djl.training.loss;
 import ai.djl.ndarray.NDList;
 import ai.djl.util.Pair;
 import java.util.ArrayList;
-import java.util.List;
 
 /**
  * {@code SimpleCompositeLoss} is an implementation of the {@link Loss} abstract class that can
  * combine different {@link Loss} functions by adding the individual losses together.
  *
- * <p>This class can be used when the losses either accept a single index of the labels and
- * predictions or the entire lists. For more complicated composite losses, extend the {@link
- * AbstractCompositeLoss}.
+ * <p>For cases where the losses use only a single index of the labels and/or predictions, use the
+ * {@link IndexLoss}.
  *
  * <p>For an example of using this loss, see <a
  * href="https://github.com/deepjavalibrary/djl/blob/master/examples/src/main/java/ai/djl/examples/training/TrainCaptcha.java">the
@@ -31,8 +29,6 @@ import java.util.List;
  */
 public class SimpleCompositeLoss extends AbstractCompositeLoss {
 
-    private List<Integer> indices;
-
     /**
      * Creates a new empty instance of {@code CompositeLoss} that can combine the given {@link Loss}
      * components.
@@ -50,7 +46,6 @@ public class SimpleCompositeLoss extends AbstractCompositeLoss {
     public SimpleCompositeLoss(String name) {
         super(name);
         components = new ArrayList<>();
-        indices = new ArrayList<>();
     }
 
     /**
@@ -61,21 +56,6 @@ public class SimpleCompositeLoss extends AbstractCompositeLoss {
      */
     public SimpleCompositeLoss addLoss(Loss loss) {
         components.add(loss);
-        indices.add(null);
-        return this;
-    }
-
-    /**
-     * Adds a Loss that applies to a single index of the label and predictions to this composite
-     * loss.
-     *
-     * @param loss the loss to add
-     * @param index the index in the label and predictions NDLists this loss applies to
-     * @return this composite loss
-     */
-    public SimpleCompositeLoss addLoss(Loss loss, int index) {
-        components.add(loss);
-        indices.add(index);
         return this;
     }
 
@@ -83,11 +63,6 @@ public class SimpleCompositeLoss extends AbstractCompositeLoss {
     @Override
     protected Pair<NDList, NDList> inputForComponent(
             int componentIndex, NDList labels, NDList predictions) {
-        if (indices.get(componentIndex) != null) {
-            int index = indices.get(componentIndex);
-            return new Pair<>(new NDList(labels.get(index)), new NDList(predictions.get(index)));
-        } else {
-            return new Pair<>(labels, predictions);
-        }
+        return new Pair<>(labels, predictions);
     }
 }
diff --git a/examples/src/main/java/ai/djl/examples/training/TrainCaptcha.java b/examples/src/main/java/ai/djl/examples/training/TrainCaptcha.java
index 766e16192..184ff3908 100644
--- a/examples/src/main/java/ai/djl/examples/training/TrainCaptcha.java
+++ b/examples/src/main/java/ai/djl/examples/training/TrainCaptcha.java
@@ -31,8 +31,10 @@ import ai.djl.training.dataset.Dataset;
 import ai.djl.training.dataset.Dataset.Usage;
 import ai.djl.training.dataset.RandomAccessDataset;
 import ai.djl.training.evaluator.Accuracy;
+import ai.djl.training.evaluator.IndexEvaluator;
 import ai.djl.training.listener.SaveModelTrainingListener;
 import ai.djl.training.listener.TrainingListener;
+import ai.djl.training.loss.IndexLoss;
 import ai.djl.training.loss.SimpleCompositeLoss;
 import ai.djl.training.loss.SoftmaxCrossEntropyLoss;
 import ai.djl.training.util.ProgressBar;
@@ -99,17 +101,18 @@ public final class TrainCaptcha {
                 });
         SimpleCompositeLoss loss = new SimpleCompositeLoss();
         for (int i = 0; i < CaptchaDataset.CAPTCHA_LENGTH; i++) {
-            loss.addLoss(new SoftmaxCrossEntropyLoss("loss_digit_" + i), i);
+            loss.addLoss(new IndexLoss(new SoftmaxCrossEntropyLoss("loss_digit_" + i), i));
         }
 
         DefaultTrainingConfig config =
                 new DefaultTrainingConfig(loss)
                         .optDevices(Engine.getInstance().getDevices(arguments.getMaxGpus()))
+                        .addEvaluators(loss.getComponents())
                         .addTrainingListeners(TrainingListener.Defaults.logging(outputDir))
                         .addTrainingListeners(listener);
 
         for (int i = 0; i < CaptchaDataset.CAPTCHA_LENGTH; i++) {
-            config.addEvaluator(new Accuracy("acc_digit_" + i, i));
+            config.addEvaluator(new IndexEvaluator(new Accuracy("acc_digit_" + i), i));
         }
 
         return config;
diff --git a/examples/src/main/java/ai/djl/examples/training/TrainSeq2Seq.java b/examples/src/main/java/ai/djl/examples/training/TrainSeq2Seq.java
index 6fd7e2c33..2e0804dcc 100644
--- a/examples/src/main/java/ai/djl/examples/training/TrainSeq2Seq.java
+++ b/examples/src/main/java/ai/djl/examples/training/TrainSeq2Seq.java
@@ -149,7 +149,7 @@ public final class TrainSeq2Seq {
                 });
 
         return new DefaultTrainingConfig(new MaskedSoftmaxCrossEntropyLoss())
-                .addEvaluator(new Accuracy("Accuracy", 0, 2))
+                .addEvaluator(new Accuracy("Accuracy", 2))
                 .optDevices(Engine.getInstance().getDevices(arguments.getMaxGpus()))
                 .optExecutorService(executorService)
                 .addTrainingListeners(TrainingListener.Defaults.logging(outputDir))
diff --git a/extensions/sentencepiece/CMakeLists.txt b/extensions/sentencepiece/CMakeLists.txt
index 54566a936..7aea0fd21 100644
--- a/extensions/sentencepiece/CMakeLists.txt
+++ b/extensions/sentencepiece/CMakeLists.txt
@@ -9,6 +9,14 @@ set(SPM_ENABLE_TCMALLOC OFF CACHE BOOL "Build sentencepiece static library")
 # disable shared library as we only link static library
 set(SPM_ENABLE_SHARED OFF CACHE BOOL "Build sentencepiece static library")
 
+if (MSVC)
+  string(REPLACE "/MD" "/MT" CMAKE_CXX_FLAGS_DEBUG          ${CMAKE_CXX_FLAGS_DEBUG})
+  string(REPLACE "/MD" "/MT" CMAKE_CXX_FLAGS_MINSIZEREL     ${CMAKE_CXX_FLAGS_MINSIZEREL})
+  string(REPLACE "/MD" "/MT" CMAKE_CXX_FLAGS_RELEASE        ${CMAKE_CXX_FLAGS_RELEASE})
+  string(REPLACE "/MD" "/MT" CMAKE_CXX_FLAGS_RELWITHDEBINFO ${CMAKE_CXX_FLAGS_RELWITHDEBINFO})
+  add_definitions("/wd4267 /wd4244 /wd4305 /Zc:strictStrings /utf-8")
+endif()
+
 find_package(JNI REQUIRED)
 find_path(UTILS_INCLUDE_DIR NAMES djl/utils.h PATHS ${PROJECT_SOURCE_DIR}/../../api/src/main/native REQUIRED)
 
diff --git a/extensions/sentencepiece/build.cmd b/extensions/sentencepiece/build.cmd
new file mode 100644
index 000000000..1253de17c
--- /dev/null
+++ b/extensions/sentencepiece/build.cmd
@@ -0,0 +1,25 @@
+set VERSION="%1"
+set FILEPATH="sentencepiece"
+
+if exist %FILEPATH% (
+    echo Found %FILEPATH%
+) else (
+    echo Couldn't find %FILEPATH%, downloading it ...
+    echo Cloning google/sentencepiece.git
+    git clone https://github.com/google/sentencepiece.git -b %VERSION%
+    echo Finished cloning sentencepiece
+)
+
+if exist build rd /q /s build
+md build\classes
+cd build
+
+javac -sourcepath ../src/main/java/ ../src/main/java/ai/djl/sentencepiece/jni/SentencePieceLibrary.java -h include -d classes
+cmake ..
+cmake --build . --config Release
+
+:: for nightly ci
+md jnilib\win-x86_64
+copy Release\sentencepiece_native.dll jnilib\win-x86_64
+
+cd ..
diff --git a/extensions/sentencepiece/build.gradle b/extensions/sentencepiece/build.gradle
index e46229873..7dd054cff 100644
--- a/extensions/sentencepiece/build.gradle
+++ b/extensions/sentencepiece/build.gradle
@@ -19,6 +19,7 @@ processResources {
     doLast {
         def url = "https://publish.djl.ai/sentencepiece-${sentencepiece_version}/jnilib/${djl_version}"
         def files = [
+                "win-x86_64"  : "sentencepiece_native.dll",
                 "linux-x86_64": "libsentencepiece_native.so",
                 "osx-x86_64"  : "libsentencepiece_native.dylib"
         ]
@@ -61,22 +62,26 @@ apply from: file("${rootProject.projectDir}/tools/gradle/cpp-formatter.gradle")
 
 task compileJNI {
     doFirst {
-        if (System.properties['os.name'].toLowerCase(Locale.ROOT).contains("mac")
-                || System.properties['os.name'].toLowerCase(Locale.ROOT).contains("linux")) {
+        if (System.properties['os.name'].toLowerCase(Locale.ROOT).contains("win")) {
             exec {
-                commandLine 'bash', 'build.sh'
+                commandLine "${project.projectDir}/build.cmd", "v${sentencepiece_version}"
             }
-
-            // for ci to upload to S3
-            def ciDir = "${project.projectDir}/jnilib/${djl_version}/"
-            copy {
-                from "${project.buildDir}/jnilib"
-                into ciDir
+        } else if (System.properties['os.name'].toLowerCase(Locale.ROOT).contains("mac")
+                || System.properties['os.name'].toLowerCase(Locale.ROOT).contains("linux")) {
+            exec {
+                commandLine "bash", "build.sh", "v${sentencepiece_version}"
             }
-            delete System.getProperty("user.home") + "/.djl.ai/sentencepiece"
         } else {
             throw new IllegalStateException("Unknown Architecture " + System.properties['os.name'])
         }
+
+        // for ci to upload to S3
+        def ciDir = "${project.projectDir}/jnilib/${djl_version}/"
+        copy {
+            from "${project.buildDir}/jnilib"
+            into ciDir
+        }
+        delete System.getProperty("user.home") + "/.djl.ai/sentencepiece"
     }
 }
 
diff --git a/extensions/sentencepiece/build.sh b/extensions/sentencepiece/build.sh
index e82dd9198..c1ddb9852 100755
--- a/extensions/sentencepiece/build.sh
+++ b/extensions/sentencepiece/build.sh
@@ -10,7 +10,7 @@ elif [[ -n $(command -v sysctl) ]]; then
 fi
 PLATFORM=$(uname | tr '[:upper:]' '[:lower:]')
 
-VERSION=v0.1.95
+VERSION=$1
 
 pushd $WORK_DIR
 if [ ! -d "sentencepiece" ];
diff --git a/extensions/sentencepiece/src/main/java/ai/djl/sentencepiece/jni/LibUtils.java b/extensions/sentencepiece/src/main/java/ai/djl/sentencepiece/jni/LibUtils.java
index 7a51d7471..09821b9cd 100644
--- a/extensions/sentencepiece/src/main/java/ai/djl/sentencepiece/jni/LibUtils.java
+++ b/extensions/sentencepiece/src/main/java/ai/djl/sentencepiece/jni/LibUtils.java
@@ -43,10 +43,6 @@ public final class LibUtils {
     private LibUtils() {}
 
     public static void loadLibrary() {
-        if (System.getProperty("os.name").startsWith("Win")) {
-            throw new UnsupportedOperationException("Windows is not supported.");
-        }
-
         String libName = copyJniLibraryFromClasspath();
         logger.debug("Loading sentencepiece library from: {}", libName);
         System.load(libName); // NOPMD
diff --git a/extensions/sentencepiece/src/main/native/ai_djl_sentencepiece_jni_SentencePieceLibrary.cc b/extensions/sentencepiece/src/main/native/ai_djl_sentencepiece_jni_SentencePieceLibrary.cc
index cf6d1c73c..de0d94261 100644
--- a/extensions/sentencepiece/src/main/native/ai_djl_sentencepiece_jni_SentencePieceLibrary.cc
+++ b/extensions/sentencepiece/src/main/native/ai_djl_sentencepiece_jni_SentencePieceLibrary.cc
@@ -96,7 +96,7 @@ JNIEXPORT jstring JNICALL Java_ai_djl_sentencepiece_jni_SentencePieceLibrary_idT
   return env->NewStringUTF(processor_ptr->IdToPiece(jid).c_str());
 }
 
-JNIEXPORT int JNICALL Java_ai_djl_sentencepiece_jni_SentencePieceLibrary_pieceToId(
+JNIEXPORT jint JNICALL Java_ai_djl_sentencepiece_jni_SentencePieceLibrary_pieceToId(
     JNIEnv* env, jobject jthis, jlong jhandle, jstring jpiece) {
   auto* processor_ptr = reinterpret_cast<sentencepiece::SentencePieceProcessor*>(jhandle);
   return processor_ptr->PieceToId(djl::utils::jni::GetStringFromJString(env, jpiece));
diff --git a/extensions/tokenizers/.gitignore b/extensions/tokenizers/.gitignore
new file mode 100644
index 000000000..41e087b17
--- /dev/null
+++ b/extensions/tokenizers/.gitignore
@@ -0,0 +1,3 @@
+Cargo.lock
+/tokenizers
+/jnilib
diff --git a/extensions/tokenizers/README.md b/extensions/tokenizers/README.md
new file mode 100644
index 000000000..66b1f2961
--- /dev/null
+++ b/extensions/tokenizers/README.md
@@ -0,0 +1,26 @@
+# NLP support with Huggingface tokenizers
+
+This module contains the NLP support with Huggingface tokenizers implementation.
+
+This is an implementation from [Huggingface tokenizers](https://github.com/huggingface/tokenizers) RUST API.
+
+## Documentation
+
+You can build the latest javadocs locally using the following command:
+
+```sh
+./gradlew javadoc
+```
+The javadocs output is built in the `build/doc/javadoc` folder.
+
+## Installation
+
+You can pull the module from the central Maven repository by including the following dependency in your `pom.xml` file:
+
+```xml
+<dependency>
+    <groupId>ai.djl.huggingface</groupId>
+    <artifactId>tokenizers</artifactId>
+    <version>0.15.0</version>
+</dependency>
+```
diff --git a/extensions/tokenizers/build.cmd b/extensions/tokenizers/build.cmd
new file mode 100644
index 000000000..3a481d33b
--- /dev/null
+++ b/extensions/tokenizers/build.cmd
@@ -0,0 +1,22 @@
+@rem https://chocolatey.org/docs/installation#install-with-cmdexe
+@rem to install rust java etc..
+@rem choco install rust -y
+@rem choco install jdk8 -y
+
+set VERSION=python-v"%1"
+
+if exist "tokenizers" (
+    echo Found "tokenizers"
+) else (
+    git clone https://github.com/huggingface/tokenizers -b %VERSION%
+)
+
+if exist build rd /q /s build
+md build\classes
+
+set RUST_MANIFEST=rust/Cargo.toml
+cargo build --manifest-path %RUST_MANIFEST% --release
+
+@rem for nightly ci
+md build\jnilib\win-x86_64
+copy rust\target\release\djl.dll build\jnilib\win-x86_64\tokenizers.dll
diff --git a/extensions/tokenizers/build.gradle b/extensions/tokenizers/build.gradle
new file mode 100644
index 000000000..017228b86
--- /dev/null
+++ b/extensions/tokenizers/build.gradle
@@ -0,0 +1,91 @@
+group "ai.djl.huggingface"
+
+dependencies {
+    api project(":api")
+
+    testImplementation("org.testng:testng:${testng_version}") {
+        exclude group: "junit", module: "junit"
+    }
+
+    testImplementation(project(":testing"))
+    testImplementation "org.slf4j:slf4j-simple:${slf4j_version}"
+}
+
+compileJava.dependsOn(processResources)
+
+processResources {
+    outputs.dir file("${project.buildDir}/classes/java/main/native/lib")
+    doLast {
+        def url = "https://publish.djl.ai/tokenizers"
+        def files = [
+                "win-x86_64/libwinpthread-1.dll": "extra",
+                "win-x86_64/libgcc_s_seh-1.dll" : "extra",
+                "win-x86_64/libstdc%2B%2B-6.dll": "extra",
+                "win-x86_64/tokenizers.dll"     : "${tokenizers_version}/jnilib/${djl_version}",
+                "linux-x86_64/libtokenizers.so" : "${tokenizers_version}/jnilib/${djl_version}",
+                "osx-x86_64/libtokenizers.dylib": "${tokenizers_version}/jnilib/${djl_version}"
+        ]
+        def jnilibDir = "${project.projectDir}/jnilib/${djl_version}"
+        files.each { entry ->
+            def file = new File("${jnilibDir}/${URLDecoder.decode(entry.key, "UTF-8")}")
+            if (file.exists()) {
+                project.logger.lifecycle("prebuilt or cached file found for ${entry.key}")
+            } else if ("extra" == entry.value || !project.hasProperty("jni")) {
+                project.logger.lifecycle("Downloading ${url}/${entry.value}/${entry.key}")
+                file.getParentFile().mkdirs()
+                def downloadPath = new URL("${url}/${entry.value}/${entry.key}")
+                downloadPath.withInputStream { i -> file.withOutputStream { it << i } }
+            }
+        }
+        copy {
+            from jnilibDir
+            into "${project.buildDir}/classes/java/main/native/lib"
+        }
+
+        // write properties
+        def propFile = file("${project.buildDir}/classes/java/main/native/lib/tokenizers.properties")
+        propFile.text = "version=${tokenizers_version}-${version}\n"
+    }
+}
+
+publishing {
+    publications {
+        maven(MavenPublication) {
+            pom {
+                name = "DJL NLP utilities for Huggingface tokenizers"
+                description = "Deep Java Library (DJL) NLP utilities for Huggingface tokenizers"
+                url = "http://www.djl.ai/extensions/${project.name}"
+            }
+        }
+    }
+}
+
+apply from: file("${rootProject.projectDir}/tools/gradle/cpp-formatter.gradle")
+
+task compileJNI {
+    doFirst {
+        if (System.properties['os.name'].toLowerCase(Locale.ROOT).contains("mac")
+                || System.properties['os.name'].toLowerCase(Locale.ROOT).contains("linux")) {
+            exec {
+                commandLine "bash", "build.sh", "${tokenizers_version}"
+            }
+        } else {
+            exec {
+                commandLine "${project.projectDir}/build.cmd", "${tokenizers_version}"
+            }
+        }
+
+        // for ci to upload to S3
+        def ciDir = "${project.projectDir}/jnilib/${djl_version}/"
+        copy {
+            from "${project.buildDir}/jnilib"
+            into ciDir
+        }
+        delete System.getProperty("user.home") + "/.djl.ai/tokenizers"
+    }
+}
+
+clean.doFirst {
+    delete System.getProperty("user.home") + "/.djl.ai/tokenizers"
+    delete "rust/target"
+}
diff --git a/extensions/tokenizers/build.sh b/extensions/tokenizers/build.sh
new file mode 100755
index 000000000..51fbf57ef
--- /dev/null
+++ b/extensions/tokenizers/build.sh
@@ -0,0 +1,38 @@
+#!/usr/bin/env bash
+
+set -e
+WORK_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+NUM_PROC=1
+if [[ -n $(command -v nproc) ]]; then
+  NUM_PROC=$(nproc)
+elif [[ -n $(command -v sysctl) ]]; then
+  NUM_PROC=$(sysctl -n hw.ncpu)
+fi
+PLATFORM=$(uname | tr '[:upper:]' '[:lower:]')
+
+VERSION=python-v$1
+
+pushd $WORK_DIR
+if [ ! -d "tokenizers" ]; then
+  git clone https://github.com/huggingface/tokenizers -b $VERSION
+fi
+
+if [ ! -d "build" ]; then
+  mkdir build
+fi
+
+rm -rf build/classes
+mkdir build/classes
+javac -sourcepath src/main/java/ src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java -h build/include -d build/classes
+
+RUST_MANIFEST=rust/Cargo.toml
+cargo build --manifest-path $RUST_MANIFEST --release
+
+# for nightly ci
+if [[ $PLATFORM == 'darwin' ]]; then
+  mkdir -p build/jnilib/osx-x86_64
+  cp -f rust/target/release/libdjl.dylib build/jnilib/osx-x86_64/libtokenizers.dylib
+elif [[ $PLATFORM == 'linux' ]]; then
+  mkdir -p build/jnilib/linux-x86_64
+  cp -f rust/target/release/libdjl.so build/jnilib/linux-x86_64/libtokenizers.so
+fi
diff --git a/extensions/tokenizers/gradlew b/extensions/tokenizers/gradlew
new file mode 120000
index 000000000..343e0d2ca
--- /dev/null
+++ b/extensions/tokenizers/gradlew
@@ -0,0 +1,1 @@
+../../gradlew
\ No newline at end of file
diff --git a/extensions/tokenizers/rust/.gitignore b/extensions/tokenizers/rust/.gitignore
new file mode 100644
index 000000000..eb5a316cb
--- /dev/null
+++ b/extensions/tokenizers/rust/.gitignore
@@ -0,0 +1,1 @@
+target
diff --git a/extensions/tokenizers/rust/Cargo.toml b/extensions/tokenizers/rust/Cargo.toml
new file mode 100644
index 000000000..f6b846f63
--- /dev/null
+++ b/extensions/tokenizers/rust/Cargo.toml
@@ -0,0 +1,15 @@
+[package]
+name = "djl"
+version = "0.1.0"
+authors = ["Frank Liu <frankfliu2000@gmail.com>"]
+edition = "2018"
+
+[dependencies]
+jni = "0.19.0"
+tokenizers = { path = "../tokenizers/tokenizers", version = "*" }
+
+[target.'cfg(target_os = "linux")'.dependencies]
+openssl = { version = "0.10", features = ["vendored"] }
+
+[lib]
+crate_type = ["cdylib"]
diff --git a/extensions/tokenizers/rust/src/lib.rs b/extensions/tokenizers/rust/src/lib.rs
new file mode 100644
index 000000000..34e532de8
--- /dev/null
+++ b/extensions/tokenizers/rust/src/lib.rs
@@ -0,0 +1,318 @@
+// Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+extern crate tokenizers as tk;
+
+use std::str::FromStr;
+use tk::tokenizer::{EncodeInput, Encoding};
+use tk::FromPretrainedParameters;
+use tk::Tokenizer;
+
+use jni::objects::{JObject, JString};
+use jni::sys::{jboolean, jlong, jlongArray, jobjectArray, jsize, JNI_TRUE};
+use jni::JNIEnv;
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_createTokenizer(
+    env: JNIEnv,
+    _: JObject,
+    input: JString,
+) -> jlong {
+    let identifier: String = env
+        .get_string(input)
+        .expect("Couldn't get java string!")
+        .into();
+
+    let parameters = FromPretrainedParameters::default();
+    let tokenizer = Tokenizer::from_pretrained(identifier, Some(parameters));
+
+    match tokenizer {
+        Ok(output) => to_handle(output),
+        Err(err) => {
+            env.throw(err.to_string()).unwrap();
+            0
+        }
+    }
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_createTokenizerFromString(
+    env: JNIEnv,
+    _: JObject,
+    json: JString,
+) -> jlong {
+    let data: String = env
+        .get_string(json)
+        .expect("Couldn't get java string!")
+        .into();
+
+    let tokenizer = Tokenizer::from_str(&data);
+    match tokenizer {
+        Ok(output) => to_handle(output),
+        Err(err) => {
+            env.throw(err.to_string()).unwrap();
+            0
+        }
+    }
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_deleteTokenizer(
+    _env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+) {
+    drop_handle::<Tokenizer>(handle);
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_encode(
+    env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+    input: JString,
+    add_special_tokens: jboolean,
+) -> jlong {
+    let tokenizer = cast_handle::<Tokenizer>(handle);
+    let sequence: String = env
+        .get_string(input)
+        .expect("Couldn't get java string!")
+        .into();
+
+    let input_sequence = tk::InputSequence::from(sequence);
+    let encoded_input = EncodeInput::Single(input_sequence);
+    let encoding = tokenizer.encode(encoded_input, add_special_tokens == JNI_TRUE);
+
+    match encoding {
+        Ok(output) => to_handle(output),
+        Err(err) => {
+            env.throw(err.to_string()).unwrap();
+            0
+        }
+    }
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_encodeList(
+    env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+    inputs: jobjectArray,
+    add_special_tokens: jboolean,
+) -> jlong {
+    let tokenizer = cast_handle::<Tokenizer>(handle);
+    let len = env.get_array_length(inputs).unwrap();
+    let mut array: Vec<String> = Vec::new();
+    for i in 0..len {
+        let item = env.get_object_array_element(inputs, i).unwrap().into();
+        let value: String = env
+            .get_string(item)
+            .expect("Couldn't get java string!")
+            .into();
+        array.push(value);
+    }
+
+    let input_sequence = tk::InputSequence::from(array);
+    let encoded_input = EncodeInput::from(input_sequence);
+    let encoding = tokenizer.encode(encoded_input, add_special_tokens == JNI_TRUE);
+
+    match encoding {
+        Ok(output) => to_handle(output),
+        Err(err) => {
+            env.throw(err.to_string()).unwrap();
+            0
+        }
+    }
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_batchEncode(
+    env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+    inputs: jobjectArray,
+    add_special_tokens: jboolean,
+) -> jlongArray {
+    let tokenizer = cast_handle::<Tokenizer>(handle);
+    let len = env.get_array_length(inputs).unwrap();
+    let mut array: Vec<String> = Vec::new();
+    for i in 0..len {
+        let item = env.get_object_array_element(inputs, i).unwrap().into();
+        let value: String = env
+            .get_string(item)
+            .expect("Couldn't get java string!")
+            .into();
+        array.push(value);
+    }
+
+    let encodings = tokenizer
+        .encode_batch(array, add_special_tokens == JNI_TRUE)
+        .unwrap();
+    let handles = encodings
+        .into_iter()
+        .map(|c| to_handle(c))
+        .collect::<Vec<_>>();
+
+    let size = handles.len() as jsize;
+    let ret = env.new_long_array(size).unwrap();
+    env.set_long_array_region(ret, 0, &handles).unwrap();
+    ret
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_deleteEncoding(
+    _env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+) {
+    drop_handle::<Encoding>(handle);
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_getTokenIds(
+    env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+) -> jlongArray {
+    let encoding = cast_handle::<Encoding>(handle);
+    let ids = encoding.get_ids();
+    let len = ids.len() as jsize;
+    let mut long_ids: Vec<jlong> = Vec::new();
+    long_ids.reserve(len as usize);
+    for i in ids {
+        long_ids.push(*i as jlong)
+    }
+
+    let array: jlongArray = env.new_long_array(len).unwrap();
+    env.set_long_array_region(array, 0, &long_ids).unwrap();
+    array
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_getTypeIds(
+    env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+) -> jlongArray {
+    let encoding = cast_handle::<Encoding>(handle);
+    let type_ids = encoding.get_type_ids();
+    let len = type_ids.len() as jsize;
+    let mut long_ids: Vec<jlong> = Vec::new();
+    for i in type_ids {
+        long_ids.push(*i as jlong)
+    }
+
+    let array: jlongArray = env.new_long_array(len).unwrap();
+    env.set_long_array_region(array, 0, &long_ids).unwrap();
+    array
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_getWordIds(
+    env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+) -> jlongArray {
+    let encoding = cast_handle::<Encoding>(handle);
+    let word_ids = encoding.get_word_ids();
+    let len = word_ids.len() as jsize;
+    let mut long_ids: Vec<jlong> = Vec::new();
+    for i in word_ids {
+        if let Some(word_id) = i {
+            long_ids.push(*word_id as jlong)
+        } else {
+            long_ids.push(-1)
+        }
+    }
+
+    let array: jlongArray = env.new_long_array(len).unwrap();
+    env.set_long_array_region(array, 0, &long_ids).unwrap();
+    array
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_getTokens(
+    env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+) -> jobjectArray {
+    let encoding = cast_handle::<Encoding>(handle);
+    let tokens = encoding.get_tokens();
+    let len = tokens.len() as jsize;
+
+    let array: jobjectArray = env
+        .new_object_array(len, "java/lang/String", JObject::null())
+        .unwrap();
+    for (i, token) in tokens.iter().enumerate() {
+        let item: JString = env.new_string(&token).unwrap();
+        env.set_object_array_element(array, i as jsize, item).unwrap();
+    }
+    array
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_getAttentionMask(
+    env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+) -> jlongArray {
+    let encoding = cast_handle::<Encoding>(handle);
+    let attention_masks = encoding.get_attention_mask();
+    let len = attention_masks.len() as jsize;
+    let mut long_ids: Vec<jlong> = Vec::new();
+    for i in attention_masks {
+        long_ids.push(*i as jlong)
+    }
+
+    let array: jlongArray = env.new_long_array(len).unwrap();
+    env.set_long_array_region(array, 0, &long_ids).unwrap();
+    array
+}
+
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_getSpecialTokenMask(
+    env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+) -> jlongArray {
+    let encoding = cast_handle::<Encoding>(handle);
+    let special_token_masks = encoding.get_special_tokens_mask();
+    let len = special_token_masks.len() as jsize;
+    let mut long_ids: Vec<jlong> = Vec::new();
+    for i in special_token_masks {
+        long_ids.push(*i as jlong)
+    }
+
+    let array: jlongArray = env.new_long_array(len).unwrap();
+    env.set_long_array_region(array, 0, &long_ids).unwrap();
+    array
+}
+
+fn to_handle<T: 'static>(val: T) -> jlong {
+    let handle = Box::into_raw(Box::new(val)) as jlong;
+    handle
+}
+
+fn cast_handle<T>(handle: jlong) -> &'static mut T {
+    assert_ne!(handle, 0, "Invalid handle value");
+
+    let ptr = handle as *mut T;
+    unsafe { &mut *ptr }
+}
+
+fn drop_handle<T: 'static>(handle: jlong) {
+    unsafe {
+        Box::from_raw(handle as *mut T);
+    }
+}
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/Encoding.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/Encoding.java
new file mode 100644
index 000000000..9d894add6
--- /dev/null
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/Encoding.java
@@ -0,0 +1,93 @@
+/*
+ * Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.huggingface.tokenizers;
+
+/** A class holds token encoding information. */
+public class Encoding {
+
+    private long[] ids;
+    private long[] typeIds;
+    private String[] tokens;
+    private long[] wordIds;
+    private long[] attentionMask;
+    private long[] specialTokenMask;
+
+    Encoding(
+            long[] ids,
+            long[] typeIds,
+            String[] tokens,
+            long[] wordIds,
+            long[] attentionMask,
+            long[] specialTokenMask) {
+        this.ids = ids;
+        this.typeIds = typeIds;
+        this.tokens = tokens;
+        this.wordIds = wordIds;
+        this.attentionMask = attentionMask;
+        this.specialTokenMask = specialTokenMask;
+    }
+
+    /**
+     * Returns the token ids.
+     *
+     * @return the token ids
+     */
+    public long[] getIds() {
+        return ids;
+    }
+
+    /**
+     * Returns the token type ids.
+     *
+     * @return the token type ids
+     */
+    public long[] getTypeIds() {
+        return typeIds;
+    }
+
+    /**
+     * Returns the tokens.
+     *
+     * @return the tokens
+     */
+    public String[] getTokens() {
+        return tokens;
+    }
+
+    /**
+     * Returns the word ids.
+     *
+     * @return the word ids
+     */
+    public long[] getWordIds() {
+        return wordIds;
+    }
+
+    /**
+     * Returns the attention masks.
+     *
+     * @return the attention masks
+     */
+    public long[] getAttentionMask() {
+        return attentionMask;
+    }
+
+    /**
+     * Returns the special token masks.
+     *
+     * @return the special token masks
+     */
+    public long[] getSpecialTokenMask() {
+        return specialTokenMask;
+    }
+}
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
new file mode 100644
index 000000000..3bf130d50
--- /dev/null
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
@@ -0,0 +1,207 @@
+/*
+ * Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.huggingface.tokenizers;
+
+import ai.djl.huggingface.tokenizers.jni.LibUtils;
+import ai.djl.huggingface.tokenizers.jni.TokenizersLibrary;
+import ai.djl.modality.nlp.preprocess.Tokenizer;
+import ai.djl.util.NativeResource;
+import ai.djl.util.Utils;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * {@code HuggingFaceTokenizer} is a Huggingface tokenizer implementation of the {@link Tokenizer}
+ * interface that converts sentences into token.
+ */
+public final class HuggingFaceTokenizer extends NativeResource<Long> implements Tokenizer {
+
+    private boolean addSpecialTokens;
+
+    private HuggingFaceTokenizer(long handle, boolean addSpecialTokens) {
+        super(handle);
+        this.addSpecialTokens = addSpecialTokens;
+    }
+
+    /**
+     * Create a pre-trained {@code HuggingFaceTokenizer} instance from hugginface hub.
+     *
+     * @param name the name of the huggingface tokenizer
+     * @return a {@code HuggingFaceTokenizer} instance
+     */
+    public static HuggingFaceTokenizer newInstance(String name) {
+        return newInstance(name, null);
+    }
+
+    /**
+     * Create a pre-trained {@code HuggingFaceTokenizer} instance from hugginface hub.
+     *
+     * @param identifier the identifier of the huggingface tokenizer
+     * @param options tokenizer options
+     * @return a {@code HuggingFaceTokenizer} instance
+     */
+    public static HuggingFaceTokenizer newInstance(String identifier, Map<String, String> options) {
+        LibUtils.checkStatus();
+
+        long handle = TokenizersLibrary.LIB.createTokenizer(identifier);
+        boolean addSpecialTokens =
+                options == null
+                        || !options.containsKey("addSpecialTokens")
+                        || Boolean.parseBoolean(options.get("addSpecialTokens"));
+        return new HuggingFaceTokenizer(handle, addSpecialTokens);
+    }
+
+    /**
+     * Create a pre-trained {@code HuggingFaceTokenizer} instance from existing models.
+     *
+     * @param modelPath the directory or file path of the model location
+     * @return a {@code HuggingFaceTokenizer} instance
+     * @throws IOException when IO operation fails in loading a resource
+     */
+    public static HuggingFaceTokenizer newInstance(Path modelPath) throws IOException {
+        return newInstance(modelPath, null);
+    }
+
+    /**
+     * Create a pre-trained {@code HuggingFaceTokenizer} instance from existing models.
+     *
+     * @param modelPath the directory or file path of the model location
+     * @param options tokenizer options
+     * @return a {@code HuggingFaceTokenizer} instance
+     * @throws IOException when IO operation fails in loading a resource
+     */
+    public static HuggingFaceTokenizer newInstance(Path modelPath, Map<String, String> options)
+            throws IOException {
+        try (InputStream is = Files.newInputStream(modelPath)) {
+            return newInstance(is, options);
+        }
+    }
+
+    /**
+     * Create a pre-trained {@code HuggingFaceTokenizer} instance from {@code InputStream}.
+     *
+     * @param is {@code InputStream}
+     * @param options tokenizer options
+     * @return a {@code HuggingFaceTokenizer} instance
+     * @throws IOException when IO operation fails in loading a resource
+     */
+    public static HuggingFaceTokenizer newInstance(InputStream is, Map<String, String> options)
+            throws IOException {
+        LibUtils.checkStatus();
+        String json = Utils.toString(is);
+
+        long handle = TokenizersLibrary.LIB.createTokenizerFromString(json);
+        boolean addSpecialTokens =
+                options != null && Boolean.parseBoolean(options.get("addSpecialTokens"));
+        return new HuggingFaceTokenizer(handle, addSpecialTokens);
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public List<String> tokenize(String sentence) {
+        Encoding encoding = encode(sentence);
+        return Arrays.asList(encoding.getTokens());
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public String buildSentence(List<String> tokens) {
+        // TODO:
+        return String.join(" ", tokens);
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    public void close() {
+        Long pointer = handle.getAndSet(null);
+        if (pointer != null) {
+            TokenizersLibrary.LIB.deleteTokenizer(pointer);
+        }
+    }
+
+    /**
+     * Returns the {@code Encoding} of the input sentence.
+     *
+     * @param input the input sentence
+     * @return the {@code Encoding} of the input sentence
+     */
+    public Encoding encode(String input) {
+        long encoding = TokenizersLibrary.LIB.encode(getHandle(), input, addSpecialTokens);
+        return toEncoding(encoding);
+    }
+
+    /**
+     * Returns the {@code Encoding} of the input sentences.
+     *
+     * @param inputs the input sentences
+     * @return the {@code Encoding} of the input sentences
+     */
+    public Encoding encode(List<String> inputs) {
+        String[] array = inputs.toArray(new String[0]);
+        return encode(array);
+    }
+
+    /**
+     * Returns the {@code Encoding} of the input sentences.
+     *
+     * @param inputs the input sentences
+     * @return the {@code Encoding} of the input sentences
+     */
+    public Encoding encode(String[] inputs) {
+        long encoding = TokenizersLibrary.LIB.encodeList(getHandle(), inputs, addSpecialTokens);
+        return toEncoding(encoding);
+    }
+
+    /**
+     * Returns the {@code Encoding} of the input sentence in batch.
+     *
+     * @param inputs the batch of input sentence
+     * @return the {@code Encoding} of the input sentence in batch
+     */
+    public Encoding[] batchEncode(List<String> inputs) {
+        String[] array = inputs.toArray(new String[0]);
+        return batchEncode(array);
+    }
+
+    /**
+     * Returns the {@code Encoding} of the input sentence in batch.
+     *
+     * @param inputs the batch of input sentence
+     * @return the {@code Encoding} of the input sentence in batch
+     */
+    public Encoding[] batchEncode(String[] inputs) {
+        long[] encodings = TokenizersLibrary.LIB.batchEncode(getHandle(), inputs, addSpecialTokens);
+        Encoding[] ret = new Encoding[encodings.length];
+        for (int i = 0; i < encodings.length; ++i) {
+            ret[i] = toEncoding(encodings[i]);
+        }
+        return ret;
+    }
+
+    private Encoding toEncoding(long encoding) {
+        long[] ids = TokenizersLibrary.LIB.getTokenIds(encoding);
+        long[] typeIds = TokenizersLibrary.LIB.getTypeIds(encoding);
+        String[] tokens = TokenizersLibrary.LIB.getTokens(encoding);
+        long[] wordIds = TokenizersLibrary.LIB.getWordIds(encoding);
+        long[] attentionMask = TokenizersLibrary.LIB.getAttentionMask(encoding);
+        long[] specialTokenMask = TokenizersLibrary.LIB.getSpecialTokenMask(encoding);
+
+        TokenizersLibrary.LIB.deleteEncoding(encoding);
+        return new Encoding(ids, typeIds, tokens, wordIds, attentionMask, specialTokenMask);
+    }
+}
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/LibUtils.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/LibUtils.java
new file mode 100644
index 000000000..a53efe6c0
--- /dev/null
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/LibUtils.java
@@ -0,0 +1,109 @@
+/*
+ * Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.huggingface.tokenizers.jni;
+
+import ai.djl.engine.EngineException;
+import ai.djl.util.Platform;
+import ai.djl.util.Utils;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.StandardCopyOption;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/** Utilities for finding the Huggingface tokenizer native binary on the System. */
+@SuppressWarnings("MissingJavadocMethod")
+public final class LibUtils {
+
+    private static final Logger logger = LoggerFactory.getLogger(LibUtils.class);
+
+    private static final String LIB_NAME = System.mapLibraryName("tokenizers");
+
+    private static EngineException exception;
+
+    static {
+        try {
+            loadLibrary();
+        } catch (RuntimeException e) {
+            exception = new EngineException("Failed to load Huggingface native library.", e);
+        }
+    }
+
+    private LibUtils() {}
+
+    public static void checkStatus() {
+        if (exception != null) {
+            throw exception;
+        }
+    }
+
+    private static void loadLibrary() {
+        String[] libs;
+        if (System.getProperty("os.name").startsWith("Windows")) {
+            libs =
+                    new String[] {
+                        "libwinpthread-1.dll", "libgcc_s_seh-1.dll", "libstdc++-6.dll", LIB_NAME
+                    };
+        } else {
+            libs = new String[] {LIB_NAME};
+        }
+
+        Path dir = copyJniLibraryFromClasspath(libs);
+        logger.debug("Loading huggingface library from: {}", dir);
+
+        for (String libName : libs) {
+            System.load(dir.resolve(libName).toString()); // NOPMD
+        }
+    }
+
+    private static Path copyJniLibraryFromClasspath(String[] libs) {
+        Path cacheDir = Utils.getEngineCacheDir("tokenizers");
+        Platform platform = Platform.detectPlatform("tokenizers");
+        String classifier = platform.getClassifier();
+        String version = platform.getVersion();
+        Path dir = cacheDir.resolve(version + '-' + classifier);
+        Path path = dir.resolve(LIB_NAME);
+        logger.debug("Using cache dir: {}", dir);
+        if (Files.exists(path)) {
+            return dir.toAbsolutePath();
+        }
+
+        Path tmp = null;
+        try {
+            Files.createDirectories(cacheDir);
+            tmp = Files.createTempDirectory(cacheDir, "tmp");
+
+            for (String libName : libs) {
+                String libPath = "/native/lib/" + classifier + "/" + libName;
+                logger.info("Extracting {} to cache ...", libPath);
+                try (InputStream is = LibUtils.class.getResourceAsStream(libPath)) {
+                    if (is == null) {
+                        throw new IllegalStateException("tokenizers library not found: " + libPath);
+                    }
+                    Path target = tmp.resolve(libName);
+                    Files.copy(is, target, StandardCopyOption.REPLACE_EXISTING);
+                }
+            }
+            Utils.moveQuietly(tmp, dir);
+            return dir.toAbsolutePath();
+        } catch (IOException e) {
+            throw new IllegalStateException("Cannot copy jni files", e);
+        } finally {
+            if (tmp != null) {
+                Utils.deleteQuietly(tmp);
+            }
+        }
+    }
+}
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java
new file mode 100644
index 000000000..cb8b68308
--- /dev/null
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java
@@ -0,0 +1,48 @@
+/*
+ * Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.huggingface.tokenizers.jni;
+
+/** A class containing utilities to interact with the Tokenizer JNI layer. */
+@SuppressWarnings("MissingJavadocMethod")
+public final class TokenizersLibrary {
+
+    public static final TokenizersLibrary LIB = new TokenizersLibrary();
+
+    private TokenizersLibrary() {}
+
+    public native long createTokenizer(String identifier);
+
+    public native long createTokenizerFromString(String json);
+
+    public native void deleteTokenizer(long handle);
+
+    public native long encode(long tokenizer, String input, boolean addSpecialTokens);
+
+    public native long encodeList(long tokenizer, String[] inputs, boolean addSpecialTokens);
+
+    public native long[] batchEncode(long tokenizer, String[] inputs, boolean addSpecialTokens);
+
+    public native void deleteEncoding(long encoding);
+
+    public native long[] getTokenIds(long encoding);
+
+    public native long[] getTypeIds(long encoding);
+
+    public native long[] getWordIds(long encoding);
+
+    public native String[] getTokens(long encoding);
+
+    public native long[] getAttentionMask(long encoding);
+
+    public native long[] getSpecialTokenMask(long encoding);
+}
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/package-info.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/package-info.java
new file mode 100644
index 000000000..710521d47
--- /dev/null
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/package-info.java
@@ -0,0 +1,15 @@
+/*
+ * Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+
+/** Contains classes to interface with the underlying Huggingface tokenizers. */
+package ai.djl.huggingface.tokenizers.jni;
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/package-info.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/package-info.java
new file mode 100644
index 000000000..fdbaaff11
--- /dev/null
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/package-info.java
@@ -0,0 +1,15 @@
+/*
+ * Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+
+/** Contains classes to interface with the underlying Huggingface tokenizers. */
+package ai.djl.huggingface.tokenizers;
diff --git a/extensions/tokenizers/src/main/javadoc/overview.html b/extensions/tokenizers/src/main/javadoc/overview.html
new file mode 100644
index 000000000..464df46ba
--- /dev/null
+++ b/extensions/tokenizers/src/main/javadoc/overview.html
@@ -0,0 +1,13 @@
+<html>
+<head>
+  <meta charset="UTF-8">
+</head>
+<body>
+<p>This document is the API specification for the Deep Java Library (DJL) Huggingface tokenizers extension.</p>
+
+<p>
+  The Tokeniszers module contains the DJL extension for Huggingface tokenizers.
+</p>
+
+</body>
+</html>
diff --git a/gradle.properties b/gradle.properties
index 0b1f66f09..58be01db8 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -17,6 +17,7 @@ trt_version=8.0.1
 onnxruntime_version=1.9.0
 paddlepaddle_version=2.0.2
 sentencepiece_version=0.1.95
+tokenizers_version=0.11.0
 fasttext_version=0.9.2
 xgboost_version=1.4.1
 
@@ -27,7 +28,7 @@ commons_logging_version=1.2
 gson_version=2.8.8
 jna_version=5.9.0
 slf4j_version=1.7.32
-log4j_slf4j_version=2.15.0
+log4j_slf4j_version=2.17.0
 awssdk_version=2.17.54
 hadoop_version=3.3.1
 javacpp_version=1.5.6
diff --git a/settings.gradle b/settings.gradle
index da6aad371..25492c51b 100644
--- a/settings.gradle
+++ b/settings.gradle
@@ -31,6 +31,7 @@ include ':extensions:fasttext'
 include ':extensions:hadoop'
 include ':extensions:opencv'
 include ':extensions:sentencepiece'
+include ':extensions:tokenizers'
 include ':integration'
 include ':model-zoo'
 include ':testing'
diff --git a/tools/gradle/publish.gradle b/tools/gradle/publish.gradle
index 25a9afb18..aff4d8de2 100644
--- a/tools/gradle/publish.gradle
+++ b/tools/gradle/publish.gradle
@@ -21,6 +21,7 @@ configure([
         project(':extensions:hadoop'),
         project(':extensions:opencv'),
         project(':extensions:sentencepiece'),
+        project(':extensions:tokenizers'),
         project(':model-zoo')]) {
     apply plugin: "maven-publish"
     apply plugin: "signing"


Test patch:
diff --git a/extensions/sentencepiece/src/test/java/ai/djl/sentencepiece/SpTextEmbeddingTest.java b/extensions/sentencepiece/src/test/java/ai/djl/sentencepiece/SpTextEmbeddingTest.java
index 811e1e216..18c4fb8c3 100644
--- a/extensions/sentencepiece/src/test/java/ai/djl/sentencepiece/SpTextEmbeddingTest.java
+++ b/extensions/sentencepiece/src/test/java/ai/djl/sentencepiece/SpTextEmbeddingTest.java
@@ -12,7 +12,6 @@
  */
 package ai.djl.sentencepiece;
 
-import ai.djl.testing.TestRequirements;
 import ai.djl.training.util.DownloadUtils;
 import java.io.IOException;
 import java.nio.file.Files;
@@ -36,8 +35,6 @@ public class SpTextEmbeddingTest {
 
     @Test
     public void testEmbedding() throws IOException {
-        TestRequirements.notWindows();
-
         Path modelPath = Paths.get("build/test/models/sententpiece_test_model.model");
         try (SpTokenizer tokenizer = new SpTokenizer(modelPath)) {
             SpTextEmbedding embedding = SpTextEmbedding.from(tokenizer);
diff --git a/extensions/sentencepiece/src/test/java/ai/djl/sentencepiece/SpTokenizerTest.java b/extensions/sentencepiece/src/test/java/ai/djl/sentencepiece/SpTokenizerTest.java
index e14360390..155290e37 100644
--- a/extensions/sentencepiece/src/test/java/ai/djl/sentencepiece/SpTokenizerTest.java
+++ b/extensions/sentencepiece/src/test/java/ai/djl/sentencepiece/SpTokenizerTest.java
@@ -39,8 +39,6 @@ public class SpTokenizerTest {
 
     @Test
     public void testLoadFromBytes() throws IOException {
-        TestRequirements.notWindows();
-
         Path modelPath = Paths.get("build/test/models/sententpiece_test_model.model");
         byte[] bytes = Files.readAllBytes(modelPath);
         try (SpTokenizer tokenizer = new SpTokenizer(bytes)) {
diff --git a/extensions/sentencepiece/src/test/java/ai/djl/sentencepiece/SpVocabularyTest.java b/extensions/sentencepiece/src/test/java/ai/djl/sentencepiece/SpVocabularyTest.java
index ec3eaddd6..fbc4d4970 100644
--- a/extensions/sentencepiece/src/test/java/ai/djl/sentencepiece/SpVocabularyTest.java
+++ b/extensions/sentencepiece/src/test/java/ai/djl/sentencepiece/SpVocabularyTest.java
@@ -13,7 +13,6 @@
 
 package ai.djl.sentencepiece;
 
-import ai.djl.testing.TestRequirements;
 import ai.djl.training.util.DownloadUtils;
 import java.io.IOException;
 import java.nio.file.Files;
@@ -37,8 +36,6 @@ public class SpVocabularyTest {
 
     @Test
     public void testTokenIdConversion() throws IOException {
-        TestRequirements.notWindows();
-
         Path modelPath = Paths.get("build/test/models");
         String prefix = "sententpiece_test_model";
         SpTokenizer tokenizer = new SpTokenizer(modelPath, prefix);
diff --git a/extensions/tokenizers/src/main/resources/.empty b/extensions/tokenizers/src/main/resources/.empty
new file mode 100644
index 000000000..e69de29bb
diff --git a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
new file mode 100644
index 000000000..4f559e430
--- /dev/null
+++ b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
@@ -0,0 +1,66 @@
+/*
+ * Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+
+package ai.djl.huggingface.tokenizers;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+public class HuggingFaceTokenizerTest {
+
+    @Test
+    public void testTokenizer() {
+        String input = "Hello, y'all! How are you 😁 ?";
+        String[] inputs = {"Hello, y'all!", "How are you 😁 ?"};
+
+        String[] expected = {
+            "[CLS]", "Hello", ",", "y", "'", "all", "!", "How", "are", "you", "[UNK]", "?", "[SEP]"
+        };
+
+        try (HuggingFaceTokenizer tokenizer = HuggingFaceTokenizer.newInstance("bert-base-cased")) {
+            List<String> ret = tokenizer.tokenize(input);
+            Assert.assertEquals(ret.toArray(new String[0]), expected);
+            Encoding encoding = tokenizer.encode(input);
+
+            long[] ids = {101, 8667, 117, 194, 112, 1155, 106, 1731, 1132, 1128, 100, 136, 102};
+            long[] typeIds = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
+            long[] wordIds = {-1, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, -1};
+            long[] attentionMask = {1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1};
+            long[] specialTokenMask = {1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1};
+
+            Assert.assertEquals(expected, encoding.getTokens());
+            Assert.assertEquals(ids, encoding.getIds());
+            Assert.assertEquals(typeIds, encoding.getTypeIds());
+            Assert.assertEquals(wordIds, encoding.getWordIds());
+            Assert.assertEquals(attentionMask, encoding.getAttentionMask());
+            Assert.assertEquals(specialTokenMask, encoding.getSpecialTokenMask());
+
+            Encoding[] encodings = tokenizer.batchEncode(Arrays.asList(inputs));
+            Assert.assertEquals(encodings.length, 2);
+        }
+
+        Map<String, String> options = new ConcurrentHashMap<>();
+        options.put("addSpecialTokens", "false");
+        try (HuggingFaceTokenizer tokenizer =
+                HuggingFaceTokenizer.newInstance("bert-base-cased", options)) {
+            String[] exp = {"Hello", ",", "y", "'", "all", "!", "How", "are", "you", "[UNK]", "?"};
+
+            Encoding encoding = tokenizer.encode(Arrays.asList(inputs));
+            Assert.assertEquals(encoding.getTokens(), exp);
+        }
+    }
+}
diff --git a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/package-info.java b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/package-info.java
new file mode 100644
index 000000000..050714fc1
--- /dev/null
+++ b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/package-info.java
@@ -0,0 +1,14 @@
+/*
+ * Copyright 2021 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+/** Contains tests for {@link ai.djl.huggingface.tokenizers}. */
+package ai.djl.huggingface.tokenizers;