Repository: deepjavalibrary/djl
Pull request #2006 (2022-09-09 20:45:28)

[tokenizers] support stride in tokenizers
## Description ##

This PR adds support for utilizing the stride parameter in truncation. Users can now pass in the stride value and get the overflowing encodings. See https://github.com/deepjavalibrary/djl/issues/1996

One thought I had is whether we should add a flag to the tokenizer that indicates whether a user wants overflows returned. I'm not sure how often these overflows are used in practice, but I imagine there are cases where tokenization results in overflow but the user doesn't care about them. We could avoid JNI calls in these situations by adding a flag. 



Patch:
diff --git a/api/src/main/java/ai/djl/nn/recurrent/LSTM.java b/api/src/main/java/ai/djl/nn/recurrent/LSTM.java
index 71fa793ca..ce4a1c57e 100644
--- a/api/src/main/java/ai/djl/nn/recurrent/LSTM.java
+++ b/api/src/main/java/ai/djl/nn/recurrent/LSTM.java
@@ -77,6 +77,16 @@ public class LSTM extends RecurrentBlock {
             // cell
             inputs.add(input.getManager().zeros(stateShape));
         }
+        if (inputs.size() == 2) {
+            int batchIndex = batchFirst ? 0 : 1;
+            Shape stateShape =
+                    new Shape(
+                            (long) numLayers * getNumDirections(),
+                            input.size(batchIndex),
+                            stateSize);
+            // cell
+            inputs.add(input.getManager().zeros(stateShape));
+        }
         NDList outputs =
                 ex.lstm(
                         input,
diff --git a/basicdataset/README.md b/basicdataset/README.md
index b56960320..c1a326f5e 100644
--- a/basicdataset/README.md
+++ b/basicdataset/README.md
@@ -41,12 +41,12 @@ images that not supported by default JDK, you can consider add the following dep
     <dependency>
         <groupId>com.twelvemonkeys.imageio</groupId>
         <artifactId>imageio-jpeg</artifactId>
-        <version>3.5</version>
+        <version>3.8.3</version>
     </dependency>
     <dependency>
         <groupId>com.twelvemonkeys.imageio</groupId>
         <artifactId>imageio-bmp</artifactId>
-        <version>3.5</version>
+        <version>3.8.3</version>
     </dependency>
     ...
 ```
diff --git a/engines/ml/xgboost/README.md b/engines/ml/xgboost/README.md
index 75ccbaf1f..d9679f20e 100644
--- a/engines/ml/xgboost/README.md
+++ b/engines/ml/xgboost/README.md
@@ -17,7 +17,7 @@ User can only create two-dimension NDArray to form as the input.
 
 ## Documentation
 
-The latest javadocs can be found on [here](https://javadoc.io/doc/ai.djl.onnxruntime/onnxruntime-engine/latest/index.html).
+The latest javadocs can be found on [here](https://javadoc.io/doc/ai.djl.ml.xgboost/xgboost/latest/index.html).
 
 You can also build the latest javadocs locally using the following command:
 
diff --git a/engines/onnxruntime/onnxruntime-engine/README.md b/engines/onnxruntime/onnxruntime-engine/README.md
index fd4427ca7..9ac022374 100644
--- a/engines/onnxruntime/onnxruntime-engine/README.md
+++ b/engines/onnxruntime/onnxruntime-engine/README.md
@@ -3,7 +3,7 @@
 ## Overview
 This module contains the Deep Java Library (DJL) EngineProvider for ONNX Runtime.
 
-It is based off the [ONNX Runtime Deep Learning Framework](https://microsoft.github.io/onnxruntime/).
+It is based off the [ONNX Runtime Deep Learning Framework](https://onnxruntime.ai/).
 
 
 We don't recommend developers use classes within this module directly.
diff --git a/extensions/aws-ai/README.md b/extensions/aws-ai/README.md
index 9d488432a..a63cf8b92 100644
--- a/extensions/aws-ai/README.md
+++ b/extensions/aws-ai/README.md
@@ -41,7 +41,7 @@ If you want to customize your AWS credentials and region, you can manually regis
 
 ## Documentation
 
-The latest javadocs can be found on [here](https://javadoc.io/ai.djl.aws/aws-ai/latest/index.html).
+The latest javadocs can be found on [here](https://javadoc.io/doc/ai.djl.aws/aws-ai/latest/index.html).
 
 You can also build the latest javadocs locally using the following command:
 
diff --git a/extensions/hadoop/README.md b/extensions/hadoop/README.md
index 23a9fc4e5..5286bd49e 100644
--- a/extensions/hadoop/README.md
+++ b/extensions/hadoop/README.md
@@ -34,7 +34,7 @@ If you want to customize your Hadoop configuration, you can manually register a
 
 ## Documentation
 
-The latest javadocs can be found on [here](https://javadoc.io/ai.djl.hadoop/hadoop/latest/index.html).
+The latest javadocs can be found on [here](https://javadoc.io/doc/ai.djl.hadoop/hadoop/latest/index.html).
 
 You can also build the latest javadocs locally using the following command:
 
diff --git a/extensions/hadoop/build.gradle b/extensions/hadoop/build.gradle
index f8236d4b9..38a2760c3 100644
--- a/extensions/hadoop/build.gradle
+++ b/extensions/hadoop/build.gradle
@@ -9,6 +9,7 @@ dependencies {
 
     testImplementation project(":engines:mxnet:mxnet-engine")
     testImplementation "org.apache.hadoop:hadoop-minicluster:${hadoop_version}"
+    testImplementation "org.mockito:mockito-core:${mockito_version}"
     testImplementation(project(":testing"))
 
     testImplementation "org.testng:testng:${testng_version}"
diff --git a/extensions/sentencepiece/README.md b/extensions/sentencepiece/README.md
index e1f5c711d..2643b16ea 100644
--- a/extensions/sentencepiece/README.md
+++ b/extensions/sentencepiece/README.md
@@ -2,7 +2,7 @@
 
 This module contains the NLP support with SentencePiece implementation.
 
-This is an implementation from [SentencePiece's](https://github.com/google/sentencepiece) C++ API.
+This is an implementation from [SentencePiece](https://github.com/google/sentencepiece)'s C++ API.
 
 ## Documentation
 
diff --git a/extensions/timeseries/README.md b/extensions/timeseries/README.md
index 8c32904d8..936a25d46 100644
--- a/extensions/timeseries/README.md
+++ b/extensions/timeseries/README.md
@@ -1,6 +1,6 @@
 # TimeSeries support
 
-This module contains the time series model support extension with [gluon-ts](https://github.com/awslabs/gluon-ts).
+This module contains the time series model support extension with [GluonTS](https://github.com/awslabs/gluonts).
 
 Right now, the package provides the `BaseTimeSeriesTranslator` and transform package that allows you to do inference from your pre-trained time series model.
 
@@ -81,7 +81,7 @@ The following pseudocode demonstrates how to create a `DeepARTranslator` with `a
 
 If you want to customize your own time series model translator, you can easily use the transform package for your data preprocess.
 
-See [examples](../src/main/java/ai/djl/timeseries/examples) for more details.
+See [examples](../../examples) for more details.
 
 We plan to add the following features in the future:
 
diff --git a/extensions/timeseries/src/main/java/ai/djl/timeseries/timefeature/Lag.java b/extensions/timeseries/src/main/java/ai/djl/timeseries/timefeature/Lag.java
index eec33dbef..e45fbf3b9 100644
--- a/extensions/timeseries/src/main/java/ai/djl/timeseries/timefeature/Lag.java
+++ b/extensions/timeseries/src/main/java/ai/djl/timeseries/timefeature/Lag.java
@@ -31,17 +31,63 @@ public final class Lag {
      * correspond to the same `season` (+/- `delta`) in previous `k` cycles. Here `delta` and `k`
      * are chosen according to the existing code.
      *
-     * @param freqStr Frequency string of the form [multiple][granularity] such as "12H", "1D" etc.
+     * @param freqStr Frequency string of the form [multiple][granularity] such as "12H", "1D", "6T"
+     *     etc.
      * @param lagUb The maximum value for a lag
      * @return a list of lags
      */
     public static List<Integer> getLagsForFreq(String freqStr, int lagUb) {
         List<List<Integer>> lags;
-        if ("D".equals(freqStr)) {
-            lags = makeLagsForDay(1);
-            lags.addAll(makeLagsForWeek(1f / 7f));
-        } else {
-            throw new IllegalArgumentException("invalid frequency: freq now must be D");
+        TimeOffset timeOffset = TimeOffset.toOffset(freqStr);
+        switch (timeOffset.getName()) {
+            case "Q":
+                if (timeOffset.getMultipleOfTimeOffset() != 1) {
+                    throw new IllegalArgumentException(
+                            "Only multiple 1 is supported for quarterly. Use x month instead.");
+                }
+                lags = makeLagsForTimeStep(3, 12, 1, 3. * timeOffset.getMultipleOfTimeOffset());
+                break;
+            case "M":
+                lags = makeLagsForTimeStep(3, 12, 1, timeOffset.getMultipleOfTimeOffset());
+                break;
+            case "W":
+                lags = makeLagsForTimeStep(3, 52, 1, timeOffset.getMultipleOfTimeOffset());
+                break;
+            case "D":
+                lags = makeLagsForTimeStep(4, 7, 1, timeOffset.getMultipleOfTimeOffset());
+                lags.addAll(
+                        makeLagsForTimeStep(3, 52, 1, timeOffset.getMultipleOfTimeOffset() / 7.));
+                break;
+            case "H":
+                lags = makeLagsForTimeStep(7, 24, 1, timeOffset.getMultipleOfTimeOffset());
+                lags.addAll(
+                        makeLagsForTimeStep(4, 7, 1, timeOffset.getMultipleOfTimeOffset() / 24.));
+                lags.addAll(
+                        makeLagsForTimeStep(
+                                3, 52, 1, (timeOffset.getMultipleOfTimeOffset() / (24. * 7.))));
+                break;
+            case "min":
+            case "T":
+                lags = makeLagsForTimeStep(3, 60, 2, timeOffset.getMultipleOfTimeOffset());
+                lags.addAll(
+                        makeLagsForTimeStep(7, 24, 1, timeOffset.getMultipleOfTimeOffset() / 60.));
+                lags.addAll(
+                        makeLagsForTimeStep(
+                                4, 7, 1, timeOffset.getMultipleOfTimeOffset() / (60. * 24.)));
+                lags.addAll(
+                        makeLagsForTimeStep(
+                                3, 52, 1, timeOffset.getMultipleOfTimeOffset() / (60. * 24. * 7.)));
+                break;
+            case "S":
+                lags = makeLagsForTimeStep(3, 60, 2, timeOffset.getMultipleOfTimeOffset());
+                lags.addAll(
+                        makeLagsForTimeStep(3, 60, 2, timeOffset.getMultipleOfTimeOffset() / 60.));
+                lags.addAll(
+                        makeLagsForTimeStep(
+                                7, 24, 1, timeOffset.getMultipleOfTimeOffset() / (60. * 60.)));
+                break;
+            default:
+                throw new IllegalArgumentException("invalid frequency");
         }
         List<Integer> ret = new ArrayList<>();
         for (List<Integer> subList : lags) {
@@ -72,25 +118,21 @@ public final class Lag {
         return getLagsForFreq(freqStr, 1200);
     }
 
-    private static List<List<Integer>> makeLagsForDay(float multiple) {
-        int numCycles = 4;
+    private static List<List<Integer>> makeLagsForTimeStep(
+            int numCycles, int multiplier, int delta, double multiple) {
         List<List<Integer>> ret = new ArrayList<>();
         for (int i = 1; i < numCycles + 1; i++) {
-            ret.add(makeLags((int) (i * 7 / multiple), 1));
+            ret.add(makeLags((int) (i * multiplier / multiple), delta));
         }
-        ret.add(makeLags((int) (30 / multiple), 1));
-        return ret;
-    }
 
-    private static List<List<Integer>> makeLagsForWeek(float multiple) {
-        int numCycles = 3;
-        List<List<Integer>> ret = new ArrayList<>();
-        for (int i = 1; i < numCycles + 1; i++) {
-            ret.add(makeLags((int) (i * 52 / multiple), 1));
+        if (multiplier == 7) {
+            ret.add(makeLags((int) (30 / multiple), 1));
+        } else if (multiplier == 52) {
+            ret.add(
+                    Arrays.asList(
+                            (int) (4. / multiple), (int) (8. / multiple), (int) (12. / multiple)));
         }
-        ret.add(
-                Arrays.asList(
-                        (int) (4f / multiple), (int) (8f / multiple), (int) (12f / multiple)));
+
         return ret;
     }
 
diff --git a/extensions/timeseries/src/main/java/ai/djl/timeseries/timefeature/TimeFeature.java b/extensions/timeseries/src/main/java/ai/djl/timeseries/timefeature/TimeFeature.java
index 21e7c39a5..11dec0346 100644
--- a/extensions/timeseries/src/main/java/ai/djl/timeseries/timefeature/TimeFeature.java
+++ b/extensions/timeseries/src/main/java/ai/djl/timeseries/timefeature/TimeFeature.java
@@ -145,7 +145,8 @@ public final class TimeFeature {
      */
     public static List<BiFunction<NDManager, List<LocalDateTime>, NDArray>> timeFeaturesFromFreqStr(
             String freqStr) {
-        return FEATURES_BY_OFFSETS.get(freqStr);
+        TimeOffset timeOffset = TimeOffset.toOffset(freqStr);
+        return FEATURES_BY_OFFSETS.get(timeOffset.getName());
     }
 
     private static Map<String, List<BiFunction<NDManager, List<LocalDateTime>, NDArray>>> init() {
diff --git a/extensions/timeseries/src/main/java/ai/djl/timeseries/timefeature/TimeOffset.java b/extensions/timeseries/src/main/java/ai/djl/timeseries/timefeature/TimeOffset.java
new file mode 100644
index 000000000..79b2e81dc
--- /dev/null
+++ b/extensions/timeseries/src/main/java/ai/djl/timeseries/timefeature/TimeOffset.java
@@ -0,0 +1,87 @@
+/*
+ * Copyright 2022 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+
+package ai.djl.timeseries.timefeature;
+
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
+
+/** This is a class use to get multiple and granularity from frequency string. */
+public class TimeOffset {
+
+    private static String oPattern = "(?<multiple>\\d*)(?<granularity>\\w+)";
+
+    private String name;
+    private int n;
+
+    /**
+     * Constructs a new {@code DateOffset} instance.
+     *
+     * @param name offset granularity including "D", "M" etc
+     * @param n offset multiple
+     */
+    public TimeOffset(String name, int n) {
+        this.name = name;
+        this.n = n;
+    }
+
+    /**
+     * Return {@code TimeOffset} object from frequency string.
+     *
+     * @param freqStr Frequency string of the form [multiple][granularity] such as "12H", "1D" etc.
+     * @return a TimeOffset containing multiple and granularity
+     */
+    public static TimeOffset toOffset(String freqStr) {
+        Matcher matcher = Pattern.compile(oPattern).matcher(freqStr);
+        matcher.find();
+        String name = matcher.group("granularity");
+        if ("".equals(name)) {
+            throw new IllegalArgumentException("Invalid frequency");
+        }
+        name = "min".equals(name) ? "T" : name;
+
+        String multiple = matcher.group("multiple");
+        if ("".equals(multiple)) {
+            multiple = "1";
+        }
+        int n = Integer.parseInt(multiple);
+        return new TimeOffset(name, n);
+    }
+
+    /**
+     * Return the granularity name of {@code TimeOffset}.
+     *
+     * @return the granularity name of {@code TimeOffset}.
+     */
+    public String getName() {
+        return name;
+    }
+
+    /**
+     * Return the multiple of {@code TimeOffset}.
+     *
+     * @return the multiple of {@code TimeOffset}
+     */
+    public double getMultipleOfTimeOffset() {
+        return n;
+    }
+
+    /**
+     * Return the formatted frequency string.
+     *
+     * @return the formatted frequency string
+     */
+    public String toFreqStr() {
+        return n + name;
+    }
+}
diff --git a/extensions/tokenizers/rust/src/lib.rs b/extensions/tokenizers/rust/src/lib.rs
index e2a62f95a..cda085df2 100644
--- a/extensions/tokenizers/rust/src/lib.rs
+++ b/extensions/tokenizers/rust/src/lib.rs
@@ -377,6 +377,25 @@ pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_
     array
 }
 
+#[no_mangle]
+pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_getOverflowing(
+    env: JNIEnv,
+    _: JObject,
+    handle: jlong,
+) -> jlongArray {
+    let encoding = cast_handle::<Encoding>(handle);
+    let handles = encoding
+            .get_overflowing()
+            .clone()
+            .into_iter()
+            .map(|c| to_handle(c))
+            .collect::<Vec<_>>();
+    let size = handles.len() as jsize;
+    let ret = env.new_long_array(size).unwrap();
+    env.set_long_array_region(ret, 0, &handles).unwrap();
+    ret
+}
+
 #[no_mangle]
 pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_decode(
     env: JNIEnv,
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/Encoding.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/Encoding.java
index e7c48f504..2979b75e3 100644
--- a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/Encoding.java
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/Encoding.java
@@ -24,6 +24,7 @@ public class Encoding {
     private long[] attentionMask;
     private long[] specialTokenMask;
     private CharSpan[] charTokenSpans;
+    private Encoding[] overflowing;
 
     protected Encoding(
             long[] ids,
@@ -32,7 +33,8 @@ public class Encoding {
             long[] wordIds,
             long[] attentionMask,
             long[] specialTokenMask,
-            CharSpan[] charTokenSpans) {
+            CharSpan[] charTokenSpans,
+            Encoding[] overflowing) {
         this.ids = ids;
         this.typeIds = typeIds;
         this.tokens = tokens;
@@ -40,6 +42,7 @@ public class Encoding {
         this.attentionMask = attentionMask;
         this.specialTokenMask = specialTokenMask;
         this.charTokenSpans = charTokenSpans;
+        this.overflowing = overflowing;
     }
 
     /**
@@ -104,4 +107,13 @@ public class Encoding {
     public CharSpan[] getCharTokenSpans() {
         return charTokenSpans;
     }
+
+    /**
+     * Returns an array of overflowing encodings.
+     *
+     * @return the array of overflowing encodings
+     */
+    public Encoding[] getOverflowing() {
+        return overflowing;
+    }
 }
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
index bbf4e35bd..fd05caea6 100644
--- a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
@@ -419,10 +419,23 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
         long[] attentionMask = TokenizersLibrary.LIB.getAttentionMask(encoding);
         long[] specialTokenMask = TokenizersLibrary.LIB.getSpecialTokenMask(encoding);
         CharSpan[] charSpans = TokenizersLibrary.LIB.getTokenCharSpans(encoding);
+        long[] overflowingHandles = TokenizersLibrary.LIB.getOverflowing(encoding);
+
+        Encoding[] overflowing = new Encoding[overflowingHandles.length];
+        for (int i = 0; i < overflowingHandles.length; ++i) {
+            overflowing[i] = toEncoding(overflowingHandles[i]);
+        }
 
         TokenizersLibrary.LIB.deleteEncoding(encoding);
         return new Encoding(
-                ids, typeIds, tokens, wordIds, attentionMask, specialTokenMask, charSpans);
+                ids,
+                typeIds,
+                tokens,
+                wordIds,
+                attentionMask,
+                specialTokenMask,
+                charSpans,
+                overflowing);
     }
 
     /** {@inheritDoc} */
@@ -621,6 +634,18 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
             return this;
         }
 
+        /**
+         * Sets the stride to use in overflow overlap when truncating sequences longer than the
+         * model supports.
+         *
+         * @param stride the number of tokens to overlap when truncating long sequences
+         * @return this builder
+         */
+        public Builder optStride(int stride) {
+            options.put("stride", String.valueOf(stride));
+            return this;
+        }
+
         /**
          * Configures the builder with the arguments.
          *
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java
index caada24c6..0e775d66c 100644
--- a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/jni/TokenizersLibrary.java
@@ -51,6 +51,8 @@ public final class TokenizersLibrary {
 
     public native CharSpan[] getTokenCharSpans(long encoding);
 
+    public native long[] getOverflowing(long encoding);
+
     public native String decode(long tokenizer, long[] ids, boolean addSpecialTokens);
 
     public native String getTruncationStrategy(long tokenizer);
diff --git a/gradle.properties b/gradle.properties
index 7d6b5547f..d2fda4d56 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -31,16 +31,17 @@ commons_cli_version=1.5.0
 commons_compress_version=1.21
 commons_csv_version=1.9.0
 commons_logging_version=1.2
-gson_version=2.9.0
-jna_version=5.11.0
+gson_version=2.9.1
+jna_version=5.12.1
 slf4j_version=1.7.36
 log4j_slf4j_version=2.18.0
-awssdk_version=2.17.151
-hadoop_version=3.3.1
+awssdk_version=2.17.269
+hadoop_version=3.3.4
 javacpp_version=1.5.6
-protobuf_version=3.19.2
+protobuf_version=3.21.5
 tablesaw_version=0.43.1
 
 antlr_version=4.9.3
 testng_version=7.5
 junit_version=4.13.2
+mockito_version=4.8.0


Test patch:
diff --git a/extensions/timeseries/src/test/java/ai/djl/timeseries/timefeature/TimeFeatureTest.java b/extensions/timeseries/src/test/java/ai/djl/timeseries/timefeature/TimeFeatureTest.java
index 3f8789180..13d8db87e 100644
--- a/extensions/timeseries/src/test/java/ai/djl/timeseries/timefeature/TimeFeatureTest.java
+++ b/extensions/timeseries/src/test/java/ai/djl/timeseries/timefeature/TimeFeatureTest.java
@@ -12,4 +12,131 @@
  */
 package ai.djl.timeseries.timefeature;
 
-public class TimeFeatureTest {}
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.ConcurrentHashMap;
+
+public class TimeFeatureTest {
+
+    @Test
+    public void testGetLagsForFreq() {
+        Map<String, List<Integer>> expectedLags = new ConcurrentHashMap<>();
+        expectedLags.put(
+                "4S",
+                Arrays.asList(
+                        1, 2, 3, 4, 5, 6, 7, 13, 14, 15, 16, 17, 28, 29, 30, 31, 32, 43, 44, 45, 46,
+                        47, 898, 899, 900, 901, 902));
+        expectedLags.put(
+                "min",
+                Arrays.asList(
+                        1, 2, 3, 4, 5, 6, 7, 58, 59, 60, 61, 62, 118, 119, 120, 121, 122, 178, 179,
+                        180, 181, 182));
+        expectedLags.put(
+                "15min",
+                Arrays.asList(
+                        1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 95, 96, 97, 191, 192, 193,
+                        287, 288, 289, 383, 384, 385, 479, 480, 481, 575, 576, 577, 671, 672, 673));
+        expectedLags.put(
+                "30min",
+                Arrays.asList(
+                        1, 2, 3, 4, 5, 6, 7, 8, 47, 48, 49, 95, 96, 97, 143, 144, 145, 191, 192,
+                        193, 239, 240, 241, 287, 288, 289, 335, 336, 337, 671, 672, 673, 1007, 1008,
+                        1009));
+        expectedLags.put(
+                "59min",
+                Arrays.asList(
+                        1, 2, 3, 4, 5, 6, 7, 23, 24, 25, 47, 48, 49, 72, 73, 74, 96, 97, 98, 121,
+                        122, 123, 145, 146, 147, 169, 170, 171, 340, 341, 342, 511, 512, 513, 682,
+                        683, 684, 731, 732, 733));
+        expectedLags.put(
+                "61min",
+                Arrays.asList(
+                        1, 2, 3, 4, 5, 6, 7, 22, 23, 24, 46, 47, 48, 69, 70, 71, 93, 94, 95, 117,
+                        118, 119, 140, 141, 142, 164, 165, 166, 329, 330, 331, 494, 495, 496, 659,
+                        660, 661, 707, 708, 709));
+        expectedLags.put(
+                "H",
+                Arrays.asList(
+                        1, 2, 3, 4, 5, 6, 7, 23, 24, 25, 47, 48, 49, 71, 72, 73, 95, 96, 97, 119,
+                        120, 121, 143, 144, 145, 167, 168, 169, 335, 336, 337, 503, 504, 505, 671,
+                        672, 673, 719, 720, 721));
+        expectedLags.put(
+                "6H",
+                Arrays.asList(
+                        1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 15, 16, 17, 19, 20, 21, 23, 24, 25,
+                        27, 28, 29, 55, 56, 57, 83, 84, 85, 111, 112, 113, 119, 120, 121, 224,
+                        336));
+        expectedLags.put(
+                "12H",
+                Arrays.asList(
+                        1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 27, 28, 29, 41, 42, 43,
+                        55, 56, 57, 59, 60, 61, 112, 168, 727, 728, 729));
+        expectedLags.put(
+                "23H",
+                Arrays.asList(
+                        1, 2, 3, 4, 5, 6, 7, 8, 13, 14, 15, 20, 21, 22, 28, 29, 30, 31, 32, 58, 87,
+                        378, 379, 380, 758, 759, 760, 1138, 1139, 1140));
+        expectedLags.put(
+                "25H",
+                Arrays.asList(
+                        1, 2, 3, 4, 5, 6, 7, 12, 13, 14, 19, 20, 21, 25, 26, 27, 28, 29, 53, 80,
+                        348, 349, 350, 697, 698, 699, 1047, 1048, 1049));
+        expectedLags.put(
+                "D",
+                Arrays.asList(
+                        1, 2, 3, 4, 5, 6, 7, 8, 13, 14, 15, 20, 21, 22, 27, 28, 29, 30, 31, 56, 84,
+                        363, 364, 365, 727, 728, 729, 1091, 1092, 1093));
+        expectedLags.put(
+                "2D",
+                Arrays.asList(
+                        1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 13, 14, 15, 16, 28, 42, 181, 182, 183,
+                        363, 364, 365, 545, 546, 547));
+        expectedLags.put(
+                "6D",
+                Arrays.asList(
+                        1, 2, 3, 4, 5, 6, 7, 9, 14, 59, 60, 61, 120, 121, 122, 181, 182, 183));
+        expectedLags.put(
+                "W",
+                Arrays.asList(
+                        1, 2, 3, 4, 5, 6, 7, 8, 12, 51, 52, 53, 103, 104, 105, 155, 156, 157));
+        expectedLags.put(
+                "8D",
+                Arrays.asList(1, 2, 3, 4, 5, 6, 7, 10, 44, 45, 46, 90, 91, 92, 135, 136, 137));
+        expectedLags.put(
+                "4W", Arrays.asList(1, 2, 3, 4, 5, 6, 7, 12, 13, 14, 25, 26, 27, 38, 39, 40));
+        expectedLags.put(
+                "3W", Arrays.asList(1, 2, 3, 4, 5, 6, 7, 16, 17, 18, 33, 34, 35, 51, 52, 53));
+        expectedLags.put(
+                "5W", Arrays.asList(1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 19, 20, 21, 30, 31, 32));
+        expectedLags.put(
+                "M", Arrays.asList(1, 2, 3, 4, 5, 6, 7, 11, 12, 13, 23, 24, 25, 35, 36, 37));
+        expectedLags.put("6M", Arrays.asList(1, 2, 3, 4, 5, 6, 7));
+        expectedLags.put("12M", Arrays.asList(1, 2, 3, 4, 5, 6, 7));
+
+        // For the default multiple (1)
+        String[] defaultFreq = new String[] {"min", "H", "D", "W", "M"};
+        for (String freq : defaultFreq) {
+            expectedLags.put("1" + freq, expectedLags.get(freq));
+        }
+
+        // For frequencies that do not have unique form
+        expectedLags.put("60min", expectedLags.get("1H"));
+        expectedLags.put("24H", expectedLags.get("1D"));
+        expectedLags.put("7D", expectedLags.get("1W"));
+
+        String[] testFreqs =
+                new String[] {
+                    "4S", "min", "1min", "15min", "30min", "59min", "60min", "61min", "H", "1H",
+                    "6H", "12H", "23H", "24H", "25H", "D", "1D", "2D", "6D", "7D", "8D", "W", "1W",
+                    "3W", "4W", "5W", "M", "6M", "12M"
+                };
+        for (String freq : testFreqs) {
+            List<Integer> lags = Lag.getLagsForFreq(freq);
+            Assert.assertEquals(lags, expectedLags.get(freq));
+        }
+    }
+}
diff --git a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
index 75c4d4492..fbb6dabf8 100644
--- a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
+++ b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
@@ -282,6 +282,52 @@ public class HuggingFaceTokenizerTest {
         }
     }
 
+    @Test
+    public void testTruncationStride() throws IOException {
+        try (HuggingFaceTokenizer tokenizer =
+                HuggingFaceTokenizer.builder()
+                        .optTokenizerName("bert-base-cased")
+                        .optAddSpecialTokens(false)
+                        .optTruncation(true)
+                        .optMaxLength(3)
+                        .optStride(1)
+                        .build()) {
+            String[] inputs = {"Hello there my good friend", "How are you today"};
+            Encoding[] encodings = tokenizer.batchEncode(inputs);
+            int[] expectedOverflowEncodings = {1, 1};
+            int[][] expectedNumberOfOverflowingTokens = {{3}, {2}};
+            for (int i = 0; i < encodings.length; ++i) {
+                Assert.assertEquals(
+                        encodings[i].getOverflowing().length, expectedOverflowEncodings[i]);
+                for (int j = 0; j < expectedOverflowEncodings[i]; ++j) {
+                    Assert.assertEquals(
+                            encodings[i].getOverflowing()[j].getTokens().length,
+                            expectedNumberOfOverflowingTokens[i][j]);
+                }
+            }
+        }
+        try (HuggingFaceTokenizer tokenizer =
+                HuggingFaceTokenizer.builder()
+                        .optTokenizerName("bert-base-cased")
+                        .optAddSpecialTokens(false)
+                        .optTruncation(true)
+                        .optMaxLength(8)
+                        .optStride(2)
+                        .build()) {
+            String text = "Hello there my friend I am happy to see you";
+            String textPair = "How are you my friend";
+            Encoding[] overflowing = tokenizer.encode(text, textPair).getOverflowing();
+
+            int expectedNumberOfOverflowEncodings = 7;
+            Assert.assertEquals(overflowing.length, expectedNumberOfOverflowEncodings);
+            int[] expectedOverflowEncodingLengths = {8, 7, 8, 7, 8, 7, 7};
+            for (int i = 0; i < expectedNumberOfOverflowEncodings; ++i) {
+                Assert.assertEquals(
+                        overflowing[i].getIds().length, expectedOverflowEncodingLengths[i]);
+            }
+        }
+    }
+
     @Test
     public void testTruncationAndPaddingForPairInputs() throws IOException {
         String text = "Hello there my friend";