Repository: deepjavalibrary/djl
Pull request #2905 (2023-12-26 23:07:07)

[llama.cpp] Adds unit-test and standardize input parameters
## Description ##

Brief description of what this PR is about

- If this change is a backward incompatible change, why must this change be made?
- Interesting edge cases to note here


codecov-commenter: ## [Codecov](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) Report
Attention: `1364 lines` in your changes are missing coverage. Please review.
> Comparison is base [(`bb5073f`)](https://app.codecov.io/gh/deepjavalibrary/djl/commit/bb5073f7ffc33f1883621fc1783dbb01e70b418f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) 72.08% compared to head [(`fc0903b`)](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) 71.85%.
> Report is 940 commits behind head on master.

| [Files](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | Patch % | Lines |
|---|---|---|
| [...va/ai/djl/modality/nlp/generate/TextGenerator.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L25scC9nZW5lcmF0ZS9UZXh0R2VuZXJhdG9yLmphdmE=) | 2.81% | [276 Missing :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [.../java/ai/djl/modality/nlp/generate/SeqBatcher.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L25scC9nZW5lcmF0ZS9TZXFCYXRjaGVyLmphdmE=) | 0.75% | [132 Missing :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [...ity/nlp/generate/ContrastiveSeqBatchScheduler.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L25scC9nZW5lcmF0ZS9Db250cmFzdGl2ZVNlcUJhdGNoU2NoZWR1bGVyLmphdmE=) | 2.97% | [98 Missing :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [...i/djl/modality/nlp/generate/SeqBatchScheduler.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L25scC9nZW5lcmF0ZS9TZXFCYXRjaFNjaGVkdWxlci5qYXZh) | 9.83% | [55 Missing :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [.../java/ai/djl/modality/cv/BufferedImageFactory.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L0J1ZmZlcmVkSW1hZ2VGYWN0b3J5LmphdmE=) | 40.96% | [47 Missing and 2 partials :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [...a/ai/djl/modality/nlp/generate/StepGeneration.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L25scC9nZW5lcmF0ZS9TdGVwR2VuZXJhdGlvbi5qYXZh) | 2.04% | [48 Missing :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [api/src/main/java/ai/djl/ndarray/NDArray.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL25kYXJyYXkvTkRBcnJheS5qYXZh) | 43.42% | [39 Missing and 4 partials :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [...n/java/ai/djl/modality/cv/output/CategoryMask.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L291dHB1dC9DYXRlZ29yeU1hc2suamF2YQ==) | 22.00% | [39 Missing :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [...i/src/main/java/ai/djl/ndarray/NDArrayAdapter.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL25kYXJyYXkvTkRBcnJheUFkYXB0ZXIuamF2YQ==) | 71.21% | [31 Missing and 7 partials :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| [.../cv/translator/SemanticSegmentationTranslator.java](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary#diff-YXBpL3NyYy9tYWluL2phdmEvYWkvZGpsL21vZGFsaXR5L2N2L3RyYW5zbGF0b3IvU2VtYW50aWNTZWdtZW50YXRpb25UcmFuc2xhdG9yLmphdmE=) | 37.50% | [35 Missing :warning: ](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) |
| ... and [75 more](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary) | |

:exclamation: Your organization needs to install the [Codecov GitHub app](https://github.com/apps/codecov/installations/select_target) to enable full functionality.

<details><summary>Additional details and impacted files</summary>


```diff
@@             Coverage Diff              @@
##             master    #2905      +/-   ##
============================================
- Coverage     72.08%   71.85%   -0.24%     
- Complexity     5126     7220    +2094     
============================================
  Files           473      718     +245     
  Lines         21970    32307   +10337     
  Branches       2351     3371    +1020     
============================================
+ Hits          15838    23215    +7377     
- Misses         4925     7489    +2564     
- Partials       1207     1603     +396     
```



</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/deepjavalibrary/djl/pull/2905?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=deepjavalibrary).


Patch:
diff --git a/api/src/main/java/ai/djl/modality/cv/translator/YoloV8Translator.java b/api/src/main/java/ai/djl/modality/cv/translator/YoloV8Translator.java
index dc160ba75..d47f7a4a1 100644
--- a/api/src/main/java/ai/djl/modality/cv/translator/YoloV8Translator.java
+++ b/api/src/main/java/ai/djl/modality/cv/translator/YoloV8Translator.java
@@ -1,103 +1,124 @@
-/*
- * Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
- *
- * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
- * with the License. A copy of the License is located at
- *
- * http://aws.amazon.com/apache2.0/
- *
- * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
- * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
- * and limitations under the License.
- */
-package ai.djl.modality.cv.translator;
-
-import ai.djl.modality.cv.output.DetectedObjects;
-import ai.djl.modality.cv.output.Rectangle;
-import ai.djl.ndarray.NDArray;
-import ai.djl.ndarray.NDList;
-import ai.djl.ndarray.types.DataType;
-
-import java.util.ArrayList;
-import java.util.Map;
-
-/**
- * A translator for YoloV8 models. This was tested with ONNX exported Yolo models. For details check
- * here: https://github.com/ultralytics/ultralytics
- */
-public class YoloV8Translator extends YoloV5Translator {
-
-    /**
-     * Constructs an ImageTranslator with the provided builder.
-     *
-     * @param builder the data to build with
-     */
-    protected YoloV8Translator(Builder builder) {
-        super(builder);
-    }
-
-    /**
-     * Creates a builder to build a {@code YoloV8Translator} with specified arguments.
-     *
-     * @param arguments arguments to specify builder options
-     * @return a new builder
-     */
-    public static YoloV8Translator.Builder builder(Map<String, ?> arguments) {
-        YoloV8Translator.Builder builder = new YoloV8Translator.Builder();
-        builder.configPreProcess(arguments);
-        builder.configPostProcess(arguments);
-
-        return builder;
-    }
-
-    @Override
-    protected DetectedObjects processFromBoxOutput(NDList list) {
-        NDArray features4OneImg = list.get(0);
-        int sizeClasses = classes.size();
-        long sizeBoxes = features4OneImg.size(1);
-        ArrayList<IntermediateResult> intermediateResults = new ArrayList<>();
-
-        for (long b = 0; b < sizeBoxes; b++) {
-            float maxClass = 0;
-            int maxIndex = 0;
-            for (int c = 4; c < sizeClasses; c++) {
-                float classProb = features4OneImg.getFloat(c, b);
-                if (classProb > maxClass) {
-                    maxClass = classProb;
-                    maxIndex = c;
-                }
-            }
-
-            if (maxClass > threshold) {
-                float xPos = features4OneImg.getFloat(0, b); // center x
-                float yPos = features4OneImg.getFloat(1, b); // center y
-                float w = features4OneImg.getFloat(2, b);
-                float h = features4OneImg.getFloat(3, b);
-                Rectangle rect =
-                        new Rectangle(Math.max(0, xPos - w / 2), Math.max(0, yPos - h / 2), w, h);
-                intermediateResults.add(
-                        new IntermediateResult(classes.get(maxIndex), maxClass, maxIndex, rect));
-            }
-        }
-
-        return nms(intermediateResults);
-    }
-
-    /** The builder for {@link YoloV8Translator}. */
-    public static class Builder extends YoloV5Translator.Builder {
-        /**
-         * Builds the translator.
-         *
-         * @return the new translator
-         */
-        @Override
-        public YoloV8Translator build() {
-            if (pipeline == null) {
-                addTransform(
-                        array -> array.transpose(2, 0, 1).toType(DataType.FLOAT32, false).div(255));
-            }
-            validate();
-            return new YoloV8Translator(this);
-        }
-    }
-}
+/*
+ * Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.modality.cv.translator;
+
+import ai.djl.modality.cv.output.DetectedObjects;
+import ai.djl.modality.cv.output.Rectangle;
+import ai.djl.ndarray.NDArray;
+import ai.djl.ndarray.NDList;
+import ai.djl.ndarray.types.DataType;
+import ai.djl.ndarray.types.Shape;
+import ai.djl.translate.ArgumentsUtil;
+
+import java.util.ArrayList;
+import java.util.Map;
+
+/**
+ * A translator for YoloV8 models. This was tested with ONNX exported Yolo models. For details check
+ * here: https://github.com/ultralytics/ultralytics
+ */
+public class YoloV8Translator extends YoloV5Translator {
+
+    private int maxBoxes;
+
+    /**
+     * Constructs an ImageTranslator with the provided builder.
+     *
+     * @param builder the data to build with
+     */
+    protected YoloV8Translator(Builder builder) {
+        super(builder);
+        maxBoxes = builder.maxBox;
+    }
+
+    /**
+     * Creates a builder to build a {@code YoloV8Translator} with specified arguments.
+     *
+     * @param arguments arguments to specify builder options
+     * @return a new builder
+     */
+    public static YoloV8Translator.Builder builder(Map<String, ?> arguments) {
+        YoloV8Translator.Builder builder = new YoloV8Translator.Builder();
+        builder.configPreProcess(arguments);
+        builder.configPostProcess(arguments);
+
+        return builder;
+    }
+
+    /** {@inheritDoc} */
+    @Override
+    protected DetectedObjects processFromBoxOutput(NDList list) {
+        NDArray rawResult = list.get(0);
+        NDArray reshapedResult = rawResult.transpose();
+        Shape shape = reshapedResult.getShape();
+        float[] buf = reshapedResult.toFloatArray();
+        int numberRows = Math.toIntExact(shape.get(0));
+        int nClasses = Math.toIntExact(shape.get(1));
+
+        ArrayList<IntermediateResult> intermediateResults = new ArrayList<>();
+        // reverse order search in heap; searches through #maxBoxes for optimization when set
+        for (int i = numberRows - 1; i > numberRows - maxBoxes; --i) {
+            int index = i * nClasses;
+            float maxClassProb = -1f;
+            int maxIndex = -1;
+            for (int c = 4; c < nClasses; c++) {
+                float classProb = buf[index + c];
+                if (classProb > maxClassProb) {
+                    maxClassProb = classProb;
+                    maxIndex = c;
+                }
+            }
+
+            if (maxClassProb > threshold) {
+                float xPos = buf[index]; // center x
+                float yPos = buf[index + 1]; // center y
+                float w = buf[index + 2];
+                float h = buf[index + 3];
+                Rectangle rect =
+                        new Rectangle(Math.max(0, xPos - w / 2), Math.max(0, yPos - h / 2), w, h);
+                intermediateResults.add(
+                        new IntermediateResult(
+                                classes.get(maxIndex), maxClassProb, maxIndex, rect));
+            }
+        }
+        return nms(intermediateResults);
+    }
+
+    /** The builder for {@link YoloV8Translator}. */
+    public static class Builder extends YoloV5Translator.Builder {
+
+        private int maxBox = 8400;
+
+        /**
+         * Builds the translator.
+         *
+         * @return the new translator
+         */
+        @Override
+        public YoloV8Translator build() {
+            if (pipeline == null) {
+                addTransform(
+                        array -> array.transpose(2, 0, 1).toType(DataType.FLOAT32, false).div(255));
+            }
+            validate();
+            return new YoloV8Translator(this);
+        }
+
+        /** {@inheritDoc} */
+        @Override
+        protected void configPostProcess(Map<String, ?> arguments) {
+            super.configPostProcess(arguments);
+            maxBox = ArgumentsUtil.intValue(arguments, "maxBox", 8400);
+        }
+    }
+}
diff --git a/build.gradle b/build.gradle
index 4b8b20cd5..ca6f7e681 100644
--- a/build.gradle
+++ b/build.gradle
@@ -44,6 +44,7 @@ configure(javaProjects()) {
         targetCompatibility = JavaVersion.VERSION_11
         options.compilerArgs << "-proc:none" << "-Xlint:all,-options,-static,-removal" << "-Werror"
     }
+    javadoc.options.addStringOption("Xdoclint:none", "-quiet")
 
     apply plugin: 'eclipse'
 
diff --git a/engines/llama/build.gradle b/engines/llama/build.gradle
index 5a7af5384..9e5d4622e 100644
--- a/engines/llama/build.gradle
+++ b/engines/llama/build.gradle
@@ -20,6 +20,7 @@ processResources {
                 "linux-aarch64/libllama.so",
                 "osx-x86_64/libdjl_llama.dylib",
                 "osx-x86_64/libllama.dylib",
+                "osx-x86_64/ggml-metal.metal",
                 "osx-aarch64/libdjl_llama.dylib",
                 "osx-aarch64/libllama.dylib",
                 "osx-aarch64/ggml-metal.metal",
diff --git a/engines/llama/src/main/java/ai/djl/llama/engine/LlamaInput.java b/engines/llama/src/main/java/ai/djl/llama/engine/LlamaInput.java
new file mode 100644
index 000000000..4b4d332fc
--- /dev/null
+++ b/engines/llama/src/main/java/ai/djl/llama/engine/LlamaInput.java
@@ -0,0 +1,430 @@
+/*
+ * Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.llama.engine;
+
+import ai.djl.llama.jni.InputParameters;
+
+import com.google.gson.annotations.SerializedName;
+
+import java.util.Map;
+
+/** A class hold input data for Llama model. */
+public class LlamaInput {
+
+    private String inputs;
+    private String prefix;
+    private String suffix;
+    private Parameters parameters;
+
+    /**
+     * Returns the input prompt.
+     *
+     * @return the input prompt
+     */
+    public String getInputs() {
+        return inputs;
+    }
+
+    /**
+     * Sets the input prompt.
+     *
+     * @param inputs the input prompt
+     */
+    public void setInputs(String inputs) {
+        this.inputs = inputs;
+    }
+
+    /**
+     * Returns the prompt prefix.
+     *
+     * @return the prompt prefix
+     */
+    public String getPrefix() {
+        return prefix;
+    }
+
+    /**
+     * Sets the prompt prefix.
+     *
+     * @param prefix the prompt prefix
+     */
+    public void setPrefix(String prefix) {
+        this.prefix = prefix;
+    }
+
+    /**
+     * Returns the prompt suffix.
+     *
+     * @return the prompt suffix
+     */
+    public String getSuffix() {
+        return suffix;
+    }
+
+    /**
+     * Sets the prompt suffix.
+     *
+     * @param suffix the prompt suffix
+     */
+    public void setSuffix(String suffix) {
+        this.suffix = suffix;
+    }
+
+    /**
+     * Returns the input parameters.
+     *
+     * @return the input parameters
+     */
+    public Parameters getParameters() {
+        if (parameters == null) {
+            parameters = new Parameters();
+        }
+        return parameters;
+    }
+
+    /**
+     * Sets the input parameters.
+     *
+     * @param parameters the input parameters
+     */
+    public void setParameters(Parameters parameters) {
+        this.parameters = parameters;
+    }
+
+    /** The input parameters class. */
+    public static final class Parameters {
+
+        @SerializedName("max_new_tokens")
+        private int nPredict;
+
+        @SerializedName("number_keep")
+        private int nKeep;
+
+        @SerializedName("number_probabilities")
+        private int nProbs;
+
+        @SerializedName("top_k")
+        private int topK;
+
+        @SerializedName("top_p")
+        private float topP;
+
+        @SerializedName("tfs_z")
+        private float tfsZ;
+
+        @SerializedName("typical_p")
+        private float typicalP;
+
+        @SerializedName("temperature")
+        private float temperature;
+
+        @SerializedName("repeat_penalty")
+        private float repeatPenalty;
+
+        @SerializedName("repeat_last_n")
+        private int repeatLastN;
+
+        @SerializedName("frequency_penalty")
+        private float frequencyPenalty;
+
+        @SerializedName("presence_penalty")
+        private float presencePenalty;
+
+        @SerializedName("penalize_nl")
+        private boolean penalizeNl;
+
+        @SerializedName("ignore_eos")
+        private boolean ignoreEos;
+
+        @SerializedName("mirostat")
+        private int mirostat;
+
+        @SerializedName("mirostat_tau")
+        private float mirostatTau;
+
+        @SerializedName("mirostat_eta")
+        private float mirostatEta;
+
+        @SerializedName("number_beams")
+        private int nBeams;
+
+        @SerializedName("seed")
+        private int seed;
+
+        @SerializedName("logit_bias")
+        private Map<Integer, Float> logitBias;
+
+        @SerializedName("grammar")
+        private String grammar;
+
+        @SerializedName("anti_prompt")
+        private String[] antiPrompt;
+
+        /**
+         * Sets the max new tokens.
+         *
+         * @param maxNewTokens the max new tokens
+         */
+        public void setMaxNewTokens(int maxNewTokens) {
+            this.nPredict = maxNewTokens;
+        }
+
+        /**
+         * Sets the number of keep.
+         *
+         * @param nKeep the number of keep
+         */
+        public void setNumberKeep(int nKeep) {
+            this.nKeep = nKeep;
+        }
+
+        /**
+         * Sets the number of probabilities.
+         *
+         * @param nProbs the number of probabilities
+         */
+        public void setNumberProbabilities(int nProbs) {
+            this.nProbs = nProbs;
+        }
+
+        /**
+         * Sets the top K.
+         *
+         * @param topK the top K
+         */
+        public void setTopK(int topK) {
+            this.topK = topK;
+        }
+
+        /**
+         * Sets the top P.
+         *
+         * @param topP the top P
+         */
+        public void setTopP(float topP) {
+            this.topP = topP;
+        }
+
+        /**
+         * Sets the tfs Z.
+         *
+         * @param tfsZ the tfs Z
+         */
+        public void setTfsZ(float tfsZ) {
+            this.tfsZ = tfsZ;
+        }
+
+        /**
+         * Sets the typical P.
+         *
+         * @param typicalP the typical P
+         */
+        public void setTypicalP(float typicalP) {
+            this.typicalP = typicalP;
+        }
+
+        /**
+         * Sets the temperature.
+         *
+         * @param temperature the temperature
+         */
+        public void setTemperature(float temperature) {
+            this.temperature = temperature;
+        }
+
+        /**
+         * Sets the repeat penalty.
+         *
+         * @param repeatPenalty the repeat penalty
+         */
+        public void setRepeatPenalty(float repeatPenalty) {
+            this.repeatPenalty = repeatPenalty;
+        }
+
+        /**
+         * Sets the repeat last N.
+         *
+         * @param repeatLastN the repeat last N
+         */
+        public void setRepeatLastN(int repeatLastN) {
+            this.repeatLastN = repeatLastN;
+        }
+
+        /**
+         * Sets the frequency penalty.
+         *
+         * @param frequencyPenalty the frequency penalty
+         */
+        public void setFrequencyPenalty(float frequencyPenalty) {
+            this.frequencyPenalty = frequencyPenalty;
+        }
+
+        /**
+         * Sets the presence penalty.
+         *
+         * @param presencePenalty the presence penalty
+         */
+        public void setPresencePenalty(float presencePenalty) {
+            this.presencePenalty = presencePenalty;
+        }
+
+        /**
+         * Sets the penalize nl.
+         *
+         * @param penalizeNl the penalize nl
+         */
+        public void setPenalizeNl(boolean penalizeNl) {
+            this.penalizeNl = penalizeNl;
+        }
+
+        /**
+         * Sets if ignore EOS.
+         *
+         * @param ignoreEos if ignore EOS
+         */
+        public void setIgnoreEos(boolean ignoreEos) {
+            this.ignoreEos = ignoreEos;
+        }
+
+        /**
+         * Sets the mirostat.
+         *
+         * @param mirostat the mirostat
+         */
+        public void setMirostat(int mirostat) {
+            this.mirostat = mirostat;
+        }
+
+        /**
+         * Sets the mirostat TAU.
+         *
+         * @param mirostatTau the mirostat TAU
+         */
+        public void setMirostatTau(float mirostatTau) {
+            this.mirostatTau = mirostatTau;
+        }
+
+        /**
+         * Sets the mirostat ETA.
+         *
+         * @param mirostatEta the mirostat ETA
+         */
+        public void setMirostatEta(float mirostatEta) {
+            this.mirostatEta = mirostatEta;
+        }
+
+        /**
+         * Sets the number of beams.
+         *
+         * @param nBeams the number of beams
+         */
+        public void setNumberBeams(int nBeams) {
+            this.nBeams = nBeams;
+        }
+
+        /**
+         * Sets the seed.
+         *
+         * @param seed the seed
+         */
+        public void setSeed(int seed) {
+            this.seed = seed;
+        }
+
+        /**
+         * Sets the logit bias.
+         *
+         * @param logitBias the logit bias
+         */
+        public void setLogitBias(Map<Integer, Float> logitBias) {
+            this.logitBias = logitBias;
+        }
+
+        /**
+         * Sets the grammar template.
+         *
+         * @param grammar the grammar template
+         */
+        public void setGrammar(String grammar) {
+            this.grammar = grammar;
+        }
+
+        /**
+         * Sets the anti prompt.
+         *
+         * @param antiPrompt the anti prompt
+         */
+        public void setAntiPrompt(String[] antiPrompt) {
+            this.antiPrompt = antiPrompt;
+        }
+
+        /**
+         * Returns the {@link InputParameters} object.
+         *
+         * @return the {@link InputParameters} object
+         */
+        public InputParameters toInputParameters() {
+            setDefaultValue();
+            return new InputParameters(
+                    nPredict,
+                    nKeep,
+                    nProbs,
+                    topK,
+                    topP,
+                    tfsZ,
+                    typicalP,
+                    temperature,
+                    repeatPenalty,
+                    repeatLastN,
+                    frequencyPenalty,
+                    presencePenalty,
+                    penalizeNl,
+                    ignoreEos,
+                    mirostat,
+                    mirostatTau,
+                    mirostatEta,
+                    nBeams,
+                    seed,
+                    logitBias,
+                    grammar,
+                    antiPrompt);
+        }
+
+        private void setDefaultValue() {
+            if (nPredict == 0) {
+                nPredict = -1;
+            }
+            if (topK == 0) {
+                topK = 40;
+            }
+            if (topP == 0) {
+                topP = 0.95f;
+            }
+            if (tfsZ == 0) {
+                tfsZ = 1f;
+            }
+            if (typicalP == 0) {
+                typicalP = 1f;
+            }
+            if (temperature == 0) {
+                temperature = 0.8f;
+            }
+            if (repeatPenalty == 0) {
+                repeatPenalty = 1.10f;
+            }
+            if (repeatLastN == 0) {
+                repeatLastN = 64;
+            }
+        }
+    }
+}
diff --git a/engines/llama/src/main/java/ai/djl/llama/engine/LlamaTranslator.java b/engines/llama/src/main/java/ai/djl/llama/engine/LlamaTranslator.java
index 52bc7345b..c8d3692b1 100644
--- a/engines/llama/src/main/java/ai/djl/llama/engine/LlamaTranslator.java
+++ b/engines/llama/src/main/java/ai/djl/llama/engine/LlamaTranslator.java
@@ -44,6 +44,8 @@ public class LlamaTranslator<I, O> implements NoBatchifyTranslator<I, O> {
     public NDList processInput(TranslatorContext ctx, I input) {
         if (input instanceof String) {
             ctx.setAttachment("out", generate((String) input));
+        } else if (input instanceof LlamaInput) {
+            ctx.setAttachment("out", generate((LlamaInput) input));
         } else if (input instanceof Input) {
             String prompt = ((Input) input).getData().getAsString();
             TokenIterator it = generate(prompt);
@@ -62,11 +64,19 @@ public class LlamaTranslator<I, O> implements NoBatchifyTranslator<I, O> {
     }
 
     private TokenIterator generate(String input) {
-        TextGenerationInput in = JsonUtils.GSON.fromJson(input, TextGenerationInput.class);
-        if (in.prefix != null && in.suffix != null) {
-            LlamaLibrary.infill(handle, in.prefix, in.prefix, in.parameters);
-        } else if (in.inputs != null && !in.inputs.isEmpty()) {
-            LlamaLibrary.generate(handle, in.inputs, in.parameters);
+        LlamaInput in = JsonUtils.GSON.fromJson(input, LlamaInput.class);
+        return generate(in);
+    }
+
+    private TokenIterator generate(LlamaInput in) {
+        InputParameters param = in.getParameters().toInputParameters();
+        String prefix = in.getPrefix();
+        String suffix = in.getSuffix();
+        String inputs = in.getInputs();
+        if (prefix != null && suffix != null) {
+            LlamaLibrary.infill(handle, prefix, prefix, param);
+        } else if (inputs != null && !inputs.isEmpty()) {
+            LlamaLibrary.generate(handle, inputs, param);
         } else {
             throw new IllegalArgumentException("Unsupported input format");
         }
@@ -91,15 +101,7 @@ public class LlamaTranslator<I, O> implements NoBatchifyTranslator<I, O> {
         @Override
         public BytesSupplier next() {
             Token token = it.next();
-            return BytesSupplier.wrapAsJson(token);
+            return BytesSupplier.wrap(JsonUtils.GSON.toJson(token) + "\n");
         }
     }
-
-    private static final class TextGenerationInput {
-
-        String inputs;
-        String prefix;
-        String suffix;
-        InputParameters parameters;
-    }
 }
diff --git a/engines/llama/src/main/java/ai/djl/llama/engine/LlamaTranslatorFactory.java b/engines/llama/src/main/java/ai/djl/llama/engine/LlamaTranslatorFactory.java
index 7a4cf5e50..089b5055b 100644
--- a/engines/llama/src/main/java/ai/djl/llama/engine/LlamaTranslatorFactory.java
+++ b/engines/llama/src/main/java/ai/djl/llama/engine/LlamaTranslatorFactory.java
@@ -35,6 +35,7 @@ public class LlamaTranslatorFactory implements TranslatorFactory, Serializable {
 
     static {
         SUPPORTED_TYPES.add(new Pair<>(String.class, TokenIterator.class));
+        SUPPORTED_TYPES.add(new Pair<>(LlamaInput.class, TokenIterator.class));
         SUPPORTED_TYPES.add(new Pair<>(Input.class, Output.class));
     }
 
diff --git a/engines/llama/src/main/java/ai/djl/llama/jni/InputParameters.java b/engines/llama/src/main/java/ai/djl/llama/jni/InputParameters.java
index 9704019a3..d13abc5ef 100644
--- a/engines/llama/src/main/java/ai/djl/llama/jni/InputParameters.java
+++ b/engines/llama/src/main/java/ai/djl/llama/jni/InputParameters.java
@@ -15,38 +15,111 @@ package ai.djl.llama.jni;
 import java.util.Map;
 
 /** A class holds input parameters. */
-@SuppressWarnings("PMD.UnusedPrivateField")
+@SuppressWarnings({"PMD.UnusedPrivateField", "PMD.UnusedAssignment"})
 public class InputParameters {
 
-    private int nPredict = -1;
+    private int nPredict;
     private int nKeep;
     private int nProbs;
-    private int topK = 40;
-    private float topP = 0.95f;
-    private float tfsZ = 1.00f;
-    private float typicalP = 1.00f;
-    private float temperature = 0.80f;
-    private float repeatPenalty = 1.10f;
-    private int repeatLastN = 64;
+    private int topK;
+    private float topP;
+    private float tfsZ;
+    private float typicalP;
+    private float temperature;
+    private float repeatPenalty;
+    private int repeatLastN;
     private float frequencyPenalty;
     private float presencePenalty;
     private boolean penalizeNl;
     private boolean ignoreEos;
     private int mirostat;
-    private float mirostatTau = 5.00f;
-    private float mirostatEta = 0.10f;
-    private int nBeams = 2;
-    private int seed = 42;
+    private float mirostatTau;
+    private float mirostatEta;
+    private int nBeams;
+    private int seed;
     private Map<Integer, Float> logitBias;
     private String grammar;
     private String[] antiPrompt;
 
     /**
-     * Returns the number of predict.
+     * Constructs new {@code InputParameters} instance.
      *
-     * @return the number of predict
+     * @param nPredict the max new tokens
+     * @param nKeep the number of keep
+     * @param nProbs the number of probabilities
+     * @param topK the top K
+     * @param topP the top P
+     * @param tfsZ the tfs Z
+     * @param typicalP the typical P
+     * @param temperature the temperature
+     * @param repeatPenalty the repeat penalty
+     * @param repeatLastN the repeat last N
+     * @param frequencyPenalty the frequency penalty
+     * @param presencePenalty the presence penalty
+     * @param penalizeNl the penalize nl
+     * @param ignoreEos the ignore EOS
+     * @param mirostat the mirostat
+     * @param mirostatTau the mirostat TAU
+     * @param mirostatEta the mirostat ETA
+     * @param nBeams the number of beams
+     * @param seed the seed
+     * @param logitBias the logit bias
+     * @param grammar the grammar
+     * @param antiPrompt the anti prompt
      */
-    public int getNumberPredict() {
+    public InputParameters(
+            int nPredict,
+            int nKeep,
+            int nProbs,
+            int topK,
+            float topP,
+            float tfsZ,
+            float typicalP,
+            float temperature,
+            float repeatPenalty,
+            int repeatLastN,
+            float frequencyPenalty,
+            float presencePenalty,
+            boolean penalizeNl,
+            boolean ignoreEos,
+            int mirostat,
+            float mirostatTau,
+            float mirostatEta,
+            int nBeams,
+            int seed,
+            Map<Integer, Float> logitBias,
+            String grammar,
+            String[] antiPrompt) {
+        this.nPredict = nPredict;
+        this.nKeep = nKeep;
+        this.nProbs = nProbs;
+        this.topK = topK;
+        this.topP = topP;
+        this.tfsZ = tfsZ;
+        this.typicalP = typicalP;
+        this.temperature = temperature;
+        this.repeatPenalty = repeatPenalty;
+        this.repeatLastN = repeatLastN;
+        this.frequencyPenalty = frequencyPenalty;
+        this.presencePenalty = presencePenalty;
+        this.penalizeNl = penalizeNl;
+        this.ignoreEos = ignoreEos;
+        this.mirostat = mirostat;
+        this.mirostatTau = mirostatTau;
+        this.mirostatEta = mirostatEta;
+        this.nBeams = nBeams;
+        this.seed = seed;
+        this.logitBias = logitBias;
+        this.grammar = grammar;
+        this.antiPrompt = antiPrompt;
+    }
+
+    /**
+     * Returns the max new tokens.
+     *
+     * @return the max new tokens
+     */
+    public int getMaxNewTokens() {
         return nPredict;
     }
 
diff --git a/engines/llama/src/main/java/ai/djl/llama/jni/Token.java b/engines/llama/src/main/java/ai/djl/llama/jni/Token.java
index 6a2b9fb00..b8d74306b 100644
--- a/engines/llama/src/main/java/ai/djl/llama/jni/Token.java
+++ b/engines/llama/src/main/java/ai/djl/llama/jni/Token.java
@@ -12,6 +12,8 @@
  */
 package ai.djl.llama.jni;
 
+import ai.djl.util.JsonUtils;
+
 import java.nio.charset.StandardCharsets;
 import java.util.Map;
 
@@ -80,6 +82,6 @@ public final class Token {
     /** {@inheritDoc} */
     @Override
     public String toString() {
-        return text;
+        return JsonUtils.GSON.toJson(this) + '\n';
     }
 }
diff --git a/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/zoo/OrtModelZoo.java b/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/zoo/OrtModelZoo.java
index 9d8037cfa..d61cb81f1 100644
--- a/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/zoo/OrtModelZoo.java
+++ b/engines/onnxruntime/onnxruntime-engine/src/main/java/ai/djl/onnxruntime/zoo/OrtModelZoo.java
@@ -31,6 +31,7 @@ public class OrtModelZoo extends ModelZoo {
     OrtModelZoo() {
         addModel(REPOSITORY.model(CV.IMAGE_CLASSIFICATION, GROUP_ID, "resnet", "0.0.1"));
         addModel(REPOSITORY.model(CV.OBJECT_DETECTION, GROUP_ID, "yolo5s", "0.0.1"));
+        addModel(REPOSITORY.model(CV.OBJECT_DETECTION, GROUP_ID, "yolov8n", "0.0.1"));
         addModel(REPOSITORY.model(Tabular.SOFTMAX_REGRESSION, GROUP_ID, "iris_flowers", "0.0.1"));
     }
 
diff --git a/engines/pytorch/pytorch-model-zoo/src/main/java/ai/djl/pytorch/zoo/PtModelZoo.java b/engines/pytorch/pytorch-model-zoo/src/main/java/ai/djl/pytorch/zoo/PtModelZoo.java
index ea70871ef..abb820cce 100644
--- a/engines/pytorch/pytorch-model-zoo/src/main/java/ai/djl/pytorch/zoo/PtModelZoo.java
+++ b/engines/pytorch/pytorch-model-zoo/src/main/java/ai/djl/pytorch/zoo/PtModelZoo.java
@@ -38,6 +38,7 @@ public class PtModelZoo extends ModelZoo {
                 REPOSITORY.model(CV.IMAGE_CLASSIFICATION, GROUP_ID, "resnet18_embedding", "0.0.1"));
         addModel(REPOSITORY.model(CV.OBJECT_DETECTION, GROUP_ID, "ssd", "0.0.1"));
         addModel(REPOSITORY.model(CV.OBJECT_DETECTION, GROUP_ID, "yolov5s", "0.0.1"));
+        addModel(REPOSITORY.model(CV.OBJECT_DETECTION, GROUP_ID, "yolov8n", "0.0.1"));
         addModel(REPOSITORY.model(NLP.QUESTION_ANSWER, GROUP_ID, "bertqa", "0.0.1"));
         addModel(REPOSITORY.model(NLP.SENTIMENT_ANALYSIS, GROUP_ID, "distilbert", "0.0.1"));
         addModel(REPOSITORY.model(CV.IMAGE_GENERATION, GROUP_ID, "biggan-deep", "0.0.1"));
diff --git a/examples/src/main/java/ai/djl/examples/inference/Yolov8Detection.java b/examples/src/main/java/ai/djl/examples/inference/Yolov8Detection.java
index 5a474c1da..3d2cfb264 100644
--- a/examples/src/main/java/ai/djl/examples/inference/Yolov8Detection.java
+++ b/examples/src/main/java/ai/djl/examples/inference/Yolov8Detection.java
@@ -16,27 +16,21 @@ import ai.djl.ModelException;
 import ai.djl.inference.Predictor;
 import ai.djl.modality.cv.Image;
 import ai.djl.modality.cv.ImageFactory;
-import ai.djl.modality.cv.output.BoundingBox;
 import ai.djl.modality.cv.output.DetectedObjects;
-import ai.djl.modality.cv.output.DetectedObjects.DetectedObject;
-import ai.djl.modality.cv.output.Rectangle;
 import ai.djl.modality.cv.translator.YoloV8TranslatorFactory;
 import ai.djl.repository.zoo.Criteria;
 import ai.djl.repository.zoo.ZooModel;
 import ai.djl.training.util.ProgressBar;
 import ai.djl.translate.TranslateException;
-import ai.djl.translate.Translator;
 
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
 
 import java.io.IOException;
+import java.io.OutputStream;
 import java.nio.file.Files;
 import java.nio.file.Path;
 import java.nio.file.Paths;
-import java.util.List;
-import java.util.Map;
-import java.util.concurrent.ConcurrentHashMap;
 
 /** An example of inference using an yolov8 model. */
 public final class Yolov8Detection {
@@ -51,82 +45,42 @@ public final class Yolov8Detection {
     }
 
     public static DetectedObjects predict() throws IOException, ModelException, TranslateException {
-        String classPath = System.getProperty("java.class.path");
-        String pathSeparator = System.getProperty("path.separator");
-        classPath = classPath.split(pathSeparator)[0];
-        Path modelPath = Paths.get(classPath + "/yolov8n.onnx");
-        Path imgPath = Paths.get(classPath + "/yolov8_test.jpg");
+        Path imgPath = Paths.get("src/test/resources/yolov8_test.jpg");
         Image img = ImageFactory.getInstance().fromFile(imgPath);
 
-        Map<String, Object> arguments = new ConcurrentHashMap<>();
-        arguments.put("width", 640);
-        arguments.put("height", 640);
-        arguments.put("resize", "true");
-        arguments.put("toTensor", true);
-        arguments.put("applyRatio", true);
-        arguments.put("threshold", 0.6f);
-        arguments.put("synsetFileName", "yolov8_synset.txt");
-
-        YoloV8TranslatorFactory yoloV8TranslatorFactory = new YoloV8TranslatorFactory();
-        Translator<Image, DetectedObjects> translator =
-                yoloV8TranslatorFactory.newInstance(
-                        Image.class, DetectedObjects.class, null, arguments);
-
         Criteria<Image, DetectedObjects> criteria =
                 Criteria.builder()
                         .setTypes(Image.class, DetectedObjects.class)
-                        .optModelPath(modelPath)
+                        .optModelUrls("djl://ai.djl.onnxruntime/yolov8n")
                         .optEngine("OnnxRuntime")
-                        .optTranslator(translator)
+                        .optArgument("width", 640)
+                        .optArgument("height", 640)
+                        .optArgument("resize", true)
+                        .optArgument("toTensor", true)
+                        .optArgument("applyRatio", true)
+                        .optArgument("threshold", 0.6f)
+                        // for performance optimization maxBox parameter can reduce number of
+                        // considered boxes from 8400
+                        .optArgument("maxBox", 1000)
+                        .optTranslatorFactory(new YoloV8TranslatorFactory())
                         .optProgress(new ProgressBar())
                         .build();
 
-        DetectedObjects detectedObjects;
-        DetectedObject detectedObject;
         try (ZooModel<Image, DetectedObjects> model = criteria.loadModel();
                 Predictor<Image, DetectedObjects> predictor = model.newPredictor()) {
-            Path outputPath = Paths.get(classPath + "/output");
+            Path outputPath = Paths.get("build/output");
             Files.createDirectories(outputPath);
 
-            detectedObjects = predictor.predict(img);
-
-            if (detectedObjects.getNumberOfObjects() > 0) {
-                List<DetectedObject> detectedObjectList = detectedObjects.items();
-                for (DetectedObject object : detectedObjectList) {
-                    detectedObject = object;
-                    BoundingBox boundingBox = detectedObject.getBoundingBox();
-                    Rectangle tectangle = boundingBox.getBounds();
-                    logger.info(
-                            detectedObject.getClassName()
-                                    + " "
-                                    + detectedObject.getProbability()
-                                    + " "
-                                    + tectangle.getX()
-                                    + " "
-                                    + tectangle.getY()
-                                    + " "
-                                    + tectangle.getWidth()
-                                    + " "
-                                    + tectangle.getHeight());
+            DetectedObjects detection = predictor.predict(img);
+            if (detection.getNumberOfObjects() > 0) {
+                img.drawBoundingBoxes(detection);
+                Path output = outputPath.resolve("yolov8_detected.png");
+                try (OutputStream os = Files.newOutputStream(output)) {
+                    img.save(os, "png");
                 }
-
-                saveBoundingBoxImage(
-                        img.resize(640, 640, false),
-                        detectedObjects,
-                        outputPath,
-                        imgPath.toFile().getName());
+                logger.info("Detected object saved in: {}", output);
             }
-
-            return detectedObjects;
+            return detection;
         }
     }
-
-    private static void saveBoundingBoxImage(
-            Image img, DetectedObjects detectedObjects, Path outputPath, String outputFileName)
-            throws IOException {
-        img.drawBoundingBoxes(detectedObjects);
-
-        Path imagePath = outputPath.resolve(outputFileName);
-        img.save(Files.newOutputStream(imagePath), "png");
-    }
 }
diff --git a/jacoco/build.gradle b/jacoco/build.gradle
index d91963932..8a2b0057e 100644
--- a/jacoco/build.gradle
+++ b/jacoco/build.gradle
@@ -10,6 +10,7 @@ repositories {
 dependencies {
     jacocoAggregation project(":api")
     jacocoAggregation project(":basicdataset")
+    jacocoAggregation project(":engines:llama")
     jacocoAggregation project(":engines:ml:xgboost")
     jacocoAggregation project(":engines:ml:lightgbm")
     jacocoAggregation project(":engines:mxnet:mxnet-engine")


Test patch:
diff --git a/engines/llama/src/test/java/ai/djl/llama/engine/LlamaInputTest.java b/engines/llama/src/test/java/ai/djl/llama/engine/LlamaInputTest.java
new file mode 100644
index 000000000..429cd5693
--- /dev/null
+++ b/engines/llama/src/test/java/ai/djl/llama/engine/LlamaInputTest.java
@@ -0,0 +1,101 @@
+/*
+ * Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.llama.engine;
+
+import ai.djl.llama.engine.LlamaInput.Parameters;
+import ai.djl.llama.jni.InputParameters;
+import ai.djl.util.JsonUtils;
+
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import java.io.IOException;
+import java.io.Reader;
+import java.nio.file.Files;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+import java.util.Map;
+
+public class LlamaInputTest {
+
+    @Test
+    public void testInputParameters() throws IOException {
+        Path file = Paths.get("src/test/resources/inputs.json");
+        try (Reader reader = Files.newBufferedReader(file)) {
+            LlamaInput in = JsonUtils.GSON.fromJson(reader, LlamaInput.class);
+            checkParameters(in);
+        }
+
+        Parameters param = new Parameters();
+        LlamaInput in = new LlamaInput();
+        in.setInputs("prompt");
+        in.setPrefix("prefix");
+        in.setSuffix("suffix");
+        in.setParameters(param);
+        param.setMaxNewTokens(2);
+        param.setNumberKeep(2);
+        param.setNumberProbabilities(2);
+        param.setTopK(2);
+        param.setTopP(2f);
+        param.setTfsZ(2f);
+        param.setTypicalP(2f);
+        param.setTemperature(2f);
+        param.setRepeatPenalty(2f);
+        param.setRepeatLastN(2);
+        param.setFrequencyPenalty(2f);
+        param.setFrequencyPenalty(2f);
+        param.setPresencePenalty(2f);
+        param.setPenalizeNl(true);
+        param.setIgnoreEos(true);
+        param.setMirostat(2);
+        param.setMirostatTau(2f);
+        param.setMirostatEta(2f);
+        param.setNumberBeams(5);
+        param.setSeed(2);
+        Map<Integer, Float> logitBias = Map.of(2, 0.4f, 3, 0.5f);
+        param.setLogitBias(logitBias);
+        param.setGrammar("grammar");
+        param.setAntiPrompt(new String[] {"User: "});
+        checkParameters(in);
+    }
+
+    private void checkParameters(LlamaInput in) {
+        InputParameters param = in.getParameters().toInputParameters();
+        Assert.assertEquals(param.getMaxNewTokens(), 2);
+        Assert.assertEquals(param.getNumberKeep(), 2);
+        Assert.assertEquals(param.getNumberProbabilities(), 2);
+        Assert.assertEquals(param.getTopK(), 2);
+        Assert.assertEquals(param.getTopP(), 2f);
+        Assert.assertEquals(param.getTfsZ(), 2f);
+        Assert.assertEquals(param.getTypicalP(), 2f);
+        Assert.assertEquals(param.getTemperature(), 2f);
+        Assert.assertEquals(param.getRepeatPenalty(), 2f);
+        Assert.assertEquals(param.getRepeatLastN(), 2);
+        Assert.assertEquals(param.getFrequencyPenalty(), 2f);
+        Assert.assertEquals(param.getFrequencyPenalty(), 2f);
+        Assert.assertEquals(param.getPresencePenalty(), 2f);
+        Assert.assertTrue(param.isPenalizeNl());
+        Assert.assertTrue(param.isIgnoreEos());
+        Assert.assertEquals(param.getMirostat(), 2);
+        Assert.assertEquals(param.getMirostatTau(), 2f);
+        Assert.assertEquals(param.getMirostatEta(), 2f);
+        Assert.assertEquals(param.getNumberBeams(), 5);
+        Assert.assertEquals(param.getSeed(), 2);
+        Map<Integer, Float> logitBias = param.getLogitBias();
+        Assert.assertNotNull(logitBias);
+        Assert.assertEquals(logitBias.size(), 2);
+        Assert.assertEquals(logitBias.get(2), 0.4f);
+        Assert.assertNotNull(param.getGrammar());
+        Assert.assertNotNull(param.getAntiPrompt()[0], "User: ");
+    }
+}
diff --git a/engines/llama/src/test/java/ai/djl/llama/engine/LlamaTest.java b/engines/llama/src/test/java/ai/djl/llama/engine/LlamaTest.java
new file mode 100644
index 000000000..7b372ee42
--- /dev/null
+++ b/engines/llama/src/test/java/ai/djl/llama/engine/LlamaTest.java
@@ -0,0 +1,143 @@
+/*
+ * Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.llama.engine;
+
+import ai.djl.ModelException;
+import ai.djl.engine.Engine;
+import ai.djl.engine.StandardCapabilities;
+import ai.djl.inference.Predictor;
+import ai.djl.llama.jni.Token;
+import ai.djl.llama.jni.TokenIterator;
+import ai.djl.modality.Input;
+import ai.djl.modality.Output;
+import ai.djl.ndarray.NDManager;
+import ai.djl.repository.zoo.Criteria;
+import ai.djl.repository.zoo.ZooModel;
+import ai.djl.testing.TestRequirements;
+import ai.djl.training.util.DownloadUtils;
+import ai.djl.translate.TranslateException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+import java.io.IOException;
+import java.net.URI;
+import java.nio.file.Path;
+import java.nio.file.Paths;
+
+public class LlamaTest {
+
+    private static final Logger logger = LoggerFactory.getLogger(LlamaTest.class);
+
+    @BeforeClass
+    public void setUp() {
+        System.setProperty("DJL_CACHE_DIR", "build/cache");
+    }
+
+    @AfterClass
+    public void tierDown() {
+        System.clearProperty("DJL_CACHE_DIR");
+    }
+
+    @Test
+    public void testLlamaVersion() {
+        Engine engine = Engine.getEngine("Llama");
+        Assert.assertEquals(engine.getVersion(), "b1696-" + Engine.getDjlVersion());
+        Assert.assertNotNull(engine.toString());
+        Assert.assertEquals(engine.getRank(), 10);
+        Assert.assertFalse(engine.hasCapability(StandardCapabilities.CUDA));
+        Assert.assertNull(engine.getAlternativeEngine());
+        try (NDManager manager = engine.newBaseManager()) {
+            Assert.assertNotNull(manager);
+        }
+    }
+
+    @Test
+    public void testLlama() throws TranslateException, ModelException, IOException {
+        TestRequirements.nightly();
+        downloadModel();
+        Path path = Paths.get("models");
+        Criteria<String, TokenIterator> criteria =
+                Criteria.builder()
+                        .setTypes(String.class, TokenIterator.class)
+                        .optModelPath(path)
+                        .optModelName("tinyllama-1.1b-1t-openorca.Q4_K_M")
+                        .optEngine("Llama")
+                        .optOption("number_gpu_layers", "43")
+                        .optTranslatorFactory(new LlamaTranslatorFactory())
+                        .build();
+
+        String prompt =
+                "{\"inputs\": \"
system\n"
+                        + "{system_message}
\n"
+                        + "
user\n"
+                        + "{prompt}
\n"
+                        + "
assistant\", \"parameters\": {\"max_new_tokens\": 10}}";
+        try (ZooModel<String, TokenIterator> model = criteria.loadModel();
+                Predictor<String, TokenIterator> predictor = model.newPredictor()) {
+            TokenIterator it = predictor.predict(prompt);
+            StringBuilder sb = new StringBuilder();
+            while (it.hasNext()) {
+                Token token = it.next();
+                Assert.assertNotNull(token.getText());
+                Assert.assertTrue(token.getToken() >= 0);
+                Assert.assertNotNull(token.getProbabilities());
+                sb.append(token.getText());
+                logger.info("{}", token);
+            }
+            Assert.assertTrue(sb.length() > 1);
+        }
+    }
+
+    @Test
+    public void testLlamaInfill() throws TranslateException, ModelException, IOException {
+        TestRequirements.nightly();
+        downloadModel();
+        Path path = Paths.get("models/tinyllama-1.1b-1t-openorca.Q4_K_M.gguf");
+        Criteria<Input, Output> criteria =
+                Criteria.builder()
+                        .setTypes(Input.class, Output.class)
+                        .optModelPath(path)
+                        .optOption("number_gpu_layers", "43")
+                        .optEngine("Llama")
+                        .optTranslatorFactory(new LlamaTranslatorFactory())
+                        .build();
+
+        String prompt =
+                "{\n"
+                        + "   \"prefix\":\"def remove_non_ascii(s: str) -> str:\n\",\n"
+                        + "   \"suffix\":\"\n    return result\n\",\n"
+                        + "   \"parameters\":{\n"
+                        + "      \"max_new_tokens\": 10"
+                        + "   }\n"
+                        + "}";
+        try (ZooModel<Input, Output> model = criteria.loadModel();
+                Predictor<Input, Output> predictor = model.newPredictor()) {
+            Input in = new Input();
+            in.add("data", prompt);
+            Output out = predictor.predict(in);
+            Assert.assertNotNull(out.getData().getAsString());
+        }
+    }
+
+    private void downloadModel() throws IOException {
+        String url =
+                "https://huggingface.co/TheBloke/TinyLlama-1.1B-1T-OpenOrca-GGUF/resolve/main/tinyllama-1.1b-1t-openorca.Q4_K_M.gguf?download=true";
+        Path dir = Paths.get("models/tinyllama-1.1b-1t-openorca.Q4_K_M.gguf");
+        DownloadUtils.download(URI.create(url).toURL(), dir, null);
+    }
+}
diff --git a/engines/llama/src/test/java/ai/djl/llama/engine/package-info.java b/engines/llama/src/test/java/ai/djl/llama/engine/package-info.java
new file mode 100644
index 000000000..b2ee78641
--- /dev/null
+++ b/engines/llama/src/test/java/ai/djl/llama/engine/package-info.java
@@ -0,0 +1,14 @@
+/*
+ * Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+/** Contains test classes for llama engine. */
+package ai.djl.llama.engine;
diff --git a/engines/llama/src/test/resources/inputs.json b/engines/llama/src/test/resources/inputs.json
new file mode 100644
index 000000000..ab77386e1
--- /dev/null
+++ b/engines/llama/src/test/resources/inputs.json
@@ -0,0 +1,33 @@
+{
+  "prefix": "def remove_non_ascii(s: str) -> str:",
+  "suffix": "    return result",
+  "parameters": {
+    "max_new_tokens": 2,
+    "number_keep": 2,
+    "number_probabilities": 2,
+    "top_k": 2,
+    "top_p": 2,
+    "tfs_z": 2,
+    "typical_p": 2,
+    "temperature": 2,
+    "repeat_penalty": 2,
+    "repeat_last_n": 2,
+    "frequency_penalty": 2,
+    "presence_penalty": 2,
+    "penalize_nl": true,
+    "ignore_eos": true,
+    "mirostat": 2,
+    "mirostat_tau": 2,
+    "mirostat_eta": 2,
+    "number_beams": 5,
+    "seed": 2,
+    "logit_bias": {
+      "2": 0.4,
+      "5": 0.6
+    },
+    "grammar": "root  ::= (expr \"=\" term \"\\n\")+\nexpr  ::= term ([-+*/] term)*\nterm  ::= [0-9]",
+    "anti_prompt": [
+      "User: "
+    ]
+  }
+}
diff --git a/engines/onnxruntime/onnxruntime-engine/src/test/resources/mlrepo/model/cv/object_detection/ai/djl/onnxruntime/yolov8n/metadata.json b/engines/onnxruntime/onnxruntime-engine/src/test/resources/mlrepo/model/cv/object_detection/ai/djl/onnxruntime/yolov8n/metadata.json
new file mode 100644
index 000000000..1e0169a25
--- /dev/null
+++ b/engines/onnxruntime/onnxruntime-engine/src/test/resources/mlrepo/model/cv/object_detection/ai/djl/onnxruntime/yolov8n/metadata.json
@@ -0,0 +1,40 @@
+{
+  "metadataVersion": "0.2",
+  "resourceType": "model",
+  "application": "cv/object_detection",
+  "groupId": "ai.djl.onnxruntime",
+  "artifactId": "yolov8n",
+  "name": "yolov8n",
+  "description": "YoloV8 Model",
+  "website": "http://www.djl.ai/engines/onnxruntime/model-zoo",
+  "licenses": {
+    "license": {
+      "name": "The Apache License, Version 2.0",
+      "url": "https://www.apache.org/licenses/LICENSE-2.0"
+    }
+  },
+  "artifacts": [
+    {
+      "version": "0.0.1",
+      "snapshot": false,
+      "name": "yolov8n",
+      "arguments": {
+        "width": 640,
+        "height": 640,
+        "resize": true,
+        "rescale": true,
+        "optApplyRatio": true,
+        "threshold": 0.6,
+        "translatorFactory": "ai.djl.modality.cv.translator.YoloV8TranslatorFactory"
+      },
+      "files": {
+        "model": {
+          "uri": "0.0.1/yolov8n.zip",
+          "name": "",
+          "sha1Hash": "9fbad7f706713843cbb8c8d6a56c81a640ec6fa2",
+          "size": 11053839
+        }
+      }
+    }
+  ]
+}
diff --git a/engines/pytorch/pytorch-model-zoo/src/test/resources/mlrepo/model/cv/object_detection/ai/djl/pytorch/yolov8n/metadata.json b/engines/pytorch/pytorch-model-zoo/src/test/resources/mlrepo/model/cv/object_detection/ai/djl/pytorch/yolov8n/metadata.json
new file mode 100644
index 000000000..399b79b48
--- /dev/null
+++ b/engines/pytorch/pytorch-model-zoo/src/test/resources/mlrepo/model/cv/object_detection/ai/djl/pytorch/yolov8n/metadata.json
@@ -0,0 +1,40 @@
+{
+  "metadataVersion": "0.2",
+  "resourceType": "model",
+  "application": "cv/object_detection",
+  "groupId": "ai.djl.pytorch",
+  "artifactId": "yolov8n",
+  "name": "yolov8n",
+  "description": "YoloV8 Model",
+  "website": "http://www.djl.ai/engines/onnxruntime/model-zoo",
+  "licenses": {
+    "license": {
+      "name": "The Apache License, Version 2.0",
+      "url": "https://www.apache.org/licenses/LICENSE-2.0"
+    }
+  },
+  "artifacts": [
+    {
+      "version": "0.0.1",
+      "snapshot": false,
+      "name": "yolov8n",
+      "arguments": {
+        "width": 640,
+        "height": 640,
+        "resize": true,
+        "rescale": true,
+        "optApplyRatio": true,
+        "threshold": 0.6,
+        "translatorFactory": "ai.djl.modality.cv.translator.YoloV8TranslatorFactory"
+      },
+      "files": {
+        "model": {
+          "uri": "0.0.1/yolov8n.zip",
+          "name": "",
+          "sha1Hash": "a868778452ef8d6d2f9cb7109a9e14a64e851d48",
+          "size": 11183356
+        }
+      }
+    }
+  ]
+}
diff --git a/examples/src/test/java/ai/djl/examples/inference/Yolov8DetectionTest.java b/examples/src/test/java/ai/djl/examples/inference/Yolov8DetectionTest.java
new file mode 100644
index 000000000..35e3fc434
--- /dev/null
+++ b/examples/src/test/java/ai/djl/examples/inference/Yolov8DetectionTest.java
@@ -0,0 +1,40 @@
+/*
+ * Copyright 2023 Amazon.com, Inc. or its affiliates. All Rights Reserved.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance
+ * with the License. A copy of the License is located at
+ *
+ * http://aws.amazon.com/apache2.0/
+ *
+ * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES
+ * OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
+ * and limitations under the License.
+ */
+package ai.djl.examples.inference;
+
+import ai.djl.ModelException;
+import ai.djl.modality.Classifications;
+import ai.djl.modality.cv.output.DetectedObjects;
+import ai.djl.testing.TestRequirements;
+import ai.djl.translate.TranslateException;
+
+import org.testng.Assert;
+import org.testng.annotations.Test;
+
+import java.io.IOException;
+
+public class Yolov8DetectionTest {
+
+    @Test
+    public void testYolov8Detection() throws ModelException, TranslateException, IOException {
+        TestRequirements.engine("MXNet", "PyTorch");
+
+        DetectedObjects result = Yolov8Detection.predict();
+
+        Assert.assertTrue(result.getNumberOfObjects() >= 1);
+        Classifications.Classification obj = result.best();
+        String className = obj.getClassName();
+        Assert.assertEquals(className, "dog");
+        Assert.assertTrue(obj.getProbability() > 0.6);
+    }
+}
diff --git a/examples/src/test/resources/yolov8_synset.txt b/examples/src/test/resources/yolov8_synset.txt
index 7139f0cc6..ffba20649 100644
--- a/examples/src/test/resources/yolov8_synset.txt
+++ b/examples/src/test/resources/yolov8_synset.txt
@@ -1,3 +1,7 @@
+# Classes for coco dataset on which yelov8 is trained
+# source config https://github.com/ultralytics/ultralytics/blob/main/ultralytics/cfg/datasets/coco.yaml.
+# COCO dataset website: https://cocodataset.org/#home
+# Ultralytics Coco doc page: https://docs.ultralytics.com/datasets/detect/coco/
 person
 bicycle
 car