Repository: deepjavalibrary/djl (java)

Issue:
[BUG]: tokenizer.json file not found
https://github.com/deepjavalibrary/djl/blob/715e6204a6d30002abda7096f9c5a210369d7470/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java#L790-L792

should be
```
} else if (!Files.exists(tokenizerPath)) {
    throw new IOException("Tokenizer file not exits: " + tokenizerPath);
}
```

Pull request description:
[tokenizer] Fixes tokenizer bug
Fixes #2840

## Description ##

Brief description of what this PR is about

- If this change is a backward incompatible change, why must this change be made?
- Interesting edge cases to note here


Patch:
diff --git a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
index f75342b7c..fa601e155 100644
--- a/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
+++ b/extensions/tokenizers/src/main/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizer.java
@@ -787,7 +787,7 @@ public final class HuggingFaceTokenizer extends NativeResource<Long> implements
                     return managed(HuggingFaceTokenizer.newInstance(vocab, merges, options));
                 }
                 throw new IOException("tokenizer.json file not found.");
-            } else if (Files.exists(tokenizerPath)) {
+            } else if (!Files.exists(tokenizerPath)) {
                 throw new IOException("Tokenizer file not exits: " + tokenizerPath);
             }
             return managed(HuggingFaceTokenizer.newInstance(tokenizerPath, options));


Test patch:
diff --git a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
index dcbef2474..2bc30d4bd 100644
--- a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
+++ b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
@@ -115,6 +115,12 @@ public class HuggingFaceTokenizerTest {
             Assert.assertEquals(encodings.length, 2);
             Assert.assertEquals(encodings[0].getIds(), ids);
         }
+
+        Assert.assertThrows(
+                () -> {
+                    Path file = Paths.get("build/tokenizer/non-exists.json");
+                    HuggingFaceTokenizer.builder().optTokenizerPath(file).build();
+                });
     }
 
     @Test