Repository: deepjavalibrary/djl
Pull request #3512 (2024-11-10 04:57:09)

Update troubleshooting.md - UnsatisfiedLinkError issue
## Description

- Rewrite
- Add setting PATH in Environment Variables

Triggered by my DJL related Issue in JabRef: https://github.com/JabRef/jabref/issues/12169


Patch:
diff --git a/docs/development/troubleshooting.md b/docs/development/troubleshooting.md
index 73cc4764f..40ecc8802 100644
--- a/docs/development/troubleshooting.md
+++ b/docs/development/troubleshooting.md
@@ -37,7 +37,9 @@ Then, right click the resources folder and select `Rebuild<default>`.
 ![FAQ1](https://resources.djl.ai/images/FAQ_engine_not_found.png)
 
 ## UnsatisfiedLinkError issue
+If a dependency is missing, it throws an `UnsatisfiedLinkError` exception. If you have encountered one, please create an issue at `https://github.com/deepjavalibrary/djl`.
 
+### Causes on Linux / MacOs
 ```
 Caused by: java.lang.UnsatisfiedLinkError: /home/ubuntu/.tensorflow/cache/2.3.1-cu101-linux-x86_64/libjni$
 ensorflow.so: /usr/lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.11' not found (required by /home/ubuntu/.te
@@ -60,33 +62,53 @@ libtorch.dylib:
 ```
 
 It shows the `libtorch.dylib` depends on `libiomp5.dylib` and `libc10.dylib`.
-If one of them is missing, it throws an `UnsatisfiedLinkError` exception, please create an issue at `https://github.com/deepjavalibrary/djl`.
 
-For Windows specifically, DJL many fail due to missing [Visual C++ 2019 Redistributable Packages](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads).
-You can install this package and reboot again to see if the issue persist. You can also check the below instruction to do further investigation.
+### Causes on Windows
 
-CN: Windows 10 加载失败常常是因为缺少 Windows Visual C++ 相关扩展包而导致的。您可以通过下面Windows的步骤来修复系统缺失依赖项。
+For Windows specifically, DJL many fail due to
 
+1. Pytorch CUDA library paths are not added to PATH in the Windows system environment variables. Usually caused by having multiple CUDAs installed on your system.
+2. missing [Visual C++ 2019 Redistributable Packages](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads).
+   You can install this package and reboot again to see if the issue persist.
 
-**Windows**
+   CN: Windows 10 加载失败常常是因为缺少 Windows Visual C++ 相关扩展包而导致的。您可以通过下面Windows的步骤来修复系统缺失依赖项。
 
+### Causes on any Operating System:
+
+   1. **Failed to extract native file issue**  
+      Sometimes you may only have read-only access on the machine.
+      It will cause a failure during engine loading because the cache attempts to write to the home directory.
+      For more information, please refer to [DJL Cache Management](cache_management.md).
+
+   2. **Package version mismatch**  
+   It happened when you had a wrong version with DJL and Deep Engines. 
+   You can check the combination [here](dependency_management.md) and use DJL BOM to solve the issue.
+
+### Solutions:
+
+1. **Visual C++ Redistributable Packages**  
+DJL requires Visual C++ Redistributable Packages. If you encounter an UnsatisfiedLinkError while building DJL on Windows, please download and install [Visual C++ 2019 Redistributable Packages](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads) and reboot.  
+
+   CN: 如果您在中国，可以使用 [DirectX 修复工具](https://blog.csdn.net/VBcom/article/details/6962388) 来安装遗失依赖项。
+
+3. **Add environment variables**.  
+The Pytorch CUDA library paths (C:\Users\USER\.djl.ai\pytorch\CUDA-VERSION) can be manually added to PATH in the Windows system environment variables by copying the folder path (search for "variable" or open Control Panel -> System -> Advanced System Settings. Click on Environment Variables. Under System variables, select Path and click Edit.). See also [how to edit environment variables on Windows 10 or 11](https://www.howtogeek.com/787217/how-to-edit-environment-variables-on-windows-10-or-11/).
+
+### Advanced Dependencies Handling:
+
+#### Visual Studio Tools CMD:
 You can run the following if you have Visual Studio tools CMD:
 
 ```cmd
 dumpbin /dependents your_dll_file.dll
 ```
 
-or install a [Dependency Walker](http://www.dependencywalker.com/).
-It's an application that can check the dependencies for a specific DLL by simply
-drag and drop.
-
-DJL requires Visual C++ Redistributable Packages. If you encounter an UnsatisfiedLinkError while building
-DJL on Windows, please download and install
-[Visual C++ 2019 Redistributable Packages](https://support.microsoft.com/en-us/help/2977003/the-latest-supported-visual-c-downloads) and reboot. 
+#### Dependency Walker
+Install a Dependency Walker. For example [Dependency Walker](http://www.dependencywalker.com/) or [a more recent fork](https://github.com/lhak/Dependencies) of it.
+It's an application that can check the dependencies for a specific DLL by simply drag and drop.
 
-CN: 如果您在中国，可以使用 [DirectX 修复工具](https://blog.csdn.net/VBcom/article/details/6962388) 来安装遗失依赖项。
-
-If the issue continues to persist, you can use the [docker file](https://github.com/deepjavalibrary/djl/blob/master/docker/windows/Dockerfile) provided by us.
+### If nothing else helps
+If the issue continues to persist, check you can use the [docker file](https://github.com/deepjavalibrary/djl/blob/master/docker/windows/Dockerfile) provided by us.
 Please note that this docker will only work with Windows server 2019 by default. If you want it to work with other
 versions of Windows, you need to pass the version as an argument as follows:
 
@@ -94,17 +116,6 @@ versions of Windows, you need to pass the version as an argument as follows:
 docker build --build-arg version=<YOUR_VERSION>
 ```
 
-### 1.4 Failed to extract native file issue
-Sometimes you may only have read-only access on the machine.
-It will cause a failure during engine loading because the cache attempts to write to the home directory.
-For more information, please refer to [DJL Cache Management](cache_management.md).
-
-
-### 1.5 Package version mismatch
-
-It happened when you had a wrong version with DJL and Deep Engines. 
-You can check the combination [here](dependency_management.md) and use DJL BOM to solve the issue.
-
 ## 2. IntelliJ throws the `No Log4j 2 configuration file found.` exception.
 The following exception may appear after running the `./gradlew clean` command:
 
diff --git a/extensions/tokenizers/rust/Cargo.toml b/extensions/tokenizers/rust/Cargo.toml
index fe9346fad..93394982d 100644
--- a/extensions/tokenizers/rust/Cargo.toml
+++ b/extensions/tokenizers/rust/Cargo.toml
@@ -13,7 +13,7 @@ candle-flash-attn = { version = "*", optional = true }
 candle-cublaslt = { git = "https://github.com/huggingface/candle-cublaslt", rev = "cf789b7dd6d4abb19b03b9556442f94f0588b4a0", optional = true }
 candle-layer-norm = { git = "https://github.com/xyang16/candle-layer-norm", rev = "e574de6a7f88bafbede8edf9ee43170c6a8ce51a", optional = true }
 candle-rotary = { git = "https://github.com/huggingface/candle-rotary", rev = "0a718a0856569a92f3112e64f10d07e4447822e8", optional = true }
-tokenizers = { version = "0.20.0", features = ["http"] }
+tokenizers = { version = "0.20.3", features = ["http"] }
 half = "2.4.0"
 tracing = "0.1.40"
 safetensors = "0.4.3"
diff --git a/extensions/tokenizers/rust/src/lib.rs b/extensions/tokenizers/rust/src/lib.rs
index 2c12559dd..59bc0a4f2 100644
--- a/extensions/tokenizers/rust/src/lib.rs
+++ b/extensions/tokenizers/rust/src/lib.rs
@@ -59,7 +59,7 @@ pub extern "system" fn Java_ai_djl_huggingface_tokenizers_jni_TokenizersLibrary_
     let mut parameters = FromPretrainedParameters::default();
     if !hf_token.is_null() {
         let hf_token: String = env.get_string(&hf_token).unwrap().into();
-        parameters.auth_token = Some(hf_token);
+        parameters.token = Some(hf_token);
     }
     let tokenizer = Tokenizer::from_pretrained(identifier, Some(parameters));
 
diff --git a/gradle/libs.versions.toml b/gradle/libs.versions.toml
index 990bad2da..509e4adf1 100644
--- a/gradle/libs.versions.toml
+++ b/gradle/libs.versions.toml
@@ -14,7 +14,7 @@ lightgbm = "3.2.110"
 tensorrt = "8.4.1"
 fasttext = "0.9.2"
 sentencepiece = "0.2.0"
-tokenizers = "0.20.0"
+tokenizers = "0.20.3"
 
 slf4j = "2.0.16"
 log4j_slf4j = "2.24.0"


Test patch:
diff --git a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
index 27fb1056b..4e11fee74 100644
--- a/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
+++ b/extensions/tokenizers/src/test/java/ai/djl/huggingface/tokenizers/HuggingFaceTokenizerTest.java
@@ -39,7 +39,7 @@ public class HuggingFaceTokenizerTest {
     public void testVersion() {
         try (HuggingFaceTokenizer tokenizer = HuggingFaceTokenizer.newInstance("bert-base-cased")) {
             String djlVersion = Engine.getDjlVersion();
-            Assert.assertEquals(tokenizer.getVersion(), "0.20.0-" + djlVersion);
+            Assert.assertEquals(tokenizer.getVersion(), "0.20.3-" + djlVersion);
         }
     }