^^^^^^^^^^
File "/var/lib/.nvidia/miniconda3/envs/facefusion/lib/python3.11/site-packages/gradio/routes.py", line 1388, in upload_file
directory.mkdir(exist_ok=True, parents=True)
File "/var/lib/.nvidia/miniconda3/envs/facefusion/lib/python3.11/pathlib.py", line 1116, in mkdir
os.mkdir(self, mode)
PermissionError: [Errno 13] Permission denied: '/tmp/gradio/c85b0047c0c710e66ef9b05ab0f4af5d86eae73c0ae12b0cf245483024eed5c2'
2024-11-12 12:13:45.388051993 [E:onnxruntime:Default, provider_bridge_ort.cc:1862 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1539 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory
2024-11-12 12:13:45.611847211 [E:onnxruntime:Default, provider_bridge_ort.cc:1862 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1539 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory
2024-11-12 12:13:45.824780621 [E:onnxruntime:Default, provider_bridge_ort.cc:1862 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1539 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory
2024-11-12 12:13:46.741355532 [E:onnxruntime:Default, provider_bridge_ort.cc:1862 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1539 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory
2024-11-12 12:13:47.361801845 [E:onnxruntime:Default, provider_bridge_ort.cc:1862 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1539 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory
2024-11-12 12:13:48.138421233 [E:onnxruntime:Default, provider_bridge_ort.cc:1862 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1539 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory
2024-11-12 12:13:48.237065986 [E:onnxruntime:Default, provider_bridge_ort.cc:1862 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1539 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory
2024-11-12 12:13:48.633369490 [E:onnxruntime:Default, provider_bridge_ort.cc:1862 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1539 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory
2024-11-12 12:13:53.765538781 [E:onnxruntime:Default, provider_bridge_ort.cc:1862 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1539 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory
2024-11-12 12:13:55.285066570 [E:onnxruntime:Default, provider_bridge_ort.cc:1862 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1539 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory

Hello! It looks like you're encountering two main issues based on the error messages you've provided:
Permission Error when accessing /tmp/gradio/ directory.
Missing CUDA library libcublasLt.so.12 needed by libonnxruntime_providers_cuda.so.
Let's address each issue step by step.
1. Permission Error When Accessing /tmp/gradio/

Error Message:

PermissionError: [Errno 13] Permission denied: '/tmp/gradio/c85b0047c0c710e66ef9b05ab0f4af5d86eae73c0ae12b0cf245483024eed5c2'
Cause:
Your application is trying to create a directory or file within /tmp/gradio/, but it doesn't have the necessary permissions.
Solutions:
Check Directory Permissions:
Run the following command to check the permissions:

ls -ld /tmp/gradio
Ensure that the directory is writable by your user or the application.
Change Ownership or Permissions:
If you're the only user, you can change the ownership to your user:

sudo chown -R $(whoami):$(whoami) /tmp/gradio
Alternatively, adjust the permissions:

sudo chmod -R u+rwx /tmp/gradio
Create the Directory if It Doesn't Exist:
If /tmp/gradio/ doesn't exist, create it with the correct permissions:

mkdir -p /tmp/gradio
chmod 777 /tmp/gradio
Note: Setting permissions to 777 gives read, write, and execute permissions to everyone, which might not be secure in a multi-user environment.
Run the Application with Sufficient Privileges:
Ensure the application is run by a user with the necessary permissions.
Temporary Directory Override:
Some applications allow you to specify a different temporary directory. Check if gradio allows setting a custom temp directory where you have write access.
Check for SELinux or AppArmor Restrictions:
If your system uses SELinux or AppArmor, they might restrict access despite correct file permissions.
For SELinux, check the status:

getenforce
If it's enforcing, you might need to adjust the SELinux policies or temporarily set it to permissive mode for testing (not recommended for production):

sudo setenforce 0
2. Missing CUDA Library libcublasLt.so.12

Error Message:

Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.12: cannot open shared object file: No such file or directory
Cause:
The ONNX Runtime is trying to use CUDA for GPU acceleration but can't find the required CUDA libraries, specifically libcublasLt.so.12.
Solutions:
Ensure CUDA Toolkit is Installed:
Verify if CUDA is installed:

nvcc --version
or

nvidia-smi
If not installed, download and install the CUDA Toolkit (version 11 or higher) from NVIDIA's website: CUDA Toolkit
Check for libcublasLt.so.12:
Locate the library:

sudo updatedb   # Updates the file name database.
locate libcublasLt.so.12
If the library is missing, ensure that you've installed the correct version of the CUDA Toolkit.
Update LD_LIBRARY_PATH Environment Variable:
Add the CUDA library path to LD_LIBRARY_PATH:

export LD_LIBRARY_PATH=/usr/local/cuda/lib64:$LD_LIBRARY_PATH
Add this line to your ~/.bashrc or ~/.bash_profile to make it persistent.
Verify that the path /usr/local/cuda/lib64 contains libcublasLt.so.12.
Install CUDA Toolkit Components:
If you installed CUDA using a package manager, ensure all components are installed. For Ubuntu users:

sudo apt-get install libcublas-11-0
Replace 11-0 with your version number.
Reinstall or Update ONNX Runtime:
Make sure you're using the GPU-enabled version of ONNX Runtime.

pip uninstall onnxruntime onnxruntime-gpu
pip install onnxruntime-gpu
Check ONNX Runtime CUDA Provider:
Verify that your code properly initializes ONNX Runtime with the CUDA provider.
Example:

import onnxruntime as ort
sess_options = ort.SessionOptions()
session = ort.InferenceSession("model.onnx", sess_options, providers=['CUDAExecutionProvider'])
Verify CUDA and cuDNN Compatibility:
Ensure that your versions of CUDA and cuDNN are compatible with each other and with ONNX Runtime.
Check the ONNX Runtime GPU requirements: ONNX Runtime Execution Providers
Avoid Symbolic Links Without Compatibility Assurance:
While creating a symbolic link from an existing version of libcublasLt.so might seem like a quick fix, it can lead to compatibility issues.

# Not recommended unless you are sure about compatibility
sudo ln -s /usr/local/cuda/lib64/libcublasLt.so.X /usr/local/cuda/lib64/libcublasLt.so.12
It's safer to install the correct version of the library.
Check Multiple CUDA Versions:
If you have multiple versions of CUDA installed, there might be conflicts.
Ensure that the CUDA_HOME environment variable points to the correct CUDA installation.

export CUDA_HOME=/usr/local/cuda
Additional Considerations

Verify GPU Support:
Ensure that your GPU supports the required CUDA version.
Conda Environments:
If using conda, make sure that the environment includes the necessary packages.

conda install -c nvidia cuda-toolkit
conda install -c conda-forge cudatoolkit-dev
System Updates:
Sometimes, missing dependencies can be resolved by updating the system packages:

sudo apt-get update
sudo apt-get upgrade
Docker Containers:
If you're running inside a Docker container, ensure that it has access to the host's GPUs.
Use NVIDIA's Docker runtime:

docker run --gpus all nvidia/cuda:11.0-base nvidia-smi
Summary

Permission Error:
Adjust permissions or ownership of the /tmp/gradio/ directory.
Ensure the application has the rights to create and modify files in that directory.
Missing CUDA Library:
Install or update the CUDA Toolkit to the version compatible with ONNX Runtime.
Ensure libcublasLt.so.12 is present and reachable via LD_LIBRARY_PATH.
Reinstall onnxruntime-gpu to ensure it links against the correct CUDA libraries.
Verify that your system's CUDA installation is correctly configured and that your GPU supports the required features.
Next Steps

Check Permissions:
Resolve the permission issues first to ensure that your application can run without being blocked by file system restrictions.
Resolve CUDA Library Issue:
Install the necessary CUDA components and configure the environment variables.
Test the Application:
After making the changes, rerun the application to see if the errors are resolved.
Provide System Information if Issues Persist:
Operating System and version.
Output of nvidia-smi and nvcc --version.
Versions of Python packages (pip freeze output).
Any additional logs or error messages.
Helpful Commands Reference

Check CUDA Installation:

nvcc --version
nvidia-smi
List CUDA Libraries:

ls /usr/local/cuda/lib64/libcublas*
Environment Variables:

echo $LD_LIBRARY_PATH
echo $CUDA_HOME
Python Environment Packages:

pip freeze | grep onnxruntime
pip freeze | grep torch
Please let me know if you need further assistance or if you have any questions about these steps!



I just read locate and got 
nvidia@0136-ict-prxmx50127:~$ locate libcublasLt.so.12
/home/Ubuntu/apps/OneTrainer/venv/lib/python3.10/site-packages/nvidia/cublas/lib/libcublasLt.so.12
/home/Ubuntu/apps/StableSwarmUI/dlbackend/ComfyUI/venv/lib/python3.10/site-packages/nvidia/cublas/lib/libcublasLt.so.12
/home/Ubuntu/apps/stable-diffusion-webui-forge/venv/lib/python3.10/site-packages/nvidia/cublas/lib/libcublasLt.so.12
/home/Ubuntu/apps/stable-diffusion-webui/venv/lib/python3.10/site-packages/torch/lib/libcublasLt.so.12