DEVELOP.md

ONNX models generation

The transformers.js library uses ONNX models. To convert a Hugging Face model to ONNX, you can use the optimum-cli tool. The following commands show how to convert a Hugging Face model to ONNX and quantize it using the optimum-cli tool.

MODEL_NAME=GIST-all-MiniLM-L6-v2

poetry run optimum-cli export onnx -m avsolatorio/${MODEL_NAME} ${MODEL_NAME}_onnx/
poetry run optimum-cli onnxruntime quantize \
  --avx512 \
  --onnx_model ${MODEL_NAME}_onnx -o ${MODEL_NAME}_onnx_quantized

We then upload this to the huggingface model repository. The onnx models must be in the onnx/ folder in the repository.

git clone https://huggingface.co/avsolatorio/${MODEL_NAME}
cd ${MODEL_NAME}

if [ ! -d "onnx" ]; then
    mkdir onnx && cd onnx

    # Copy the onnx model
    rsync -avP ../../${MODEL_NAME}_onnx/model.onnx onnx/

    # Copy the quantized onnx model
    rsync -avP ../../${MODEL_NAME}_onnx_quantized/model_quantized.onnx onnx/
    rsync -avP ../../${MODEL_NAME}_onnx_quantized/ort_config.json onnx/

    # Commit and push
    git add onnx
    git commit -m "Add onnx models"
    git push origin main
fi