opus-mt-fr-ca-onnx

ONNX export of Helsinki-NLP/opus-mt-fr-ca, for use with optimum.onnxruntime.ORTModelForSeq2SeqLM.

Licence

Apache-2.0, taken from the base model card (Helsinki-NLP/opus-mt-fr-ca).

Export command

optimum-cli export onnx --model Helsinki-NLP/opus-mt-fr-ca --task text2text-generation-with-past --no-post-process fr-ca

int8 dynamic quantization was then applied to each ONNX file with onnxruntime.quantization.quantize_dynamic (QUInt8).

File layout

encoder_model.onnx
decoder_model.onnx
decoder_with_past_model.onnx
config.json / generation_config.json
source.spm / target.spm / vocab.json / tokenizer_config.json / special_tokens_map.json
int8/
  encoder_model.onnx
  decoder_model.onnx
  decoder_with_past_model.onnx

Sanity check

3 short sentences, PyTorch original vs this ONNX export, num_beams=4, max_new_tokens=48:

input	PyTorch	ONNX fp32	ONNX int8
Bonjour, comment vas-tu?	Hola, com estàs?	Hola, com estàs?	Hola, com estàs?
Le chien court dans le parc.	El gos corre cap al parc.	El gos corre cap al parc.	El gos corre cap al parc.
Bonne journee a tous.	Que tingui un bon dia.	Que tingui un bon dia.	Que tingui un bon dia.

fp32 matches PyTorch on: 3/3. int8 output recorded above (not gated).

Usage

from transformers import AutoTokenizer
from optimum.onnxruntime import ORTModelForSeq2SeqLM

model_id = "TigreGotico/opus-mt-fr-ca-onnx"
tok = AutoTokenizer.from_pretrained(model_id)
model = ORTModelForSeq2SeqLM.from_pretrained(model_id)

inputs = tok("Bonjour, comment vas-tu?", return_tensors="pt")
out = model.generate(**inputs, num_beams=4, max_new_tokens=48)
print(tok.decode(out[0], skip_special_tokens=True))

For the int8 quantized version, load from the int8/ subfolder:

model = ORTModelForSeq2SeqLM.from_pretrained(model_id, subfolder="int8")