opus-mt-ca-it-onnx

ONNX export of Helsinki-NLP/opus-mt-ca-it, for use with optimum.onnxruntime.ORTModelForSeq2SeqLM.

Licence

Apache-2.0, taken from the base model card (Helsinki-NLP/opus-mt-ca-it).

Export command

optimum-cli export onnx --model Helsinki-NLP/opus-mt-ca-it --task text2text-generation-with-past --no-post-process ca-it

int8 dynamic quantization was then applied to each ONNX file with onnxruntime.quantization.quantize_dynamic (QUInt8).

File layout

encoder_model.onnx
decoder_model.onnx
decoder_with_past_model.onnx
config.json / generation_config.json
source.spm / target.spm / vocab.json / tokenizer_config.json / special_tokens_map.json
int8/
  encoder_model.onnx
  decoder_model.onnx
  decoder_with_past_model.onnx

Sanity check

3 short sentences, PyTorch original vs this ONNX export, num_beams=4, max_new_tokens=48:

input	PyTorch	ONNX fp32	ONNX int8
Hola, com estas?	Ehi, come stai?	Ehi, come stai?	Ehi, come stai?
El gos corre al parc.	Il cane va al parco.	Il cane va al parco.	Il cane va al parco.
Bon dia a tothom.	Buongiorno a tutti.	Buongiorno a tutti.	Buongiorno a tutti.

fp32 matches PyTorch on: 3/3. int8 output recorded above (not gated).

Usage

from transformers import AutoTokenizer
from optimum.onnxruntime import ORTModelForSeq2SeqLM

model_id = "TigreGotico/opus-mt-ca-it-onnx"
tok = AutoTokenizer.from_pretrained(model_id)
model = ORTModelForSeq2SeqLM.from_pretrained(model_id)

inputs = tok("Hola, com estas?", return_tensors="pt")
out = model.generate(**inputs, num_beams=4, max_new_tokens=48)
print(tok.decode(out[0], skip_special_tokens=True))

For the int8 quantized version, load from the int8/ subfolder:

model = ORTModelForSeq2SeqLM.from_pretrained(model_id, subfolder="int8")