⚠️ See Disclaimer below before using.

opus-tatoeba-en-ja 🇬🇧 → 🇯🇵

A Teradata-compatible Translation Model

A sequence-to-sequence translation model that translates text from English 🇬🇧 to Japanese 🇯🇵. This repository hosts an ONNX-converted version of Helsinki-NLP/opus-tatoeba-en-ja, packaged for use with the Teradata mldb.ONNXSeq2Seq BYOM function.

This repository does not redistribute the original model weights. It contains only:

• onnx/model-fp32.onnx — full-precision ONNX graph
• tokenizer.json — repacked Marian tokenizer suitable for BYOM
• config.json — model architecture metadata, copied unchanged from the upstream repo
• generation_config.json — generation defaults, copied unchanged from the upstream repo

A weight-only int8-quantized variant is also published as onnx/model-int8.onnx. Use model-fp32.onnx unless deployment size is a constraint.

For the original PyTorch weights and training details, see the upstream model: Helsinki-NLP/opus-tatoeba-en-ja.

Specifications

Source language	English 🇬🇧 (en)
Target language	Japanese 🇯🇵 (ja)
Architecture	MarianMT (encoder-decoder)
Max input tokens	512
Max output tokens	512
ONNX file size	801 MB (fp32) / 404 MB (int8)
ONNX opset	14
ONNX IR version	8 (BYOM 7.0+ compatible)
License	Apache-2.0 (from upstream)
Reference	https://huggingface.co/Helsinki-NLP/opus-tatoeba-en-ja

Generation parameters are configurable at SQL time via the mldb.ONNXSeq2Seq USING clause through Const_* keys: Const_min_length, Const_max_length, Const_num_beams, Const_length_penalty, Const_repetition_penalty. They are not fixed in the ONNX graph. (num_return_sequences is the exception — it's baked into the graph as 1.)

Known quality limitations

This model is published with a quality caveat. It is usable for short, plain sentences and unreliable on idioms, on text containing embedded email addresses or similar non-prose tokens, and on some proper-noun constructions.

Its upstream pedigree is modest: the Tatoeba-Challenge benchmark reported for Helsinki-NLP/opus-tatoeba-en-ja is chr-F 0.258 / BLEU 15.2 on Tatoeba-test.eng-jpn (10,000 sentences). That is materially lower than most other pairs in this collection, and the behaviour below is consistent with it.

The examples are real outputs from this repository's ONNX artifacts, run through mldb.ONNXSeq2Seq on Teradata with exactly the Const_* values used in the Quickstart below (Const_num_beams(4) Const_max_length(64) Const_min_length(1)). They are reproduced verbatim, including spacing.

English input	fp32 output	int8 output	Problem
Tokyo is the capital of Japan.	東京は日本都である.	東京は日本の首都である.	fp32 emits 日本都, which is not a word (日本の首都 is correct). int8 happens to get this one right.
It is raining cats and dogs.	猫 や 犬 が 雨 を 降らせ て い ます。	猫や犬の雨が降っている.	The idiom is translated literally ("cats and dogs are making it rain"). Grammatical, but wrong.
Please send the report to john.smith@example.com by Friday.	2016年10月10日閲覧. ^ "ジョンに報告を". スミス@example.com ^ "金曜".	2016年3月18日閲覧. ^ "ジョン・スミス@example.com".	The email address derails the output into Japanese Wikipedia citation boilerplate. The translation is lost entirely.

Two further things to be aware of:

Token spacing is inconsistent between outputs. Some outputs come back with spaces between tokens (猫 や 犬 が 雨 を) and some without (東京は日本の首都である.), for the same model on the same run. This reflects the model's SentencePiece vocabulary surface and matches the upstream PyTorch checkpoint's output exactly — it is not introduced by the ONNX conversion. Downstream consumers that care about presentation should normalise whitespace themselves.

fp32 and int8 agree less often here than on the other models in this collection — 6 of 10 outputs identical at the Quickstart Const_* values above, 7 of 10 at the model's own native generation defaults. Where they differ, the difference is semantic rather than cosmetic (see rows 1 and 2 of the table). If you deploy model-int8.onnx, spot-check it against model-fp32.onnx on your own inputs rather than assuming the two are interchangeable.

For English → Japanese work where these limitations matter, evaluate the model on your own data before relying on it. The reverse direction, Japanese → English, is served by a stronger model in this collection.

Quickstart: Deploying this Model in Teradata

Requires Teradata 17.20+ with BYOM 7.0.0.4 or newer (the conversion targets ONNX IR version 8, which BYOM 7.0.x requires).

Note on schema name: the SQL example below uses mldb.ONNXSeq2Seq. On modern Teradata deployments BYOM is installed in the td_mldb database — adjust the schema prefix in the SQL accordingly.

import getpass
import teradataml as tdml
from huggingface_hub import hf_hub_download

repo_id = "Teradata/opus-tatoeba-en-ja"
model_id = "opus-tatoeba-en-ja"  # used as BYOM model_id

# 1. Download artifacts from this repo
hf_hub_download(repo_id=repo_id, filename="onnx/model-fp32.onnx", local_dir="./")
hf_hub_download(repo_id=repo_id, filename="tokenizer.json", local_dir="./")

# 2. Connect to Teradata
tdml.create_context(
    host=input("host: "),
    username=input("user: "),
    password=getpass.getpass("password: "),
)

# 3. Load model + tokenizer into BYOM tables
tdml.save_byom(
    model_id=model_id, model_file="onnx/model-fp32.onnx", table_name="translation_models"
)
tdml.save_byom(model_id=model_id, model_file="tokenizer.json", table_name="translation_tokenizers")

# 4. Translate
query = f"""
SELECT id, sequences
FROM mldb.ONNXSeq2Seq(
    ON (SELECT id, txt FROM your_input_table) AS InputTable
    ON (SELECT model_id, model FROM translation_models
         WHERE model_id = '{model_id}') AS ModelTable DIMENSION
    ON (SELECT model AS tokenizer FROM translation_tokenizers
         WHERE model_id = '{model_id}') AS TokenizerTable DIMENSION
    USING
        Accumulate('id')
        ModelOutputTensor('sequences')
        SkipSpecialTokens('true')
        OutputLength(512)
        OverwriteCachedModel('true')
        Const_min_length(1)
        Const_max_length(64)
        Const_num_beams(4)
        Const_length_penalty(1.0)
        Const_repetition_penalty(1.0)
) AS t
"""
print(tdml.DataFrame.from_query(query))

How this model was converted

This model was produced with the open-source teradata-opus-translate package, which exports the encoder/decoder, stitches in the BeamSearch op, applies weight-only int8 quantization, and verifies parity against PyTorch on a small sample set.

Note: the same package can convert any Helsinki-NLP MarianMT model (including ones not in this collection) to a BYOM-ready ONNX bundle. If you have a translation pair that's not published here, install the package and run:

from teradata_opus_translate import convert_model, convert_tokenizer

convert_model(
    "Helsinki-NLP/<your-model>",
    output_path="model-fp32.onnx",
)
convert_tokenizer(
    "Helsinki-NLP/<your-model>",
    output_path="tokenizer.json",
)

The resulting model-fp32.onnx and tokenizer.json are ready to deploy with the Quickstart flow above.

Disclaimer

DISCLAIMER: The content herein ("Content") is provided "AS IS" and is not covered by any Teradata Operations, Inc. and its affiliates ("Teradata") agreements. Its listing here does not constitute certification or endorsement by Teradata.

To the extent any of the Content contains or is related to any artificial intelligence ("AI") or other language learning models ("Models") that interoperate with the products and services of Teradata, by accessing, bringing, deploying or using such Models, you acknowledge and agree that you are solely responsible for ensuring compliance with all applicable laws, regulations, and restrictions governing the use, deployment, and distribution of AI technologies. This includes, but is not limited to, AI Diffusion Rules, European Union AI Act, AI-related laws and regulations, privacy laws, export controls, and financial or sector-specific regulations.

While Teradata may provide support, guidance, or assistance in the deployment or implementation of Models to interoperate with Teradata's products and/or services, you remain fully responsible for ensuring that your Models, data, and applications comply with all relevant legal and regulatory obligations. Our assistance does not constitute legal or regulatory approval, and Teradata disclaims any liability arising from non-compliance with applicable laws.

You must determine the suitability of the Models for any purpose. Given the probabilistic nature of machine learning and modeling, the use of the Models may in some situations result in incorrect output that does not accurately reflect the action generated. You should evaluate the accuracy of any output as appropriate for your use case, including by using human review of the output.