Pantogloss 500-to-English

Model status: released publicly as version 0.1.0.

This repository contains the model artifacts for the Pantogloss TensorFlow/Keras many-to-English translation package. The model remains separate from the Python distribution, so its approximately 2.18 GB of weights are never included in the wheel.

Usage

from pantogloss import Translator

translator = Translator.from_pretrained("pantogloss-500-en", device="auto")
print(translator.translate("Comment allez-vous ?"))
print(translator.translate("Comment allez-vous ?", beam_size=4, length_penalty=0.6))

Install pantogloss[cuda] for NVIDIA acceleration on Linux or pantogloss[metal] for Apple Silicon. Pantogloss 0.3.0 and later resolve this public repository anonymously. Cached and explicit tokens remain supported for private or gated repositories.

Validation

All 308 learned PyTorch tensors map to 307 Keras variables because the target embedding and output projection are tied. CPU greedy token IDs match the archived RTG implementation across a ten-language, six-script batch. The maximum absolute difference in inference-critical tensors is 1.24e-5. With the original beam size 4 and length penalty 0.6, the complete decoded four-best candidate sets match on all ten examples; one near-tied example changes top-candidate rank because of framework floating-point ordering.

CUDA placement and inference have been validated on an NVIDIA GeForce RTX 3080 Ti Laptop GPU. Metal placement and inference have been validated on an Apple M3 Max with 128 GB unified memory, macOS 26.5.2, Python 3.12.9, and TensorFlow 2.18.1. The first model variable was confirmed on /GPU:0, and translation matched CPU.

Pantogloss 0.2.0 uses graph-compiled greedy decoding and decoder attention caches. Warm throughput for batches 1, 8, 16, and 32 was 11.6, 62.1, 96.8, and 143.0 sentences/second on Linux CPU; 14.2, 94.9, 160.6, and 330.0 on CUDA; and 3.83, 29.15, 60.20, and 115.28 on Metal. These figures use one repeated short input and characterize runtime behavior, not translation quality.

Provenance

The weights were converted from the massively multilingual model described by Thamme Gowda, Zhao Zhang, Chris A. Mattmann, and Jonathan May in Many-to-English Machine Translation Tools, Data, and Pretrained Models, ACL-IJCNLP 2021 System Demonstrations, DOI 10.18653/v1/2021.acl-demo.37.

Coverage and translation quality vary by language and domain. “500-to-English” describes training provenance, not equal quality across every language.

Intended use and limitations

This model is intended for research and general-purpose translation of text into English. It has not been evaluated for safety-critical, legal, medical, or high-stakes decisions. Users should review translations before relying on them.

Training coverage does not imply reliable language identification or equal quality for every language, script, dialect, or domain.
Inputs outside the training distribution can be mistranslated, shortened, hallucinated, or returned as low-quality English.
The current model does not return confidence, detected language, citations, or calibrated uncertainty.
The parity corpus validates checkpoint conversion, not broad translation quality, fairness, or safety.
Source data may contain biases that can be reproduced in generated text.

Artifact integrity

Pantogloss pins model version 0.1.0 to immutable Hugging Face revision 250fc3b4122d79ac0734b28b368d2c1d68f72f7e. manifest.json records byte sizes and SHA-256 digests for every runtime artifact. The two weight shards are:

model_00000.weights.h5: 388,347,832 bytes, SHA-256 6eb629f8f797b73d8a0cbf34211f22eec515aa0dd3da709df183f918ead643b8
model_00001.weights.h5: 1,769,751,744 bytes, SHA-256 c07ca4cc88b1f21a51a8ae4ee35414dee284f29f05382765200b8608fd5476b1

The source archive checksum, source checkpoint checksum, conversion mapping, and validation results are recorded in manifest.json and validation.json.

License

The Pantogloss implementation and converted model are licensed under the Apache License, Version 2.0. See LICENSE and NOTICE.

Downloads last month: 106