Joseph31415926/nllb_hieroglyph_final

Introduction

Almost all Egyptian hieroglyph translators on the internet just map the 24 uniliteral hieroglyphs to an English letter. The hieroglyphs produced are still English words but in a different character set. In the other direction, the 1000+ multiliteral hieroglyphs are ignored and the result is a subset of the consonant sounds of the original Egyptian text.

This model extends the facebook/nllb-200-distilled-600M base model by adding egy_Latn as a new language token and fine tuning the model with texts from various sources. The first iteration of this project used unicode strings from the Egyptian hieroglyphs block but performance was poor for two reasons: 1) The low availability of translations where the hieroglyphs were in a machine readable format and 2) Hieroglyphic spelling of words is variable due to phonetic complements and the freedom to use determinatives and ideograms as desired. Egyptologists solved this second problem by transliterating the words into a standard form using Latin and phonetic characters. By using transliterations, more training data is available.

Model Overview

This model has been trained using texts where the hieroglyphs are encoded using MdC (Manuel de Codage - a standard encoding of Egyptian transliteration using purely Latin characters). Where the source material is already transliterated, it is converted to MdC and extra symbols (e.g. ()[]<>.) are removed. Where the source has the original hieroglyphs, these are transliterated by a Python script that removes the phonetic complements, standardises plurals and duals to the Middle Kingdom style, converts numbers to digits, preserves determinatives using their Gardiner code preceded by a dash, undoes honorific transposition and adds spaces after determinatives and multiliterals.

Tokenizer Training

Spaces are not represented in hieroglyphic texts but adding them to the transliteration helps the tokenizer arrive at a good set of tokens. The tokenizer is trained by creating a new SentencePiece (SP) model with all the training sentences and adding the SP tokens to the base NllbTokenizer. As all the language tokens come after the SP tokens, they get moved to higher id values. When the base model is resized for the new SP tokens, we need to copy all the embeddings for the language tokens from their old id value to their new one (this step is missed in all tutorials resulting in loss of existing translation functionality).

Training Data

The training data required extensive cleansing as the source material is only intended for human readers. Lots of explanations of terms are included, usually in parentheses but not always. Pronouns are followed by a description of what they refer to. Dictionaries often include grammatical comments in the definitions. Hieroglyphic texts have annotations of missing/omitted/incorrect entries. Some sources invent their own Gardiner codes to capture some variant hieroglyph. A common abbreviation seen in german translations is LHG which turns out to stand for 'Leben, Heil, Gesundheit', the translation for 𓋹 𓍑 𓋴 . The same phrase was found to be abbreviated in english translations as LPH standing for 'Life, Prosperity, Health'.