GlyphMind

GlyphMind reads text as images. It draws each line of text as a 64 × 768 pixel image and embeds the image with a vision transformer. It has no tokenizer and never sees a character label. Its only supervision is which drawings share a meaning: pretraining pulls together drawings of the same sentence in different writing systems, and also has the model predict the features of masked parts of each drawing.

This repository holds two models from our paper and a demo to test them. The paper is under review; a link to the preprint will follow.

File
GlyphMind-205M.pt	GlyphMind-205M after pretraining on 27 writing systems
GlyphMind-205M-Lao.pt	GlyphMind-205M adapted to Lao, a language and writing system absent from pretraining, without any Lao translations: the Lao model reported in the paper
glyphmind_demo.py	the model, the renderer and three commands to try them
fonts/	the Noto fonts the models were trained and tested with

Both files hold the float32 weights of the encoder and its projection head, 202.8M parameters: 16 layers of width 1,024 over 8 × 8 pixel patches. (GlyphMind-205M is named after its size during pretraining, which included a prediction head that is not needed for the features.)

Try it

pip install torch numpy pillow huggingface_hub
hf download Leon5201314/glyphmind --local-dir glyphmind
cd glyphmind
python glyphmind_demo.py similarity --model GlyphMind-205M.pt

compares three sentences in nine languages with their English versions. To compare your own sentences:

python glyphmind_demo.py similarity --model GlyphMind-205M.pt "The cat is asleep on the sofa." "Кошка спит на диване." "Stock markets fell sharply on Monday."

python glyphmind_demo.py render --text "ສະບາຍດີ" --out lao.png saves the image the model sees. The demo uses a GPU when there is one. On a CPU it is slower, and the last digit of a result may differ.

The paper's retrieval results

The test set is NTREX-128 (CC BY-SA 4.0):

git clone https://github.com/MicrosoftTranslator/NTREX
N=NTREX/NTREX-128
python glyphmind_demo.py retrieve --model GlyphMind-205M.pt --queries $N/newstest2019-ref.rus.txt --candidates $N/newstest2019-ref.eng-US.txt
python glyphmind_demo.py retrieve --model GlyphMind-205M-Lao.pt --queries $N/newstest2019-ref.lao.txt --candidates $N/newstest2019-ref.eng-US.txt --pure

For each of the 1,997 Russian or Lao sentences, the demo ranks all 1,997 English sentences by the cosine similarity of their features and counts how often the right translation comes first (top-1) or among the first five (top-5). --pure keeps only the Lao letters of each query, as in the paper's Lao tests. Run with either model, the two commands give:

Model	Test	Top-1	Top-5	In the paper
GlyphMind-205M	Russian → English	56.53%		56.5% (Fig. 2c)
GlyphMind-205M	Lao → English, --pure	1.45%	3.41%	top-5 3.4% (Fig. 3a)
GlyphMind-205M-Lao	Lao → English, --pure	6.56%	14.27%	top-5 14.3% (Fig. 3a)
GlyphMind-205M-Lao	Russian → English	44.42%		44.4% (Extended Data Table 1)

These were measured with PyTorch 2.12.0 and Pillow 12.2.0 on an NVIDIA A100-SXM4-80GB GPU.

How a line is drawn

Each line is drawn black on white at 64 × 768 pixels, in the Noto font that pretraining used for its writing system (Noto Sans Lao for Lao). The font size is fitted between 44% and 70% of the height. A line that is still too wide is cut at a word boundary, so the model sees only the start of a long sentence. The demo picks the font from the letters of each line; --font chooses another. Pillow needs libraqm to shape Arabic, the Indic scripts, Thai, Lao and other writing systems, and the demo warns when it is missing. The features are the mean of the final tokens after the last layer norm, scaled to unit length.

Limits

These are research models. Lao retrieval is far from good enough for translation. Each writing system was seen in one font, and only the first line-width of a sentence is read.

Not included yet

The code for training and adaptation, the adaptation corpora and the other models of the paper will be released on publication.

Licences

Model weights: CC BY 4.0. glyphmind_demo.py: MIT (LICENSE-demo.txt). Fonts: SIL Open Font License 1.1 (fonts/OFL.txt).