Emergent Semantics Beyond Token Embeddings: Transformer LMs with Frozen Visual Unicode Representations
Authors: Andrey Bochkov
Venue: Accepted by TMLR ()

Abstract: Understanding the locus of semantic representation in large language models (LLMs) is crucial for interpretability and architectural innovation. The dominant paradigm posits that trainable input embeddings serve as foundational "meaning vectors." This paper challenges that view. We construct Transformer models where the embedding layer is entirely frozen, with vectors derived not from data, but from the visual structure of Unicode glyphs. These non-semantic, precomputed visual embeddings are fixed throughout training. Our method is compatible with any tokenizer, including a novel Unicode-centric tokenizer we introduce to ensure universal text coverage. Despite the absence of trainable, semantically initialized embeddings, our models converge, generate coherent text, and, critically, outperform architecturally identical models with trainable embeddings on the MMLU reasoning benchmark. We attribute this to "representational interference" in conventional models, where the embedding layer is burdened with learning both structural and semantic features. Our results indicate that high-level semantics are not inherent to input embeddings but are an emergent property of the Transformer's compositional architecture and data scale. This reframes the role of embeddings from meaning containers to structural primitives. We release all code and models to foster further research.

Published in Transactions on Machine Learning Research (10/2025)
Emergent Semantics Beyond Token Embeddings: Trans- former LMs with Frozen Visual Unicode Representations
A. Bochkov Moscow Institute of Physics and Technology (MIPT), Moscow, Russio
andrey.bocckon@@gmal..o
Reviewed on OpenReview: https: // opwnreveee..........mmmu net/ htll//ormmmt id= Odh81yn01o
Abstract
Understanding the locus of semantic representation in large language models (LLMs) is cru- cial for interpretability and architectural innovation. The dominant paradigm posits that trainable imput embeddings serve foundational 'meaning vectors.' This paper challenges that view. We construct Transformer models where the embedding layer is entirely frozen, with vectors derived not from data, but from the visual structure of Unicode glyphs. These non-semantic, precomputed visual embeddings are fixed throughout training. Our method compatible with any tokenizer, including a novel caicode-centric tokenizer we introduce to ensure universal text coverage. Despite the absence of trainable, semantically initialized emboddings, our models converge, generate coherent text, and, critically, outperform archi- lect uralur identical models with trainable embeddings the MMLU reasoning benchmark. We attribute this to 'representational interference" in conventional models, where the em- bedding layer is burdened with Icarning both structural and semantic features. Our results indicate that high-level semantics are not inherent to input embeddings but are an emergent property of the Transformer's 's compositional architecture and data scale. This reframes the role of embeddings from meaning containers to structural primitives. We release all code and models to foster further rescarch.
1 INTRODUCTION
Understanding where and how semantic abstractions arise in transformer language models is both theoreti- cal and practical concern for the development of scalable, robust, and interpretable Al systems. Traditionally, input token embeddings- meaning: their algebraic properties have been cited as evidence of encodable semantic relationships (*king man + woman = queen"), and ad vancing architectures routinely attribute model improvements smarter or larger embedding matrices. This paradigm has shaped assumptions about what is required for "intelligent" reprosentation learning.
representations from large corpora- -re viewed as the primary locus of
The robustness of modern Large Language Models (LLMs) to orthographic variations, such as in 'I can wRiTee' highlights a fundamental question. The model's understanding is not derived from single, seman- tically rich token for "write'. Instead, it often relics on composing meaning from a sequence of semantically poor character- or subword-level tokens (e-g. 'w', 'R', 'I, 'T', 'e"). If the input vectors for these atomic units lack inherent semantic content, where does the modcl's understanding originato? This paper argucs and em- pirically demonstrates that semantics are not a property of input embeddings but an emergent phenomenon of the Transformer architecture itself.
However, recent advances in byte and character-level modeling, as well as modular and multi-lingual archi- tectures, suggest that the relationship between embeddings and high-level meaning is far subtler. In pursuit of a deeper understanding, we investigate the extreme case: Transformer language models where the input embedding layer is never trained and, fact, deliberately devoid of semantic optimization. Instoad, we fix this layer to deterministic mapping based on visual Unicode glyph representations and token-level n-gram image composites.

---

Published in Transactions on Machine Learning Research (10/2025)
Our experimental framework draws embeddings not from data-driven optimization, but from precomputed character images spanning the entire Unicode range. For multicharacter tokens (as in byte-pair and unigram models), we employ compositional image techniques to generate a fixed-length visual feature vector per token, achieving universal text coverage. The tokenizer is similarly Unicode-centric, balancing character- level granularity with multi-scripl n-gram tokens to maximize overlap across languages.
Instead of pursuing state-of-the-art benchmark scores, which often depend on massive scale, our primary goal is to isolate and test fundamental hypothesis: can language abstraction emerge without trainable, semantically-rich imput embeddings? We deliberately constrain model and dataset sizes to create controlled environment for comparing the learning dynamics of frozen- versus trainableeemeeddngg models. Our focus is on the conditions under which meaning emerges, not on outperforming large-scale systems.
Critically, we observe that despite this radical departure from conventional practice, these "frozen- embedding" models not only converge but display robust convergence and, surprisingly, outperform architec- turally identical counterparts with trainable embeddings on reasoning benchmarks like MMLU, suggesting representational interference in conventional approaches. This suggests that emergent abstraction appears within the transformer blocks themselves, implying that semantic structure is an architectural phenomenon, not an initialization artifact. Our main contributions are:
demonstrate that semantic understanding in Transformers can emerge without trainable input em- beddings
introduce universal visual embedding scheme compatible with any tokenizer
identify 'representational interference' as potential limitation of trainable embeddings
release open-source implementations enabling reproducibility
Section reviews related work; Section III details our method for constructing visual embeddings and Unicode tokenization; Section TV describes the experimental setup: Scction V presents results and analysis; Section VI discusses implications for future architectures; and Section VII concludes. Code and all artifacts are released to foster further progress in both science and industry
The rest of this paper is organized as follows:
2 RELATED WORK
Semantic representation in neural language models has received extensive attention. Early word2vec MMikolov ct al. 2013) and GloVe (Pennington et al., 2014) models established the conceptual centrality of "mean- ingful" dense vector spaces, with well-documented algebraic properties in vector arithmetic. Subsequent transformer-bast LMs (Radford et al. 2019) extended this paradigm to context-dependent embeddings, further entrenching the notion that trainable input vectors are the source of semantic capability. However, several lines of rescarch challenge this view.
Character- and byte-level models, such as ByT5 (uue et al. 2022 and CANINE (Clark al. 2022 forego word-level tokenization for universal coverage; yet, even here, embedding matrices are trainable and morph into semantic vector spaces during pretraining, Cross-lingual and modular approaches pursue shared embedding spaces for transfer [Lample et al.] 2018 Lample & Conneau 2019). but rely on substantial alignment learning. Mixturre-f-exxpets (Shazeer et al., 2017 Fedus et al. 2022) and adapter fusion (Pfeiffer eet al.] 2020) aim modularize reasoming, yet remain dependent on trainable emboddings. Our input
embedding matrix is never trained and strictly visual/surface oriented.
More radically, recent work in visual-language models (Tan & Bansal 2022 investigates multimodal repre- sentations by grounding language in visual contexts. Models like CLIP (Radford et al. 2021 or Flamingo [Alayrac et al. 2022) learn joint embeddings for text and images. However, these approaches typically com- bine separate, powerful encoders for each modality and aim to align two rich semantic streams, rather than enforcing a non-semantic, frozen vector space for language as we propose.
Repository: https://ghthubb.com///e http:///hhbbbcoo////////eeeeeeee
hutps: (nttps//nuugigggaarrccoo Bochkkkooo/oooo////o
2

---

Published in Transactions on Machine Learning Research (10/2025)
The idea of incorporating visual information from text glyphs is not entirely new. Prior work has suc- cessfully utilized visual features to enhance specific tasks. For instance, visual representations have been used to improve machine translation, especially for handling rare words and achieving robust cross-lingual transfer (Saeesky et al. [2023] Wang et al. (2020). Other studies have leveraged glyph-aware embeddings for better represenlation of logographic languages like Chinese Dai & Cai (2017) or to learn character-level compasitionality Liu et al. (2017).
Our work differs from these approaches in two fundamental aspects. First, whereas prior studies use visual features a trainable component or auxiliary signal to enrich semantic embeddings, methodology employs them as a completely frozen, non-semantic substrate. Sccond, our primary goal is not to improve a specific downstream task, but to use this constraint as a tool to investigate foundational question: whether semantic understanding can emerge exchusively within the Transformer's compositional layers, without any semantic information provided at the input. In our work, this reframes the role of visual features from an enhancement to a basic, structural primitive.
3
METHOD
3.1
FROZEN VISUAL UNICODE EMBEDDINGS
Our method generates fixed embedding matrix E of shape (V, d d_model), where V is the vocabulary size and d model is the model's hidden dimension. The vector for each token is derived through a deterministic, multi-step process:
Glyph Rendering: Each token in the vocabulary is rendered into a bitmap. Singlo-character tokens are rendered directly. Multi-character n-gram tokens are rendered by horizontally concatenating the glyphs of their constituent characters. For example, the token 'ing" is rendered a single image containing the sequence of T, 'n', "g' glyphs. We use standardized font (e.g., unifout-14.0.01) at fixed-point size.
Image Standardization: The resulting wariable-width bitmaps are resized to fixed square resolution (e.g.. 32x32 for d model=1024) using bilinear interpolation. This creates a standardized visual representation for every token.
Vectorization: Each standardized HxH bitmap is flattened into a HF-dimensional raw vector (binary com- ponents).
Projection: To project the high-dimenssonal image vector into the model's 's latent space, we apply Principal Component Analysis (PCA). A PCA model is fit on the raw vectors of the entire vocabulary, and the top d model principal components are used to transform each raw vector into a d model-diiensioonll embedding. This preserves the maximum variance of the visual features within the target dimensionality (flotting points components). PCA was chosen for its determinism, compulational efficiency, and ability capture dominant visual variance in a fixed-dimensional space without introducing trainable parameters. This aligns with our core hypothesis of investigating emergent semantics absent of any data-driven feature learning at the input layer. While learned visual encoder like CNN could potentially extract richer features, it would re-introduce trainable parameters, confounding our experimental goal of isolating the Transformer's compositional layers.
The resulting embedding vectors are L2-normalized. The final matrix E is loaded into the Transformer's embedding layer, and its weights are frozen (i.e., Bequirrs__rrad-FPee for all subsequent training stages.
Normalization and Freezing:
The approach supports tokenizers from arbitrary LMs (e.g., Mistral Nemo tokens text used in bvv241-nemo tokenizer), as new tokens are mapped via their textual form.
To investigate the influence of PCA and normalization, as well as to understand the fundamental siguif- icance of visual representations of token texts, a maximally simplified approach was implemented. For a vocabulary of 65,536 tokens, a frozen pre-computed tensor was constructed based on binary components of a 16-dimensional vector for each token token- essentilll, each token was represented by its index as 16-dimensional vector where each component is single bit, thereby eliminating the influence of PCA algorithms, normaliza-
3

---

Published in Transactions on Machine Learning Research (10/2025)
tion, and additional information about the visual representation of the token text (model 'Model_16. [i_ee_____iit)) Similarly, to study the influence of PCA and normalization algorithms separately, another pre-computed tensor of 16-dimensional vector representations was constructed, but this time with the application of PCA and normalization (model 'Model_16 10______foool float')
For ablation study purposes, three additional variants with binary components based on noise were cre- ated for 64-dimensional, 256-dimensional, and 1024-dimeniional vectors. Three corresponding variants with applied PCA and normalization were also developed to investigate the impact convergence and metrics in the absence of information about the visual representation of the token text itself. ("Model 1024 bit', "Model [024 float', 'Model [0d bit', 'Model 256 float', 'Model 64 "ut', "Model 64 float').
3.2 SUICODE-CENTRCE TOKENIZER CONSTRUCTION
To test our hypothesis across diverse languages and existing token schemes, we required a method gen- erate visual embeddings for any given vocabulary. We developed Unicode-centric tokenization framework, bvv241, to serve this purpose.
3.2.1 N-gram Extraction
The first part is n-grams extraction: automatic extraction of the most. frequent. n-grams (sequences of n characters) from Wikipedia and Wiki-source text corpora across 32 world languages. For each language, the algorithm extracts n-grams of four different sizes (n = 2, 3, 4, 5 characters), counts their frequency using the Counter data structure, and preserves the top 1000 most frequent n- n-grams for each value of n. Determination of whether character belongs to CJK languages is performed by checking if it falls within Unicode ranges: U-4E500 9F for Chinese ideographs, U+30400 U-309F for Hiragana, U+30A0 U+30FF for Katakana, and A--ACOOOUDDAA for Korean Hangul. Results are saved in a structured format containing the language code, its full name, and a dietionary with top n-grams for each length, enabling subsequent use of this data for language identification tasks, morphological pattern analysis, or language model creation.
3.2.2 Tokenizer Implementation and Vocabulary
The second part is tokenizer implementation: algorithm implements the creation of a multilingual tokenizer with fixed vocabulary size of 65,536 tokens, based on statistically significant n-grams from the previous stage. The algorithm employs a direct mapping strategy for Unicode characters in the range 0x0000 OxD7FF (55,296 positions) to corresponding token indices, ensuring complete coverage of the Unicode Basic Multilin- gual Plane. The most frequent n-grams (8,177 unique sequences in total) are distributed across two ranges: the first 2,048 n-grams are placed in the Unicode surrogate pairs zone used for regular characters, while the remaining ones occupy the range OxF11000xF8FF Special tokens, including sequence start and end markers, dialogue markup tokens, and technical markers, are reserved in the range OxEOF0-OxEOFF (256 positions). Unused positions are filled with dummy tokens to achieve the target vocabulary size of 65,536 elements.
which is not
So, the core approach is based on bijective mapping of Unicode codepoints to individual tokens, whenever possible, ensuring a 1:1 relationship between characters and tokens across a wide range of scripts. Reserved, surrogate, and private Unicode ranges are systematically exploited to store custom and multi-character tokens, allowing for efficient representation of commonly occurring substrings and entire words pands-pprticulally beneficial for high-frequency terms shared across models and languages.
3.2.3 Vocabulary Expansion
The next version or expansion of the tokenizer from 65,536 to 131,072 tokens represents a qualitative tran- sition from basic Unicode coverage lo a hybrid architecture capable of efficiently processing modern multi- lingual corpora and specialized domains. In the base 65.536-token version, virtually the entire vocabulary is occupied by direct Unicode character mapping (approxinately 55,000 positions) and statistically identified n-grams (about 8,000), leaving minimal space for additional optimizations. Doubling the vocabulary size to 131,072 preserves all advantages of bijective Unicode mapping while simultaneously integrating approx-
4

---

Published in Transactions on Machine Learning Research (10/2025)
imately 19,000 high-frequency multi-character tokens extracted from analysis of leading language models. These additional tokens include whole words, stable collocations, technical terms, and programming con- structs that frequently appear across various languages and domains. This approach dramatically reduces tokenization fragmentation for specialized texts, code, and low-resource languages, where traditional BPE tokenizers may split semantically coherent. units into numerous small fragments.
Architecturally, the extended tokenizer utilizes the additional address space 0x10000 OxIFFFF accom- modate these high-frequency patterns, systematically exploiting reserved and private Unicode ranges. This achieves optimal balance between universality (every Unicode character is guaranteed to have token) and efficiency (frequent sequences are encoded with a single token). The hybrid design is particularly effective for processing mixed texts containing code-switching, technical documentation with embedded code snippets, and texts in non- non-Latin scripts, where standard tokenizers often demonstrate suboptimal performance due to bias toward English-languuge corpora during training.
Anac
2.46
2:03
1:43
2,83
2.37
1.44
1.72
2.03
Amesiin
1.02
0.59
0.59
2.40
1.05
1.49
1.79
2.45
Crasse
1.51
0.66
0.89
100
1.38
1.11
1.20
L0)
Souch
2.50
191
2,28
2.76
221
184
136
2.74
4.21
407
3.96
3.5
Fresch
2.78
2332
2.50
2.56
2.49
1.0
221
2.85
2.79
122
172
2.66
Groek
2.15
0:95
117
2447
1.20
123
1.54
247
Hebrow
2.36
0.90
106
2.35
2.33
1.31
1.7
195
3.65
2669
3.26
3.55
3.16
1.75
224
2.55
Belee
5.40
2.75
525
541
511
1.74
2:14
1.4
Jpanese
1.43
0.6
0.99
1.32
1.34
121
1.37
1.32
Latin
313
2.83
Nanwegin
1.53
2.59
2:11
2:46
2m
177
708
7
Pertian
2.30
0.87
133
2.76
1.67
1.48
1.7
2.78
2665
RuSSion
2:51
098
197
2.87
2.46
1.7
1.5
Sanskitt
1.1
0.60
0.53
1.2
100
1.24
1.47
L62
Sparish
3.71
2.30
2008
252
2.77
1.75
Vetrumese
2.47
1:40
2.23
2.97
2.87
1.50
1.58
2.0
DeepSeck- R1
CPT2
GF74
Mara
Oker
Bw941-23
bo244)-7a
Figure Average characters per token. Lower values indicate lower compression efficiency. Our bvv241 variants prioritize charactor-level granularity.
The bvv241-nemo variant is provided as a demonstration of the flexibility of the proposed methodology. While the underlying Unicode-centric mapping scheme is applicable to any tokenizer, bvv241-nemo specifically replicates the vocabulary, token structure, and exact. token enumeration of the SOTA Mistral Nemo tokenizer, using its public vocabulary files as template. This showcases how the approach can serve as a drop-in replacement or augmentation for existing BPE or unigram schemes.
Furthermore, since all token indices and their corresponding istic and cxhaustively specified, this methodology enables the direct crcation of pre-initialized embedding matrix of shape vocab size x n embed. Such a matrix can be either used as fixed (non-trainable) em- bedding or as a high-quality cold start, reducing the need for extensive embedding pre-training. Combining strict Unicode mapping with multi-character tokens, the bvv241 family achieves highly stable performance for Asian, Arabic, and underrepresented where differences from SOTA are minimal. For Indo-E aado-European (Tatin and partially Cyrillic) languages compression less aggressive.
representations determin-
5

---

Published in Transactions on Machine Learning Research (10/2025)
3.3 MODEL ARCHITECTURE
We employ standard decoder-only Transformer architecture, similar GPT-2. All models use hidden dimension d model=1024, n layer = 16, multi-head attention, head 32, rotary embeddings, and GELU aclivations. The key modification is that the token embedding layer (un. Euneeddndd has its weights frozen to the precomputed visual matrix (labeled as 'Model UNI GLIF", total parameters: 335.8M, trainable parameters: 268. 7M, frozen parameters: 67.1M). For comparison, we train baseline model with identical architectures but with a standard, randomly initialized and trainable embedding layer (labeled as 'Model unfrozen', total parameters: 335.8M, trainable parameters: 335.8M, frozen parameters: 0.0M). For ablation study; model with 16-dimensional binary frozen embeddings (labeled as "Model 16 6in',,nn total parameters: 269.7M, trainable parameters: 268.7M frozen paramcters: 1.0M, d modcl=1024, n cmbod=16)) model with 16-dimensional float frowen embeddings (labeled 'Model 16 float', total parameters: 269.7M 7M, trainable parameters: 2M frozen parameters: 1.0M, d model==1244 projected into the model's 1024-dimensional space via repetition (repeat_interlest interleave(64, dim=-1)), creating a simple, -raiaaaineele mapping.) ). The model does not use tied input/output output embedding design; the output projection head is trained independently.
embed=16). The 16-dimensional vector is
4 EXPERIMENTAL SETUP
4.1 TRAINING DATASETS
The training corpus comprises multilingual mix including subsets of Wikipedia (en, ru, zh), and SFT datasets (10% of iterations) SQuAD-v2, TriviaQA, HotpotQA, NQ-Open, BoolQ, CommonsenseQA, and ARC: approximately 4B tokens (train 4'121'073'286 tokens + SFT train 419'136 tokens, validation 157'897'03 tokens).
4.2 BASELINES AND COMPARISONS
Baselines include classic transformer LMS with trainable embedding layers matched for architecture, param- eter count, and tokenizer coverage. Our primary comparisons are between models with 'Frozen' embeddings and their 'Trainable' counterparts (e.g. 'Model unfrozen' vs. 'Model UNI ANI_GLFF) which share identical architecture and training data and tokenizer "bvv241-2-3' (vocab size: 65536).
4.3 EVALUATION
We report standard language model metries:
Train and validation loss/perplexity,
MMLU,
ARC and CommonsenscQA.
All metrics are computed on held-out test sets.
4.4 IMPLEMENTATION DETAILS
Training was performed on H100 80 Gb with batch sizes up to 18 and block sizes of 1024. Evaluation interval was 500 (cach 500 iterations MMLU, ARC, C-SENSE metrics recalculated), gradient accumulation steps parameter was 250, total iterations was set 400'000 due to strict computation and time limits (1.7 epochs). The final model checkpoint after 400'000 iterations used for final evaluation.

---

Published in Transactions on Machine Learning Research (10/2025)
5 RESULTS
Before presenting our results, it is crucial to frame their context. This study does not aim to achieve state- of-the-art performance on benchmarks like MMLU. Our models are intentionally constrained in scale (up to 0.3B parameters) and trained on a small dataset. The primary objective is to demonstrate the feasibility of learning with non-semantic embeddings and to compare the learning dynamics between our proposed frozen- embedding models and traditional trainableeembedding baselines under identical conditions. The following metrics should therefore be interpreted evidence of emergent abstraction, not challenge to large-scale models.
5.1 MODEL CONVERGENCE
A primary concern was whether a model with non-semantic, frozen embeddings could converge at all. Figure 2 and Figure 3 show the training loss curves and MMLU, ARC-e performance for 0.3B frozen-embedding model("Model UNI (LFF" and its trainable-embedding counterpart ('Model unfrozen'). Both models ex- hibit stable convergence profiles, with nearly identical loss trajectories.
Of particular nole is the convergence of models with 16-dimenssioall binary embeddings (Model 16 bit) and 16-dimensional float embeddings (Model 16 float), which effectively carried information only about the token index distinguish one token from another, without any information about the visual represcntation of the token text.
Loss over [Log Scalo)
Model UNN GLF Triis
Model Luntrozen Taan
Model Lio Tan
Model, 16, flot Trah
Los (rg)
Perplexity Scale)
PPL (og)
207
10a
300
700
303
430
500
600
700
800
Figure 2: Learning curves for frozen trainable embedding models.

---

Published in Transactions on Machine Learning Research (10/2025)
NMLU (smoothed)
25
Actrery
500
600
700
ARC-e % (smoothed)
Acorty %
15
00
100
200
100
400
500
600
700
Figure 3: MMLU and ARC-e curves for frozen vs. trainable embedding models.
5.2 PERFORMANCE OF FROZEN VS. TRAINABLE EMBEDDING AND SOTA MODELS
Our primary finding challenges the conventional wisdom that trainable, semantically-rich embeddings are necessary for high-level reasoning. As shown in Figure g and Figure 5 while our frozen-embedding models demonstrate robust convergence across all tasks, they show a striking and consistent performance advantage on the MMLJU reasoning benchmark compared to their architecturally identical counterparts with trainable embeddings. We also conducted comparison with state-of-the-arr (SOTA) models SmolLM (Allal et al. 2024) and SmolLM2 (Allal et al. 2025) (360M and 135M) that have comparable number of parameters.
For instance, 'Model UNI GLIF' (0.3B params, frozen embeddings) achieves an MMLU score of 23.81, whereas 'Model unfrozen' (0.3B params, trainable embeddings) scores only 12.54 - nearly 2x performance gap.
We hypothesize this phenomenon, which we term 'representational interference, stems from the dual burden placed on trainable embeddings. A trainable embedding layer must simultancously learn low-level token identification lask in terms of vector representation and high-level semantic abstractions within the parameter space. This creates conflict during optimization, leading to suboptimal representation that is neither perfect structural encoder nor pure semantic vector.
In contrast, our frozen visual embeddings provide a stable, information-rich structural foundation. By offloading the task of representing token form to a fixed, deterministic layer, we free the Transformer's attention and leed-forward layers to focus exclusively on their core competency: composing these structural vector primitives into high-level meaning. The model does not waste capacity learning what a token looks like; it only learns what to do with it.
This hypothesis is further supported by the (-SNE visualizations of the embedding spaces below.
8

---

Published in Transactions on Machine Learning Research (10/2025)
[wweica_lsoowlddgeee jnowledge
MMLU [oomputer__eee security]
MNLU fformal logic
23.2
11.5
9.8
12.1
26,0
30.2
175
21.1
NOLLL (giobal facts)
24.0
NMLL [high, school geographyl
24.7
12.7
11.3
13.7
11.6
17.7
14.7
26.5
MNLU (high, school government, and polttcs)
23.8
10.2
10.6
9.7
18.6
20.7
12.4
29.5
MMLU (righ, school mac ueconomiss
25.2
14.5
10.9
14.3
11.0
22.3
11.3
24.9
school psychology]
WMLU School us history]
22.1
8.5
10.2
8.1
27.0
24.0
20.6
22.6
MMLU thihh_chool_vcco worrl, historyl
22.6
10.5
9.0
8.5
26.1
23.6
20.7
24.3
(human, aginal
MMLU sexuslity]
24.4
MML [ursprudence)
23.0
11.0
11.0
12.0
26.4
21.3
15.7
27.8
MMLU [management)
24,4
10.3
10.2
10.9
21.4
24.8
11.7
29.2
beling]
MMW [medical genetics)
23.7
15.5
12.4
13.6
27.0
35.0
14.0
25.0
MMLU (meral, disputes)
24.7
12.6
10.3
15.5
24.9
24.3
12.1
22.2
MMLU (phsloemphyl
23.1
14.0
13.0
16.2
18.3
11.9
[professione _accounting]
MMLL [professional medicinel
23.9
MMLU [public jelat ons]
24.0
14.4
12.0
10.2
18.2
21.8
11.8
28.2
(security by
[tsuuies studies]
18.4
MHLU policy]
Model (baseline)
Model (trczen)
Modal (trozen)
Figure 4: Performance Comparison: MMLU sub tasks with score greater than 25%.
C-SENSE
19.75
19.75
19.16
19.89
32.70
35.30
33.90
MMLU
23.81
12.54
10.93
11.35
28.30
30.60
29.30
Model UNI (frozen)
Model (baseline)
Model_12 (frozen)
Model (fozzen)
SmolLM
SmolLM-
SmolLM2
Figure 5: Performance Comparison: Frozen vs. Trainable Embedding and SOTA SmolLM Models.

---

Published in Transactions on Machine Learning Research (10/2025)
It is critical to contextualize our baseline's performance against the SOTA models used in comparisons. The SmolLM models (~28% MMLUU) were trained on up to 600B tokens (smollm-corpuss) whereas our experiments were deliberately constrained to a 4B token dataset a 150x difference in compute and data, Our goal was not to match SOTA but to ensure a perfectly fair comparison between our proposed mothod and the baseline under identical, resource-consttrinee conditions (the only difference was the frozen mowen/unfrozen embedding layer).
Under these severe data constraints, it is the conventional trainable baseline [rodell_nffrzeen 12.5% MMLU) that dramatically underperforms. In contrast, our frozen- crozen-embedddng model (Model UNI GLIF, 23.8% MMLU) achieves score comparable to established models like GPT-2 137M 25.8%%-6..9% MMLUU.
5.3 EMERGENT SEMANTICS VISUALIZATION OF INPUT TOKEN EMBEDDINGS AND DEEP TRANSFORMER LAYERS
Our fixed visual embedding vectors naturally cluster by surface/Tormal fealures (e.g., length), rather than by meaning (Figure In contrast, trainable embeddings demonstrate soft semantic grouping, though the distributed latent space remains globally unstructured (Figure
t-SNE Embedding Visualization by Semantic Group (Mistral Nemo: Original Trainable Embeddings)
80
animals
fur
fruits
three
dities
colors
two
SiX
60
dog
professions
emotions
doctor
digits
teacher
40
apple
chef
ooe
driver
artist
writer
0-
mouse
gold
one
ten
red
anger
white
black
babii
pear
green
blue
Gray
cat
yellow
horse
-10
Ox
oon
love
orange
orange
-60
doy
china
-80
-60
-40
-20
20
40
60
80
Figure 6: t-SNE visualization of input token embeddings. In the trainable Mistral Nemo embeddings, semantic groups (e.g., numbers, professions, animals) tend to lie closer to one another, yet, globally, the point cloud remains mostly uniform.
10

---

Published in Transactions on Machine Learning Research (10/2025)
t-SNE Visualization by Semantic Group (Predefined Frozen Embeddings, bvv241)
20
wo bx
fruits
animals
sieffo
oy
cities
one sed
colors
dog
professions
black
emotions
gat
10-
digits
mouse anger Spple three
White
china
gray
blue
tea techer
jon
on
ero pear
cher
orangege
abbit
diVer
ellowriteer
artist
-30
-40
-30
-20
-10
20
40
Figure 7: t-SNE visualization of the frozen visual embeddings. The prominent clustering directly corresponds to token length. This phenomenon is direct artifact of our embedding generation process: multi-character tokens, when rendered and resized to a fixed-resolttion bitmap, produce images with higher average pixel density. As Principal Component Analysis (PCA) is designed to capture axes of maximum variance, it naturally identifies this structural feature as a dominant component. The resulting clusters are therefore based on a physical token property (length/densstty) not semantic content, reinforcing the conclusion that semantic understanding must emerge entirely within the subsequent Transformer layers.
The following part (Figures language model through extraction and visualization of activations from various neural network layers. The study examined 70 words organized into seven semantic groups (royalty, animals, cities, actions, objects, feelings, professions). Individual words were fed as input the model, and vector representations were extracted using forward hooks mechanism from key architectural components: input embeddings, attention projections (v proj and proj), and transformer block outpuls across all layers of the model.
10
presents a method for analyzing internal representations of a
The collected high- high--imennsonnal vectors were normalized using StandardScaler, followed by t-SNE algorithm application for dimensionality reduction to a two-dimensional space for visualization purposes. This approach enabled tracking the evolution of semantic representations as information propagates through the model layers. The analysis reveals how initially dispersed tokens gradually organize into semantically related clusters in deeper network layers, demonstrating the model's capability for hierarchical extraction and structuring of semantic features. This approach provides a tool for interpreting the "black box' of transformer models and understanding the mechanisms of semantic representation ion formation during natural language processing.
11

---

Published in Transactions on Machine Learning Research (10/2025)
Input embeldingg
Claa
Hettige
Figure 8: t-SNE visualization of input token emboddings for bascline ('Model unfrozen'). In this trainable embeddings, semantic groups (e.g., numbers, professions, animals) tend to lie closer one another and clustering.
Inpur embbedding
Realyy
Figure 9: t-SNE visualization of frozen input token embeddings for 'Model UNI GLIF'. In the this frozen embeddings, semantic groups (e.g., numbers, professions, animals) located randomly and without clustering.
12

---

Published in Transactions on Machine Learning Research (10/2025)
1ar bap
aad byer
becc ouppt
(mn kaw 11
p Jeye
bosk 15 cotpar
Figure 10: t-SNE visualization of first two (layer 0 layer 1) and the last two (layer 14 layer 15) attention projections (v_proj and o_proo), and transformer block outputs for bascline ('Model unfrozen'). In this picture semantic groups tend to clustering.
13

---

Published in Transactions on Machine Learning Research (10/2025)
by
ape 14
Und eitee
byw
Sper
Figure 11: t-SNE visualization of first two (layer 0 layer 1) and the last two (layer 14 layer 15) attention projections (v_proj and proj), and transformer block outputs for 'Model_ GLIF with frozen input eubedding layer. In this picture semantic groups (e.g,, nunbers, professions, animals) tend to make clusters after frozen embedding layer demonstrating emergence of semantic structure. This visualization provides direct evidence that semantic clustering, absent in the frozen input, is formed by the model's compositional blocks.
14

---

Published in Transactions on Machine Learning Research (10/2025)
5.4 ABLATION STUDY: THE ROLE OF EMBEDDING INITIALIZATION
This ablation study compares the performance of traditional trainable embeddings, frozen embeddings ini- tialized with Unicode-based visual representations, and frozen embeddings initialized with random black- and-white bitmaps, which serve as a randomized baseline.
Model JALLUL rrsh
Lion Chei
Hocell LCL, Soat mainn
300
200
800
800
Perpiesity Scale)
SLB
Nocel L004 Tea
20d Than
faat
132
Figure 12: Learning curves for the baseline model 'Model unfrozen' vs frozen EmEmmddding layer model 'Model UNI GLIF and randomly initialized frozen nn. Embeddinn layer models
MMLU Ss tomotthed)
Hedet Fast
ARC. %a tomoothed
HE0AL 204,
Falt
Figure 13: MMLU and ARC-e performance for the baseline model 'Model unfrozen' vs frozen nn. Emmeddinn layer model 'Model UNT GLIF' and randomly initialized frozen nn. Faneeddin layer models
This ablation study reveals an insight: even when frozen embeddings are initialized from random noise, the model still converges (Figure 12 Figure 13 Figure 14h. This finding suggests that the semantic structures
15

---

Published in Transactions on Machine Learning Research (10/2025)
ARC Ssmothedl
1004 Uea
Figure 14: ARC-c and C-SENSE performance for the bascline model 'Model unfrozen' vs frozen Embedding layer model 'Model UNI "LIF" and randomly initialized frozen nn Embeddinnn layer models
observed in trainable embeddings are not prerequisite for learning, but rather emergent artifact. We hypothesize that this phenomenon stems from trainable embedding layer being implicitly forced to solve two conflicting tasks: 1) Token Discrimination, the primary goal of assigning unique vector to each token, and 2) Semantic Proximity, the secondary goal of clustering vectors for contextually similar tokens. These objectives are often at odds. In essence, during backpropagation, gradients from high-level contextual errors "imprint" semantic relationships onto the embedding vectors, forcing them to serve as low-dimensional projection surface for knowledge learned by the entire network. Our proposed frozen-embedding approach resolves this conflict by decoupling the tasks. By providing stable, unique identifier for each token, the fixed vectors (whether visually or randomly initialized) perfectly satisfy the discrimination objective. This liberates the Transformer's compositional layers the attention and feed-forward networks to exclusively manage the learning of semantic relationships. We posit that this separation of concerns reduces optimization conflicts and allows the architecture to learn more efficiently, as evidenced by the superior performance on reasoning benchmarks.
6 DISCUSSION
Our findings suggest that the role of input embeddings in Transformers may be widely misunderstood. Meaning appears not to be property of the embeddings, but a process enacted by the architecture.
6.1 RE-EVALUATING EMBEDDINGS: FROM MEANING CONTAINERS TO STRUCTURAL PRIMITIVES
The success of our frozen-embeddding models, particularly MMLU, challenges the view of embeddings 'meaning vectors." Instead, they can be seen as "structural primitives.' The 'representational interference' hypothesis suggests a fundamental inelficiency in standard LLMs. By tasking the embedding layer with both structural and semantic learning, we may be creating an optimization bottleneck that harms downstream reasoning. Forcing the model learn, for instance, that the token "king" semantically close to "queen" but structurally dissimilar from "monarch" within the same vector space is a difficult, potentially conflicting task.
16

---

Published in Transactions on Machine Learning Research (10/2025)
Our method resolves this by offloading the structural representation to fixed, deterministic layer. The model doesn't waste capacity learning what token looks like; it only learns what to do with its form in context. This may be particularly advantageous for tasks requiring orthographic or morphological awareness, which could be implicitly tested in MMLU's diverse sub-tasks.
This suggests a hidden inefficieccy in standard models. Consider task requiring orthographic reasoning (e.g.. 'How many 'e's in "strawberries"?) If "strawberries" is a single token, its semantic embedding contains direct information about its character composition. The model must perform costly 'internal detok- enization,' using its deep layers to implicitly reconstruct the word's structure from memory. Our approach mitigates this by providing structural information at the input layer, potentially making the model more computationally efficient for a class of reasoning tasks. The performance gap on MMLU suggests that de- coupling structural representation from semantic composition more effective learning strategy, especially under constrained data and compute.
6.2 IMPLICATIONS FOR ARCHITECTURE AND EFFICIENCY
This paradigm opens several avenues for future architectures:
Modular and Standardized Embeddings: A universal, algorithmicaall-deriiee visual embed- ding matrix could become a standard component, allowing rescarchers to swap out different Trans- former backbones for direct comparison without confounding factors from different embedding ini- tializations.
Efficient MoE and Model Fusion: Mixture-of-Expent (MoE) models, which typically shard their MLP layers, could additionally benefit from a shared frozen embedding layer, reducing total parameter count and simplifying the routing logic by providing a common input space for all experts.
Improved Cross-Lingual Transfer: A single Unicode-based visual embedding space is inherently multilingual. This could provide a more robust foundation for zero-shot cross-lingual transfer, as the model learns to operate on visual script features common to many languages.
6.3 LIMITATIONS AND FUTURE WORK
This study was intentionally constrained in scale. Future work should test whether these findings hold for models at the 10B+ parameter scale. Additionally, while our method covers the entire Unicode text space, its generalization to truly novel modalities (e.g., mathematical formulas, chemical structures) remains an open question.
7 CONCLUSION
We have demonstrated that transformer LMs with froven, purely visual Unicode-based embeddings can learn rich linguistic and logical abstraction. Semantics in neural LMs is not a property of the embedding matrix, but an emergent phenomenon of the model's layered composition and self-attention. This insight opens new avenues for model standardization, modularity, and efficient scaling in langullingual and multi-domain NLP.
References
Jcan-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Mioch, Tain Barr, Yana Hasson, Karel Lone, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Nando de Freitas, Oriol Vinyals, and Andrew Zisserman. Flamingo: visual language model for few-shot learning, 2022.
Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. Smollm blazingly fast and remarkably powerful, 2024. URL.
17

---

Published in Transactions on Machine Learning Research (10/2025)
Loubna. Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martin Blázquez, Guilherme Penedo, Lewis Tun- stall, Andrés Marafioti, Hynek Kydliček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathicu Morlon, Colin Raffel, Leandro von Werra, and Thomas Wolf. Smollm2: When smol goes big- data-centric training of small language model, 2025. URL https::/ //axxiv org/abs/25022
02737
Jonathan H. Clark, Dan Garrette, Iulia Ture, and John Wicting. CANINE: Pre-training an cffi- cient tokenization-Iree encoder for language representation. In Transactions of the Association for Computational Linguistics, volume 10, pp. 726 745, 2022.
Falcon Dai and Zheng Cai. Glyph-aware embedding of Chinese characters. Manaal Faruqui, Hinrich Schuetze, Isabel Trancoso, and Yadollah Yaghoobzadeh (eds.), Proceedings of the First Workshop on Subword and Character Level Models in NTP, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. URL https://aclantthology http://ccaaatoooooy.oor//////-00 org/W17-1109/
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120): 2022.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. In Advances in Neural Information Processing Systems (NeurIPS), volune 32, 2019.
Guillaume Lample, Alexis Comneau, Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In International Conference on Learning Representations, 2018. URL
htttps://ppenreiiet net/foormmii
Frederick Liu, Han Lu, Chich Lo, and Graham Neubig. Learning character-level compositionality with visual features. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada, July 2017. Association for Computational Linguistics. URL https:
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space, 2013.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word repre- sentation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1532-1543, 2014.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rüccklé, Kyunghyun Cho, and Iryna Gurevych. AdapterFusion: Non-destructive task composition for transfer learning. Tn Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pp. 3637 77500,50 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Lan- guage models are unsupervised multitask learners. OpenAL, 2019. URL https::/cdat openai..cmm. better- waaggugge-mddes//aappae 2024-11-15.
mode_moolelll.eeee are unsupervised http://wiiedmmtitteddeeeeeee multitask learners, pdf Accessed:
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Πya. Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning (ICML), pp. 8748-8763, 2021.
Elizabeth Salesky, Neha Verma, Philipp Koehn, and Matt Post. Multilingual pixel representations for translation and effective cross-lingual transfer. In Houda Bouamor, Juan Pino, and Kalika. Bali (ods.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Singapore, December 2023. Association for Computational Linguistics.
18

---

Published in Transactions on Machine Learning Research (10/2025)
Noam Shazeer, Azalia Mirhoscini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. In International Conference on Learning Representations (ICLR), 2017.
Hao Tan and Mohit Bansal. Vokenization: Improving language understanding with contextualized, visual- grounded supervision. In Findings of the Association for Computational Linguisties: EMNLP 2022, pp. 4141 4116, 2022.
Haohan Wang, Peiyan Zhang, and Eric P. Xing. Word shape matters: Robust machine translation with visual embedding, 2020. URL https://arxiv: www.oo/ggg//.oo..../rr
Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. ByT3:: Towards a token-free future with pre-trained byte-to-byte models. In Transactions of the Association for Computational Linguistics, volume 10, pp. 291-306, 2022.
8 APPENDIX A: Model Parameters and Vocabulary Statistics
Table 1: Model Parameters and Vocabulary Statistics. Overview of models used experiments. 'Frozen' models use our fixed visual embeddings. "Trainable" models are baselines with standard, randomly initialized, trainable embeddings. All other architoctural parameters (d m0dcl=1024, layers, etc.) are held constant between frozen and trainable embeddings mode.
Model Name
Parameters and Vocab Size
Embedding Type
Model LUL_N__CC GLIF
0.3 B, 65536
Frozen (Visual Unicode)
Model unfrozen
0.3 B, 65536
Trainable
Model 16 bit
0.3 B, 65536
Frozen (Sequential 16-bit)
Model 16 float
0.3 B, 65536
Frozen (Sequential 16-float PCA)
Model 64 bit
0.3 B, 65536
Frozen (Random 64-bit)
Model 64 float
0.3 B, 65536
Frozen (Random 64-float PCA)
Model 256 bit
0.3 B, 65536
Frozen (Random 256-bit)
Model 256 float
0.3 B, 65536
Frozen (Random 256- float PCA)
Model 1024 bit
0.3 B, 65536
Frozen (Random 1024-bit)
Model 1024 float.
0.3 B, 65536
Frozen (Random 1024-float PCA)
9 APPENDIX B: Training loss and perplexity data
Table 2: Training loss and perplexity data
Model
Model
Model unfrozen
Model unfrozen
Model
Model
Model
Model
UNL_GI GLIF
UNI GLIF
16 bit
16 bit
16 f_ooat
16 float
Step
Loss
PPL
Loss
PPL
Loss
PPL
Loss
PPL
11.46
95123.07
11.30
80778.27
11.20
73204.74
11.23
75269.51
050000
3.42
30.51
2.96
19.21
3.10
22.17
3.08
21.69
100000
2.50
12.13
2.28
9.75
2.41
11.17
2.35
10.47
150000
2.10
8.20
1.97
7.19
2.08
8.04
2.08
7.98
200000
1.93
6.89
1.82
6.18
1.99
7.30
1.98
7.22
250000
1.81
6.10
1.74
5.72
1.88
6.53
1.86
6.45
300000
1.72
5.57
1.67
5.30
1.74
5.70
1.75
5.74
350000
1.62
5.05
1.62
5.08
1.63
5.12
1.62
5.07
400000
1.54
4.68
1.53
4.64
1.56
4.78
1.55
4.73
19


Reviews:
[{"review_id": "3AzY1L8uYK", "reviewer": "Reviewer_6yAd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose that semantic understanding in LLMs is an emergent property of the Transformer architecture itself, rather than a feature learned and stored in the input embedding layer. To prove this, they make Transformer models where the embedding layer is entirely frozen and derived from a non-semantic source: the visual appearance of Unicode glyphs.\nThe core experiment compares these \"frozen-embedding\" models against architecturally identical baseline models that use standard, randomly initialized, and trainable embeddings. The frozen-embedding models converge successfully and they significantly outperform their trainable-embedding counterparts on the MMLU reasoning benchmark.\n\nThe authors argue that a single trainable embedding layer is inefficiently burdened with two conflicting tasks: learning a token's low-level structural form and its high-level semantic meaning. By offloading the structural representation to a fixed visual layer, the Transformer's compositional layers are free to focus solely on learning semantics, leading to more efficient learning for reasoning tasks.\n\n\nStrenghts:\n\n- The central idea that meaning emerges purely from the Transformer's architecture, not its inputs, could be an exciting contribution to the field.\n- The methodology is clever and provides a clean test of the hypothesis. The direct A/B comparison between frozen visual embeddings and standard trainable embeddings under identical conditions is sound. The ablation study using frozen random embeddings is very interesting, as it strengthens the argument that the embedding's primary role may simply be to provide a unique and stable identifier for each token, rather than any specific visual or semantic content.\n- The fact that the frozen-embedding models not only converge but significantly outperform their trainable counterparts on a complex reasoning benchmark like MMLU is a striking result. This finding provides strong empirical backing for the \"representational interference\" hypothesis and forces the reader to seriously reconsider the role of embeddings.\n\n\nWeaknesses:\n\n- The paper looks more like a report than a proper paper, it presents a fascinating result and names a hypothesis (\"representational interference\") but does not formalize it. It lacks of depth and math grounding. \n- The choice of visual features (derived from a specific font, size, and PCA projection) feels arbitrary. The paper provides no study on how sensitive the model's performance is to these choices, or whether a different non-semantic (but still structured) embedding might work better or worse.\n- The paper is easy to ready, but unfortunately it's a bit redundant (some concepts are repeated two or 3 times), and there are some \"cliche\" things that I think could undermine the credibility of the paper.\n-", "claims_and_evidence_yn": "Yes", "claims_explanation": "I think the core idea is very interesting and the author back ther claims with proper experiments.", "audience_yn": "Yes", "audience_explanation": "The core idea is very interesting, and I think it could have an impact for the field.", "requested_changes": "Tecnical:\n- Fromalize what \"interference\" means in terms of gradient dynamics or information theory and add more formal description on the observed phenomenon.\n\nWriting:\n- Remove all the redundancies and maybe polish a bit the language to make it sound less cliche (see things like \"This has profound implications\").\n- Tab 1 could be moved in the appendix to make the reader focus more on the core details.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9s9mkC7Pr8", "reviewer": "Reviewer_LcuB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the question \"What if we didn't train embeddings?\". Instead, embeddings are computed using visual representations of chunked text and then assigned a fixed embedding using PCA such that semantics are broadly not taken into account. This is experimentally compared to systems with the more typical learned word embeddings and also randomized visual embeddings. In the experiments a massive increase in quality is observed on a 200M parameter model with the non-learned embeddings (~2X absolute score on MMLU).\n\nThe authors offer a hypothesis for this: Their \"representational inference\" hypothesis states that embeddings being burdened with semantic features can cause optimization bottlenecks such that models underperform.\n\nThese studies also offer evidence that Transformers can store semantic knowledge almost entirely in non-embedding layers (something previously known, at least for encoder models).", "claims_and_evidence_yn": "No", "claims_explanation": "Not *yet* -- my primary concern is the baseline with learned embeddings.\n\n## Baseline\n\nThe authors state the very proper goal of `“Instead of pursuing state-of-the-art benchmark scores, which often depend on massive scale, our primary goal is to isolate and test a fundamental hypothesis: can language abstraction emerge without trainable,\nsemantically-rich input embeddings? We deliberately constrain model and dataset sizes to create a controlled environment…” `. This is, in my opinion, exactly the sort of science TMLR should promote.\n\nThe flip side of this coin (avoiding simply competing for SOTA) is that it comes with a much higher danger that the baseline may be flawed in some subtle way (since it's less likely that SOTA results come from buggy models); this imposes a higher burden of evidence on non-SOTA baselines that they are sufficiently strong so as to offer a fair comparison and that the comparison will yield durable conclusions. The most convincing methods I've seen to demonstrate that one's baseline is strong:\n* Using well-established public baselines (e.g. OLMO) and then scaling them down, showing publicly-demonstrated results, and then the scaled-down created for the purposes of the paper's experimentation such that the gap can be seen in the absence of other modifications; or\n* comparing your baseline with another well-trusted system that's directly comparable (e.g. in terms of parameter scale) and demonstrating that they achieve a similar level of quality.\n\nAs-is, I'm left wondering if there's some experimental confounder that could be causing the massive quality change (doubling the absolute score) between the baseline with learned embeddings and the experimental system with fixed embeddings as such gains from embeddings alone would be very out of the ordinary (though admittedly, not strictly impossible).\n\nFor example, could it be that the embeddings weren't initialized with sufficiently small/large values of an appropriate variance with standard normalization techniques at initialization and during training? Were the optimizer hyperparameters set such that embeddings would be easily learnable given the model scale?\n\nWhile eventually, community replication of the results will be the bar, some additional evidence is needed within this work.\n\n## Other Evidence\n\nBesides the question of the baseline, I really like the design of the experiments such that they include both learnable tokens and randomly initialized embeddings and show both perplexity and other evaluation results.", "audience_yn": "Yes", "audience_explanation": "The question of whether modern generative LLMs benefit from learnable embeddings is very interesting and the authors' methods of exploring this with embeddings that use PCA on visual representations is also creative and likely to provide new knowledge for many folks in the field (myself included).\n\nNote: This question has been previously answered for earlier generations of Transformers (see Requested Changes below).", "requested_changes": "## Primary Concerns\n\n* Evidence for the trustworthiness and correctness of baseline system (see above) since this could dramatically change the conclusions that can be drawn from the presented experiments.\n\n## Needed Changes\n\n* Additional prior work on visual embeddings should be cited as this is not entirely novel:\n  * “Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfer” Salesky et al, 2023.\n  * “Learning Character-level Compositionality with Visual Features”, Liu et al, 2017\n  * “Glyph-aware Embedding of Chinese Characters”, Dai 2017\n  * “Word Shape Matters: Robust Machine Translation with Visual Embedding”, Wang 2020\n* Additional items to specify about both baseline and experimental systems, since the tokenizer is being modified:\n  * How many passes over the *same data* were made in training? (Differing sequence lengths with packing can change this, even at the same batch size)\n  * How were differing sequence lengths handled? What were the average sequence lengths under each tokenization scheme?\n  * How were evaluation checkpoints selected and what was the training termination criterion?\n* Having Figure 2 and 3 separate is very hard to compare; it would be much better to overlay them.\n* In addition to Figure 2/3, it would be good to see numeric perplexities in a text table.\n* In Section 5.2, the authors claim that current representations must learn character shapes and token boundaries. I don't believe this is true (unless character shapes are somehow required by a specific training task). Token boundaries are typically determined offline by SentencePiece. I recommend removing this claim (or providing evidence for it).\n* Figure 4/5: It's difficult to tell which columns are baseline (non-frozen) vs experimental (frozen); please label explicitly.\n\n\n## Minor Suggestions\n\n\n* The claim in 3.2 that this method is more linguistically inclusive is somewhat dubious: Having an embedding without learned semantics is likely not useful.\n* Period on last sentence of intro\n* In section 2, dash should be attached to \"Character -\"", "broader_impact_concerns": "N/A", "additional_comments": "Overall, I think the ideas in this paper are great, that they will be broadly interesting to the community, and that the general experiment design will allow drawing very interesting conclusions.\n\nI would like to see this paper get to the point of publication by working through the items above.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HsGsT8NZWW", "reviewer": "Reviewer_KxDw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper studies and implements a frozen word embedding based on the visual information of the characters. In details, it first visualizes the characters and using PCA to project the binary map to features. \n\nThe contributions are listed as below:\n1. The paper shows the success of non-learnable word embedding in language model training, which can potentially reach good (even better) performance when evaluating on downstream tasks.\n2. The word embedding scheme is based on visual information and is not learned (i.e., using PCA instead). Thus this method can be potentially applied to nearly all character systems. \n3. It proposed \"representational interference\" as an explanation of why this frozen embedding can improve over the learnable embeddings.\n4. The paper claimed open-source and the code are submitted into supp.", "claims_and_evidence_yn": "No", "claims_explanation": "I think that the paper almost aligns with the claims. For several points, it would be better to further extend the explanation.\n\nFor 2 (universal tokenizer), please provide a detailed implementation of the bvv241 tokenizer. Although it is not listed in the main claims of contribution, understanding the detailed implementation (or have a detailed reference) would help readers to understand the details.\n\nFor 3 (representational interference), I would request more analysis to convince the reader. The hypothesis is currently mostly explained in plain words without strong experimental results. The result of \"random frozen embedding\" is interesting. It might be extensible to create a more solid claim here. \n\nFor 1 (better performance of frozen embedding), I think that the evidence is not enough to claim \"better\". However, I would not force that since it would be an endless game. I hope that the author can provide more analysis to help readers to understand \"why\" it is better. For example, provide some examples of winning cases and show some statistics on why it works.", "audience_yn": "Yes", "audience_explanation": "I think that the findings in the paper is interesting, and I personally enjoyed the reading. The tokenization and embedding methods are both novel. The results to show that frozen embedding actually works is also quite interesting. \n\nI hope that this paper can be published after some revisions.", "requested_changes": "As I am not sure about the compute resource of the authors, please let me know if some requests beyond the compute budget. We could find way to get around it. \n\nMajor:\n1. I would like to see more analysis and explanation towards the reason why a frozen random embedding can \"converge\" but is much worth than the PCA embedding. Would that mean some grouping based on the \"shape\" of the character help the training a lot?  \n\n2. The semantic embedding and non-semantic embedding might drive to different behaviors (e.g., attention maps, the scale of the residuals, weight norms...). It would be great to see more studies over it. \n\n3. I would like to understand more differences in claims between this work and char-LMs. Char-LMs embed characters thus fundamentally can not have mapped the characters to semantically-rich features. \n\n4. The uniform representation of characters in different languages is impressive. It would be preferred to extend the explanation for bvv241 tokenizer. I tried to follow the context to understand the construction of such tokenizer and found it a little bit hard. \n\n\n\nMinor:\n- Would it be possible to try a frozen CNN to extract the feature? \n\n- L2-normalized feature: does that introduce a different to standard model’s embedding scale? As transformer (sometimes) initialized with N(0, 1). \n\n- Fig 1 is not cleaned off; there are some remaining noise at the bottom of the figure. Fig 4 has the same issue. \n\n- Fig 8 and 9 are not referenced in the paper. \n\n- For Fig 2 and Fig 3, it is better to draw the plots of frozen visual emb and trainable emb together to understand the difference. It is the same for Fig 8/9/10.\n\n- I did not get the point “Mixture-of-Experts (MoE) models could share a common frozen embedding layer” in the discussion. MoE usually sharded over the MLP layers instead of word embedding layers?", "broader_impact_concerns": "NA", "additional_comments": "Questions:\n\n- Does the model architecture use a tied input/output embedding design? (i.e., the output projection layer from final model output to word logits shares the weight with the input word embedding.)\n\n- What would be the range of number of characters per token. Would 32x32 be enough for the longest token?\n\n- As the model perform PCA over the HxH vector. If H=32, I think that the maximal number of features (i.e., model dim) is bounded by 1024?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)