Growing Transformers: Modular Composition and Layer-wise Expansion on a
  Frozen Substrate
Authors: name: A. Bochkov
Published: 2025-07-08 20:01:15

Abstract: The prevailing paradigm for scaling large language models (LLMs) involves
monolithic, end-to-end training, a resource-intensive process that lacks
flexibility. This paper explores an alternative, constructive approach to model
development, built upon the foundation of non-trainable, deterministic input
embeddings. In prior [1], we established that high-level semantic reasoning can
emerge in Transformers using frozen embeddings derived from the visual
structure of Unicode glyphs. Here, we demonstrate that this fixed
representational substrate acts as a universal "docking port," enabling two
powerful and efficient scaling paradigms: seamless modular composition and
progressive layer-wise growth.
  First, we show that specialist models trained on disparate datasets (e.g.,
Russian and Chinese text) can be merged into a single, more capable
Mixture-of-Experts (MoE) model, post-training, with zero architectural
modification. This is achieved by simply averaging their output logits. The
resulting MoE model exhibits immediate performance improvements on reasoning
benchmarks like MMLU, surpassing its constituent experts without catastrophic
forgetting. Second, we introduce a layer-wise constructive training
methodology, where a deep Transformer is "grown" by progressively stacking and
training one layer at a time. This method demonstrates stable convergence and a
clear correlation between model depth and the emergence of complex reasoning
abilities, such as those required for SQuAD.
  Our findings suggest a paradigm shift from monolithic optimization towards a
more biological or constructive model of AI development, where complexity is
built incrementally and modules can be composed freely. This opens new avenues
for resource-efficient scaling, continual learning, and a more democratized
ecosystem for building powerful AI systems. We release all code and models to
facilitate further research.