TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters
Authors: Haiyang Wang
Yue Fan
Muhammad Ferjad Naeem
Yongqin Xian
Jan Eric Lenssen
Liwei Wang
Federico Tombari
Bernt Schiele
Venue: ICLR 2025 Spotlight (2025)
Keywords: Fully Attention-based Neural Network
Large Language Model
Model Scaling
Tokenized Model Parameters
Primary area: foundation or frontier models, including LLMs
TL;DR: Designing a fully attention-based neural network for efficient model scaling through treating model parameters as tokens.

Abstract: Transformers have become the predominant architecture in foundation models due to their excellent performance across various domains. However, the substantial cost of scaling these models remains a significant concern. This problem arises primarily from their dependence on a fixed number of parameters within linear projections. When architectural modifications (e.g., channel dimensions) are introduced, the entire model typically requires retraining from scratch. As model sizes continue growing, this strategy results in increasingly high computational costs and becomes unsustainable. To overcome this problem, we introduce Tokenformer, a natively scalable architecture that leverages the attention mechanism not only for computations among input tokens but also for interactions between tokens and model parameters, thereby enhancing architectural flexibility. By treating model parameters as tokens, we replace all the linear projections in Transformers with our token-parameter attention layer, where input tokens act as queries and model parameters as keys and values. This reformulation allows for progressive and efficient scaling without necessitating retraining from scratch. Our model scales from 124M to 1.4B parameters by incrementally adding new key-value parameter pairs, achieving performance comparable to Transformers trained from scratch while greatly reducing training costs. Code and models are available at {\color{red}\url{https://github.com/Haiyang-W/TokenFormer.git}}

Published as a conference paper at ICLR 2025
TOKENFORMER: RETHINKING TRANSFORMER SCAL- ING WITH TOKENIZED MODEL PARAMETERS
Haiyang Wang" 133+ Yue Fan'; Muhammad Ferjad Naeem², Yonggin Xian", Jan Eric Lenssen', Liwei Wang", Federico Tombari', Bernt Schiele Max Planck Institute for Informatics, SIC *Google &Peking University
ABSTRACT
Transformers have become the predominant architecture in foundation models duc to their excellent performance across various domains. However, the substantial cost of scaling these models remains a significant concern. This problem arises primarily from their dependence on a fixed number of parameters within linear projections. When architectural modifications (e.g.. channel dimensions) intro- duced, the entire model typically requires retraining from scratch. As model sizes continue growing, this strategy results in increasingly high computational costs and becomes unsustainable. To overcome this problem, we introduce Tokenformer, natively scalable architecture that leverages the attention mechanism not only for computations among input tokens but also for interactions between tokens and model parameters, thereby enhancing architectural flexibility. By treating model parameters as tokens, we replace all the linear projections in Transformers with our -orem-parrmeter attention layer, where input tokens act as queries and model parameters keys and values. This reformulation allows for progressive and efficient scaling without necessitating retraining from scratch. Our model scales from 124M to 1.4B parameters by incrementally adding new key-value parameter pairs, achieving performance comparable to Transformers trained from scratch while greatly reducing training costs. Code and models are available https:/ htps:///gihub.. wwt.ggtthhhhooo www.uub.comHHia. comealla.omm/aaaagggg W/ TokenF ormer. git
INTRODUCTION
Designing a powerful neural network architecture is a long-standing goal in machine learning. Recent developments in foundation models (FMs) have shown the potential of Transformers (Vaswani ct al. 2017) as a universal computational architecture. Thanks to their flexibility and scalability, Transformers have achieved state-of-thh-aa performance across various domains, including natural
language processing (NLP) (Radford et al.
2018 Alee al. 2019 Brown al.) 2020), visual
modeling (Dosovitskiy et al. 2021: Jiu et al.
2021 vision-language
(Lu et al. 2023 Wang et al.
2024 graph representation Ying et al. 2021 and 3D vision (Wang al.) 2023a
Transformers typically divide the computation required to process a single token into two distinct parts: interactions with other input tokens (token-token interaction) and computations involving the model's 'ss parameters parkeneprrmmetr interaction). The attention mechanism (Vaswani et al. 2017) facilitates token-token interactions, allowing modern general-purpose foundation models to encode multi-modal data into a unified token sequence and effectively capture complex dependencies among them (Liu al., 2023 Zhu et al., 2203 Wang al. 2023d). Conversely, token- -erammprraetter computations rely heavily on linear projections (Dunford & Schwartz 1988). where input tokens are multiplied by a fixed set of parameters. This preseribed design limits scalability because increasing the model size requires altering core architectural components, often necessitating retraining the entire model from scratch. As models grow larger, this results in excessive resource consumption, making it increasingly impractical. In this paper, introduce novel architecture that enhances the flexibility of token-pane parameeeree interactions, allowing for incremental scaling of model parameters and effectively reusing previously trained models, thus significantly reducing the training burden.
"Equal contribution.

---

Published as a conference paper at ICLR 2025
Scaling Transforner from scretch Scaling Takenfarmer incremertally
Tokem-Param Attn
Param
Value Param
15
Perplexity
124M 354M
Token
Token-Token Attn
24
V
QKV Projection
Token-Param Attn
Input
Input
Key Param
Value Param
2004
4003
8000
10000 12000
Training cost TPU hours
Transformer
Tokenformer
Figure 1: Traditionally, large transformer architectures are trained from scratch without reusing previous smaller-scale models (represented by blue dots on the left). In this paper, we propose novel fully attention-basse architecture that allows scaling model incrementally, thus greatly reducing the overall cost of training large transformer architectures (depicted by red dots the left). The right panel delineates a comparison between conventional Transformer and our Tokenformer.
To achieve this objective, we introduce Tokenformer, a novel architecture that unifies the computations of token-token and token- ooken-parameter interactions by entirely employing the attention mechanism. The flexibility of our token- token-paramedd parameter attention layer, along with its ability to handle a variable number of parameters, inherently enhances the model's 's scalability, facilitating progressively efficient scaling.
As shown in Figure [ we extend the Transformer architecture by preserving the computational patterns between input tokens while reformulating all the linear projections using a cross-attention mechanism. Specifically, to project features with input and output dimensions Dj and D2, we employ two sets of parameters, each comprising N learnable tokens with channel dimensions of Dj and D2, respectively. In this formulation, input tokens serve queries, and model parameters keys and values. This ficxibility renders our model's parameters inherently scalable with variable N, allowing for efficient expansion by continuously adding new key-value parameter pairs. Figure our model can be scaled incrementally from 124M to 1.4B parameters, achieving performance similar to training from scratch while saving more than half of the training cost.
[ shows that
The key contributions of this work are summarized as 1) As shown in Figure 1 we propose Token- former, fully attention-driven neural network that treats model parameters tokens, maximizing the flexibility of token-parameter computations while achieving competitive performance on standard benchmarks across both language and vision domains. 2) Thanks this design, our model can be naturally scaled by progressively adding new key-value parameter pairs. Compared with the approach (Biderman et al. 2023 Kaplan et al. 2020), our method achieves nearly the same performance while greatly reducing training costs.
We hope that the idea of tokenizing watttthiinnghheeee be data, parameter, or memory-and utilizing attention mechanisms to build connections between them will introduce unique perspective on model architecture, inspiring innovative designs for future networks.
RELATED WORK
Transformer (Vaswani al. 2017) has emerged as foundational architecture in deep learning due to its versatile attention mechanism, enabling it to process any tokenized data and adapt to numerous domains, including language modeling (Radford al. 22088] Touvron et al. 2023). image
processing (Dosovitskiy et al. 2021). multi-modal understanding (Liu et al. [023 Wang et al.) 2024 2023b 2022), decision making (Chnn et al., 2021b), graph learning (Yun et al.) 2019), among others. While the Transformer effectively handles interactions among input tokens with flexibility, this property does not extend to computations involving model parameters, which are conducted via prescribed linear projections. In this work, seek to restructure loken-parameter interactions by developing a fully attention- network that unifies both token-token and parameer computations through attention mechanisms, thus further extending the network's 's flexibility.
2

---

Published as a conference paper at ICLR 2025
Large Scale Training has proven to be effective approach for developing powerful foundation models. As demonstrated by models like the GPT series (Radford et al. 2018 Alec et al. 2019) Brown et al. 2020), simple architectures-wwiit when supported by larger training datasets and increased model sizes (measured parameters) is generally more cost-effective because it is independent of the model's architecture and allows for the continuous integration of new data through fine-tuning existing models (Kaplan et al. 2020). In contrast, increasing the model size often incurs extremely high costs, as it alters architectural details and usually requires retraining the entire dataset from scratch at each scaling step (Biderman et al. 2023). This significantly raises the expenses for building progressively larger models in the industry.
outperform more complex algorithms. Scaling up data
Model Reusing. Previous methods for reusing models have typically involved initializing larger models with pre-trained smaller models by duplicating (Chen et al. 2013] 2021a), stacking (Gong et al., 2019), or combining (Wang et al., (023c) model weights. While these approaches can be effective, they often disturb the pre-established distribution of the smaller model, increasing the risk of losing pre-trained knowledge and slowing convergence. In contrast, our model allows for parameter scaling in a natural and seamless manner and preserves the integrity the existing model.
METHODOLOGY
In this section, we first revisits the conventional attention mechanism in Section 3.1 Then, Section 3.2 introduces Tokenformer, a natively scalable architecture centered around flexible token-parameter attention layer. Finally, incremental model scaling of Tokenformer is detailed in Section 3.3
3.1 PRELIMINARIES
Transformer models (Vaswani et al. 2017) have established themselves as fundamental architectures in deep learning, demonstrating outstanding performance across wide range of tasks. The corner- stone of their success is the self-attention mechanism, which allows the model to dynamically assess the importance of each token, efficiently modeling complex dependencies among them.
Given a set of 7 input tokens X E RTxd with channel dimension d, the self-attention block first derives input-dependent query Q, key K, and value V, with three distinct linear projections as
x Wº
K-X-WA.
where the W W E dddx and WV E _ddx are learnable weight matrices. The attention scores are calculated by measuring the similarity between query and key vectors, followed by softmax function to obtain normalized weights. These scores are subsequently used to compute the output of the scaled not-product attention as,
Attention (Q, K, V) = softmax Q-K K 1-V,
(2)
Vi
where Vd is a scale factor for alleviating small gradients caused by softmax. Finally, the output is,
O = Xan W
with Xan being the attention output and O E Rd. xd as the output projection matrix.
The above architectural design enables the model to flexibly manage interactions between tokens of varying lengths, thereby allowing modern general models to concurrently process any form and quantity of tokenized multi-mode data. This capability markedly enhances the development of current Al domain and is fundamental to the success of transformer-based systems.
3.2 TOKENFORMER
Although transformers excel across various domains, their scalability is limited by high computational overheads resulting from prescribed laken-parameter interactions (i.e., linear projections). As result, scaling strategies that adjust architectural components (e.g., channel dimensions) typically require retraining the entire model from the beginning, leading to inefficient of computational resources.
To overcome this challenge, we propose Tokenformer, an architecture entirely based on attention attention (Pattention) layer,
mechanisms. The central innovation of Tokenformer is
3

---

Published as a conference paper at ICLR 2025
Input Tokens
Pattention Layer
Key Tokens
Value Tokens
Norm
Input Tokens
Matching
Pattention
Pattention
Pattention
TokenForme.
Attention Scores
Output Tokens
Taken-Tokee Token
Weighted Sum
Pattention
Parameter Reusing Pattention
Key
Norm
Input
Old
New
Old 4
New d
Pattention
Token-Parameter Cross-Attention
Tokenformer is fully attention-driven architecture featuring a new token-Paan Parameter attention (Pattention) layer. The Pattention parameters and lets the imput tokens attend to them. As the model scales, Tokenformer adds new learnable tokens to expand the existing key-value parameter sets, while keeping the feature dimension constant and leaving the rest of the computation unaffected.
Figure 2:
set of learnable tokens represent model
which incorporates set of trainable tokens functioning model parameters and then employs cross-attention to manage interactions between input tokens and these parameter tokens. this way, the Pattention layer introduces operates independently of the input and output channel dimensions. This decoupling enables input data to dynamically interact with variable number of parameters, providing the flexibility required for incremental model scaling by reusing pre-trained models. Consequently, training larger models is greatly accelerated while achieving performance par with transformers trained from scratch.
additional dimension- the number of parameter likens-whice
Notation
Description
Notation
Description
Q, K, V softmax
Input-dependent query, key, value. Standard softmax: exp + Linorm
(K2, V (KK (KY. (KK.V))
Param tokens for generating Q, K, V. Modified softmax: Lznorm + gelu
Let the input tokens and output tokens be represented as I Txdd and O E xd2, where T' is the sequence length, and dj and d2 are the input and output dimensions, respectively. To implement our Pattention mechanism, we introduce two sets of n learnable parameter tokens: Kp E Roxdd representing the keys, and Vp E Raxd2 representing the values. The output O from the scaled dot-product Pattention layer is computed as:
Pattention Layer
Pattention X, Kp, Vp) θ (X.-K))-V Vp,
where θ is modified softmax operation for stable optimization of Pattention layer. Standard softmax involves two steps: applying an element-wise exponential the input matrix and then performing LI normalization along the token dimension. Our modified softmax replaces this with L2 normalization along the token dimension, followed by applying the GeLU activation each element. This design improves gradient stability in our architecture, smooths the attention scores, and results in better performance compared to the standard softmax operation (see Appendix A and Table for details).
layer employs cross-attention mechanism to manage interactions between tokens and parameters, thereby fully preserving the adaptability characteristic of attention mechanisms. Similar to how self-attention in Transformer models handles sequences with variable lengths, our Pattention layer is designed to process a flexible number of parameters independently of the input and output channel dimensions used in feature projection. This allows network parameters to be expanded seamlessly along the parameter token axis, enabling the effective reuse of pre- trained weights and offering naturally incremental manner for model scaling.
Our Pattention
4

---

Published as a conference paper at ICLR 2025
Overall Architecture. Figure Segurreiilustrattee illlstrates the architecture of Tokenformer. Given the input tokens Xin E 12'x we follow the design of the pre-norm transformer, the computation for the output of a Tokenformer layer is represented as follows:
X Xin + MHA(LN(X
(5)
X Xinter + FFN(LN(XMmee)
(6)
where LN denotes the layer normalization (Ba 2016, Zhang Sennrich, 2019), and MHA and FFN refer to our modified multi-head self-attention and feed-forward layer, respectively.
In the multi-head self-attention block, for simplicity, we consider a single-head variant and set both du and d, equal to d. Then we replace all the linear projections with our Pattention layers. Let (N( be denoted as X, this block is formulated as follows:
Q = Pattention( X, kp, Vg), K Pattention (X, KP VF), V Pattention Nattention(XAAe kY, V
(7)
Q K
X art = softmax
V,
(8)
vd
0at Pattention (Xau, (X, V9),
(9)
where Eq. and represent roken-parameter attention while Eq. represents token-token attention. The key-value parameter tokens for the QKV projections are (Kp, vp) E RT, xd, (K V
R xd (KY, V E R while (KP, VP) E R used for the output projection layer.
For consistency and simplicity, the feed-forward block in Tokenformer utilizes a single Pattention Layer. Denote LN(Xinter) as Xtins and the FFN computation given by:
O Pattention (X K-f v-ft)
(10)
where p. V/h E -unmxxd are learnable key-value pairs for FFN block.
By designing the architecture in this manner, we represent all fundamental components-including both input data and model parameters- as tokens within the computational framework. This token-centric perspective allows the utilization of successful attention mechanisms to unify two primary computa- tions within the transformer, token-token and moken-parameter interactions, thereby establishing a fully attention-bann based neural network characterized by exceptional flexibility.
Architecture Configurationss Our model meticulously mirrors the hyper-pane rrper-parameter configuration of the standard Transformer architecture. Taking GPT-2 (Radford et al. (0188) as exemplar, which features 12 Transformer layers and hidden dimension 768, our model replicates this configuration with identical layer counts and dimensionality. The number of key-value parameter pairs in both the guery-key-valus and output projections corresponds directly to the hidden dimension. In contrast, the FFN module utilizes four times the number of parameter pairs relative to the hidden size. This architectural alignment facilitates the initialization of our model's parameters using a pre-trained Transformer, thereby ensuring seamless integration into the Transformer pre-training ecosystem.
3.3 PROGRESSIVE MODEL SCALING
Our model demonstrates strong suitability for large-scale model training along the parameter axis, at- tributable the versatile design of the Pattention layer, which allows for the incremental development of larger models by reusing parameters from smaller, pre-trained counterparts.
To facilitate understanding without compromising generality, we employ single Pattention layer to exemplify the intricacies of model scaling. Consider an existing Tokenformer model equipped with a set of pre-trained key-value parameter tokens, denoted as Kold, Vidld Ba As shown in Figure 2 to scale the model, we augment this set by appending new key-value parameter tokens K
new e mpmxd xd as
K P
[Kppd Keew]
Vigalee
[Vgld
Vnew
(11)
where .old _ means the concatenation operation along the token dimension and Pepaae V R ((m+n))x xd are scaled parameter sets. The forward pass of the scaled model is then defined as
neww]
0 Pattention (X, Kacalo V/saae).
(12)
5

---

Published as a conference paper at ICLR 2025
Scaling Transformer from scratch with 3008
16
Sca ing transformer from scratch with 30c3
Incrementally scaling Tokenformer incrementally scaling Tokenformer with 308
from 306
incrementally scaling Tekentormer with 303
incrementally scaling Tokenformer with 60B
Perples leplextty
Perplexityy
124M 354M 757M
1.48
12
2000
4000
6008
0000
10000
19000
2003 9000
E000
2920
10020 12:00
Training cost
Training cost TPU hours
Figure 3: Evaluating model scaling costs through cumulative computational budgets. The Trans- former baseline incurs expenses for each individ- ual scaling step performed independently from scratch, whereas Tokenformer aggregates costs across all scaling stages, including training 124M model initially, progressively scaling to 354M, 757M, and 1.4B parameters.
Figure 4: Evaluating model scaling costs by mea- suring the budget required at each scaling stage. The Transformer baselines used are consistent with those depicted in Figure 3 trained with 30B and 300B tokens. Similarly, for Tokenformer, the cost is the budget required for each incremental scaling step from smaller one. All the experi- ments were conducted on TPU v4 hardware.
This scaling scheme permits the integration an arbitrary number of parameters without altering the input output dimensions. As demonstrated in Figure this approach notably enhances training efficiency for models greater scales without degrading performance. Importantly, if we choose to initialize KP with zero and 1/pe with random values, similar to LoRA technique (Hu et al. 2022), our model can effectively resume from pre-training phase without losing the well-learned knowledge, facilitating faster convergence and accelerating the overall scaling process. Moreover, this manner provides new dimension for scaling beyond just the hidden dimension, which helps maintain manageable token-token interaction costs and is well-suited for long-context modeling.
EXPERIMENTS
In this section, we present experimental results for the techniques described above. Section 4.1 validates the continual expansion capability of our model. Section 4.2 highlights the model's 's efficacy in handling tasks within both language and vision domains. Section4.3) offers an in-depth comparison, highlighting our model's advantages over standard Transformer models. Finally, Section (.4 details the ablation experiments conducted to assess the significance of each module in Tokenformer.
4.1 PROGRESSIVE MODEL SCALING
Datasets. Our models are trained using the WebText Corpus described in (Gokaslan Cohen 2019). This corpus serves as widely recognized open-source approximation of OpenAl's proprictary WebText dataset, which was employed in the development of GPT-2 (Alec al. 2019). The dataset comprises textual data extracted from 8,013,769 Reddit-shared documents. During training, we randomly sample segments from these documents.
Baseline Transformer Training from Scratch. To evaluate the effectiveness of our progressive model scaling strategy, we established a baseline by training a Transformer model from scratch. Following the training procedures outlined in Karpathy (2022); Kaplan et al. (2020), we employed the AdamW optimizer (Loshchilov Hutter 2019) with a batch size of 512 sequences, each containing 1024 tokens. For fair comparison with our incremental scaling approach, we configured two training variants based on the total number training tokens. The first variant underwent 6 x 10 steps (approximately 300B tokens), consistent with the training steps utilized by Karpathy (2022) to replicate GPT-2 performance. The second variant was limited to 6 x 10 steps (approximately 30B tokens) to ensure comparability with each stage of progressive scaling. In all trainings included

---

Published as a conference paper at ICLR 2025
Model
4Param
ppl.
Pilc
LAMRADA
ppl
LAMBADA
ace
HellaSwag
acc 1
ase
PIQA
Are-E
asc
Arc-C:
aoc 7
WinoGrande axc 1
Average aue
Pythis- 160M Biikrman et al. 3023
160M
29.61
37.25
35.4
30.3
62.3
43.6
23.6
51.3
40.1
Ours (Tokearormee-
150M
10.45
16.38
4500
35.5
64.9
47.3
24.9
30.4
44.7
Pylhii=410M Biderman al. 2023
410M
9.95
51.4
40.6
66.9
52.1
Ours (Tokzarormezs-
450M
8.38
7.69
57.3
47.5
69.5
56.2
26.7
Pythis 18 BBdermmn et al. 2023
18
7,82
7,92
56
47.2
70.7
57.0
27.1
51.5
51.9
Ours (TdkcaFoorm22599000
900M
7.38
5.46
64.0
55.3
72,4
59.9
30.6
56.4
56.4
GPT.Neo 1.38 (Black
2021
7.50
57.2
48.9
71.1
56.2
OPT-1.3B
58.0
53.7
72.4
56.7
59.5
Pythis- 1.38
Bidkrman et di 2023
1.38
7.51
6.08
61.7
52.1
71.0
60.5
28.5
57.2
55.2
Ours ( Iak.enoormec--aa. 1.58)
1,58
6.91
5.24
64.7
60.0
74.8
64.8
32.0
59.7
59.3
Table 1: (Zero-shot Evaluations.) The best performance for each model size is highlighted in bold.
Our comparisons are made with publicly available transformer-base LMs with various tokenizers. Following Pythia (Biderman et al. 2023), our model trained for up to 300B tokens on pile dataset.
in our analysis, unless otherwise indicated, a learning rate of 6 x 10 was employed, featuring a 2000-step warmup followed by cosine decay zero.
Tokenformer with Progressive Model Scaling, Building upon the above training protocols, we testify the performance of our model scaling with parameter sizes ranging from 124M to 1.4B. Unlike the aforementioned scratch-training approach, each scaling iteration leverages pre-trained smaller Tokenformer to partially initialize the weights of the larger one described in Section 3.3 The scaling procedure begins with training the initial source model from scratch on approximately 300B tokens, mirroring the Transformer baseline. For scaling, we select the pre-trained model closest in parameter count to the target size for weight initialization. For example, to train a model with 354M parameters, we employ the 124M model as a partial initializer and retrain the entire model using a reduced computational budget (e.g., 15B, 30B, or 60B tokens). This iterative process continues for scaling to 757M and then to 1.4B parameters. Notably, to simplify the scaling procedure, both new and existing parameters are trained equivalently with identical training hyperparameters throughout the process. Additionally, since the learning rate strategy in scaling follows a cosine schedule, the newly added parameters are randomly initialized to ensure stable training process.
Our training optimizes the autoregressive bog-likelhhod (i.e., cross-entropy loss) averaged over a 1024-token context and the log perplexity evaluated the test set the test score.
Experimental Analysis. As illustrated in Figure 3 our progressive scaling methodology employing Tokenformer achieves performance comparable to that of a Transformer model trained from scratch, while substantially reducing the training budget. Specifically, Starting with a 124M parameter model trained 300B tokens, we progressively scaled to 354M 757M, and 1.4B parameters, requiring only an additional 30B tokens-just one-tenth of the computational budget compared to the scratch-trained Transformer. This scaling process achieved test perplexity of 11.77 the 1.4B parameter level. In comparison, a Transformer model of the same size trained from scratch achieved similar perplexity of 11.63 but with 3x the training cost. Importantly, our approach reports cumulative training costs, encompassing all scaling stages, unlike the Transformer baseline that only accounts for individual stages. Even with this comparison, our method demonstrates a substantially lower computational cost than training Transformer from scratch, thereby validating the effectiveness of our approach.
Figure 4 presents the training costs at each scaling stage for both our model and the standard Transformer. When compared to Figure 3 the cost savings are even more significant. Specifically, our model requires only one-tenth of the training costs associated with Transformer baselines. To mitigate the effects of varying training data, we also included the performance curve of Transformer trained from scratch using an equivalent computational budget of 30B tokens. Under the same computational constraints, our progressively scaled model achieves lower perplexity of 11.77 compared to the Transformer's ' 13.34, thereby highlighting the superior efficiency and scalability of our approach.
4.2 BENCHMARKING OF MODEL EXPRESSIVENESS
Language Modeling. We assess the efficacy of our proposed architecture through standard au- toregressive language modeling tasks, benchmarking against existing transforrmerbaseed models. Evaluations are conducted using both pre-training metrics, specifically perplexity, and zero-shot performance measures. Training is performed on the Pile dataset (Gao et al. 2020), following the training protocol described in Biderman et al. (2023). Detailed training procedures and model sizes (depth and width) are provided in the Appendix F
7

---

Published as a conference paper at ICLR 2025
Method
Image Size
*Param
Top-1 acc
VIT-8/16 (Dsoovisklyy
384
86M
DeIT-B/15 loavron
2011
224
86M
81.8
VIT-8/16 (MAE) (He
224
86M
82.3
Ours-WI6 W/16
224F
109M
82,5
VTT-L/16
Dosovisshiv
3842
VIT-L/16 (MAE) (193 et
224
307M
82.6
Ours-L7Tb
224
307M
83,0
Ours-1/16
407M
Table 2: (Image Classification.) Comparison of standard vision transformer ImageNet-1K. The training hyperparameters are completely consistent (batch size, learning rate, etc.) with He et al. (2022). denotes models where the parameter size has been matched that of the standard ViT.
Parumeter
Training FLOPss
Operation
Transformer
Ours
Transfrnce
Embed
Rrearloudd
Attention: QKV Project
Mmayehhaen (2l
Emayeeereat 4migrtheasal T
Attention: Taken- Token
4mhoy Planee
Atlention: Output Project
Onger
Amtaeethaa Rig
Feedforward
2te
16muye Goods
Iitgeedhatn "ey
Devembed
(obeddnbedingg)
12
Maseee 1ig 1.y th my))
2NT
40
Table 3: Parameter counts and training compute estimates for Transformer and our Tokenformer. Sub-leading terms such as nonlinearities, biases, and layer normalization are omitted.
Table | presents the performance of Tokenformer across various widely-recognized zero-shot down- stream tasks. Comparisons are drawn against leading open-source transformer models of equivalent scale, notably Pythia (Biderman et al.) 2023). which utilizes the same tokenizer, dataset, and training duration (300B tokens) as our models. As shown in this table, our model achieves competitive performance compared to the standard Transformer, demonstrating the potential of our architecture in terms of expressive power foundation model.
Visual Modeling, Table 2 validates the expressiveness of our model visual tasks. We compare our approach against the standard Vision Transformer (ViT) (Dosovitskiy et al. (021) trained with supervised learning on the ImageNet- dataset (Deng et al. 2009). For a fair comparison, we used the MMDetection code base (MMDetection Contributors (018) and followed the hyperparameters and training strategy used in He et al. (2022). As shown in the table, our model achieves the same performance ViT in visual modeling, confirming its expressiveness in visual tasks.
4.3 COMPARISON WITH STANDARD TRANSFORMER
Transformer can also achieve model reuse to certain extent. Net2Net (Chen et al. 2015). a classical model growth method, proposes technique expand the width of neural networks by duplicating neurons. In this method, the pre-trained weight matrix of transformer layer in the smaller model denoted Woold e Rd. xd, is used to create a larger weight matrix Wmew gd xd: (dj >> d,) to fill the larger model. This expansion is formulated as follows,
Wold
Whew [(12)
Wnew
(13)
Whew ((21)
Whew [(22)
where Winew ((12) E dd(d Weew Idd x (d.-.,) and W [g(4-d.)x(d:-- are new ((21)
((22)
parameters for expansion. The scaling procedures are the same schemes introduced in Section 4.1
Controllable cost of token-token interaction for long-context modeling, Recent advancements in Mhain-off-hhughh (CoT) modeling (Wei al., 2022) have emphasized the critical importance of efficiently processing lengthy textual sequences (Tay et al.) (020) within Large Language Mod- els (LLMs). As delineated Section [ the training costs transformer architectures are primarily divided into two components: interactions involving model parameters and interactions among input sequences. Table demonstrates that the computational complexity of transformer based models exhibits quadratic dependence on text length, scaling linearly with ooken-parameter interactions and quadratically with token-toke token interactions. Consequently, it is imperative to expand model parameters while controlling the computational burden of token-token interaction part.
8

---

Published as a conference paper at ICLR 2025
Trandormer (15-4M)
2500
Transformer (2579)
3500
Trandformer (1.40)
1500
Thkento mer (254M)
Telenformer (757M)
3000
Tekenformer (1.48)
2000
Flops trillion
1500
2000
750
300
1000
300
250
500
20
40
00
00
300
120
20
40
80
00
300
120
20
40
00
00
300
120
Sequence Length /
Figure 5: The relationship between FLOPs and text length for both Transformer and Tokenformer. As shown Table Transformer exhibits an increase computational cost for token-token interactions as dmoxee scales upwards. Our Tokenformer model, however, offers flexible parameter scaling mechanism that maintains deoenn at a constant value. This strategy results in controllable computational costs for token-token interactions and markedly enhances the efficiency of long-text modeling.
Transfomerr
Incrementally scaling Trareforme (Ne:2Netl
lokenformer
Incrementally scaling Tolenformer
Test loss
Perplexity
2500
3020
4003
Training iterations
Training TPU hours
Figure 6: Loss curves comparing pre-trained Transformer and Tokenformer as their parameters scaled during continued training on enwik8.
Figure Performance benchmarking on incre- mental model scaling between Transformer with Net2Net scheme and our Tokenformer.
Conventionally, scaling transformer models involves increasing the channel dimension. For fixed text length, this results in higher computational costs, mainly because dominant token-token interactions become more intensive, which hampers the model's performance with long texts. Our proposed model takes a different approach by decoupling the computation cost of token-token interactions from model scaling. We increase the parameter size without changing the token channel dimension, thereby maintaining the computational cost associated with token-token interactions. As shown in Figure 5) our model exhibits increasingly significant computational advantages Transformers as the number of parameters grows, especially when processing longer sequences.
Scaling without losing the well-learned distribution. Our Tokenformer can maintain the existing output distribution when new key parameters are initialized to zero. This characteristic is beneficial for continuously scaling models to incorporate additional data, facilitates an increase in model capacity without disrupting the ongoing training process, thereby promoting rapid convergence.
To evaluate Tokenformer's 's scaling efficacy, we compare the loss curves of Net2Net-based transformer scaling against Tokenformer scaling. Both models, initially with 354M parameters, were pre-trained on the Open WebText dataset. We then introduced the EnWik8 dataset and continued training with one epoch, expanding the models to 757M parameters accommodate Tokenformer not only converges more rapidly but also reaches a lower final loss, attributable to its ability to preserve the output distribution during the resumption of training.
data. 6
Performance benchmarking on incremental scaling, In this study, we progressively scale the standard Transformer using Net2Net approach detailed earlier. For fair comparison, aligned all hyperparameters, including the parameter size, learning rate, dataset, and so on. As shown in Figure 7 our model performs better in scaling compared to the standard Transformer.

---

Published as a conference paper at ICLR 2025
4.4 ABLATION STUDY
Nonlinear Function
Normalization
Top-I acc
Learnable Weight (7)
Learnable Bias (5)
Top-I acc
gx
L1 Norm
79.6
82.6
GeLU
L1 Norm
81.7
GeL.U
La Norm
82.5
82.5
Table 4: Ablation of Softmax part on ImageNet classification with base model.
Table 5: Ablation of non-parametric layer normaliza- tion on ImageNet classification with base model.
Optimized Softmax Function in Pattention Layer. Within the token-parannn parameter attention layer, we address training instabilities arising from the diminished gradients associated with the traditional softmax function. The conventional softmax operation comprises two primary steps: computing the exponential attention scores followed by L1 normalization. As shown in Table 4 to mitigate the issue of small gradients, substitute the exponential non-linearity with the GeLU function, resulting in a performance enhancement of +2.1 points on the ImageNet classification benchmark. Subsequently, replace the L1 normalization with L2 normalization, yielding additional improvement of +0.8 points, allowing our model to achieve performance parity with the standard VIT.
Non- Non-Parametric Layer Normalization. In pursuit of enabling model expansion and the merging of two separately trained parameter token sets for subsequent studies, we modified the Transformer's layer normalization to non-parametric variant by removing its trainable weights and biases. This adjustment guarantees that only the key-value parameters are subject to learning within the model. Empirical results presented in Table (able-Jdemon demonstrate that the model maintains comparable performance after discarding the learnable weights and biases.
FUTURE WORK
Extending the Mixture-of-Experte Paradigm. We interpret Tokenformer as an extreme instantiation of the Mixture of Experts (MoE) framework (Fedus al. 2022 Zhu et al. 2024), where each key- value parameter pair functions as an individual expert. This innovative MoE-like like architecture has the potential significantly reduce the computational costs associated with loken-parameter interactions.
Advancing Crrameter-ffficieee Tuning. The scaling approach of Tokenformer, which involves integrating additional key-value parameter pairs, exemplifies strategy for parameter-efficien tuning. When confronted with new tasks or datasets, the model can augment its pre-trained parameters by incorporating these new parameter tokens, thereby adapting to specific task requirements quickly.
Integrating Vision and Language Models. Leveraging the carameter-efficiee tuning capabilities of Tokeformer, we can achieve seamless integration visual and linguistic modalities. This can be ac- complished by unifying the key-value parameter tokens derived from pre-trained visual Tokenformer and language Tokenformer into single parameter set. Then, the new learnable tokens are introduced to perform vision-language alignment and instruction tuning.
Enhancing Model Interpretability. As Tokenformer is entirely based on attention mechanisms, it inherently benefits from the interpretability associated with attention in loken-parameter interac- tions (Geva et al. 202T). This characteristic enhances the model's explainability, contributing to the AI community s efforts to develop more transparent and understandable models. Please see G.2 for more discussions.
CONCLUSION
This paper introduces Tokenformer, a naturally scalable architecture that leverages the attention mechanism to facilitate not only inter-token computations but also interactions between tokens and model parameters, thereby enhancing architectural flexibility. By representing model parameters as tokens, replace all linear projections in Transformer with our Pattention, allowing for seamless incremental scaling without retraining from scratch. We believe that this architecture, offering greater flexibility than Transformers, will further contribute the development of foundation models.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Radford Alec, Wu Jeffrey, Child Rewon, Luan David, Amodei Dario, Sutskever Ilya, et al. Language models are unsupervised multitask learners. OpenAl blog, 2019.
Jimmy Lei Ba. Layer normalization. arXiv preprint NKKV:1607.0045333 2016.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In ICML. PMLR, 2023.
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. scale autoregressive language modeling with mesh-tensorflow. 2021. URL https://api
Gpt-neo: Large
semanticscholar: htt.:/aaa...e//oooras00:22225350
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, al. Gpt-neox-20b: An open-source autoregressive language model. ACL Workshop, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. In NeurlPS, 2020.
Cheng Chen, Yichun Yin, Lifeng Shang, Xin Jiang, Yujia Qin, Fengyu Wang. Zhi Wang. Xiao Chen, Zhiyuan Liu, and Qun Liu. bert2bert: Towards reusable pretrained language models. In ACL, 2021a.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In NeurlPS, 2021b.
Tianqi Chen, Ian Goodfellow, and Jonathon Shlens. Net2net: Accelerating learning via knowledge transfer. ICLR, 2015.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021.
Nelson Dunford and Jacob T Schwartz. Linear operators, part 1: general theory. John Wiley & Sons, 1988.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. JMLR, 2022.
Daniel Fu, Tri Dao, Khaled Kamal Saab, Armin W Thomas, Atri Rudra, and Christopher Re. Hungry hungry hippos: Towards language modeling with state space models. In ICLR, 2023.
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. Transformer feed-forward layers are key-value memories. In EMNLP, 2021.
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http::/9Kkylion07..gta i OpeenNebTextCorpus 2019.
Linyuan Gong, Di He, Zhuohan Li, Tao Qin, Liwei Wang, and Tieyan Liu. Efficient training of bert by progressively stacking. In ICMI., 2019.
11

---

Published as a conference paper at ICLR 2025
Albert Gu and Tri Dao. Mamba: Sthear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752, 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1rXiv:6606.084155. 2016.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXhv:2001.0836/ 2020.
Andrej Karpathy. Nanogpt, 2022. URL
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurIPS, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
I Loshchilov. Decoupled weight decay regularization. In ICLR, 2019.
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
Matt Mahoney, Large text compression benchmark, 2011. URLnntt:://netttmhhneey.y. hhmll htmll
dc/text
MMDetection Contributors. Open Detection Toolbox and Benchmark, August 2018. URL
https::/giitub..com open mmmabb www.aa//mdeettttiit....
Bo Peng, Eric Alcaide, Quentin Gregory Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Nguyen Chung, Leon Derczynski, Xingjian Du, Matteo Grella, Kranthi Kiran GV, Xuzheng He, Haowen Hou, Przemyslaw Kazienko, Jan Kocon, Jiaming Kong, Bartlomiej Koptyra, Hayden Lau, Jiaju Lin, Krishna Sri Ipsit Mantri, Ferdinand Mom, Atsushi Saito, Guangyu Song, Xiangru Tang, Johan S. Wind, Stanistaw Wožniak, Zhenyuan Zhang, Qinghua Zhou, Jian Zhu, and Rui-Jie Zhu. RWKV: Reinventing RNNs for the transformer era. In FMNLP, 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, al. Improving language understanding by generative pre-training. OpenAl, 2018.
Mattia Rigotti, Christoph Miksovic, Ioana Giurgiu, Thomas Gschwind, and Paolo Scotton. Attention- based interpretability with concept transformers. ICLR, 2021.
Mohammad Samragh, Iman Mirzadeh, Keivan Alizadeh Vahid, Fartash Faghri, Minsik Cho, Moin Nabi, Devang Naik, and Mehrdad Farajtabar. Scaling smart: Accelerating large language model pre-training with small model initialization. arXiv preprint arXiv:2409. 12903, 2024.
Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. Aug- menting self-attention with persistent memory. In ICLR, 2020.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. arXiv preprint ar XXX::20009090732 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
12

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017.
Haiyang Wang, Chen Shi, Shaoshuai Shi, Meng Lei, Sen Wang, Di He. Bernt Schiele, and Liwei Wang. Dsvt: Dynamic sparse voxel transformer with rotated sets. In CVPR, 2023a.
Haiyang Wang. Hao Tang, Shaoshuai Shi, Aoxue Li, Zhenguo Li, Bernt Schiele, and Liwei Wang. Unitr: A unified and efficient multi-modal transformer for bird's bird's-eye --eee-view representation. In ICCV, 2023b.
Haiyang Wang. Hao Tang, Li Jiang, Shaoshuai Shi, Muhammad Ferjad Nacem, Hongsheng Li, Bernt Schiele, and Liwei Wang. Git: Towards generalist vision transformer through universal language interface. In ECCV, 2024.
Peihao Wang, Rameswar Panda, Lucas Torroba Hennigen, Philip Greengard, Leonid Karlinsky, Rogerio Feris, David Daniel Cox, Zhangyang Wang, and Yoon Kim. Learning to grow pretrained models for efficient transformer training. In ICLR, 2023c.
Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-to-sqqeene learning framework. In ICML, 2022.
Wenhui Wang, Hangbo Bao, Li Dong. Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for vision and vision-language tasks. In CVPR, 2023d.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thougg prompting elicits reasoning in large language models. NeurIPS, 2022.
Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie- Yan Liu. Do transformers really perform badly for graph representation? In Neur/PS, 2021.
Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo Kim. Graph transformer networks. In NeurIPS, 2019.
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In NeurIPS, 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01066 2022.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
Xingkui Zhu, Yiran Guan, Dingkang Liang, Yuchao Chen, Yuliang Liu, and Xiang Bai. Moe jetpack: From dense checkpoints to adaptive mixture of experts for vision tasks. 2024.
13

---

Published as a conference paper at ICLR 2025
A GRADIENT OF PATTENTION LAYER
Our token- parameter attention mechanism employs Izznormalization followed by the GeLU (Hendrycks & Gimpel 2016) activation function, in contrast to the conventional SoftMax func- tion in standard loken-token attention layers, which utilizes an exponential transformation followed by -n-normalization. This design choice motivated by our experimental observation that SoftMax tends to increase the magnitude of outputs, often pushing them into regions where the gradients become extremely small, leading to an inferior overall performance (see Table
Specifically, given a query token and key-value pairs with dimension d, let the similarity scores between the query and key tokens be represented as A E 121x" In standard SoftMax attention, the attention scores S R x are computed as follows:
exp(A./Vd)
S Σー= (xp(AA/Vd)
ViE I...,
(14)
The derivative of the SoftMax function with respect to A given by:
S - S3)
i=j
0s. -
vd
Sq) -g) Vd
(15)
S,S,
i i j.
In contrast, our activation function uses 12-normalization followed by the GeLU function. Denoting GeLU as f. the attention scores Z R 1xn computed as:
A x v
-(Z)) = /(
ViEL....
(16)
Eー= A,1
For the derivative of our attention function when = j. we have:
as; 0A
V' || 2 - A A
TA
(17)
||A
||A|12 //2 - A? ||ll
V'
(18)
|Al13
||A 12
AA
f' Vn
(19)
11
||A|
ZZ
Vn'
(1
(20)
|1
1
1
8'
(n 223)
(21)
Vn
||Al12
When * j. the derivative becomes:
as: 0A,
A TAA &a
8' Vn
(22)
A:A
f Vn
(23)
||A||2 la
||A|
1
=-f' /n ||A||2 Z,Z,
(24)
Thus, the derivative of our attention function is:
1
Vn ||Al|2 (n -277g)
i=
as
(25)
daj
1
f' Vn ||A||2 2iZ1
14

---

Published as a conference paper at ICLR 2025
Comparing the gradients of SoftMax (Eq. 15 and method (Eq. 25 the key difference is that our gradient depends on the product z,Zy, whereas SoftMax relies on S. Due to the exponential nature of SoftMax, the distribution of S; tends to be sharper and more concentrated, which often drives the gradients toward zero. Conversely, our activation function produces a smoother distribution of Z, mitigating the vanishing gradient problem and enabling more stable training dynamics.
B ZERO INITIALIZATION IN TOKENFORMER
during model scaling, initializing new key parameters zero allows the model to continue training with minimal disruption. This is because zero initialization preserves the model's original output distribution, preventing significant interference to the learned representations.
As shown in Section 3.3
In this section, we demonstrate that replacing the exponential function in the standard softmax with GeLU makes the Pattention layer invariant to newly added parameters when they are zero-ime inoiinitialized. This behavior cannot be achieved with the standard exponential-baas initialize new parameters a way that guarantees the output scores of the newly added components exactly zero after applying the exponential function, thereby preventing any influence the existing parameter tokens.
softmax, it is impossible to
Let X R be the input vector, and let the Pattention layer have key-value pairs, represented as Kr, Vp E Raad We fix the scale factor r and, for simplicity of formulation, temporarily set to 1. The output of the Pattention layer computed
A = Kp X,
(26)
A
S = f(
(27)
Σ, A3
0 VØ S.
(28)
where A is the score derived from Kp- X and f a non-linearity function, which our formulation is set to the GeLU function (Hendrycks & Gimpel (016).. When scaling the model by adding m new key-value pairs with zero initialization, output becomes:
KP
A
0
0
0
X
(29)
0,
S
0
$ f(
(30)
Σ, A3
= [VT V. 1] 0.
(31)
Since the newly added key parameters are initialized to zero, the attention mechanism does not modify the original output. Therefore, the output Ô remains identical to O. This property is advantageous for scaling models because increases the model capacity without interrupting the well-learned distribution and the ongoing training process, leading to faster convergence.
C TABULAR MAIN RESULTS
Here, we provide the tabulated results corresponding to Figure 3 from the main paper. Table 7 presents the perplexity on the validation set of OpenWebText. The Transformer models are trained from scratch, while Tokenformer models leverage parameter reuse from smaller models (except for the first Tokenformer model with 124M parameters, which also trained from scratch). We observe that Tokenformer achieves on-par performance Transformers trained from scratch, but
15

---

Published as a conference paper at ICLR 2025
Perplexity 354M
#Param
WPraam
Tokenformer
Transformer
757M
1.4B
Tokens
Perplexity
Tokens
Perplexity
Transformer
124M
16.41
300B
17.06
13.02
11.99
11.63
Train from scratch 300B
354M
15B
14.58
Transformer
354M
30B
14,02
300B
13.02
15.97
13.78
13.34
354M
13.59
Train from scratch 30B
757M
15B
13.08
Tokenformer
14.02
12.59
11.77
757M
30B
12.59
300B
11.93
Purameter Reusing 30B
757M
60B
12,28
1.4B
15B
12.14
Table 6: Tabular results of Figure 4 The perplex- ity of models trained with different numbers of schemes is compared. Transformers are trained from scratch, while Tokenformer are progres- sively scaled via parameter resuing. When trained with the same number of tokens (30B), Tokenformer demonstrates superior performance. of training tokens when scaling up model sizes.
1.4B
30B
11.77
300B
11.63
1.4B
60B
11.60
Table 7: Tabular results of Figure 3 Due to pa- rameter reusing. Tokenformer achieves the same performance while using a much lower number
124M (60B)
354M (15B)
757M (1SB)
method
model
PPL
Flops
PPL
Flops
PPL
Flops
Net2Net
Transformer
16.4
3.6
14.3
12.1
10.1
HyperCloning
Transformer
16.4
3.6
14.1
12.0
10.1
Tokenformer
3.6
13.7
11.5
3.6
Table 8: More comparison results of model scaling. The unit of Flops is 104 x T Our method
outperforms HyperCloning (Samragh et al.) (024) and Net2Net (Chen et al. 2015) while requiring less computational effort for token-token interactions.
with significantly reduced training costs due to parameter reuse. Notably, the largest Tokenformer (1.4B) achieves even a slightly better perplexity than its Transformer counterpart (11.60 vs. 11.63).
Table/occoppares Transformer models trained from scratch and Tokenformer trained by parameter reusing with varying amounts of seen tokens during training. It is evident that Transformers trained with the same number of seen tokens do not reach the performance level of Tokenformer with parameter reusing. This demonstrates that parameter reusing successfully transfers knowledge from smaller models, reducing training time without sacrificing performance.
D EXPERIMENTAL DETAILS ON PROGRESSIVE MODEL SCALING
In this experiment, we utilize the OpenWebTen. dataset Gokaslan & Cohen 2019) evaluate the model scaling capability. The dataset comprises 8,013,7 3,769 documents, from which we randomly select 5% serve as the validation set and report perplexity on this subset. We investigate four model sizes: 124M, 354M, 757M, and 1.4B parameters. Please find model specifications in Table 9 During parameter reusing of Tokenformer, we partially resume the old model parameters and add new key-value pairs to the Pattention layers and do not alter the number of layers or feature dimensions.
The training recipe is the same for both Transformers and Tokenformer: We do not implement dropout in either model and the logits are computed at the final layer using the embedding layer. The tokenizer from OPT-NeoX-205B (Black et al., 2022). We employ the AdamW optimizer (Loshchilov 2019) with 31 0.9 and 32 0.95. A learning rate of 6 10 is employed, with linear warmup over 2000 steps followed by a cosine decay to zero. The training is conducted with a batch size of 512 and sequence length of 1024.
16

---

Published as a conference paper at ICLR 2025
Atiention KV Paizs
FFN Pairs
Model
Layers
Hidden size
Heads
#Params
Language Modeling
Tokenformer- 150M
12
768
768
3072
150M
Tokenformer- 450M
24
1024
450M
Tokenfoorer-990m 900M
32
1280
1280
5120
16
900M
Tokenformer-L.35
40
1536
1536
6144
1.58
Visual Modeling
Tokenformer Base!
12
768
576
2304
86M
Tokenformer- Base
12
768
768
3072
109M
Tokenfonmer-Lange
24
1024
307M
Tokenformer- Large
24
1024
1024
4096
407M
Porameter Reusing
Tokenformer- 124M
768
2304
124M
Tokenfomerr 354M
12
768
2140
8560
354M
Tokenforee--7511
12
768
4850
19400)
757M
Tokenformer-L.4B
768
34480
Table 9: Details of Tokenformer model variants used in Section indicates models whose key-value pairs are chosen to match the parameter numbers of Transformer of equivalent sizes.
E EXPERIMENTAL DETAILS ON SCALING WITHOUT LOSING THE WELL-LEEARNED DISTRIBUTION
In this experiment, we utilize the EnWik8 (Mahoney 2011) dataset to evaluate the model's capacity for continued adaptation new data. The EnWik8 dataset comprises the first 100 million bytes of English Wikipedia in XMI. format.
We begin with model size of 354M parameters, which has been pre-trained on OpenWebTexx (Gokaslan & Cohen, 2019), and then scale it to 757M parameters for both the Transformer and Tokenformer models. In the case of the Transformer, the 354M and 757M models differ solely in feature dimension and the number of heads in the multi-head attention mechanism, and we follow Net2Net(Chen et al. 2013) methodology for parameter expansion. For Tokenformer, we increase the number of key-value pairs from 2140 to 4850.
We employ a constant learning rate of 6 x 10-4 and process the dataset for a single pass, resulting in total of 2204 training steps. The AdamW optimizer (Loshchilov) 2019) is utilized, and do not resume the optimizer's internal state from any previous runs. The batch size set to 512, consistent with the batch size used during pre-training.
F EXPERIMENTS ON LANGUAGE MODELING BENCHMARKING
We evaluate the expressiveness and performance of our proposed architecture using standard au- toregressive language modeling benchmarks, comparing results to those of existing open-source LLMs, including RNN-baser methods (Peng et al.) 2023 Gu & Dao 2023). in Table T0 Evaluations are conducted using both pre-training metrics, specifically perplexity, and zero-shot performance measures. For training, we used the Pile dataset (Gao et al. 2020) over single epoch, adhering to the training strategy used by Biderman et al. (2023). The Adam optimizer is applied with 6 x 10 learning rate, and each batch contains 1024 samples with sequence length of 2048 tokens, mirroring the setup Mamba. The training process consists of 14,300 steps, with 1430-step warmup, equivalent to 1 % of total training steps. Detailed model specifications are in Table 9
G DISCUSSIONS
G.1 COMPARISON WITH A TWO-LAYER MLP
We aim to establish a novel and unified concept for model computation by treating all components, including parameters, as tokens. This token-based design allows the attention mechanism to perform dynamic interactions among them, such as data and parameter tokens, fostering greater flexibility and scalability in computational models.
17

---

Published as a conference paper at ICLR 2025
Model
4Param
ppl.
Pilc
ppl.
LAMBADA
ace
HellaSwag acc
PIQA
Arc-E
asc
a0c
axc 1
Average
avc
Hybrid H3-130M Fu es 2023
130M
89.48
25.77
31.7
611.2
44.4
21.2
50.6
40.1
Pylhis- 160M Bickerman
160M
29.64
37.25
35.4
30.3
62.3
43.6
23.6
51.9
40.6
Mumbh-
10.56
48,0
44.7
(TokenFormee-
Hybrld H3-WiM Pu
2023
360M
12.58
48.0
41.5
68.1
51,4
24.7
54.1
48.0
Pythiz-410M [Biderman er 2024
410M
9.95
10.34
51.4
40.6
66.9
52.1
24.6
53.8
48.2
Mamba: 3708T
Duot
370M
8.28
55.6
46.5
69.5
55.1
28.0
55.3
50.0
(TokenFormee
56.2
Pythix [Bdcrman et
2024
1B
7.82
7.92
56.1
47.2
70.7
57.0
27.1
53.5
51.9
Mambaa 79081 (OB & Don 21M4
790M
7.33
6.02
62.7
S5.1
72.1
61.2
29.5
56.1
$6.1
Ours (TokenForrtte->000))
900M
7.38
5,46
64.0
55.3
72,4
59.9
30.6
56.4
56.4
GPT-Neo 1.38
2021
Hybrid H3.1
11.25
49.6
71.3
59.2
23.1
OPT-1.3B
Zhail Cal
puze
1.38
6.64
53.0
33.7
72.4
56.7
29.6
59.5
55.0
1.thi-11338
Biotmmnn ca
1.38
7.51
6.08
61.7
52.1
71.0
60.5
28.5
57.2
55.2
RWKV-1.5h 1.5
trenn P024)
1.58
7.70
7.04
56.4
52.5
72.4
60.5
29.4
$4.6
54.3
Mamba-1.44
6.80
5.04
64.9
74.2
65.5
GPT-Neo
61.1
Hybrid H13-1
ai 5026
1.30
11.25
19.6
52.6
71.3
59.2
28.1
61.4
58.0
OPT-22.B Zearl 2ra3 K0ree
2.78
5.12
63.6
60.6
74.8
60.8
31.3
61.0
58.7
Pythie2.888 2,88
2,88
5.04
64.7
59.3
74.0
64.1
32.9
39.7
59.1
Tokeen--rmeer-....
Table 10: (Zero-shot Evaluations.) The best results for each model size are highlighted in bold. We compare our models against open-source language models (LMs), including RNN-based methods, with various tokenizers, trained for up to 300B tokens.
Specifically, We begin by reformulating two-layer MLP an attention mechanism (Geva (091||Sukbbatte et al.) [2020), using this module as the foundational computational unit for neural network design, instead of the traditional linear projection, maximizing the flexibility of Transformers.
Based on this all-attention framework, we treat each key-value parameter pair as a learnable unit. By incrementally expanding this parameter set, the model grows progressively from a pre-trained smaller model to larger one. This strategy achieves comparable performance training a large model from scratch while greatly reducing computational costs.
The concept of tokenizing everything and leveraging attention for establishing connections is highly versatile. For example, it can be extended to the hidden states in linear models, allowing the memory expand incrementally. This incremental memory expansion has the potential to mitigate excessive information loss, a common limitation in models such as Mamba (Gu & Dao 2023) and RWKV (Peng
al. 2023)
G.2 INCREASING INTERPRETABILITY
As TokenFormer is fully grounded in attention mechanisms, it leverages the inherent interpretability provided by attention, especially in token-parameter interactions. Geva et al. (202T) and Rigotti et al.) (202I) studied Feed-Forward Networks (FFNs) and demonstrated that they could be interpreted as key-value memory structures. These memory structures encode learned patterns, which are selectively activated depending on the specific input. Our TokenFormer uses pattention the most fundamental computational unit in the network design, rather than the less interpretable linear projection. Token- Parameter interactions provide an additional way of visualizing the interaction between the model parameters and the input via the attention score map, which may be helpful to further understand the behavior of the model.
G.3 PRACTICAL LIMITATIONS
The complexity of token-token interactions is manageable, while token-parameter computations scale linearly with network size. This creates challenges in expanding parameters without increasing inference overhead. However, TokenFormer's inherent compatibility with sparse inference makes an ideal candidate for integrating MoE techniques in future work, aiming to build more efficient and powerful networks. A remaining challenge is achieving efficient communication between experts when each key-value parameter token functions as an independent expert.
18


Reviews:
[{"review_id": "i8c4Vvv8iV", "reviewer": "Reviewer_sjss", "summary": "This work proposes a new architectural component that can be used in transformers to replace linear projections present in the attention and feed-forward layer. A Pattention layer executes an attention mechanism between the input and some learned key and query tokens to output a transformed token. Both the channel dimensions and the number of key-query tokens can be scaled flexibly, in the sense that smaller models can be used as initial configurations for larger models. Iteratively enlarging the model results in significantly smaller training costs and similar performance to comparable architectures.", "questions": "- Table 1: \"The best performance highlighted for each model\" seems to be misleading. It seems it is highlighted in bold for each dataset and model size what the best value is, and even then it is unclear why e.g. LAMBADA acc highlights ours-1.5B and not Pythia 1.8B (this is also the case for several other columns)\n- Figure 6: Would you say the (Token)Transformer converged on the test loss respectively (left)?\n- It would be nice if the practical relation between parameters, dataset size, scaling stepsize and performance was further investigated. Is there a minimum model size for which this approach only starts being useful? What were your practical considerations in choosing the steps and initializations you did? \n- Is there a different effect of scaling the channels and the dimensions?  \n- Are there artifacts of the scaling steps in the final model? E.g. do the norms of the new channels (new tokens) that were initialized at zero have a different magnitude than those from previous time steps?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- Innovative and flexible idea of tokenizing model parameters, that will probably inspire further experimental work in this direction \n- Clearly written, well-motivated and contextualized\n- Demonstrated lower training cost in comparison of the proposed approach compared to a number of different open-source architectures and datasets", "weaknesses": "- The computational complexity of inference and the model itself is only dicussed late in the work. \n- The work would benefit from an explicit discussion of the architectures practical limitations.\n- See questions below", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qb2AZUjDF9", "reviewer": "Reviewer_M5Lm", "summary": "This paper proposes an innovative modification to the standard transformer architecture by replacing all linear layers with learned tokens via attention operations. By making this alteration, the proposed architecture can effectively remove all learned parameters (except embedding layers), instead replacing them with learned tokens. This token-parameter-attention mechanism (Pattention) therefore provides an efficient mechanism for scaling parameter counts. The authors validate their architecture with several scaling experiments, on both causal language modeling and vision tasks, demonstrating comparable performance with a significantly reduced training budget. \n\nOverall, this paper makes a significant architectural contribution to the lineage of transformer architectures. However, the authors primarily focus on the empirical validation of their methodology without sufficient theoretical backing or exploring limitations of this approach. I am therefore inclined to reject, but would be willing to reconsider pending further discussion.", "questions": "1. Your main argument for scaling is based on adding parameter tokens to the Pattention layers. However, this necessarily requires that you fix A) the hidden dimension, and B) the model depth. \n    - 1a. Fixing the hidden dimension can lead to representation crowding, which typically does not occur when increasing it (e.g. larger vector space for representations). Could you investigate this with your scaled models? As a limitation it should result in an upper limit to capacity, where adding new parameter tokens will no longer improve performance.\n    - 1b. Fixing the model depth can have a significant impact on the range of functions the transformer can learn. It’s possible that this was not apparent due to the model size and benchmarks, but would severely limit feasibility for more complex problems, which often require larger scales (>70B). Could you address this limitation?\n    - 1c. By fixing the hidden dimension, achieving improved performance would then depend on expanding the representation span within a fixed vector space (i.e. approaching 100% channels to capture the variance). This will likely have a negative impact on quantization methods, which may offer substantial computational improvements beyond what Pattention can provide alone. Could you address this?\n2. The parameter token scaling method raises several questions that should be addressed.\n    - 2a. When scaling the number of parameter tokens, do you observe representation collapse, or do you apply a regularization term to prevent this? Adding fixed parameters without enforced symmetry breaking often leads to vector consolidation into tight groupings. Alternatively, have you confirmed that this behavior is not occurring?\n    - 2b. In Figure 2, you suggest keeping the previously learned parameter tokens unfrozen for model growth; is this necessary? It may be more beneficial to keep the previously learned parameters frozen to essentially learn “residuals”. \n3. While you present the mechanism by which Pattention works, there is no explicit connection to linear layers, or is it closer to an FFN? Could you clarify this connection in the appendix? Intuitively, if the GELU scoring is one-hot, then you have a K-V lookup.\n4. Your scaling conclusions in Section 4.3, Table 3, and Figure 5 are contingent upon which term dominates in the asymptotic analysis. You require that $l \\cdot n \\cdot d \\cdot T < l \\cdot d \\cdot T^2$, otherwise Pattention will become more expensive than simply using linear layers. The scaling argument should be more precise, and both acknowledge this point and demonstrate which regime the different model scales are in. Notably, a significant number of parameters were added to the 1.4B model (6.6M per matrix vs. 2.4M), where it would also be appropriate to report the FLOPs required for a forward pass of sequence length $T$, verified with empirical measurement. Furthermore, relying on a fixed $l$ while scaling $n$ may significantly impact parallelism during inference. \n5. To confirm, did you retain the original softmax attention for the MHA layers, or did you apply the GELU scoring there as well? This should be made clear to avoid confusion.\n6. Questions about the Future Work section. While it is understandable to suggest exploratory directions in the future work section, some of the proposed ideas would benefit from either preliminary results or a more concrete implementation plan.\n    - 6a. You mention adjustable computational cost: can you elaborate on how this could be done? Your current methodology seems to be restricted to $n_\\mathrm{new}$ once scaled up, without any way to “adjust” the computational cost. \n    - 6b. The idea of combining vision and language parameter tokens is interesting; however, I am highly skeptical whether this would work. Do you have preliminary results to back up this idea? If not, perhaps the statement should be weakened, as it's not improbable that some features may have significant overlap, allowing for $\\mathrm{max}(n_\\mathrm{lang}, n_\\mathrm{vision}) < n < n_\\mathrm{lang} + n_\\mathrm{vision}$, where the vision and language tasks are not simply combined after independent training. \n    - 6c. The paragraph on “Increasing interpretability” is vague. How exactly will Pattention improve interpretability? Can you make a connection to Sparse Auto-encoders (SAEs)? This seems like something that should have been added to the appendix.\n\n\n\n\nAdditional Comments:\n\n-\tIt appears you are missing a factor of 2 in the parameter counts in Table 3. According to your description, you learn $n$ Key and $n$ Value tokens, resulting in a total of $2n$. The parameter counts and cost would then be identical to a traditional transformer when $n=d_\\mathrm{model}$ and $d_\\mathrm{token}=d_\\mathrm{model}$. \n-\tPlease include a discussion connecting the GELU scoring function to your scaling ability. Scaling by setting new keys to zero relies on the GELU function, where GELU(0)=0; in contrast, exp(0)=1, which causes a norm shift.\n\n\nYao, Yiqun, Zheng Zhang, Jing Li, and Yequan Wang. ‘Masked Structural Growth for 2x Faster Language Model Pre-Training’. In The Twelfth International Conference on Learning Representations, 2024.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a unique approach to scaling parameters and demonstrates compelling results under those conditions, including vision and language tasks. The authors further support their claims with extensive experimentation, demonstrating a substantial reduction in training costs. Additionally, the authors propose an interesting attention mechanism for their parameter tokens, which solves the issue of vanishing gradients caused by the softmax function in traditional attention. Furthermore, this paper is well-written and easy to follow, with clear figures and tables.", "weaknesses": "-\tThroughout the paper, the authors seem to imply that the scaling issue is unique to transformers. However, the increased training cost with scale applies to most DNN architectures. Furthermore, they contradict themselves when attributing training cost to increasing parameters while later showing that cost is dominated by self-attention (which is well known). The proposed training improvements are therefore solely due to fixing the hidden dimension size and resuming, rather than directly due to the Pattention mechanism. The authors should make their motivation more precise, and avoid overly vague and potentially misleading arguments to support their approach.\n-\tThe paper lacks a theoretical foundation as to why their approach works, only briefly mentioning a potential explanation in Section 5, Future Work. Although scaling parameters generally improves performance, this is typically done while also scaling the hidden dimension, which has other benefits beyond parameter count (e.g. larger representation space). While the empirical results are promising, discussing deeper theoretical insights would allow application to other (hybrid) architectures beyond TokenFormer. \n-\tThe authors compare against an old baseline (Net2Net) to demonstrate their model's effectiveness in growth when resuming from a smaller model. It would be more appropriate to compare with newer methods, which have been shown to perform much better (Yao et al., 2024). Furthermore, the practice of initializing extended queries to zero may negatively impact potential performance due to symmetry, where the proposed method instead relies entirely on the value token initialization. The authors should clarify why Net2Net was chosen over newer methods, or why the newer methods would not be a more suitable comparison.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tRxFEzhXt4", "reviewer": "Reviewer_eA2h", "summary": "The propsoed TokenFormer replaces dense linear layers of transformers with learnable token parameters to achieve progressive model scaling and reduce training cost compared with retraining from scatch of larger models with full data. The original matrix vector production is also replaced with the novel cross-attention between input token with tokenized parameters. Therefore, more token parameters can be easily added to well-trained smaller models incrementally due to flexibility of attention. The training cost is significantly reduced by 3x with similar or even better model performance. This paper proposes a interesting direction to scale LLMs and uses attention for almost all components in LLMs.", "questions": "same as weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This paper proposes the novel TokenFormer that replaces the original linear layers in transformers with Pattention layers to enable more efficient and progressive model scaling without retraining from scratch. Pattention tokenizes the dense weights of query, key, value, and output layers with flexible and scalable token parameters. It then introduces cross-attention between input tokens with those token parameters. Instead of increasing the hidden dimension in transformers, TokenFormer achieves scaling by increasing the number of token parameters, incrementally adds new token parameters, and train all old and new token parameters with less training data. Therefore, it enables effectively reusing smaller model knowledge or weights and faster training of larger models with similar architecture. Experimental results show that the training cost of a 1.4B model is significantly reduced by training a 124M model initially and progressively scaling to 1.4B parameters, compared with training from scratch of a 1.4B model with 300B tokens.", "weaknesses": "1. In this paper, only results of scaling only up to 1.5B parameters are given. What are the complexity and effectiveness of scaling to larger scales such as 7B, 13B, 70B, or even more? In which case, the number of token parameters will increase multiple times and may be larger than token or model dimension.\n2. The hidden dimension is relatively constrained to enable progressively scaling and reusing token parameters. Will it limit the capability of LLMs as FFN layers contain knowledge?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ad6w2ubBhO", "reviewer": "Reviewer_UJd1", "summary": "A novel growing architecture called \"tokenformer\" is introduced. The idea is to replace linear layers in the transformer architecture with a cross attention mechanism where input dependent queries cross attend to key and value parameters. This framing allows for the architecture to accommodate new parameters concatenated to the original key and value parameters. The architecture and training method are validated empirically by pretraining a 124M parameter tokenformer and incrementally growing it to 1.4B parameters. The resulting model performs competitively on evaluations compared to other models of similar size pretrained from scratch.", "questions": "Have the authors considered a linear attention version of Pattention, i.e. a version without the gelu activation in the cross attention? Also, it seems that the tokenformer replacement of linear layers can be made in any architecture, i.e. not just transformers. Have the authors experimented with alternative backbone architectures?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 4, "strengths": "The main strength of the proposed method in the paper is the flexibility of the architecture to accommodate additional parameters without jeopardizing the capabilities of the original model. Consequently there is no additional cost to \"healing\" the performance of the original model before continually training and allowing the model to exploit the added parameters. The experimental results reported in the paper are compelling and the key ideas are explained adequately clearly.", "weaknesses": "The presentation of the mathematical form of the \"Pattention\" operation can be improved - specifically, the notation in equations 4 and 5 are confusing - the function \\Theta is described as a \"a modified softmax operation for stable optimization of Pattention layer\" when in fact the function used in subsequent sections and experiments isn't really a softmax and what's more the function name \\Theta doesn't appear. Also it is unclear how equation 5 connects to equation 4, i.e. what exactly the function f has to do with \\Theta. Perhaps it would be better to remove any reference to \\Theta, and to solely describe the operation in terms of the function f, and to make explicit the choice of the specific form of f clear. Ideally a glossary defining all mathematical symbols in the paper would really help. \n In fact, the authors should make it clear that the structure of the Pattention operation can be seen as an MLP with standard activations: GeLU(RMSNorm(XK)V). The analogy with attention is interesting, but it somewhat obscures the key mathematical form of the operation - especially since there are no tokens (i.e. data) involved in the K and V parameters in Pattention. Also it isn't made clear how the V parameters are expanded - are they also padded with 0? \n\nAlso, in addition to the comparison with the Net2Net method, have the authors considered comparing to say the method in  arXiv:2409.12903v2 [cs.CL]  which also aims directly at growing transformer models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)