Transformers at a Fraction

Authors: aritra  mukhopadhyay
rucha  bhalchandra joshi
nidhi  tiwari
subhankar  mishra
Venue: NLDL_org_2025_Conference
Published: 2024-09-07 07:20:01

Abstract: Transformer-based large models, such as GPT, are known for their performance and ability to effectively address tasks. Transformer-based models often have many parameters, which are trained to achieve high-performance levels. As a result, they cannot be run locally on devices with smaller memory sizes, such as mobile phones, necessitating the use of these models remotely by sending the data to the cloud. This exposes us to privacy concerns over sending confidential data to the server, among others. In this work, we propose a method to make these large models easier to run on devices with much smaller memory while sacrificing little to no performance. We investigate quaternion neural networks, which can reduce the number of parameters to one-fourth of the original real-valued model when employed efficiently. Additionally, we explore sparse networks created by pruning weights as a method of parameter reduction, following the Lottery Ticket Hypothesis.

We perform the experiments on vision and language tasks on their respective datasets. We observe that pruned quaternion models perform better than the real-valued models in severely sparse conditions.

Main paper: 
 introduction: 
 The advent of powerful transformer-based models such as GPT-4 [1] , LLaMA [2] , Mistral [3] [4] , Vision Transformer (ViT) [5] , ViTDet [6] , and DINO [7] has revolutionized various domains, including natural language processing, computer vision, and image classification. However, the sheer scale of these models poses significant deployment challenges, with their massive parameter counts and computational requirements rendering them impractical for edge devices. For instance, running the smallest LLaMA 3 model (8B parameters) on a Raspberry Pi 5 (8GB) yields a meager 1-2 tokens per second (experiment done using ollama).Hence, we take up addressing this challenge of reducing the number of parameters, thereby reducing the burden of tuning a large number of parameters.Techniques like Mixture of Experts (MoE) models like Mixtral [4] and knowledge distillation have been explored to alleviate these challenges. MoEs selectively utilize multiple sub-models based on input data, enabling faster inference while keeping the output quality and total number of parameters high.The pursuit of efficient model deployment has led to the exploration of various techniques, including quantization, which reduces model weight precision [8] . While effective, these methods often compromise on accuracy. On the other hand, pruning methods like SparseGPT [9] have demonstrated one-shot pruning to around 50-60% of the weights while maintaining similar performance. However, given the large size of transformer-based models, even a 50-60% reduction may not be sufficient to fully address the computational challenges they present.To further reduce the number of parameters, we explore the application of the Lottery Ticket Hypothesis (LTH) [10] to transformers [11] [12] [13] , and the use of quaternion-valued weights in transformers [14] [15] [16] , both of which have been implemented separately in previous studies. For instance, LTH has been successfully applied to pre-trained BERT networks [12] and vision transformers [13] , while quaternion networks have shown promise in various tasks such as facial expression recognition [16] , NLP [14, 17] , and hyperspectral image classification [15] . In this work, we combine these approaches, applying LTH to quaternion transformers, and make the following research contributions:1. Apply Lottery Ticket Hypothesis (LTH) to Quaternion transformers.2. Demonstrate that quaternion-valued lottery tickets perform better than real-valued lottery tickets at higher sparsities across various computer vision as well as natural language processing tasks.3. Release a PyPi package called qytorch1 for easy implementation of quaternion layers in PyTorch.  
background: 
 In this section, we introduce quaternions and the Lottery Ticket Hypothesis, which serve as the foundation for our work.EQUATIONThe self-attention mechanism in transformers computes a weighted representation of the input by first generating the Query (Q), Key (K), and Value (V ) matrices through linear transformations of the input x ∈ R (n,d) using three real weight matrices, W r Q , W r K , and W r V ∈ R (d,d ′ ) . Attention scores, calculated as the cosine similarity between each vector in Q and each vector in K, are obtained by multiplying the Q and K matrices, followed by normalization and the application of the softmax function. The resulting values are then multiplied by the V matrix to produce the final output. This process is summarized by the equation 1.Quaternions are mathematical constructs widely used in computer graphics, simulations, and other applications. These can be represented as fourdimensional extensions of complex numbers, expressed as q = a+bi+cj +dk, where a, b, c, and d are real numbers, and i, j, k are imaginary components. In neural networks, quaternion representations reduce computational requirements. Calculations are facilitated by representing quaternions in matrix notation, as shown in Eq 2.EQUATIONAs illustrated in Figure 1 , using quaternions to represent the weights reduces the number of parameters by four times while maintaining performance at least as good or sometimes even better [18] . A real weight matrix W r with m × n weights, where m and n are multiples of 4, can represent four m 4 × n 4 dimensional quaternion weight matrices W qr , W qi , W qj and W q k (representing r, i, j, and k components of quarternion weight matrix W q ). This results in a total of 4 × m 4 × n 4 = m×n 4 weights, which is four times fewer than the real-valued representation. W qr , W qi , W qj and W q k matrices are concatenated as shown in Eq 3 and Fig1 to create the W r . W r =     W qr W qi W qj W q k W qi W qr W q k W qj W qj W q k W qr W qi W q k W qj W qi W qr     (3)The Lottery Ticket Hypothesis (LTH) proposed by Frankle et al. [10] states that within a randomly initialized neural network, there exists a subnetwork that can match the original network's test accuracy when trained in isolation. This subnetwork, called the "winning ticket", achieves comparable accuracy after training for at most the same number of iterations as the original network. Mathematically, this means that for a network Q with minimum validation loss l 1 at iteration j 1 , there exists a subnetwork Q ′ with fewer parameters that reaches its minimum validation loss l 2 at iteration j 2 , where j 2 ≤ j 1 and l 2 ≤ l 1 . 
methodology: 
 We designed custom Quaternion Layers for Linear, Convolutional, Self Attention, and Transformer Layers, mirroring the functionality of their PyTorch real-valued counterparts. This is released as a PyPi package called qytorch. For our experiments, we developed quaternion versions of Vision Transformer (ViT) and nanoGPT using qytorch, which allowed us to maintain consistency with the real counterpart of the existing architecture and implementation.In the context of Quaternion Transformers, the self-attention mechanism involves three weights: Query (W q Q ), Key (W q K ), and Value (W q V ). Each of the three contain 4 small weight matrices corresponding to the r, i, j annd k components of a quaternion as shown in Fig 2 . Using these 12 quaternion weights (W q Q,K,V ) we construct the 3 real weight matrices(W r Q,K,V ) following Fig 1.We conducted a series of reset-train-prune experiments using L1 unstructured pruning. At each pruning iteration, a fraction p (prune rate) of the weights in the r, i, j, k components of the threeW q Q,K,V d q n d 4 d 4 d 4 d q 4 d k 4 d v 4 d k n d v n d n n n softmax & scaling n n X W q V W q K W q Q Q K V QK σ(QK) d v n Y Figure 2.Self attention in quaternion setting weights was pruned. Each of the 12 matrices were pruned independently. This process, detailed in Algorithm 1, was repeated n p times. By fixing the seed before reinitializing the model, we ensure that the model parameters are reset to the same values each time. The pruning was designed such that (1 -p) np resulted in a sparsity level below 1%.Algorithm 1 Reset-Train-Prune Process for Transformers Set the value p (p ∈ (0, 1)) for i in 0 to n p -1 do 1. Fix seed and Reinitialize model parameters 2. Train model for required number of epochs 3. Prune the W q Q , W q K , W q V weight matrices, removing a fraction p of the weights compared to what was left before pruning end for• Quaternion: Following [19, 20] , we kept the norm layers, bias, and embedding and class tokens (in ViT) as real-valued, since they collectively contribute less than 1% to the total number of parameters. Moreover, norm weights and bias being 1D arrays, we generally cannot convert them to quaternion matrices which require last two dimentions to be multiple of four. Additionally, existing literature on quaternionbased norm layers are slow and do not reduce parameter counts.• Pruning: We strategically excluded specific layers from pruning, including norm layers, biases, positional embedding, and the class token in Vision Transformer (ViT), as they barely contribute to the overall model size. But in case of 1 . Summary of all the experiments performed, the models used, their parameter counts, and the pruning percentages used for each.smaller models like our ViT (4.1.1), these layers account for approximately 1-2% of the model's parameters, which although becomes significant in extremely sparse conditions, do not cause much problems. We experimented with pruning norm weights and bias weights, observing that pruning norm weights hurts performance more than pruning bias weights.• Resetting: All the weights and biases were reset without any deviation. 
experiments: 
 In this section, we give the details of the experiments we conducted. We explored the two most popular tasks where transformers frequently find use in Computer Vision and Language Modelling. On the vision front, we tried the well-known model ViT, which generally has many parameters and performs pretty well in various vision tasks. To evaluate the performance of our approach on the language task, we use the nanoGPT model. The summary of the experiments performed, the models used, and their parameter count has been provided in table 1.We performed the experiments to show the efficiency of our proposed method on vision tasks. The pri- We modified the Vision Transformer (ViT) [21] models for our experiments, adjusting the number of transformer layers and embedding dimensions to optimize accuracy. For the small CIFAR10 dataset, we used a compact model with 6 transformer layers, an embedding dimension of 64, 8 heads, and an MLP size of 512, resulting in approximately 1.2 million initial parameters in real for both CIFAR10 and CIFAR100. The quaternion-based model had around 0.3 million parameters. We observed that, ViT models easily overfit on small datasets like CI-FAR, which is why we took this approach. Figures 3(a ) and 3(b) illustrate the performance of our model on the CIFAR10 and CIFAR100 datasets. Due to the quaternion layer limitation that input and output neurons must be a multiple of 4 [22] [23] [24] , we added two dummy classes to the CIFAR10 dataset, making the model output three quaternions corresponding to 12 classes. This approach had no distinguishable impact on the learning statistics but allowed us to use purely quaternion models.We conduct experiments on the MedMNIST dataset [25] [26], a comprehensive repository of 12 medical image datasets. The dataset's diversity, featuring varying numbers of classes, images, and channels, makes it well-suited for evaluating our models' performance under different conditions.We use the Vision Transformer (ViT) model with modified specifications: 8 transformer blocks, each with 8 attention heads, a ViT patch size of 7, selfattention embedding dimension of 768, prediction head MLP size of 768, and dropout regularization with a rate of 0.1.We deliberately designed the model to have a lower weight count for the MLP layers compared to the transformer weights. Initially, we found that the MLP had a significant weight count in the ViT-Base configuration [5] . We reduced the MLP size to address this, minimizing its contribution to the total weight count. While increasing transformer properties (e.g., number of heads or layers) didn't significantly impact accuracy, boosting MLP layer characteristics substantially improved performance. However, we prioritized transformer characteristics, maintaining a low MLP size and high transformer properties.The results, as shown in Figures 3(c ) and 3(d), are consistent with our MedMNIST experiments, demonstrating that the quaternion model's performance remains high till a much higher sparsity level compared to the real-valued model.We implement Quaternion and LTH pruning on the NanoGPT model using the Shakespeare [27] dataset, which contains 40,000 lines of Shakespeare's plays. The nanoGPT model [28] generates the next most probable token based on a given context.We trained the nanoGPT model with original hyperparameters and seeds to reproduce the results from the original work. The dataset has approximately 64 characters (rounded to a multiple of 4), where each character is treated as a token.The full real model has about 10.65M parameters, while the quaternion version has about 2.7M parameters. We prune the nanoGPT model using three rates: 0.3, 0.5, and 0.7.Perplexity score is calculated as PPL(X) = exp(-p(x) log q(x)) (p(x) and q(x) denotes output and true probabilities respectively). A lower perplexity score indicates better performance, while a higher score suggests that the model is less accurate or more "perplexed" by the query. 
computer vision: 
 Figures 3 show that the quaternion model's performance is comparable to the real-valued model inspite of having one-fourth the number of weights, achieving similar accuracy with fewer parameters. Additional results confirming similar observations are also shown in figure 9 in Appendix which contains our results related to the MedMNIST dataset.In figure 4 , we observe the different values of weights and their frequency of occurrence during pruning iterations. The insignificant weights closer to zero are pruned, revealing the effective weights contributing to inference. As we prune the model, the number of remaining weights decreases significantly. The near-zero weights are pruned in every iteration, making the distribution two peaks away from zero. Similar results can be seen in figure 6 , which corresponds to the MedMNIST dataset. The performance of the nanoGPT model on the Shakespeare dataset for all prune rates is shown in Figure 5 using perplexity scores. The figure compares the perplexity scores of quaternion and realvalued models with respect to the percentage of weights remaining relative to the unpruned realvalued model. It demonstrates that quaternionvalued models, which originally have approximately four times fewer weights, maintain their performance even when further pruned, resulting in lower perplexity scores compared to the real-valued versions. In unpruned models, a significant portion of weights are concentrated near zero, indicating they don't contribute significantly to predictions. As we prune, the remaining weights adjust themselves, converging to a value not very close to zero. This suggests the model finds an alternative optimum that doesn't rely on very small weights.Quaternion weights exhibit a higher standard deviation than real-valued counterparts, indicating diverse values. They respond better to pruning, with larger middle gap sizes in quaternion models compared to real models.Unlike Figure 4 , we don't see a clear Gaussian distribution. Individual ViT layers follow Gaussian distributions with mean close to zero but different standard deviations. The sum of these distributions results in a non-Gaussian overall weight distribution, more pronounced in the third column (prune iteration 4), where two distinct peaks are visible.This trend is consistent across all MedMNIST datasets and CIFAR datasets used in our experiments. 
conclusion: 
 In this work, we explore the use of Quaternion variations in Transformer models to reduce weight parameters by one-fourth compared to traditional realvalued Transformers. We also test the effectiveness of the Lottery Ticket Hypothesis in pruning these Quaternion-based models and compare their performance with real-valued Transformers.One limitation of our approach is that Quaternion neural networks require the number of input and output features to be multiples of four. While this constraint is generally not problematic for hidden layers, as large neural networks often use feature sizes that are large powers of two, it can pose a challenge for input and output layers where feature sizes may not align with this requirement. We are actively exploring methods to overcome this limitation, ensuring broader applicability of Quaternion-based models across different layers and architectures.Looking ahead, combining our approach with methods like quantization [8] , pruning [29] [30] [31] , and a mixture of experts [32] can lead to even more efficient and deployable models that maintain performance on various devices. We are also interested in investigating the impact of other complex space transformations of weights, such as octonions, and the direct conversion of real trained weights to quaternions without or with minimal training. 
appendix: 
 In this section, we expand on the experiments conducted in the paper. Further results on nanoGPT and vision transformer on MedMNIST are reported here.The performance of the nanoGPT model during each pruning iteration is detailed in Figure 7 . This figure compares the perplexity scores of quaternion-and real-valued models at each iteration, illustrating how the perplexity score increases as more weights are pruned. The results highlight that, despite an increase in perplexity due to the pruning, quaternionvalued models tend to maintain lower perplexity scores compared to real-valued models, particularly in later iterations. This behavior suggests that quaternion models are more resilient to pruning, maintaining performance even as the sparsity level increases. The results of the MedMNIST experiments are compiled in Figure 9 , which displays the highest accuracy achieved on the full training dataset after each pruning iteration. The figure compares the performance of the real-valued and quaternion versions of the Vision Transformer (ViT) across various MedMNIST datasets, using a prune rate of 0.3. The consistent patterns observed across all experiments further reinforce the resilience of quaternion models, as they tend to maintain higher accuracy levels compared to their real-valued counterparts, even as pruning progresses. d_prime = Wq . shape [1] if Wq is not None else x . shape [1] 13 14 a t t e n t i o n _ w e i g h t s = F . softmax ( Q @ K . T / d_prime **0.5 , dim = -1)15 output = a t t e n t i o n _ w e i g h t s @ V for QKV in [ " Q " , " K " , " V " ]: 34 for rijk in [ " r " , " i " , " j " , " k " ]: for QKV in [ " Q " , " K " , " V " ]: 43 for rijk in [ " r " , " i " , " j " , " k " ]:  
 
 Appendix: https://pypi.org/project/qytorch/ 


Decision: In this paper the authors show that quaternion neural networks and sparse networks created by pruning weights can reduce the number of parameters in transformer-based models, making them easier to run on devices with smaller memory. Experiments on vision and language tasks show that pruned quaternion models perform better than real-valued models in severely sparse conditions. The proposed method is implemented in a PyTorch package called qytorch.

Quaternion variations in Transformer models reduce weight parameters by one-fourth compared to real-valued Transformers. Pruning these Quaternion-based models using the Lottery Ticket Hypothesis maintains performance comparable to real-valued Transformers. Combining them with other techniques like quantisation and pruning can lead to more efficient and deployable models.

These are some neat results worth publishing and presenting at the conference.

Reviews: review: title: title: Review for transformers at a fraction
paper_summary: paper_summary: This manuscript proposes a method to reduce the number of parameters so that the transformer-based large models can be used in on-device environments without significant performance degradation. By constructing the weight matrices of query, key, and value of transformer from quaternion components, the authors decrease the parameters of the transformer by $1\over4$. Also, they further diminish the number of parameters by pruning based on the Lottery Ticket Hypothesis (LTH). They demonstrate that their model shows similar or slightly lower performance than the original models with much fewer parameters on computer vision and language modeling tasks using Vision Transformer (ViT) and nanoGPT.
main_review: main_review: The authors explain the weaknesses I pointed out well, and their arguments are valid. The lacking experiments have been supplemented in the rebuttal, and their ideas are novel.
strength_weakness: strength_weakness: The quaternion method you applied requires that the real weight matrix be divisible by a multiple of 4, so adjustments such as dummy class padding are required. \
If the number of parameters after pruning is provided, the parameter reduction performance can be shown more clearly. \
What criteria were used to select the prune rate $p$, such as $p=0.4$ for ViT on CIFAR10 datasets? \
The models used in the experiments are insufficient to be called large models. It was good to show the results for each task, but it is unclear whether their method can work sufficiently in a large model. \
The contribution is not enough. Methods for reducing the parameters of the transformer based on the quaternion method have been proposed previously [1], and also LTH-based pruning methods are previously used [2, 3]. \
Furthermore, the proposed method requires iterative pruning with $n_p$ steps since it is based on the initial version of LTH-based pruning, which increases the model learning time. If you use the methods for obtaining ‘winning tickets’ faster [4, 5], the efficiency of your model can be improved. \
It is difficult to fully understand what the authors claim, like model pruning or parameter reductions from the current title. 

Minor
* Table 1: There is no result about OpenWebText dataset
* (typo) line 259: “CIFAR” -> “MedMNIST”
* (typo) line 227: Prune rate might be “0.3, 0.5, and 0.3” -> “0.3, 0.5, and 0.7”
* The reference page needs to be rearranged

[1] Tay, Y., Zhang, A., Tuan, L. A., Rao, J., Zhang, S., Wang, S., ... & Hui, S. C. (2019). Lightweight and efficient neural natural language processing with quaternion networks. arXiv preprint arXiv:1906.04393. \
[2] Morcos, A., Yu, H., Paganini, M., & Tian, Y. (2019). One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers. Advances in neural information processing systems, 32. \
[3] Yu, H., Edunov, S., Tian, Y., & Morcos, A. S. (2019). Playing the lottery with rewards and multiple languages: lottery tickets in rl and nlp. arXiv preprint arXiv:1906.02768. \
[4] Lee, N., Ajanthan, T., & Torr, P. H. (2018). Snip: Single-shot network pruning based on connection sensitivity. arXiv preprint arXiv:1810.02340. \
[5] Wang, C., Zhang, G., & Grosse, R. (2020). Picking winning tickets before training by preserving gradient flow. arXiv preprint arXiv:2002.07376.