Complexifying BERT using LoRA Adapters
Authors: Fabio Tamburini (Università di Bologna, 2024)
Topic: Multimodal

Abstract
This paper presents the first results of a pilot study for transforming a real-valued pre-trained transformer encoder into a complex-valued one. Following recent findings about pre-training using LoRA, the main idea is to employ complex-valued LoRA adapters to make the trick and continue the pre-training of a given Italian model for setting up the adapters. After pre-training, the proposed complex-valued model has been evaluated on a standardised benchmark for Italian natural-language understanding obtaining very encouraging results.

Introduction
The works from Arjovsky et al. [1], Trouillon et al. [2] and Trabelsi et al. [3] proposing complex-valued Deep Neural Networks (DNNs) rose an increasing interest on this type on Neural Networks for their intrinsic ability to manage problems defined on complex-valued features. For example, in the fields of signal and image processing, speech, signal and audio data are naturally complex-valued after Fourier, Laplace or Complex Wavelet transforms. Yang et al. [4] and Eilers and Jiang [5] presented state-of-the-art Automatic Music Transcription systems and Wang et al. [6] evaluated their complex-valued embeddings in text classification, machine translation and language modeling with promising results. Quantum-inspired Machine Learning, an emerging topic of research in NLP and AI, is completely based on complex-valued features and tensors. Liu et al. [7] presented a survey of novel quantum-cognitively inspired models that solved the task of sentiment analysis with good performances and Tamburini [8] proposed a Quantum WSD system based on static complex-valued embeddings obtained modifying the ‘word2vec’ [9] code. The transformer encoder is a crucial component in transformer architectures [10]: primarily designed for processing input text and producing intermediate representations of input sequences, it consists of multiple layers of self-attention mechanisms and feed-forward neural networks, each contributing to the encoding process of both single words and entire sequences.LoRA (Low-Rank Adaptation) [11] is a technique recently introduced to efficiently fine-tune transformer models. Instead of updating all the parameters of a large pre-trained model, LoRA introduces a small set of additional trainable parameters. These parameters are incorporated into the transformer layers through low-rank matrices, allowing the model to adapt to new tasks with significantly reduced computational and storage requirements. This method preserves the original model’s performance while enabling quick and cost-effective customisation for specific applications. A very recent work [12] suggested that, by applying LoRA adapters, it is possible to pre-train large transformer models from scratch obtaining comparable performance with respect to regular pre-training. The main idea and contribution of this work consists in using LoRA adapters to convert a real-valued pre-trained transformer model into a complex-valued one being able to produce as output complex-valued word and sequence embeddings to be used in subsequent tasks. This process will require to continue the pre-training stage of a realvalued transformer model for setting up complex-valued LoRA adapters and train the global model to produce meaningful complex-valued embeddings. Section 2 describes the state-of-the-art about complex-valued transformers; Section 3 presents the proposed model describing the internal details of our complex-valued LoRA-based transformer. Section 4 illustrates the obtained results when testing our complex-valued model on a benchmark for evaluating Natural Language Understanding (NLU) systems for the Italian Language [13] and Section 5 discusses the results and draws some conclusions.

Conclusion
In our evaluation experiments we adopted the hyperparameters proposed in Basile et al. [13] for maintaining comparability, but our models are bigger and more complex and, maybe, need more training epochs and/or different learning rates to achieve a full convergence during the fine-tuning phase for evaluation. For example, we were forced to reduce the learning rate to 1e-5 for each model evaluated on TE benchmark to favour convergence. Again, we clarify that the goal of this work is not to beat other systems in the leaderboards, but to show the effectiveness of this approach for complexifying transformer architectures and we think that the results confirm our initial research question. Having complexified BERT matrices by adding LoRA adapters, we have no guarantee, in principle, that the system will not converge to the original BERT-based model setting all adapters to zero and nullify all imaginary part in the complex-valued model. We checked this in various ways and, as shown in Figure 1, some randomly chosen complex-valued components of token embeddings for the CmplxBERTLoRA_16 model show to cover the entire complex space in a uniform way, supporting the idea that the pre-training phase consistently adapted the starting real-valued model to produce reliable complex-valued embeddings. We did also some experiments with a real-valued LoRA model containing about the same number of parameters of CmplxBERTLoRA_8, adding real-valued adapters of rank 16, to investigate if a complex-valued transformer is able to produce better results that an equivalent realvalued one, but such experiments did not show any relevant performance differences between the two models. This work presented a relevant set of experiments for testing the idea of being able to complexify a Transformer encoder architecture like BERT by using complex-valued LoRA adapters. The obtained results on Italian models are very encouraging showing in a clear way that this technique is effective in transforming a real-valued pretrained model into a complex-valued one maintaining the same level of performance. We have to say that the UINAUIL benchmark is not without problems: TE dataset is very small and such large models struggle to reliably converge to a reasonable minimum during training leading to very unstable results. FactA is very problematic as well: classes are strongly skewed and the Max_Freq_Baseline, always choosing the highest-frequency class, is able to achieve an accuracy of 0.967! For all these reasons, we think that these two benchmarks should be excluded from any real evaluation. This pilot study presents only the first step for proposing building blocks based on LoRA adapters for complexifying any kind of transformer, either for representation learning or for text generation or for both processes together. All the complex-valued models were pre-trained on various GPUs for speeding up the experiments, but a general CmplxBERTLoRA model can be trained on a single 12/16GB GPU without problems, while the pretraining of a complex-valued BERT model from scratch would have required at least 4 NVIDIA A100 64GB GPUs for obtaining results in reasonable time. Using LoRA for ‘complexifying’ a model mitigates the need of complex and expensive computational infrastructures not easily available to any scholar. Code and models are available on github. .