# Enhancing Diffusion-Based Music Generation Performance with LoRA
## 1 Introduction  
The rapid advancement of generative artificial intelligence (AI) has catalyzed a significant transformation across creative technologies, with AI-based music generation emerging as an active research field due to its broad range of applications. Traditional methods relying on rule-based composition systems or simple probabilistic methods often lack the ability to represent diverse musical structures and contextual expressiveness [1]. However, with the emergence of deep learning-based generative models, particularly latent diffusion models (LDMs), it is now possible to generate high-quality music that more accurately reflects the user's intent [2]. Among these models, text-to-music generation has drawn considerable interest, enabling users to generate music from natural language prompts without prior musical expertise. Various models, such as MusicLM [3], MusicGen [4],and AudioLDM [5], which are built on the LDM framework, have demonstrated the ability to map textual inputs into coherent musical outputs. Despite these advances, current text-to-music models still face several unresolved limitations that hinder their scalability in real-world applications. There are three significant problems. First, training large-scale text-to-music models requires massive, paired datasets. Most musical data are subject to copyright restrictions, limiting access and legal use. Furthermore, the ownership of AI-generated music remains a legally ambiguous area, raising concerns over fair use, licensing, and authorship [6]. Second, existing models are effective at generating short musical phrases; however, they struggle to maintain structural coherence over longer durations (e.g., several minutes). Many models prioritize music quality over musical structure, resulting in outputs that may sound realistic but lack meaningful development or narrative form, limiting their usability in professional media production [7]. Third, most models are generalist by design, aiming to cover a wide range of musical genres. However, they struggle to capture accurate, fine-grained stylistic elements of specific genres. For example, boom bap, trap, and gangsta rap within the hip-hop genre differ significantly in rhythm, instrumentation, and tempo. The current models often fail to reflect these distinctions, producing generic outputs that may not align with the user's creative intent [8]. As a representative text-to-audio generation model based on the LDM, AudioLDM aims to address several challenges in the domain. First, this model enables generation without relying on large-scale explicit text-audio paired datasets by applying contrastive language-audio pretraining (CLAP) embeddings to capture the semantic alignment between text and audio, partially alleviating data collection burdens and copyright concerns. Second, this model improves the coherence of the generated music by aligning the semantic meaning between the text and audio rather than relying solely on simple conditional prompts. Third, this model incorporates architectural flexibility that allows it to respond to a wide range of text conditions. However, despite these improvements, several limitations remain. For instance, the model still struggles with fine-grained music generation tailored to specific domains or genres. Moreover, its ability to control detailed musical style remains limited, and it lacks an efficient fine-tuning mechanism that enables lightweight adaptation to specialized data. This paper aims to address these challenges by introducing a Parameter-Efficient Fine-Tuning (PEFT) strategy by integrating low-rank adaptation (LoRA) into the self-attention blocks of the AudioLDM U-Net. The AudioLDM system lacks the ability to express accurate fine-grained subgenres. Instead, this model maps textual semantics into conditioning embeddings using CLAP and generates audio waveforms via a U-Net-based diffusion process.