# Enhancing Diffusion-Based Music Generation Performance with LoRA
## 1 Introduction
Although CLAP is effective at modality-level semantic alignment, it struggles to capture subtle musical elements, such as rhythmic patterns, instrumentation, and timbre, that distinguish subgenres. More critically, most generative operations are performed within the U-Net, which is trained to reflect conditions uniformly; hence, the model fails to adapt its generation process to subgenre-specific characteristics. For example, within the hip-hop genre, styles (e.g., boom bap, trap, and gangster rap) differ significantly in rhythm and instrumentation, but AudioLDM tends to produce musically generic outputs that fail to reflect these distinctions. Similar limitations have been observed in MusicGen, which has a diffusion-based architecture [3; 4]. Second, the model exhibits inconsistency in reflecting user-defined conditions. While AudioLDM incorporates CLAP embeddings via cross-attention layers in the U-Net, it fails to consistently align high-level conditions (e.g., emotion or atmosphere) with musical components across the entire generation process. For instance, prompts containing emotional terms (e.g., \"melancholic\" or \"dreamy\") may appropriately influence chord progressions but often contradict the tempo or instrumentation choices. This misalignment results in a perceptual gap between the user expectations and the generated output, forcing users to revise prompts or perform manual post-processing repeatedly. MusicLM exhibits similar problems when translating emotional input into coherent musical structure [9]. Third, the use of full fine-tuning imposes significant barriers to fast adaptation and reusability. The AudioLDM framework is built on a large-scale pretrained diffusion model, and any adaptation to new genres or user preferences requires completely retraining the U-Net. This retraining results in inefficiency in real-world deployment. For commercial applications that require rapid music generation, full-model retraining is impractical due to its associated high computational costs. Furthermore, the lack of modularity requires each style to be trained from scratch, delaying the development pipeline. Finally, no mechanism exists to transfer the learned parameters across related domains, limiting reuse, experimentation, and task switching. These limitations have been well-documented in prior work, which has called for lightweight adaptation methods to improve training efficiency and transferability [10]. The AudioLDM framework generates audio in the latent space of a variational autoencoder based on mel spectrograms and conditioned on the CLAP embedding, which learns the semantic alignment between text and audio. However, fine-tuning the original AudioLDM is challenging due to the difficulty of updating all model parameters, which incurs high computational costs and complicates fine-grained adaptation for specific genres. In this paper, we propose a method to address the challenges of genre-specific music generation by applying LoRA to AudioLDM, a text-to-music generation model based on the LDM. The proposed method applies LoRA only to the core components of AudioLDM (i.e., the self-attention and projection layers) while selectively adding learnable parameters. The LoRA method adapts a low-rank matrix approach to update only a small portion of the parameters while keeping the remaining model parameters fixed. This approach enables efficient fine-tuning for specific domains or styles while maintaining the overall model structure, making it practical for environments with limited computational resources. By reducing the number of learned parameters, the method offers advantages in terms of faster training speed and lower memory usage. Therefore, this method allows for quick adaptation to new genres or styles, making it effective for user-customized music generation.