Poster: Evaluation of Membership Inference Attacks on LoRA with Differential Privacy 1st Takuya Higashi Mitsubishi Electric Corporation Kanagawa, Japan Higashi.Takuya@da.MitsubishiElectric.co.jp 2nd Tsunato Nakai Mitsubishi Electric Corporation Kanagawa, Japan Nakai.Tsunato@dy.MitsubishiElectric.co.jp Abstract—While the utilization of large language models (LLMs) is expected in various industries, concerns have been raised regarding information leakage and privacy issues related to training data. The application of differential privacy has been actively researched as a solution to the problem. Differential privacy has been applied during the fine-tuning stage of pre- trained LLMs, and the information leakage risk concerning training data has been evaluated through membership inference attacks (MIAs). Although parameter-efficient fine-tuning (PEFT) has recently gained attention, it has also been studied for the application of differential privacy. An evaluation of information leakage risk through MIAs has not yet been reported. Therefore, in this paper, we focused on the representative PEFT method Low-Rank Adaptation(LoRA) , and performed an MIA evalua- tion of a model trained with differential privacy-enhanced LoRA. We demonstrate the effectiveness of applying differential privacy to LoRA through MIA evaluations. Through the experiments, it was revealed that fine-tuning via LoRA has an effect in mitigating privacy risks. However, no distinct impact on privacy risk reduction was observed through the application of differential privacy. This was consistent even when varying the privacy parameter ϵ in the validation. Index Terms—Large Language Model, PEFT, Differential Pri- vacy, Membership Inference Attack I. INTRODUCTION Transformer-based large language models (LLMs), such as BERT and GPT [1] have been developed and have achieved state-of-the-art performance in various natural language pro- cessing tasks. Generally, such models are pre-trained on large and diverse publicly available datasets and then fine-tuned for specific tasks. Fine-tuning is a technique in which a pre- trained model is retrained on a new dataset to improve the task performance. Recently, as the models become larger, parameter-efficient fine-tuning (PEFT) has gained attention. PEFT is a technique used to customize the LLM, where a small number of parameters or layers are added to the original LLM and trained with case-specific data. The weights of the original LLM remain fixed, resulting in significantly fewer parameters being updated during training. This improves performance in specific cases with limited computational resources. LLMs have a high capacity to memorize training data, leading to concerns regarding the information leakage risks associated with training data [2]. Recently, membership in- ference attacks (MIAs) have been regarded as an information leakage evaluation tool that reveal the extent to which a model memorizes individual samples from a training dataset and the associated risks. Ensuring differential privacy (DP) in the training of LLMs has gained attention as a countermeasure to address the information leakage risks associated with training data. DP is a powerful theory in data privacy and a technology that was originally proposed for statistical data. Recently, it has been commonly employed when applying privacy guarantees to machine learning models. Research is underway to apply DP to the training methods of pre-training, fine-tuning, and PEFT in LLMs. MIA evaluations have begun to be performed on large-scale language models that apply differential privacy, but PEFT has not been evaluated. Du et al. conducted information leakage risk evaluations of training data using MIAs on full fine-tuned models with DP [3]. However, models that are fine-tuned by DP-enhanced PEFT have not been evaluated. In this study, we investigate whether applying DP to repre- sentative PEFT, LoRA can mitigate information leakage risks associated with training