Title: AC-LoRA: (Almost) Training-Free Access Control Aware Multi-Modal LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Information access control

Secure LLM

corporate LLM

LoRA
Abstract: Corporate LLMs are gaining traction for efficient knowledge dissemination and management within organizations. 
However, as current LLMs are vulnerable to leaking sensitive information, it has proven difficult to apply them in settings where strict access control is necessary.
To this end, we design AC-LoRA, an end-to-end system for access control-aware corporate LLM chatbots that maintains a strong information isolation guarantee.
AC-LoRA maintains separate LoRA adapters for permissioned datasets, along with the document embedding they are finetuned on.
AC-LoRA retrieves a precise set of LoRA adapters based on the similarity score with the user query and their permission.
This similarity score is later used to merge the responses if more than one LoRA is retrieved, without requiring any additional training for LoRA routing.
We provide an end-to-end prototype of AC-LoRA, evaluate it on two datasets, and show that AC-LoRA matches
or even exceeds the performance of state-of-the-art LoRA mixing techniques while providing strong isolation guarantees.
Furthermore, we show that AC-LoRA design can be directly applied to different modalities.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces AC-LoRA, a system that regulates which document(s) an LLM can use (include as finetuned  LoRAs) for users with different roles. AC-LoRA stores a set of LoRAs with the corresponding documents used to fine-tune these LoRAs in a form of sentence embeddings. At inference time, AC-LoRA considers only the LoRAs/documents that are permissible to the user, and compares the embedding of the user’s query to the available document embeddings to dynamically load the relevant LoRAs. After selecting the relevant LoRAs, the model generates an answer by mixing the retrieved LoRAs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n* The paper targets an important problem with the aim of making the use of LLMs more practical \n* The paper considers both text and image modalities (the latter is admittedly not comprehensive (cf. Limitations))\n\nWeaknesses:\n* The work is incremental. From my understanding, AC-LoRA adds only a filtering step to select only relevant LoRAs based on the cosine similarity. The mixing is already introduced in other works. \n* It is not clear how AC-LoRA compares to a baseline where all the permissible LoRAs for the user are used. \n* Hinting is introduced as part of AC-LoRA, but is never evaluated and its benefits are not clear (it only shows the users that there is something they don't see)\n\nQUESTIONS:\n### Questions:\n* Q1: Table 2: what is meant with IID and OOD?\n* Q2: Why not use a baseline with only permissible LoRAs?\n* Q3: Is AC-LoRA practical in case of hundreds of (relevant) documents?\n\n\n### Comments:\n\nL165: “Isolated LoRA Fine-tuning and Knowledge Injection”: Please motivate the experiment better at the very beginning (\"LoRAs can reliably inject domain-specific knowledge\") \nTable 2: The selection baseline seems competitive. Please add this to the discussions (L310-319)\n\nSome parts can be made more consistent (e.g., \"knowledge merging\" is used in 4.1, and the corresponding paragraph in 4.2 is called \"combining knowledge\")\n\nL206: I believe $Q$ is not defined before\n\nFig. 5 caption: “Retrieval of FlanV2 target domains (**left**) and corresponding retrieved domains (**left**).” (one should be \"right\")\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents AC-LoRA, the first end-to-end access control-aware inference serving system to equip LLMs with strong access control management. AC-LoRA control the sensitive information access by fine-tuning different LoRAs on predefined data groups and later uses embedding similarity to compute the LoRA based on user permission and user query. The design of similarity combination based on embedding makes AC-LoRA almost training-free. The author conducts comparison experiments with the previous method on two datasets, and the result demonstrates that AC-LoRA show better or on-par performance than SoTA LoRA mixing techniques, which highlights the effectiveness and efficiency of AC-LoRA.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. I like the content of the paper a lot. The paper is well motivated and written in an organised format.\n2. The design of using embedding similarity to implement access control is novel and light-weight. This design not only reduces the additional training requirement but also ensures high performance compared to the previous method.\n3. The experiment is comprehensive and sufficient to illustrate the effectiveness, efficiency and potential application of multimodality. Basically, I have no question about the experiment set-up, as the result is pretty clear to support the author's claim.\n\nWeaknesses:\n1. The \"gap in prior work\" section in the introduction is well motivated, but I think there is a lack of the limitation of RAG to the usage of LoRA or LoRA mixing techniques.\n2. It seems that the effectiveness of AC-LoRA is built on the sparse distribution of the knowledge base. I am wondering whether AC-LoRA can achieve the same high performance on embedding space that is not well-separated (maybe math or coding)?\n3. What if there is partial overlapping between knowledge groups? Can the author illustrate whether the overlapping will affect the LoRA selection?\n4. What if one query requires access to more than 5 knowledge groups, for example, 10 or embedding space is pretty large, will GPU memory become a major bottleneck?\n\nQUESTIONS:\nSee weaknesses 3 and 4.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes AC-LoRA, a system for developing access control-aware LLM chatbots to prevent leakage of sensitive/confidential data to unauthorized parties. In doing this, the authors attempt to address a gap in existing work where maintaining individual models (or LoRA adapters) for different roles becomes infeasible as the number of permission zones grows, while training a single foundation model compromises data confidentiality. Other approaches such as access-controlled RAG are also computationally expensive and/or detrimental to performance. Instead, the authors propose a method based on fine-tuning LoRAs for individual permission zones. These LoRAs are then mixed together in a permission-ware manner, ensuring that a user does not retrieve information from LoRAs they are not authorized to use; non-permissible LoRAs are then used to provide a hint to the user that they might be able to receive a better answer by applying for relevant permissions. Evaluations are performed using three different datasets: Flanv2 , RepLiQA, and Wikiart. Specifically, the authors show that the proposed LoRA mixing offers competitive performance to perfect selection (using the specific LoRA associated with a given query) and outperforms other baselines in 8 out of 10 evaluated domains.\n\nSTRENGTHS AND WEAKNESSES:\nThis was an interesting paper addressing an important gap in the usage of LLMs in access controlled environments. The authors propose a plausible training-free solution, and the evaluation results show that AC-Lora performance comparably to perfect selection, while outperforming existing methods in performance and/or latency.\n\nThe technical novelty is modest, largely involving a thoughtful combination of existing methods rather than introducing fundamentally new techniques. I would have liked to see additional evaluations, particularly a computational complexity comparison with perfect selection, as well as an ablation study to quantify the performance gains achieved by AC-LoRA compared to simply mixing all permissible LoRAs without the retrieval step. It is also not clear how the framework performs when a query is relevant to multiple domains/LoRAs.\n\nOther more minor comments:\n- I suggest the authors revise the Figure captions to make them more self-contained.\n- The evaluation using the WikiArt dataset is limited to Figure 7, which does not include any quantitative performance metrics. I recommend either expanding the evaluation for WikiArt or removing it from the evaluation set to make room for other improvements.\n\nQUESTIONS:\n- In Table 2, what metric is being reported? Please clarify in the caption.\n- Have the authors performed evaluation using queries relevant to multiple domains (i.e., permission zones)?\n- How much performance improvement does AC-LoRA offer compared to a simpler strategy of mixing all permissible LoRAs without retrieving relevant ones first?\n\nLIMITATIONS:\nThe limitations are adequately discussed, with the possible exception of multi-domain queries that require retrieving information from multiple permission zones.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new method called AC-LORA, which tackles the problem of letting a single large model serve many users who each see only the data they are entitled to. Instead of fine-tuning one big model on all documents, or leaving those documents into the prompt, it fine-tunes a lightweight LoRA adapter for every permission segment, stores each adapter’s embedding in a vector index, and at inference time retrieves only the adapters that are semantically relevant to the query and  allowed for the current user. These adapters are loaded on-the-fly and blended by cosine-similarity weighting, so no extra routing network or retraining is needed when new data arrive. Experiments on text (RepLiQA, Flan-V2) and image generation (WikiArt styles) show that AC-LORA matches or beats task-specific LoRA baselines while slashing context length, latency, and leakage risk. The system therefore offers a practical, nearly “zero-additional-training” way to combine access control, retrieval, and knowledge fusion for multi-modal LLM applications, with remaining challenges around hint privacy and adapter-swap overhead in very large deployments.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. This paper combines LoRA tuning, vector search, and cosine mixing into a single, low-overhead framework, which is a novel integration. 2. This mode works across modalities, which is shown to run on both text (LLAMA-2/3) and image models (Stable-Diffusion, Qwen-VL). \n3. It provides fine-grained access control and hot-swappable adapters, matching real-world deployment needs. \n4. Leakage-extraction attacks confirm that per-adapter isolation greatly reduces verbatim data leaks, demonstrating the security.\n\nWeaknesses:  \n1. No confidence intervals or significance tests, so robustness of the gains remains uncertain. \n2. Results stop at ten adapters; latency and memory for hundreds are not reported. \n3. This paper offers little analysis of when linear LoRA mixing could fail due to conflicting updates.\n\nQUESTIONS:\n1. Please give analytical or empirical evidence that simple cosine-weighted blending is robust when two adapters make conflicting updates to the same weight tensor, visualise conflict rates or run controlled ablations. \n\n2. Please attach 95 % confidence intervals or formal significance tests (e.g., paired t-tests or bootstrap) for the main RepLiQA, Flan-V2, and WikiArt results and state the random seeds used.\n\nLIMITATIONS:\nThe paper briefly admits that the Hint mechanism could leak sensitive information but never formalizes any threat model or measures what is actually exposed, and it overlooks broader risks such as adapter-fusion bypassing content filters, the environmental cost of large-scale hot-swapping, and governance concerns around data revocation or labor displacement. A concise yet concrete treatment of these issues like quantitative leakage metrics, energy estimates, and responsible-deployment guidelines would make the societal-impact discussion far more convincing.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-04"}