AC-LoRA: (Almost) Training-Free Access Control Aware Multi-Modal LLMs
Authors: Lara Magdalena Lazier
Aritra Dhar
Vasilije Stambolic
Lukas Cavigelli
Venue: NeurIPS 2025 poster ()
Keywords: Information access control
Secure LLM
corporate LLM
LoRA
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: AC-LORA provides a training-free secure access control for LLMs using separate LoRAs fine-tuned with sensitive data, and merge them based on the user query and permission.

Abstract: Corporate LLMs are gaining traction for efficient knowledge dissemination and management within organizations. 
However, as current LLMs are vulnerable to leaking sensitive information, it has proven difficult to apply them in settings where strict access control is necessary.
To this end, we design AC-LoRA, an end-to-end system for access control-aware corporate LLM chatbots that maintains a strong information isolation guarantee.
AC-LoRA maintains separate LoRA adapters for permissioned datasets, along with the document embedding they are finetuned on.
AC-LoRA retrieves a precise set of LoRA adapters based on the similarity score with the user query and their permission.
This similarity score is later used to merge the responses if more than one LoRA is retrieved, without requiring any additional training for LoRA routing.
We provide an end-to-end prototype of AC-LoRA, evaluate it on two datasets, and show that AC-LoRA matches
or even exceeds the performance of state-of-the-art LoRA mixing techniques while providing strong isolation guarantees.
Furthermore, we show that AC-LoRA design can be directly applied to different modalities.

AC-LORA: (Almost) Training-Free Access Control-Aware Multi-Modal LLMs
Lara Magdalena Lazier'
Aritra Dhar'
Vasilije Stambolic
Lukas Cavigelli
*Computing System Labs, Huawei Technologics Switzerland AG
EPPL
Abstract
Corporate LLMs are gaining traction for efficient knowledge dissemination and management within organizations. However, as current LLMs are vulnerable to leaking sensitive information, has proven difficult apply them in settings where strict access control is necessary. To this end, we design AC-LORA, an end-to-end system for access control-aware corporate L.M chathots that main- tains a strong information isolation guarantee. AC-LORA maintains separate LoRA adapters for permissioned datasets, along with the document embedding they are finetuned on. AC-LORA retrieves precise set of LoRA adapters based on the similarity score with the user query and their permission. This similarity score is later used to merge the responses if more than one LoRA is retrieved, with- out requiring any additional training for LoRA routing. We provide an end-to-end prototype of AC-LORA, evaluate it on two datasets, and show that AC-LORA matches or even exceeds the performance of state-of-the-at LoRA mixing tech- niques while providing strong isolation guarantees. Furthermore, show that AC-LOR design can be directly applied to different modalities. AC-LORA A is open-source and is available https:/ :/pp:///g htt.:/cco/mmaaveeaaa/////
Introduction
LLMs are increasingly utilized for search, summarization, and knowledge retrieval, and are being rapidly adopted both personal and corporate use cases. Despite their benefits, the security risks [1, 2] introduced by including sensitive IP data in training retrieval-augmentee generation (3) threaten the widespread deployment of such tools. Therefore, LLM inference must adhere to strict access control rules, such allowing only authorized users ensuring safety by preventing users from accessing harmful content.
Gap in prior work. While RAG can fetch new data (grounding the LLM response) with existing access control methods, it has slower inference due to retrieval from storage media, or inter- net [4], diminishing inference performance (latency, memory and accuracy) in long context infor- mation extraction [5], low accuracy in multi-hop retrieval[6], embedding space collapse [7] due to high dimensionality, and vulnerable to poisoning attacks [8-10]. A recent study [11] shows that RAG-based solutions can make models even more unsafe than their non-RAG counterparts. Fine- tuning adds task capabilities to base models [12] and can incorporate new knowledge [13]. Not having to include relevant information in each request's context achieves lower inference latency. However, once trained or finetuned, it is challenging isolate remove [14] information due to memorization of training data [15, 16]. Notably, the absence of reliable unlearning techniques[17] significant issue when dealing with proprietary corporate data with strict access control require- ments. Maintaining isolated models finetuned each on sensitive non-overlapping datasets (n) is also feasible due to exponentially increasing (2") possible permission zones.
i Work done during an internship at Computing System Labs, Huawei Technologies Switzerland
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Generalization
User
Query: How many cores are CPU?
Document 1
The CPU has 8 cores.
Models
Hicrarchy
Document 2 The CPU has 6 corcs.
Model
Response:
The CPU may
Document 3
have 6 & cores.
The CPU has high- performance and 2 low-performance
However, there are other configurations that you don't 't have access to.
Fine-tune
Data
corcs.
Specialization
Domains/Access Groups
Apply here.
Hint
Training Dataset
(b)
Figure 1: A corporate LLM overview: (a) shows the corporate information and role hierarchy, highlighting the complexity of managing access control. (b) Shows the expected inference result combining multiple knowledge domains while adhering to the permission rules, including the hint.
This work. We present AC-LORA, the first end-to-end access control-aware inference serving system for LLMs with strong access control by construction. AC-LORA A compartmentalizes sensi- tive information by fine-tuning different LoRAs on data from different access control groups (c.g., projects or departments). AC-LORA uses a retriever that retrieves set of the most relevant (al- lowed) LoRAs, and combines them on top of the base model, based on the input prompt and the user's permissions. AC-LORA effectively summarizes information from multiple information do- mains (cf., Fig. b), while providing helpful guidance to the users in case the access to the document requires additional permission. Importantly, unlike most existing mixtures of LoRA approaches [18, 191, AC-LORA A requires no additional training. This tackles exponentially increasing (2") pos- sible permission zones without requiring the effort of training and the maintenance of an exponential number models.
We evaluate AC-LORA multiple models and datasets and show its adaptability to different modalities: LLAMA2/3 for text, STABLE-DIFFUSION-VE for text-to-image, and QWEN2-VL for text-image-to-text. We compare AC-LORA's dynamic LoRA mixing mechanism with existing works [20] using the Flanv2 dataset [21]. AC-LORA achieves competitive performance on all tasks, matching outperforming prior works in 8 out of 10 domains. We evaluate AC-LORA on RepLiQA [22] dataset, which consists of a wide range of knowledge-specifut questions across 3591 documents spanning 17 different domains, and wikiarts [23], image dataset that consists of 27 different style domains. AC-LORA 's retriever consistently achieves high (>> 90%) accuracy retrieving the correct fine-tuned adapter (without we highlight that fine-tuned adapters can actively inject domain-specific knowledge. To evaluate the knowledge augmentation via LoRA mixing, using RepLiQA, we create dataset by partition- ing knowledge. We show that AC-LORA not only leverages the information included in individual LoRAs but can combine knowledge across multiple LoRAs to give a unified answer. AC-LORA's time-to- first-token generation latency is lower compared to the RAG due to the shorter context. Ad- ditionally, we demonstrate that besides text, AC-LORA can extend access control to other modali- ties: text-to-image and text-image to text.
retrieving more than LoRAs). Further,
Our Contribution. In summary, our contributions the following:
Access control-aware inference serving. We present AC-LORA, an efficient end-to-end access control-aware inference serving system for corporate LLMs.
LoRA retrieval and training-free free LoRA mixing. We design, implement, and evaluate multi-LoRA retrieval and mixing based on user queries, allowing users to retrieve information across datasets without the complexity of maintaining exponentially many models.
Comparative study, We conduct an in-depth comparative study of existing LoRA mixing and merging techniques and their effectiveness in corporate access control. We demonstrate the sever- ity of the information leakage from the LLM memorization. This shows that designing an access control-aware LLM is critical for the successful adaptation of corporate Al chatbots.
Multi-modal demonstration, We demonstrate AC-LORA on text and multi-modal L.Mss, and we show that AC-LORA is effective for practical use-cases.

---

Table I: Comparison of our proposed method AC-LORA with existing LoRA mixing techniques.
St #ioput sequence
Rank of router LoRA
#LoRA
D: Layer dimension
not supported
LY: #Layers
Nmod' modules Transformer block
Text
Vision
Supported
Routing
Elliieasyy
Triining tro
Updan
Solocdion
Medlemiua
Cuse vize (C)
paramitor (P)
Lokt Aocating
Menory
Triningg Elfert
systems
SMORA (35)
Top
Bank
N"SD
G LY
OIN)
O1P)
O12*)
OLN)
N*SD
OIN)
012")
Doflasia-MMoLE [36]
Topsk
N282
NSD2
[(N]
(
0000
M.ELORA [19]
X
Top-k
Token
ND
G
Rugieeal-Aggneeeel (20)
Top-k
Token
2A5D
001
LLaVA MoLE
Top
Token
ND
Gx
ON
DynMoLL [38]
Top Koop
O(N)
O(1)
O12")
0(N)
HDDMLLB [39]
Top
O(N)
012)
O12*)
0(N)
LORA-LEGO (40)
unit chasering
merge
None
OU1
012%)
0(1)
MiLoRA [41]
Top
Seq
NaN
NdW
0(1)
O11)
012")
O(N)
MetcoRA [42]
X
Top-k
Tokm
ND
G EY
OLN]
O(P)
012
O(N)
SMEAR (41)
Topsk
Sag
ND
G CY
O[N]
0(P)
012")
O(N)
LocaRatizeer
Aug
Token
Non:
AC-LORA paper)
Top-k
Seg
None
013)
None
((N)
2 Motivation, Problem Statement, and Related Works
Besides documentation and code bases, corporate LLMs are trained with employee-specific data such meeting records, emails/chat records on project progress, wiki entries, etc. Fig. shows that information access typically follows the organization hierarchy. Users should only be able to access their data and the projects they participate in or manage. Naively, organizations can train sep- arate models with non-overlapping sensitive documents. Maintaining these models is prohibitively expensive, as organization with n permission zones has 2n distinct permission groups.
Challenges with Single Foundation Model Training. A single foundation model trained with all organizational data is easy to manage but poses security risks. LLMs retain some part of the training dataset, known as memorization, which can reproduced or confirmed via membership inference attacks [24]. In practice, censorship methods are employed monitor inputs and outputs to prevent sensitive data leaks. However, studies [25-27] show these mechanisms are often inad- equate, as attackers can bypass them with jailbreaking [28, 29] and harmless-looking inputs [30- 33]. Information leaks in corporate chatbots [1, 2, 34] threaten Al adoption in such contexts.
To show the real-world implica- tions of memorization, we fine-tune LLAMA3.1-82 using two LoRAs on arXiv papers about confidential com- puting (CC) and quantum computing (QC), all published after the model's s training cutoff. We evaluate verbatim memorization of the training dataset inference time. Fig. 2 shows the information leakage (8, 12, 15, and 18 subsequent grams) from the train-
8-gram
12-gram
15-gram
18-gram am
sample
CC[MP, T-0.7)
2
QCIMP, T-0.7)
21
2
OCISP, T-0)
QQSP, T-0]
CCB, T=0)
QC18, T=0)
ing set. A large segment of text match
Inference samples
4600
Inference samples
(usually 2 12 grams) is a telltale sign of memorization. We observe that memorization amplifies with a higher temperature (T' 0). We perform three inference runs (MP) at T 0.7.
Figure 2: LLAMA-3.1 8B Memorization in multiple predic- tion (MP) with T = 0.7, single prediction (SP) and base model (B) with T O. on confidential computing (CC) & quantum computing (QC) datasets.
In our 12- 2-gram experiment, the substring leakages from MP and SP above the base model are in CC, 9.9k and 3.7k, and in QC, 15.6k and 4.7k words. From this observation, we conclude that a single foundation model trained with all sensitive data is detrimental to maintaining information isolation and safety. Appx. A provides more details on the memorization experiment.
Related Works and Drawbacks. We discuss the drawbacks of several existing techniques.
3

---

RAG: RAG with access control [45] can allow or deny user access to docu- ment. However, retrieving from external sources (storage, the internet) is expensive. Additionally, retrieving set of large documents and putting them into context is detrimental to the performance of the L.Mss, as shown by [5], as L.MS often fail to extract relevant information from long con- text. Too many documents can also lead to inferior performance [46]. Longer context significantly increases inference time and memory consumption. Recent research shows that RAG can produce unsafe LLM responses compared to the base model answer [11], fine-tune scores higher in accuracy compared to RAGs [47], and RAGs are ineffective in multi-hop queries [6].
1.
2. Separate adapters for permission roles: Maintaining separate LoRAs for non non-overlapping, per- missioned datasets is feasible only if all users have a single permission role associated with a single dataset. Users of multiple permission zones, which reflect the organization's hierarchical structure, require fine-tuning new LoRAs with the merged datasets. However, this is prohibitively expensive as an exponential number of LoRAs (n permission zones lead to ((2)) LoRAs) is needed.
3. Training-free free LoRA Mixing: There are two primary existing methods to combine multiple LoRAs without requiring any trained gating or routing mechanism. One approach is to mix the outputs of the LoRAs by averaging their results. Given the up and down projection of n LoRAs as A = {AI,..... An) and B- = [BI,..., B.). The average output (Y) for input x from the mixture of LoRAs is: Y = E. B,A,T. Another method produces new LoRA with merged weights of the LoRAs as: Wmerged
B,AR
- B,A,, Y Wmergedd. In both cases, the inference accuracy diminishes severely [44, 48] with increasing number of LoRAs
To improve the above techniques, MoLE [18] uses trained gate to merge the entire output sequence from every MLP layer to combine tasks from every LoRA expert. The gate merges the output sequences based on the input encountered during the training phase. The gate parameter size increases linearly with the number of LoRAs and input sequence size. Like MoE [49], MoELoRA [19] utilizes sparse MoE activation with a trained gate. After every attention layer, the expert gate diverts a single token (unlike the sequence in MoLE) to an expert MLP. The routers/gates MoE models act as load balancers, trained jointly on all experts' data, which risks including confidential information, even if an expert is disabled due to the permission set. Alternatively, the routers can be trained for every possible permission set ( (O)2), bringing us back to the same challenge of training many LoRAs. This shows that MoE-style LoRA mixing techniques are not directly suitable for strict access control.
4. Training-hased LoRA Mixing:
Threat Model. AC-LORA assumes that the attacker can remotely access the LLM chatbot and send unlimited queries, aiming to maximize the retrieval of unauthorized information. They can inject arbitrary system commands or special tokens into queries and modify documents they can write, like personal records (corporate email, chat accounts, meeting recordings, etc.), and project data. We also assume the attacker cannot steal the identity to impersonate user.
Requirements. Given the above-mentioned problem space, we summarize the following require- ments for a secure corporate AI chatbot with strict information access control:
RR I: Strict access control policy. A user without the proper access rights cannot access restricted information or bypass the access control through means such as prompt injection.
RR 2: Arbitrary permission rules. The model can handle users' requests with new permission
rules never encountered before, while maintaining the access control policy.
RR 3: Efficient update. Information can be added, updated, or deleted with minimal effort.
RR 4: Efficiency. Ensuring that all of the above changes can be addressed without adding
significant number of parameters to the model to avoid a significant increase in inference latency.
3 AC-LORA: Permission-Aware LoRA Retrieval and Mixing
We present AC-LORA: end-to-end access control-aware LLM inference system. It integrates LoRA-based retrieval with dynamic LoRA mixing to efficiently support an exponential number of permission rules, while ensuring users can access all information to which they are authorized.
Main Observation. AC-LORA finetunes and maintains separate LoRAs for different permission zones. We assume a permission zone consists of projects or topics. We use three open-source datasets: Flan-v2 [21], RepLiQA [22] for text, and wikiart [23] for molti-modal. Figs. B.1 to B.3 show that topic embedded spaces are separable. Tasks such as anl i r1 and anli x2 in the Flan -2

---

(Fig. B.1) are variants of the same task, and their embeddings overlap. This observation is further reinforced by the pairwise cosine similarity score depicted in Figs. B.4 and B.5.
Isolated LoRA Fine-tuning and Knowledge Injection. Our memorization observation
(ef. Sec. 2) suggests that ensuring information isolation between different permissioned data zones requires fine-tuning on individual, isolated datasets with separate LoRAs (RQ. D). The base LLM contains public knowledge, while a specific LoRA(s) is required for the base model when user queries a re- stricted topic. We fine-tune 17 LoRAs using the RepLiQA [22] dataset, which contains human-evaluatedd mock news dataset that our base model, LL.MMA-3.1-8B, has never seen during its training. We use GEMMA-3-27B to grade the responses (on scale of [0, 5).
LoRA Rank (RapliQA Cybersecurity)
16
128
Grade Imprevement
0.75
0.50
0.25
100
200
400
500
600
700
Fire tuninginnn Steps
cf. Appx. D.3.2). Fig. 3(a) shows the grade of Cybersecurityy ty topic from the RepLiQA over different ranks (r) and finetune steps. We ob- serve that LoRAs can reliably inject domain- specific knowledge into the base model, assum- ing the base model has not been trained on a similar dataset. We observe that the most cru- cial factor is the dataset size; a smaller dataset can lead to model overfitting. Across ranks, Lo- RAS perform the best at a fine-tune step size of 2220. Fig. 3(b) shows summarized result for all RepLiQA datasets, and it shows that the finetuned model performs better in every sub- ject than the base model with an average of 0.959 grade improvement. Our observation aligns with existing works [13], which evaluate the effect of knowledge injection using LoRA.
(a) RepliiQA Cybersecurity topic
2 3
10
11 12
13
14 15
16 17
Grade Improvement
1.5
1.0
0.5
(h) RepLiQA: all topics (x-axis: Tab, C.2).
Figure 3: Knowledge injection of RepLiQA (split
0) dataset in LLAMA-3.1-8B.
Secure LoRA Retrieval and Similarity-based LoRA Merging. AC-LORA maintains two vector databases. A-RA-DoC EMBED contains the mapping between the LoRA and their corresponding fine-tuned document embeddings (chunked in 100 tokens). LORA-PERMISSION contains the per- mission information of the users. Each user (uniquely identified by their User-ID attribute) is asso- ciated with an n-dimennsional (n LoRAs) vector, where the vector elements denote deny (0) or access (1) to a specific LoRA. Given a tuple: [query, User-ID} from the user, AC-LORA A retrieves a set of candidate LoRAs based on the cosine similarity between the embeddings of the query and the training dataset. We denote the set of candidate LoRAs along with their cosine similarity scores as O. The User-ID retrieves the permission vector from LORA-PERMISSIAE
We denote the set of permissible LoRAs of the given user as P. AC-LORA retrieves and loads the set of relevant LoRAs f = O nP from L-RA-Dos EMBED. The similarity scores of the LoRAs in L. are also passed to the mix-gate after each MLP layer. Given the LoRAs in L = {In, L2, Iz,... Lb} where k S and their corresponding normalized similarity score (S1, S2,,,.... ..... Sk). such that E x== S, 1, then for a query Q the output for each LoRA each layer La E L is y4 = Q(A, (Iy's low rank components: A,, B,). The final output for each layer is then y W + ΣS S.h where W is the base model weight. Note that the mixing is completely training-rree e, i.e., the model owner does not need to retrain it for every possible combination of LoRAs (RQ. 2), and therefore enables faster and memory- inference (RQ. 4) due to the absence of gate parameters. Not loading any LoRAs outside the user's permission also ensures that sensitive data is leaked (RQ. 1).
Combining Knowledge from LoRAs Fig. shows that corporate AI chatbots should be able to combine information from different datasets. The example query "How many cores are in the CPU?" might have different answer depending, for example, on the platform or generation. There- fore, it is important to collect and combine the relevant information across different permission zones (given the user has the correct access rights). AC-LORA'S similarity- -based LoRA merging captures domain-specific knowledge across non-overlapping permission zones. This knowledge combination enables AC-LORA to avoid maintaining all possible permission zone LoRAs (RQ. 2).
5

---

Answer Hinting, The hint set are determined as H = -- of the LoRAs in H. are retrieved from LORA-DoC EMBED and given to the user as hint that there might be better answers given the queries and how to apply permission for them. This acts as valuable guidance for the users to apply for the correct permission to further refine their response. A curious/malicious user can gain knowledge of the possible existence of the information. Disabling the hint for sensitive LoRAs (c.g., specification of an upcoming product) will prevent AC-LORA from mentioning the existence of specific dataset to not-permitted users.
The metadata
Update Operations. Unlike the majority of existing works on LoRA mixing (see Tab. 1), AC- LORA is more flexible. To remove a dataset, the model owner must only remove one entry (0(1) operation) from both the LORA-Doc EMBED and CORA-PERMISSION databases. To modify an existing permission zone (addldeletc/moodify)) the model owner needs to fine-tune the specific LoRA with updated data, recompute the embedding of the fine-tune dataset, and update the LORA DDo EMBED vBctor-IB with the updated LoRA and the document embeddings. This does not affect the LoRA mixer, as it is only dependent on the individual cosine similarity score of the query and the document embedding vectors (RQ. 3). Updating the access control vector of the user only requires updating one entry in LORA-PERMISSION, which is also an (1) operation.
Summary of the Secure LoRA Retrieval and Merging. The step-by-step process of our proposed system AC-LORA, depicted in Fig. 4 are: ① The user passes their query and credential information to the system. First, the query goes to an embedding model to produce vector embedding. This embedding then passed to LORA-DOC EMBED for a top-k similarity search. @2 The top-k similar- ity search produces O: top-k LoRA along with their cosine similarity scores with the user query. 3 The user permission passes as the input to the LOR A PERMISSION that retrieves the set of permitted LoRAs. ④ The permitted LoRAs are then passed to the base model. 5) The outputs of the LoRAs are mixed with the same proportion of the similarity score of O. C The merged model outputs the main Response, which abides by the strict access control policy in the LORA-PERMISSION The Hint is derived from based on the non- non-permitted LoRAs with a higher similarity score.
?
AC-LORA scaling with number of documents. AC-LORA is independent of the number of relevant LoRAs. We can summarize the effects into two scenarios: (I) If all relevant documents are from the same permission domain (c.g., one LoRA), we choose to assign the average similarity score of the documents to that specific LoRA. (II) the documents are spread over many permission domains, we assign the average similarity score of the relevant document to each permission zone, corresponding to respective LoRA. In both cases, AC-LORA only calculates the similarity score for the permission domain if the user has access rights to it. Therefore, unlike RAG, AC-LORA does not suffer from lower performance (due to increased context size) when retrieving many relevant documents. However, in rare scenario, a query from very high-privilege user (e.g.. a CEO) may require many As simultaneously. In such a case, the GPU memory will be a bottleneck, and can be mitigated by using suitable parallelization strategy, such as pipeline parallelization.
Multi-modalities. AC-LORA extends beyond text-based models. Similar to the text, a tight access control mechanism is applied to such multi-modal scenarios. Existing work [36] utilizes mixing of LoRAs on stable diffusion to enhance the overall image quality when using LoRAs specialized on partial human features. AC-LORA uses similar mechanism to train isolated multi-modal LoRAs based on QWEN2- VT and stable diffusion model: STABLE- DIFFUSIONNVV-I4
4 AC-LORA Evaluation
This section describes AC-LORA's 's end-to-end evaluation. We run our experiments on two worksta- tion GPUs with 48GB GDDR6 VRAM. Additional details regarding the setup and implementation can be found in Appx. E. This section summarizes the key results of the AC-LORA evaluation. Further results are discussed in Appx. C.
4.1 Methodology
Datasets. In the following, we describe our setup for three different datasets:
1. RepLiQA: RepLiQA (split 0 cf. Fig. B.2) consists of several small artificial articles covering various topics, along with multiple question-answer pairs per article. We split it into an 80-20 training-test set, ensuring with stratification that each article is seen at least once in the training

---

Query embedding
3.5
User
-8
Query
User-ID ID
3
Embedding model
Transfommer
LoRA
LoRA
0.88
Permitted LoRAs
Attention
User- ID
Doc Embedding
restitnn Loot
22
-5
-7
0.5
-9
0.67
Forbidden
LoRAs
LoRA-Doc
embed
Penn-sernsss
Hint
Response
Figure 4: A high-level overview of AC-LORA showing LoRA-Dos EMBED and LORA-
PERMISSION vectorDBs. LoRAs are retrieved and mixed with the base model based on the user's query embedding and permission. Relevant and non-per onn-eermttted LoRAs can be returned as hint.
set. We finetune 17 LoRAs (with rank r INSTRUCT as the base model. As seen in Fig. 3(a), we keep the finetune step size ~ 200 to avoid overfitting, The training set comprises four data points per question: two with the document and two without, each pair once with a short and long answer. Including question-only pairs improved results as it aligns with the test set. To build the embedding database for AC-LORA, use the ALL-MNET- BASE-V2 [50] sentence transformer and split the training set for each LoRA into chunks of 100 tokens, adding the corresponding LoRA tag.
64), one for each topic, using LLAMA3.1-8 1-BB-
2. Flan: F2anV2 contains datasets of 10 task domains (cf. Tab. 2). We use the identical setup of [44], including the LoRAs shared with parameters (r 8 and = 16). We utilize their test set, which consists of data points per task. As the training set used for the different LoRAs was not shared, we constructed one based on the official Flan V2 dataset for the retriever. In particular, we take the first 30k (or fewer for smaller tasks) samples of each selected task as the training set and build the database as described above. As in LoRARetriver translation, ROUGE for the STRUCT-TO-TESS TASKS, and EXACT MATCH for the rest.
the BLEU score to evaluate the
3. WikiArts: We query QW to generate descriptions of the images from the wikiarts [23] dataset (see Appx. D) to construct the text-embedding for the retrieval. We then finetune STABLE- DIFFUSION-VI with the images to generate 27 LoRAs separated by the style attribute.
Combining Knowledge. Combining different LoRAs across different permission zones is impor- tant for AC-LORA(CE, Sec. 3). Although existing works [18, 20] show that combining LoRA can be used combine tasks from different LoRAs, e.g., translating from English to Spanish and then from Spanish to German, to answer queries for English to German. However, to our knowledge, no existing work shows that combining different LoRAs can increase model's information recall. Retrieving more than just single (best-fitting) LoRA and combining them introduces new infor- mation and increases the response quality. To demonstrate, we create dataset (CS-COMBI) from Cybersecurity it News category in RepLiQA. For each text, we ask reasoning model (DEEPSEEK- R1-32B) to extract the most (between 3 and 12) relevant facts. We then divide these facts randomly into two groups. From these two groups, we generate:
1. Context: We ask the model to write a text that exclusively includes the facts given, creating two new articles with parts of the information missing.
Combined QA Pairs: Taking one fact from the first group and one from the second, we ask the model to generate a question and answer pair that requires both facts to answer.
3. Single QA Pairs: Taking two facts from the same groups, we ask the model to generate a question
and answer pair that requires both facts to answer.
We built one test and two disjoint training sets (with and without context), each containing at least one question for each context in its corresponding group. We then fine-tune LLAMA3. 8B-INSTRUCT for three epochs and produce two As on these two training sets using different r (= a) E {4, 8, 16, 32, 64, 128). The prompts for extracting facts and additional details are in Appx. D.
7

---

RepliOA
RepligA
mandoro
8.8
LoRAc
zorelerance
Correct @ereccteetteeeee
1000 ww0 www.ooonooooonnn......
nt
cored book DA
Ussel boligg
123
23
15
(a)
(b)
Figure 5: (a) LoRA retrieval performance in RepLiqA and FlanV2 for different top-k. Left: % of queries per field for which the correct LoRA E the set of retrieved LoRAs. Right: # retrieved LoRAs. (b) Retrieval of Flan target domains (left) and corresponding retrieved domains (right).
Table 2: AC-LORA evaluation on F2anV2 dataset and comparison with other LoRA approaches. The baselines are extracted from LoRARetriever [44] for comparison. The best result is bold, while the second best is underlined.
Selection
Mixture
0OD>
MoE: Top1
Top3
(95%--CI (Km3, fetch_k=10))
Struct fmggl
64.0
61.3
50.1
49.4
45.9
55.9
50.4
45.6
46.8
47.9
61.7 (58.2 64.7)
Rogaz2
37.0
(33.--33)
Struct tex: frgge
57.0
54.5
43.9
43.6
40.3
49.5
44.0
39.8
40.7
41.3
(51.-511)))))3)
Translation
13.1
12.8
12.0
12.2
12.3
12.8
12.2
9.5
10.3
10.7
13.6 (1..7-15)))
Commonsense
62.5
55.5
46.0
$1.0
48.0
61.5
50.0
54.5
52,0
51.5
65.10 (58.5-72.0)
Sentiment
90.0
89.5
89.
79.0
78.5
89.5
90.5
70.0
75.0
74.5
90.0 (86.0-94))
Reading Comp
67.3
51.7
47.3
$1.3
47.3
48.7
(49.0 60.6)
Closel book
4010
48.5
40.5
(30.5-43)))
coadd.mmp.iicmmmm.... silammocsenee
31.0
(49.0-69.0)
Parsphrase
$3.0
42.0
36.0
(56
NLJ
72.3
70.0
60.6
51.4
53.8
67.9
64.3
50.3
49.6
48.3
68.3 (64.2-72.5)
IDD: secess any LoRA for sample, encompassing the LoRA specifie the sample's malk spacific during the phuse masked.
O0D: for each test sample, the LoRA associsted with
4.2 Main Results
Retriever Performance. We demonstrate that our retriever achieves high LoRA when given query. As depicted in Fig. 5(a), with increasing & (for the top-k
discuss two main results:
first,
accuracy in retrieving the correct
3.5
retrieved documents), the accuracy of having the correct LoRA in the set of retrieved Lo- RAs approaches one, while keeping the num- ber of retrieved LoRAs under for RepLiqQ and 5 for Flan. Fig. 5(b) confirms this by dis- playing the connection between a query from domain (left) and the retrieved LoRA domains (right) that answer the query (in Flan More detailed results and discussion are provided in Appx. C for both V2 and RepLiQA
Average Grode
LC-L
Single
(Perfect Selection)
2.5
2.0
1.5
13
14 15
16
IQA
(a) AC-LORA inference grades w.r.t single LoRAs.
Grade Improvement
0.2
0.0
Over lower performing LoRA
Over higher performing LoRA
provide AC- Inferece Results. We RepliiQ: Dur-
0.2
LORA's inference results. 1. ing inference, AC-LORA retrieves the rel- evant LoRAs (A: based on the cosine similarity with the query. Fig. 6(a) shows the AC-LoRA's mixed LoRA inference grades (judged by GEMMA-3-2 3-27B model), compared to the single finetuned LoRAs (perfect selection) specific for the given query.
48
32
64
128
LoRA
10) and mixes them
(b) Mixed LoRAs grade improvement CS-COMBL
Figure 6: AC-LORA evaluation on RepLiQA.
AC-LORA performs very close to the perfect selection on most topics, and in some, exceeds the perfect selection due to mixing with relevant LoRAs from other domains.

---

Dama331188 with IOR context
3amma3:1-889Whh333 conteRt
2
Liama 1-88 with 260 context
2-a
10
15
Figure 7: AC-LOORA multi-modal LoRA mixing on Wikiart dataset. (Prompt 8)
Number of active LORAS
Figure 8: AC-LORA's time-to-first-tkken latency
(TTFT) in 90 input-tokens on varying LoRAs.
Flan V2: Unlike RepLiQA, for Flan, we follow similar accuracy metrics as state-of-the-an Lo- RARetriever [44] to evaluate AC-LORA. Tab. 2 shows that AC-LORA matches or exceeds Lo- RARetriever's accuracy without requiring any optimization or training for LoRA mixing. More details are in Appx. C. F2anV2 focuses on different formats (tasks) rather than information. There- fore, AC-LORA also effectively isolates tasks.
3. modll: Fig. highlights AC-LORA multi-modal performance where images are generated using the prompt Prompt 8 with increasing top-k (k {1,2.3)) and using (in the retrieval order) the LoRAs of Ukiyo Impressionism, and Symbolism. Additional multi-modal results are in Appx. C.4.1.
AC-LORA effectiveness on overlapping knowledge groups. AC-LORA utilizes an off-the-shelf embedding model calculate the embedding space and the similarity score. For datasets with overlapping permission domains, a fine-tuned embedding model can ensure separation. Overlapping knowledge groups
detrimental AC-LORA. If one more overlapping datasets included in the user's permitted list, the specific LoRA(s) will not be used to answer the question. If one or more are permitted, they will be merged based on the similarity score. A1 in Flan V2 are very similar and have a large overlapping embedding space.
Combining Knowledge. We fine-tune (cf. 4.1) two LoRAs disjoint datasets. While single LoRA can answer some test questions, most require information from both. In Fig. 6(b), we illustrate, in blue, the average improvement in answers using both LoRAs compared to the lower- scoring LoRA, and in orange, the improvement over the higher-scoring LoRA. Although combining LoRAs improves performance over the weaker one, it generally performs worse than the higher- performing LoRA. The query still requires both LoRAs to answer, but not with the same weight, leading the LoRA with less information the subject to introduce noise. We observe 7.71 % of test queries show improvements over both, and depict a similar behavior to the one described in Fig. 1. Specific examples of such query- gesponses pairs are provided Appx. C.2.2.
Effeet on Inference Latency. Fig. shows the time to first token generation latency with an in- creasing number of active LoRAs. We construct prompt (90 input tokens long) such that with increasing k we can retrieve an increasing number of LoRAs (RepLiQA). As a comparison, we also provide vanilla LLAMA3. 1-8B's latency with 260, 5K, and 10K context sizes to visualize the effect of an equivalent RAG-like solution that retrieves and sets the entire relevant documents to the context. This shows AC-LORA is efficient and satisfies RQ,
Effect on memory. Loading the 8B base model with 17 LoRAs (r = 64) requires ~40GB of mem- ory without quantization. Similarly, Flan with 48 LoRAs (r 8) only slightly more and still fits on a single 48GB GPU. These requirements depend on the base model size, LoRA configuration (targeted modules, number of layers, and rank), and the prompt (and context) length. However, par- allelization techniques such as pipeline parallelization can allow AC-LORA to load more LoRAs. Additionally, in memory-constoe constraiied environment, could keep frequently used LoRAs on the GPU while loading less common ones on demand, trading off slight latency increase.
Discussions
Societal impacts. There are other scenarios where AC-LORA can enforce strict access control while maintaining high inference quality, besides corporate Al chatbots.

---

1. Safeguard users from unsafe content (e.g., illegal advice or violent images); One can isolate the training sets (of the said contents) and finetune separate LoRAs, which could, for example, only be accessed by authorized personnel (c.g., law enforcement..
2. Foundation models with IP data: As recent reports [51] indicate, unlawful usage of IP data in training may have severe legal implications; AC-LORA could allow using such data by keeping on licensed LoRAs and loading it with the base model for specific users. However, such use cases require further investigation, and the details of how such systems could be implemented using AC-LoRA are out of the scope this paper.
3. Bypass content filter: If the content filtering is implemented by model alignment [52], there are three scenarios in which they could be affected by AC-LORA:
No LoRAs are retrieved: If the user query does not match any LoRA, none will be retrieved, and the base model will answer the query. Therefore, the original alignment of the model remains unchanged.
LoRA(s) are retrieved: challenging to ensure model alignment without considering this during fine-tuning.
Specific LoRAs for bypassing filters: A set of LoRAs could be specifically fine-tuned to bypass the content filter (e.g., LoRAs used by law enforcement), making the bypass of the filter intentional.
However, the broader impact of such a system complex, requires further analysis, and is beyond the scope of this paper.
Limitations. We now discuss some of the limitations of AC-LORA.
1. General Limitation of LLMs and finetuning: LLMs perform well on some tasks but have notable shortcomings like hallucinations, context scaling issues, and limited reasoning abilities. Reasoning models help address some gaps, like multi-hop reasoning, but major issues persist. Importantly, AC-LORA relies LLMs' capacity learn and integrate new data during fine-tuning, making its design agnostic to future advancements in reasoning models, as it is applicable on top.
The hinting mechanism in Sec. 3 can introduce new attack vectors. Although useful in specific scenarios, it risks membership-inferene-lll attacks that could expose confidential data. We recommend using it cautiously, ideally with LoRAs on non-sensitive datasets.
2. Hinting:
3. Combining Knowledge: While we have presented a first experiment and dataset suggesting that different LoRAs can combine knowledge, a more extensive analysis required to better understand the extension (and limitation) of such capabilities.
Frequent Swapping of LoRAs: We assume that either all LoRAs on the devices, or LoRA swap- ping between inference rounds minimal, If not, the time first token increases significantly, as shown in Fig. C.9. However, we believe this assumption is reasonable. In the improbable worst-case scenario, where multiple (or all) LoRAs must loaded in each inference round, performance could, for example, be improved by reducing the value of k or optimizing the batching algorithm.
5. Multi-Modal Our evaluation other modalities serves proof of concept rather than com- prehensive analysis. thorough assessment would require more resources, including a new dataset and more robust evaluation methods, which are beyond the scope of this work (see Appx. C.4).
6 Conclusion
In this paper, we propose AC-LORA, a multi-modal, access-control aware LoRA serving system that requires no additional training for mixing responses by different LoRAs. AC-LORA is efficient, can retrieve and mix relevant LoRAs based on the user's query, while maintaining strict organization information access control policies. AC-LORA evaluation shows that deploying and providing high-quality responses is practical.
10

---

References
[1] Adam Marshall. Study: Using AT Chathots Can Expose Sensitive Business Data, 2025. [Ac- cessed 09-04-2025).
[2] Ernestas Naprys. Hello, this is your chatbot leaking: WotNot exposes 346K sensitive customer files, 2024. [Accessed 09-04-2025).
[3] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Retrieval- augmented generation for knowledge-intensive nlp tasks. Advances in neural information pro- cessing systems, 33:9459 997449999444 2020.
[4] Benjamin Reichman, Kartik Talamadupula, Toshish Jawale, and Larry Heck. Reading with intent. arXiv preprint wwxxv:2408.1/1888 2024.
[5] Ali Modarressi, Hanieh Deilamsalehy, Franck Deronncuur, Trung Bui, Ryan A Rossi, Se- unghyun Yoon, and Hinrich Schütze. Nolima: Long- evaluation beyond literal match- ing. arXiv preprint arXiv:2502.051677 2025.
[6] Yixuan Tang and Yi Yang. Multihop-rag:
Benchmarking retrieval- -uggrente generation for multi-hop queries. arXiv preprini urXiv:2401.1539), 2024.
[7] Mario Köppen. The curse of dimensionality, In 5th online world conference on soft computing in industrial applications (WSC5), volume 1, pages 4-8, 2000.
[8] Gelei Deng, Yi Liu, Kailong Wang, Yuekang Li, Tianwei Zhang, and Yang Liu. Pandora: Jailbreak gpts by retrieval augmented generation poisoning. arXiv preprint arXiv:2402.08416, 2024.
[9] Hyeonjeong Ha, Qiusi Zhan, Jeonghwan Kim, Dimitrios Bralios, Saikrishna Sanniboina, Nanyun Peng, Kai-wei Chang, Daniel Kang, and Heng Ji. Mm-poisonrag: Disrupting mul- timodal rag with local and global poisoning attacks. arXiv preprint 252.:2502..7832, 2025.
[10] Fatemeh Nazary, Yashar Deldjoo, and Tommaso di Noia. Poison-rag: Adversarial data poi- soning attacks on retrieval-augmess arXiv:2501.117599 2025.
generation in recommender systems. arXiv preprint
[11] Bang An, Shiyue Zhang, and Mark Dredze. RAG LLMs not safer: A safety analysis of retrieval-augmeentd generation for large language models. Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: LLong Papers), pages 5444-5474, Albuquerque, New Mexico, April 2025, Association for Computational Linguistics. ISBN 1799-8891176-88-66 URL https: /aapes//aalaaate laclaanhhooogyyyy org/ 2025 2025.naac1-long 281/
281/
[12] Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Roge- rio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2019.
[13] Nick Mecklenburg, Yiyou Lin, Xiaoxiao Li, Daniel Holstein, Leonardo Nunes, Sara Malvar, Bruno Silva, Ranveer Chandra, Vijay Aski, Pavan Kumar Reddy Yannam, Tolga Aktas, and Todd Hendry. Injecting new knowledge into large language models via supervised fine-tuning, 2024. URL htttps://axxiv.rrtt abs/ httpg/aabb224400221111 2404.00213.
[14] Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-Chie Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 202/ IEEE symposium on security and privacy (SP), pages 141-159. IEEE, 2021.
[15] Jiaheng Wei, Yanjun Zhang, Leo Yu Zhang, Ming Ding, Chao Chen, Kok-Leong Ong. Jun Zhang, and Yang Xiang- Memorization in deep learning: A survey, arXiv preprint arXiv:2406.085 03880, 2024.

---

[16] Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceed- ings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pages 954-959, 2020.
[17] Hongsheng Hu, Shuo Wang, Tian Dong, and Minhui Xue. Learn what you want unlearm: Unlearning inversion attacks against machine unlearning. In 2024 IEEE Symposium on Secu- rity and Privacy (SP), pages 3257-3275. IEEE, 2024.
[18] Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. In The Twelfth International Conference on Learning Representations, 2024.
[19] Tongxu Luo, Jiahe Lei, Fangyu Lei, Weihao Liu, Shizhu He, Jun Zhao, and Kang Liu. Moelora: Contrastive learning guided mixture of experts on parameter-efficient fine-tuning for large lan- guage models. arXiv preprint 2xXi::2402.128511, 2024.
[20] Ziyu Zhao, Leilei Gan, Guoyin Wang, Yuwei Hu, Tao Shen, Hongxia Yang, Kun Kuang, and Fei Wu. Retrieval-augmente mixture of lora experts for uploadable machine learning. arXiv preprint arXiv:2406. 16989, 2024.
[21] Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. International Conference on Machine Learning, pages 22631 1- 22648. PMLR, 2023.
[22] Joao Monteiro, Pierre-Andre Noel, Etienne Marcotte, Sai Rajeswar, Valentina Zantedeschi, David Vazquez, Nicolas Chapados, Christopher Pal, and Perouz Taslakian. Repliqa: A question-answering dataset for benchmarking ilms unseen reference content. arXiv preprint arXiv:2406.1 1, 2024.
[23] Wei Ren Tan, Chee Seng Chan, Hernan Aguirre, and Kiyoshi Tanaka. Improved artgan for conditional synthesis of natural image and artwork. IEEE Transactions on Image Processing, 28(1):394 409, 2019. doi: 10.1 1..1109/TIP.00 20t.:221122228 8.2866698. URL https://doi.org//lt 10.//..10// TIP
[24] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3-18. IEEE, 2017.
[25] Zhiyuan Yu, Xiaogeng Liu, Shumning Liang, Zach Cameron, Chaowei Xiao, and Ning Zhang. Don't listen to me: understanding and exploring jailbreak prompts of large language models. In 33rd USENIX Security Symposium (USENIX Security 24), pages 4675 4675-4692, 2024.
[26] Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang. Ruoxi Jia, and Weiyan Shi. How johnny can persuade 1lms to jailbreak them: Rethinking persuasion to challenge ai safety by human- izing 1lms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 14322-1432 14350, 2024.
[27] Maanak Gupta, CharanKumar Akiri, Kshitiz Aryal, Eli Parker, and Lopamudra Praharaj. From chatgpt to threatgpt: Impact of generative ai in cybersecurity and privacy. IEEE Access, 11: 80218-80245 2023.
[28] Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, and Yang Zhang. Com- prehensive assessment of jailbreak attacks against 1lms. arXiv preprint arXiv:2402.05663 2024.
[29] Bo Hui, Haolin Yuan, Neil Gong, Philippe Burlina, and Yinzhi Cao. Pleak: Prompt leaking attacks against large language model applications. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 3600-3614, 2024.
[30] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. 30th USENIX security symposium (USENIX Security 21), pages 2021.
12

---

[31] Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski. Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion mod- els. In 32nd USENIX Security Symposium (USENIX Security 23), pages 5253-5270, 2023.
[32] Renhong Huang, Jiarong Xu, Zhiming Yang. Xiang Si, Xin Jiang, Hanyang Yuan, Chunping Wang, and Yang Yang. Extracting training data from molecular pre-trained models. Advances in Neural Information Processing Systems, 37:97948- 979711 2024.
[33] Weichen Yu. Tianyu Pang, Qian Liu, Chao Du, Bingyi Kang, Yan Huang, Min Lin, and Shuicheng Yan. Bag of tricks for training data extraction from language models. In Inter- national Conference on Machine Learning, pages 40306-40320. PMLR, 2023.
[34] Elizabeth Montalbano. Microsoft Copilot Studio Exploit Leaks Sensitive Cloud Data. [Ac- cessed 09-04-2025).
[35] Ziyu Zhao, Yixiao Zhou, Didi Zhu, Tao Shen, Xuwu Wang, Jing Su, Kun Kuang, Zhongyu Wei, Fei Wu, and Yu Cheng. Each rank could be an expert: Single-ranked mixture of experts lora for multi-task learning, 2025. URL attps::/arrii..rg//aa//////
[36] Jie Zhu, Yixiong Chen, Mingyu Ding, Ping Luo, Leye Wang, and Jingdong Wang. Mole: Enhancing human-centric text-to-image diffusion via mixture of low-rank experts. NurlPS 2024, 2024.
[37] Shaoxiang Chen, Zequn Jie, and Lin Ma. Llava-mole: Sparse mixture of lora experts for mitigating data conflicts in instruction finetuning milmss. arXiv preprint 26X1v:2401.16166, 2024.
[38] Dengchun Li, Naizheng Wang, Zihao Zhang, Haoyang Yin, Lei Duan, Meng Xiao, Yuanchun Zhou, and Mingjie Tang. Dynmole: Boosting mixture of lora experts fine-tuning with hybrid routing mechanism. 2024.
[39] Bingshen Mu, Kun Wei, Qijie Shao, Yong Xu, and Lei Xie. Hdmole: Mixture of lora experts with hierarchical routing and dynamic thresholds for fine-tuning llm-based asr models. arXiv preprint arXiv:2409. 19878, 2024.
[40] Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwu Wang, Kun Kuang, and Fei Wu. Merging loras like playing lego: Pushing the modularity of lora extremes through rank-wise clustering. arXiv preprint mxXv:22409.6616) 2024.
[41] Jingfan Zhang, Yi Zhao, Dan Chen, Xing Tian, Huanran Zheng, and Wei Zhu. MiLoRA: Efficient mixture of low-rank adaptation for large language models fine-tuning. In Yaser Al- Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Findings of the Association for Com- putational Linguistics: EMNLP 2024, pages 17071-17084 Miami, Florida, USA, November 2024. Association for Computational Linguistics.
[42] Jingwei Xu, Junyu Lai, and Yunpeng Huang. Meteora: Multiple-tasks embedded Iora for large language models. arXiv preprint arXiv:2405.13033 2024.
[43] Mohammed Muqeeth, Haokun Liu, and Colin Raffel. Soft merging of experts with adaptive routing, Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https: //oppnreeieew.ee/fforrree wtt.fforuunnnd1119991111
Featured Certification.
[44] Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu. LoraRetriever: Input-aware LoRA retrieval and composition for mixed tasks in the wild. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Associ- ation for Computational Linguisticy: ACL 2024, pages 4447-4462, Bangkok, Thailand, Au- gust 2024, Association for Computational Linguistics. URL https: hatp:/aaaaaaaaa //taaalaaannooogoo.oooooo org/ fo24.findings-- ac1.26//.
[45] Pengcheng Zhou, Yinglun Feng, and Zhongliang Yang. Privacy-aware rag: Secure and isolated knowledge retrieval. arXiv preprint arXiv:2503.15548 2025.
13

---

[46] Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost the middle: How language models use long contexts, 2023. URL
[47] Aleksander Ficek, Jiaqi Zeng, and Oleksii Kuchaiev. Gpt retro: Exploring the intersection of retrieval and parameter-efficient fine-tuning. arXiv preprini urXiv:2407.04528. 2024.
[48] Zirui Song, Bin Yan, Yuhan Liu, Miao Fang, Mingzhe Li, Rui Yan, and Xiuying Chen. Inject- ing domain-specific knowledge into large language models: A comprehensive survey. arXiv preprint arXiv:2502.107000 2025.
[49] Yanqi Zhou, Tao Lei, Hanxiao Liu, Nan Du, Yanping Huang, Vincent Zhao, Andrew M Dai, Quoc V Le, James Laudon, et al. Mixture- cixture-of-experts with expert choice routing. Advances in Neural Information Processing Systems, 35:7103-7114 2022.
[50] HuggingFace. Sotteuuc-trrassoommeeraaaaapeees http://nnaccc.oosetttteeeee crraateeneetransffrrin htt.a//eeeeatttooooomaaamamaaaaa base--aaeee
huggingface. URL https:// //
[51] Michael Han Daniel Han and
Unsloth team. https//arsteetmica.com/eeeh batte
2023.
[52] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, al. Llama guard: Llm-based ouput- safeguard for human-ai conversations. arXiv preprint wrXiv:2322.06644 2023.
[53] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiing llama model. https:/ ////gttuuba cttauu.com/ttttuus labbssu-labsse haab/sttafoon ard.aaaca,, 2023.
[54] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Kather- ine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large language models, 2021. URL https:
[55] Avi Schwarzschild, Zhili Feng, Pratyush Maini, Zachary C. Lipton, and J. Zico Kolter. Re- thinking I1m memorization through the lens of adversarial compression, 2024. URL https: // wwwprvv.org/b///200000
[56] Peter Weiner. Linear pattern matching algorithms. 14th Annual Symposium on Switching and Automata Theory (swat 1973), pages 1-11, 1973. doi: 10. 100/SWAT.1933.13
[57] Hanwen Cheng- Find all the common substrings. https: http:///mdduumme cwm httmm/////00wee10////////// 2020. Accessed: 24.04.2025.
Find-all-tae 1all-the-come commonoooo----ooeeoooooo wwwssrsinssssotteeeeeoooo
(58) Zekun Li, Xianjun Yang, Kyuri Choi, Wanrong Zhu, Ryan Hsieh, HyeonJung Kim, Jin Hyuk Lim, Sungyoung Ji, Byungju Lee, Xifeng Yan, Linda Ruth Petzold, Stephen D. Wilson, Woosang Lim, and William Yang Wang. Mmsci: dataset for graduate-level multi-discipline multimodal scientific understanding, 2025. URL https: http://aaxxvv...aa///////// /0002//0777000....
[59] Steven Bird, Ewan Klein, and Edward Loper. Natural Language Processing with Python.
O' O'Reilly Media, Inc., 1st edition, 2009. ISBN 0596516495.
[60] Michael Han Daniel Han and Unsloth team. Unsloth, 2023. URL http://github.com unslothai/unslth..
[61] Patrick von Platen, Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj, Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. diffusers, 2022.
(62] LangChain. Ranrieval-Iangchainn
http:://ppttao..ancchai....////
aattlles/datt__comeeee 2025. Accessed: 30.04.2025.
14

---

[63] Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Pierre- Emmanuel Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
[64] Dun Zhang, Jiacheng Li, Ziyang Zeng, and Fulong Wang. Jasper and stella: distillation of sota embedding models, 2025. URL http::::/axxiv.or/abbb////////// 19048.
15

---

7 NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction accurately reflect the pa- per's contribution. AC-LORA ensure access-control-awant LoRA merging to ensure high- quality response without training any mixing/routing gate. Sec. 2 additionally provides motivation, the gap in the prior works, and an attacker model that concretely establishes the scope of this paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:[Yes]
Justification:
Sec. discusses limitation of AC-LORA detail.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The au- thors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the ap- proach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to ad- dress problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA].
Justification: This paper does not contain theoretical results. This paper proposes new access-control aware inference serving system that we call AC-LORA. We provide exper- imental proofs in Sec. 4.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims and/or conclu- sions of the paper (regardless of whether the code and data are provided or not)?
16

---

Answer:[Yess
Justification: We provide full details the supplemental data/code on reproducing the paper.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our supplemental material includes detailed instructions to reproduce the results of this paper.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All the evaluation details discussed in Sec. 4.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropri- ate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All the results (where it is applicable), such as Figs. 5(a), 6 and C.2 to C.4 and tabs. 2 and C.I provide statistical significance.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
[Yes]
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:///aaaecccc//aaa//cccccccaa
Answer:[Yes]
Justification: This paper fully adheres to the NeurlPS Code of Ethics
Broader Impacts
Question:
Does the paper diseuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:[Yes]
Justification: AC-LORA ensures strict information access control in corporate LLMs. These LLMs are trained on corporate data and are specifically designed to serve the em- ployees efficiently and get accurate and helpful information while maintaining strict infor- mation access control guidelines. In Sec. 5, we discuss how AC-LORA can filter unsafe responses from Al agents. However, also consider this specific use case be out of the scope of this paper.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
12. Justification: This paper does not include such artifacts.
13. Licenses for existing assets
17

---

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: Appx. F provides a list of the assets along with their licenses.
14. New Assets
Question: Are new assets introduced in the paper well documented and is the documenta- tion provided alongside the assets?
Answer: [Yes].
15. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the pa- per include the full text instructions given to participants and screenshots, applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: No experiments in this paper involve human subjects.
16. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA].
Justification: The paper does not require IRB approvals due to human subject involve- ment.
18

---

Appendix
Table of Contents
A LLM Memorization Evaluation
19
Dataset Preparation
A.2 Fine- Fine-Tuning
20
A.3 Inference ................................
20
A.4 Prediction Evaluation
20
B Dataset Embedding
21
Additional AC-L.RA Results
22
Flan ................................
22
C.2 RepLiQA
24
C.3 Mixing all LoRAs
27
C.4 Multi-Modal
28
C.5 Latency
30
Templates
30
D.1 Knowledge Combination
30
D.2 WikiArts
34
D.3 Grading
34
E Implementation detail
35
Evaluation Setup
................................
35
F.2 Finetuning
................................
35
E.3 LoRA Mixing
36
F List of Assets
36
A LLM Memorization Evaluation
We construct an experimental pipeline consisting of several stages: preparing the dataset, fine-tuning the base model, performing inference, and comparing the model's predictions against the training data. In the following sections, we provide a detailed description of each step in the pipeline.
A.1 Dataset Preparation
To create the datasets, we use the arXiv API to download research papers on two topics: confidential computing (CC) and quantum computing (QC). The specific URLs used retrieve the papers listed in Tab, A.I.
Topic
aiXiv API URL (docursentation: sttps://intt http://wwwwggeeeepp/////aaeeeeee bomlitusin))
Computing
idencial -1- comput ingtran hag.apuunnnngtaaaarraaeeeeeeo B80000000000 brtps: //oppor///tttt..// wwapprt..axivvee arxiv.arigg.. www.grrg/ap...... httg//g//aaaaa.uuurrrr/aa wtw.araall.uuauuurrrraaa...a. quattuu-tttattaaaaaaa... cosputinggaa resui tanbou
hetpa: batpa://eeppe //wppott.ttttteooeoeooeo arriv
Quantum Compuring
Table A.1: arXiv API URLs used for data retrieval
After downloading the papers, we filter out the ones published before 2024. Then, we convert the PDF files into plain text and split the resulting text into smaller chunks. Each chunk is then input into the LLAMA3.1 1.3..118B 1-8 model, accompanied by a prompt instructing it to generate five question-answer pairs based on the given text context:
19

---

USER: Write the questions and corresponding answers, and do not repeat the given context any final answer. Generate five questions and their corresponding answers from the given context. {context}
The final dataset comprises 15,459 question-answer pairs related to confidential computing and 15,466 question-ane -nestion-answer answer pairs related quantum computing. Finally, both datasets partitioned into training and test sets using an 80-20 split.
A.2 Fine- Mine-Tuning
The second step the experimental pipeline involves fine-tuning an LLM for text generation using LoRA. We follow this approach to assess the extent to which LoRA adapters memorize training data, iee, to evaluate how much of the original input retained within the adapted parameters during fine- tuning. We load the base model, LLAMA3. 1-8B, using 4-bit precision, nested (double) quantization, with normalized 4-bit quantization type and bfloat16 as the compute data type. We configure the LoRA adapters with attention dimension 16, scaling factor 64, and dropout probability 0.1.
For the training itself, we adopt the same hyperparameter configuration used in the Stanford Alpaca project [53]. due the similarity between our datasets and those used in Alpaca of size and structural format (instruction-ans wnsweer pairs). This also helps to rule out overfitting as contributing factor memorization. We also use the same prompt format as in Alpaca.
both in terms
Fine-tuning was performed using the same experimental setup described in Sec. 4.
A.3 Inference
In the inference phase, we combine the individual LoRA adapters with the base model and prompt the fine-tuned models using inputs from the test dataset. In real-world scenario, the model will likely encounter prompts that resemble those from the training set. Therefore, we use the test set, which shares similar context with the train set since they originate from the same source, but differ enough to simulate real-world conditions. Our objective is to evaluate the model's memorization after fine-tuning, without having direct access to the training set, while still using similar prompts. We repeat the experiment in three distinct variants.
In the first variant, we apply the greedy search decoding strategy to obtain a single deterministic prediction per query. These predictions are consistent and can always be reproduced.
In real-world scenarios, attackers can prompt model as often as they like. Consequently, repeated prompting can increase the likelihood of the model revealing memorized content, amplifying the risk of information leakage. We adopt a second experimental variant using the multinomial sampling decoding strategy reflect this threat model. Specifically, we modify the default LLAMA3. 1-......B. generation configuration by slightly increasing the temperature from 0.6 to 0.7, and setting the top-p parameter 1.0 instead of 0.9. This approach enforces more diverse predictions. We prompt the model three times, generating multiple prediction candidates.
In the third variant, we apply the greedy search decoding strategy again, but this time using only the base LLAMA3.1 1-8B model, without combining it with any LoRA adapters. This helps isolate the contribution of the LoRA adapters, allowing us assess how much newly introduced knowledge is memorized by the adapters versus what the base model retains.
A.4 Prediction Evaluation
The final stage involves a quantitative measurement of LLM memorization by comparing each gen- erated prediction from the prediction set P' against each entry in the corresponding model's training dataset S. Unlike previous work that relies on concepts such as eidetic memory [54] and adversarial compression [55] to define and measure LLM memorization, tion using simple string comparison techniques directly.
work aims to quantify memoriza-
We compare each prediction p E P against each question-answer pair E S from the corresponding model's 's training dataset by searching for all the common substrings between y and s. Importantly, treat p and as sequences of words (rather than characters), where tokens are defined by whites-
20

---

QUESTION:
What the relationahip between the entropy and trace of the logarithm of the map
entropy pain map undera completely positive map is related to the trace of the logerithm of the map by dS(F) trif log
QUESTION:
How entropy gain logarithm of quantum channel?
PREDICTION: entropy gain directly related the trace of the logarithm of the map which is meesure the emount epplication of Specifically the entropy gain be expressed es the difference between the logerithnic and the logarithmic which is given by the trace of logarithm of the map This relationship between entropy gain trace of the lagarithm of the map Jf a key result in the context quantum channels provides useful meesure the amount of information gained through the application the channel.
gained through
QUESTION:
meaning
HX represente the entropy of random variable which Information that contalins
ANSWER:
after the spplication of T
the
What the distance measure used the quantur kernel
QUESTION:
ANSWER:
The distancc measure used in the quantum fidelity
encuding states which is given by the traze the product of the two encoding states
Figure A.1: An example of prediction generated by the quantum computing LoRA using the greedy search decoding strategy (left) and training set entries whose segments are contained within the prediction (right). The highlighted text indicates matching sequences with length greater than or equal to words. The prediction an absolute memorization score 43 and a relative score of 0.387.
pace separation. We further enforce minimum substring length measured in consecutive words, to ensure that only meaningful overlaps are considered the analysis. We repeat the experiment for {8, 12, 15, 18).
For this purpose, we generalize the Longest Common Substring (LCS) Suffix Tree algorithm [56], to search not only for the longest common substring, but also for all common substrings between two strings [57]. We additionally adapt the algorithm include only the substrings of length greater than or equal to words.
This process results in of [S] overlapping intervals for each prediction p, where each range cor- responds to the overlap with specific training example s. To quantify memorization, we aggregate the intervals across all S compute a global overlapping interval the union of all sequences within that are directly and exactly memorized from the training set. We then compute two mem- orization scores for each prediction: words within the global interval, and relative score, calculated as the ratio of captured words to the total number of words in the prediction. Fig. A.1 shows an example of prediction alongside training set entries whose segments are memorized verbatim. In the case of multinomial sampling, where we generate three predictions per test query, we additionally aggregate the global intervals from all three predictions. We avoid double-counting when merging the intervals, such when captured substring from one prediction is partially or entirely contained within longer overlapping substring from another. We then compute the cumulative absolute score based the total number of memorized words within the merged interval.
absolute score, defined as the total number of memorized
It is important to note that finding all common substrings of two strings is prohibitively expensive, with time complexity of O(m + n). where m and are the lengths of the two strings. Due the number of experiments, the size of the training and test datasets, and the lengths of the generated predictions, the comparison stage required significant computational resources. The whole process took over 15 days to complete when executed in parallel across seven nodes.
B Dataset Embedding
Figs. B.I to B.3 show the embedding space for F2an RepliQA, and wikiart dataset respectively. As mentioned previously, ALL--NNET-RASF-V- [50] was used to generate the dataset embeddings.
21

---

-20
-40
transiate
yelp, relviws roviews
reviuws
translate
openbookqu
pows wiki
sil_
multir
sqpend sd_
w _entr
translatee_ffee
Jransiste_roen
sst2
copa
glue mipp
plea
wmil6 transiate csen
mni matched
wmit6, translatee nuen
record
crawl onne
sentiment140
bool_
are challinee
nutural_uestoons
antirs
gep
c2c, nig
definite proneun reso
wab wab_nig.on mig.cn
story_cloze cloze
wSc
coimas
hillaswag
Figure B.I: Embedding space of the Flan-v2 [21] Dataset.
-40
20
40
Local
Locai_Arts, AAtss, _rttssssddd,a.... Culbure Culture
Local Politics, Local_Polities and Gevernance
Neighborhood Stories
Local Health and Wellness Regional Folidore Local Fronomy _ad____aieeee Incident Report
Policies News,
Small Medium
Technology, Innovation Local,
Regional Cuisine and Recipes
Market
Local Sports and Activities
Local, Envronmental Issues
Local Education, Education_Syttems
Cybersecurity_
Figure B.2: Embedding space of the RepliQA [22] Dataset.
Appx. B and fig. B.5 additionally show the pairwise cosine similarity of these three datasets. To cal- culate the cosine similarities between each pair of topics in dataset, we first calculate the centroid of all document embeddings topic. This embedding centroid is then used as the representative embedding for the specific topic and is used to derive the pair-wise cosine similarity.
C Additional AC-LORA Results
In the following subsection, we will provide more detailed results from AC-LORA evaluation.
C.1
Flan
In Tab. C.1 we showcase the full comparison to [44] on Flan-v2. Similarly to the briefer version, we can see that AC-LORA matches or outperforms other methods in most tasks. Unlike the results on RepLiQA, if the wrong LoRA is retrieved, the output format will (in some cases, drastically) change and thus receive a worse exact match score, even if the content of the answer correct. Given the nature of AC-LORA, it is unsurprising that its performance is poorer tasks evaluated by exact
22

---

60
20
20
-40
20
40
Realsm
Abstract Expressioniss niam
Synthetic, Cubism
Contemporary. Realism
Art, Nouvea
Barocue
Naive Primitvism
Mannerism Renaissance
Pop,
Ukiya_
Pointilism
Impressioniom
Cubism
AnalyBial, Cublim Post, Impressioniam
Espressiorism
Mnimmaism
Highh Reraissance
Renaissance Action painting
Romanticism
Pauvism
Field Painting
Figure B.3: Embedding space of the generated Wikiart prompts.
Camari tigltss
iieo Seiis
Settl Eaer encerm toam
you
Figure B.4: All pair cosine similarities of Flan-v2 [21] and RepliQA [22] datasets.
Figure B.5: All pair cosine similarities of generated prompts from Wikiart [23] dataset.
23

---

(a) Domain grouped
(b) Tasked grouped The thickness shows the frequency of the connection.
Figure C.1: Actual retrieved LoRAs for given domain or task.
match. This is due to its vulnerability to additional retrieved LoRAs, where retrieving the correct LoRA and irrelevant ones can sufficiently lower the exact match score. In general, AC-LORA consistently retrieves at least one LoRA that belongs to the same domain, though this is not always exclusive for certain domains. As illustrated in Fig. C.I, for queries originating from domains such as CLOSED BLOSED_BOOK QA and COMMONSENSE, AC-LORA occasionally retrieves LoRAs from other domains. This is intuitive, given that these domains have less distinct boundaries compared other tasks. If the primary goal of AC-LORA were to retrieve LoRAs based on the task, rather than knowledge, performance in these cases could be improved by emphasizing the requested format more during retrieval.
One can notice similar behavior in Fig. C.2 and Fig. C.3, where show extensive version of Fig. 5. The plots show the results with increasing threshold (horizontally) and fetch_k parameter (vertically). The threshold indicates that LoRAs with a retrieved average similarity score lower than the given threshold are disregarded. The threshold does not significantly affect the results, except for thresholds higher than 0.5, where we start to retrieve fewer, if any, LoRAs, and thus also not the correct one. On the other hand, fetching more documents has a minimal impact on the results. While the plots per task (instead of per domain) are bit noisier, they show similar trend. The worst-performing one is mnli mismatched, which is not surprising, as we have not included it in our database (since the entire idea of this task is see how performs out of distribution to the matched ones), and therefore cannot be retrieved. In the event that we consider mnli_matched as the correct LoRA this case, we achieve, for example, a 92% accuracy for the mnli mismatched queries with hyperparameters k = 10, fetch.k= 200, and threshold 0.0. Similarly, the task arc_eat _easy and are challenge, or r33,,,),, whose accuracy increases when considering any of the options as correct.
C.2 RepLiQA
In Fig. C.4 we present a more detailed version of the plot presented in Fig. The plots show the results with increasing threshold (horizontally) and increasing fetch k parameter (vertically). Simi- larly to Flan, one can see that the fetch parameter does not appear to affect the results significantly. At the same time, once we increase the threshold to 0.5, retrieval results degrade significantly as this leads AC-LORA to disregard often all retrieved LoRA. We show the actual retrieved LoRAs per domain and their frequency in Fig. C.5.
C.2.1 Hinting
To evaluate the Hinting mechanism described Sec. 3 we run AC-LORA once by masking (not permitting) the LoRA of the corresponding topic and once with all permissions if there has been

---

Table C.I: Full comparison with LoRARetriver.
Task/Liama275
Perfect Selection
Selection
Fusion
Mistlree
MoF. Top1
Mol: Top3
MoF.
SME-
Adapter Soup
LoRa Hub
AC-LORA (95%--C1
IID
00D
0OD
IID
00D
(k=3, fetch
Struct to Text
WebNL WebeLGRAage
71.2
67.0
53.9
49.4
45.4
57.8
53.9
45.1
47.6
49.1
51.1
3.9
32,5
69,8 (65.7-75.2)
50.6
44.5
25.9
24.1
33.5
29.4
22.6
25.8
26.1
27.9
0.9
17.3
48.4
WebNLe
64.4
60.9
49.1
45.5
41.0
52.3
49.6
40.0
41.9
43.3
45.4
3.9
31.1
61.9 (56 (6.6667.6)
71.7
67.9
58.4
534
55.4
56.3
3.3
40.0
72.5
DART
49.1
45.8
34.9
52.3
30.6
36.6
35.4
30.3
31.0
30.8
33.0
1.3
20.1
49.1 (42 (42-4-55.0)
DART
64.6
611
47.9
56.3
50.8
$4.8
3.3
6400 (58.7-69..)
E2ENLGMMaS1
06.1
65.8
59.3
62.2
57.2
66.0
58.7
$2.9
54.0
55.3
53.2
4.2
50.1
66.1 (62 7-70.3)
E2ENLGMeasd
40.0
39.4
34.1
34.7
32.0
38.8
32.1
26.9
27.6
28.8
27.5
2.4
26,3
39.6 (35 (-3.5
E2ENLG
56.7
55.7
50.2
52.7
49.1
56.9
49.0
45.1
45.0
47.0
45.1
4.2
42.2
$6.4 5-0.6)
CommonGen
46.9
44.7
29.0
29.9
27.7
36.5
29.0
29.0
29.3
30.1
27.6
6.6
19.8
38.3 (30.9-44.0)
CommonGGen 2rag-2
18.8
18.3
7.3
9.9
7.2
11.1
8.6
7.7
7.1
9.3
8.4
0.0
6.9
11.1 (6. 16.2)
42.5
40.5
24.0
25.8
23.3
32.7
24.88
24.4
25.1
26.3
24.3
6.6
18.0
34.8 (28.2, 40.4)
Trunslation
Paractawl lenes
21.2
22.3
22.8
22.1
18.0
16.4
26,3 (19.0-33.6)
16-tren
3.2
3.1
3.7
2.6
3.5
3.2
2.0
3.4 (0.2-8.3)
WMT 16-run
10.8
10.4
9.8
9.2
9.3
11.0
10.8
6.2
7.8
8.3
7.3
0.0
4.8
11.3 (6.0- (2.2)
WMT 16-deon
18.9
18.7
20.3
17.9
18.8
18.8
18:7
11.6
14.0
14.7
16.6
1.1
11.4
17.9 (12. 1-24.5)
6.5
6.5
7.0
7.2
7.1
7.3
7.8
6.2
6.2
6.1
6.5
0.7
4.3
2.2 (2.6-12.9)
WMT 16-men
13.9
14.0
12.3
12.8
13.3
13.1
12.2
9.8
10.7
10.1
10.3
0.3
8.0
20.720
WVT 14-enf
T6.1
18.0
15.9
17.9 (12.3-211
WMT 16-esen
10.7
2.4
7.0
6.2
8.3
4.7
6.3
6.1
(5.4-13.)
Commonseuse
StoryClooe
72.0
62.0
42.0
72.0
68.0
$1.0
58.0
74.0
70.0
70.0
68.0
62.0
48.0
86,0 (76 (7600-96.0)
PIQA
46.0
46.0
32.0
34.0
36.0
38.0
34.0
40.0
38.0
38.0
36.0
38.0
0.0
44.0 (31 9-58.0)
COPA
36.0
74.0
68.0
78.0
70.0
80.0
68.0
72.0
70,0
72.0
70.0
56.0
22.0
80.0 (72.0-92. (2.00)
HellaSwag
40.0
18.0
44.0
32.0
0.0
50,0 (6600-6..))
Sentiment
SST-2
98.0
98.0
96.0
74.0
78.0
96.0
94.0
56.0
68.0
66.0
66.0
74.0
0.0
98.0 (944 100.0)
Yelp
98.0
94.0
94.0
96.0
96.0
98.0
9%.0
66.0
90.0
86.0
84.0
80.0
0.0
98.0 (90 100.0>
IMDB
96.0
96.0
96.0
92.0
82.0
96.0
96.0
76.0
80.0
80.0
$4.0
80.0
0.0
96.0 (90.0- 100.0)
68.0
70.0
70.0
54.0
58.0
68.0
74,0
62.0
62.0
66.0
62.0
60.0
2.0
68.0 (5600-8..0)
READING Comp.
MulcRC
$4.0
52.0
6.0
60.0 (46.0-
SQuADv2
62.0
56.0
12.0
30.0
20.0
22.0
16.0
24.0
24.0
26.0
22.0
16.0
0.0
34.0 (24.0-50 50.00)
SQuADv]
68.0
66.0
58.0
61.0
61.0
62.0
58.0
58.0
70.0
66.0
66.0
54.0
4.0
56.0 (42 0- 70.0)
OBQA
82.0
68.0
58.0
64.0
60.0
78.0
66.0
62.0
64.0
66.0
60.0
40.0
0.0
20.0 (56 0-0.0-
Bool))
$4.0
60.0
60.0
68.0
30.0
$0.0
76.0
74.0
68.0
76.0
70.0
72.0
6.0
84.0 (74.0-94.0)
8.0
12.0
180
10.0
0.0
28.0 (1400-38.0)
CLOOSEDBOOK QA
NQ
18.0
16.0
10.0
16.0
14.0
16.0
10.0
12.0
12.0
12.0
4.0
12.0
0.0
10.0 (2.00 10.00-18..00
ARC-c
50.0
56.0
70.0
54.0
56.0
66.0
82.0
58.0
58.0
60.0
58.0
48.0
0.0
64.0 (46 (-7.0--7440)
ARC-c
46.0
42.0
46.0
34.0
34.0
50.0
46.0
46.0
42.0
42.0
42.0
24.0
0.0
38 (24 0-48.
TriviaQu
66.0
46.0
46.0
60.0
46.0
48.0
56.0
46.0
42.0
46.0
24.0
42.0
4.0
41.0 (26 0 54.0)
CORKFERENCE
50.0
60.0
64.00
2.0
54.0
WSC
50.0
50.0
42.0
38.0
46.0
55.0
42.0
53.0
58.0
52.0
54.0
40.0
0.0
54.0 (40.0-6888
READ. COMP. W/ COMMONNENNE
Cosmossaa
68.0
63.0
34.0
46.0
32.0
50.0
46.0
44.0
46.0
44.0
38.0
14.0
6.0
72.0 (S8.0-824 (2.00)
recurd
70.0
70.0
26.0
24.0
6.0
42.0
34.0
18.0
12.0
14.0
8.0
14.0
0.0
51.0
(32.0-62.0)
PARAPHRASK
Pows
64.0
42.0
56.0
56.0
50.0
2.0
78.0 0-6.0-88.10
QOP
74.0
74.0
68.0
66.0
60.0
80.0
58.0
50.0
40.0
36.0
28.0
54.0
0.0
74.0 (62 86.0)
MRPC
60.0
58.0
58.0
60.0
62.0
60.0
58.0
42.0
44.0
40.0
42.0
60.0
2.0
60.0 (44 0-74.1)
STSB
38.0
36.0
16.0
12.0
12.0
30.0
20.0
20.0
20.0
23.0
14.0
12.0
0.0
40,0 (26.0. (-6.0-54.0
NLI
57.8
$6.7
64.4
13.3
8 0-222-886.6
60.0
56.0
56.0
52.0
0.0
62.0 (50.0-
ANLIsl
50.0
50.0
50.0
40.0
42.0
40.0
42.0
40.0
40.0
36.0
38.0
38.0
24.0
44.0 (30 60.0)
ANLI-2
46.0
46.0
46.0
52.0
36.0
46.0
46.0
40.0
36.0
38.0
32.0
46.0
20.0
12.0 (26 0- 51.0)
ANLI-3
46.0
42.0
38.0
38.0
40.0
44.0
50.0
28.0
32.0
34.0
38.0
40.0
24.0
46.0 (26 0-5..0
MNLI
$8.0
$4.0
85.0
62.0
66.0
80.0
88.0
48.0
54.0
50.0
56.0
76.0
0.0
78.0 (66.0-92) 066.0-99..0
MNLI-mI
88.0
2.0
90.0 (54.0)
SNLI
$4.0
56.0
580
90.0
$4.0
52.0
0.0
(90.0- 100.0)
ONLI
94.0
94.0
26.0
46.0
48.0
74.0
35.0
56.0
56.0
54.0
60.0
70.0
0.0
78.0 (64 0 88.0)
RTF
$2.0
62.0
73.0
54.0
58.0
70.0
76.0
64.0
58.0
56.0
64.0
80.0
22.0
74.0 (62 0)..6)
Table C.2: Mapping index for Figs. 3(b) and 6(b) to RepLiQA domain. 2
Index
5
8
Domsin
Regional Folk and Mytis
Local Health and Wellness
Lavall Foviron- mentall Ivnes
Neighborhood Storics
Lavall Sports and Activilies
Local Technology and Inaovation
Locil Arts and Cultae
Cyhersecnity News
Index
10
15
16
Domaln
Local Politiss and Governance
Malium Emerprises
Small
Local News
Local Eoonomy and Market
Local Educat- ion Systems
News Sturies
Company Policies
Regional Cuisine and Recipes
Lackdent
Repor
25

---

Fank2 Domains
strut
matting comp deast
rnstoom COnTrOSSeer prsphras
12 threshold
20 01
20
threshald
Neth Bhreshdd
fetchk 50 threshlld 0.0
fetch_k 50 threshaid 01
fesch_ka 50 threshold 02
Serch.k = 50 threshhad 6.3
Sekdhik j 50 thresheld 0.5
7.44
budded corveny
feichuk
feint
2.8
Iech x 150 Ltteshold 0.
Tetch k 150 threhhldd 51
leichkk 150 (hnechlld 02
teich 150 Ltheshold 0.3
Ietch k- 150 Uhreshold 25
7.5-
Ieateed
2.2
2.8
0.2
115
1:3
10
Figure C.2: Flan- V2 task retrieval grouped per domain.
hint the masked run. In Fig. C.6 we present the results. As one can see, on average, the hinted LoRA improve the answer given by AC-LORA.
C.2.2 Combining Knowledge
QUESTION: What are two significant contributions Dr. Chase has made to Al-powered cyberse- curity?
REFERENCE ANSWER: October 15, 2023, Dr. Chase presented her groundbreaking work on Al-powered cybersecurity at the Chicago Cyber Security Summit. Additionally, in Novem- ber 2023, her Al-powered security system detected an attempted zero-day vulnerability exploit against financial institution in Chicago before it could cause significant damage.
LORAI: Dr. Chase has developed an Al-powered cybersecurity platform that uses machine learning algorithms detect and respond to eyber threats in real-time. Additionally, she has introduced a new cybersecurity curriculum at the university level, which includes topics such as AI ethics and security protocols.
26

---

Flar12 Tasks
DEALA
recodd
white, Lrassive,
sclariyy FestONs
drpp
pana
quitiion
guc ogp
wHite Dorsloz
guccmpp
WhiLe,
story_tace
WHiLe, torskone
promun
fetchkk 50) threhhldd 0.0
latch k 50 Bhreshold - a.
Tetch k- 50 Phrestold 02
Tecch k- 50 thesshold 0.3
latch x 50 Bhreahlld - DS
2.
5.5
thrsshald
fetsh
thrishlld
E30ee Cormectty
12tkk 150 tireshlld 0.8
totchkk 150) threshald - 81
fetchkk 150 (threshold - 02
fotorkk 150 tiveshlld 0.3
fotch.k 150 threshald
05
tatt_k threshold 0.0
200 threshold
200 jthresheld
200 threshold
threshald
2.6
Figure C.3: Flan- 12 task retrieval.
LoRA2: Dr. Chase has contributed to the development of Al Al-powered cybersecurity by creating framework that integrates machine learning and human expertise detect and respond to cyber threats. Additionally, she has demonstrated the effectiveness of Al in detecting and mitigating cyber attacks, showcasing its potential to revolutionize the field. LORA12. 12: Dr. Chase has developed an Al system capable of detecting and neutralizing cyber threats real-time, and she has also created an Al-powered cybersecurity platform that uses machine learning to predict and prevent cyber attacks.
C.3 Mixing all LoRAs
The primary advantage of using only the relevant LoRAs instead of just mixing all permissible ones the reduction of inference latency. However, we also observed an average grade improvement of 0.066 (95% interval: (-0.00748, 0.138) over mixing all possible LoRAS for the RepliQA dataset.
27

---

Repa2A Oatamet
Locauho aos
Lock, Diwtormennn
poiiee
Look Eduotton, Sysvers
fetch.k threshoodde 9.0
fescn_k threshld 6.1
Tetchuk ctressol
fetdh thresnald 03
ferch (thesheld -
1.0
100000 C0mOO
9.6
Nechukk
fatdukk
Roch
fece, (tresseld
00300
fetchkk 100 threshold 00
ferhuk 100 thresheld 0.l.
Teocn x 100 threshid 9.2
Reech J 102 threshold 03
fenchuk 100 threstats 05
10
70.0
100000
0.64
04
Guch.k 200 thresncdd 0,
fowhkk 200 Uhrishold - 9.1
fe 200 Urrcho d 02
fetch k 200 thraahold 0.
fezsh = 200 tives hold 0.
0.8
830000
0.6
0.04
123
123
20
335
123
20
123
25
Figure C.4: RepliQA retrieval.
This effect could come from several factors; we suspect it's primarily due to the relatively high rank used with relatively small dataset. We expect that the AC-LoRA improvement will increase further as the number of LoRAs increases. Flanv2, however, behaves differently. Since Flanv2 primarily finetuned influence output format, mixing all Flan LoRAs significantly degrades performance. This is evident in Tab, C.3, which shows the improvement of AC-LORA over the mixing of all Flan LoRAs.
C.4 Multi-Modal
In the following we present some additional information about our setup for the stable diffusion experiments and show some additional results. We then briefly provide some information regarding the capabilities to use AC-LORA also with other modalities, such as text-image to text.
In general, we view the results presented in this section more of a proof of concept rather than a comprehensive evaluation. more thorough analysis would require significantly more resources to accurately assess the capabilities of the base model and the specific contributions made by the finetuning. To ensure fair and precise evaluation, one would need to create new dataset (to guarantee that the base model has not previously been trained on it). However, even with this step,
28

---

LoCAl Envirummentt besues
L Loraa Teessrolog eorrog andd Jn ceeeeraaaa uoe
Looal Sportd and Adiiiiid
Aroidant Repon
Regienal Fottory taftt
Newn, Stonis
Small and Medium Entorpiace
Neighborhood,
News
Ragional, Caisime and, Rocies
Local Polliice gamt Govemante
Lccil Novs
Look Edtccation Systeme
Local Cammryy and Martost
Company Proicies
Looal Health and Wethesss
Are and Cutture
Figure C.5: Actual retrieved LoRAs for given domain. The thickness shows the frequency of connection.
Table C.3:
Improvement of AC-LoRA over just mixing all Flan V2 LoRAs
Task
Improvement
Struct to text (rouge-1)
34.64 (31.95, 37.19)
Struct to text (rouge-2)
26.94 (23.97, 30.17) 30.64 (28.02, 33.64)
Struct to text (rouge-l)
Translation
9.76 (8.22, 11.43)
Commonsense
65.50 (58.50, 72.50)
Sentiment
90.00 (85.50, 94.00)
Reading Comprehension
54.67 (49.32, 60.00)
Closed book QA
36.50 (30.00, 43.00)
Coreference
54.00 (44.00, 64.00)
Read.com comp. w/commonsense
59.00 (49.97, 68.00)
Paraphrase
63.00 (56.00, 69.50)
NLI
68.28 (64.44, 72.32)
29

---

3.00
2.75
2.50
Average Grade
2.25
2.00
1.75
1.50
masked hint
1.25
2
10
13
16
RepliQA Dataset
Figure C.6: RepliQA evaluation with the expected LoRA masked and once with it if part of the Hint.
evaluating the model would remain challenging, as images are inherently more difficult to grade than text. Given these considerations, we believe such an in-depth evaluation is beyond the scope of this work.
C.4.1 Stable Diffusion
We trained different LoRA models on the different styles in (WikiArts(23)). For this, we asked QWEN2-VL VL to generate generation prompt given the image, the style, and the artist. From these prompts and the images, we fine-tune DTABBE-DIFFUSION-V and alpha 16, learning rate le-04, and utilizing the diffusers Huggingface library. We then use the generated prompts to build our embeddings for the retriever.
each of the 27 styles, using rank
Fig. C.7 shows six example AC-LORA image generation along with their generation prompts and the corresponding retrieved (and mixed) LoRAs
C.4.2 Text-Image Text (Qwen2-VL.)
We evaluate AC-LORA also on text-image to text models. We finetune 10 LoRAs using QWEN2- VL-7B-INSTRUCT. Starting from the MMSci dataset [58], create 10 smaller datasets (5k data points each) as shown in Tab. C.4. We show the retrieval results in Fig. C.8. We describe how we embed the text and image for the retrieval mechanism in Appx.
C.5 Latency
In Sec. we provide our evaluation result of the AC-LORA's time to first token generation with inereasing number of active LoRAs (i.e., isolated permission zones). However, this assumes that the LoRAs and the base model are already loaded into the device's memory (such as the GPU). This is valid assumption, switching the model frequently can adversely affect token generation, specifically the latency to first token generation. We evaluate the worst-case scenario, where every user query requires the LoRAs As to be loaded into the device memory from seratch. Fig. C.9
D Templates
D.1 Knowledge Combination
As described Sec. 4.1 we built our own dataset starting from RepLiQA split to showcase the capabilities of combining LoRAs to combine knowledge. Starting from all the documents of the CYBERSFCURITY NEWS category, we ask deepseek-rl ::bbb to extract the most relevant facts using the following prompt:
30

---

(a) "A dog in style of Da Vinci" LoRAs: 'Early Re- naissance', "Mannerism Late Renaissance', 'North- em Renaissance'
(b) "draw dog by Picasso" LoRAs: 'Symbolism', 'Expressionism', 'Cubism'
(d) "please generate rococo dog" LoRAs: 'Ro- coco
"a dog in pop art style" LoRAs: 'Pop Art'
(e) "Please generate image of dog if gogh would have drawn it" LoRAs: 'Realism', "oos Im- pressionism'
(1) "a dog by Schiele". LoRAs: 'Rococo', 'Pop Art', 'Romanticism'
Figure C.7: AC-LORA
-4
31

---

Table C.4: Composition of the different training-sets starting from the MMSci [58] dataset.
LoRA
Subject
Number of datapoints
Ecology
3051
Biogcochemistry
466
environmental earthscience
Hydrology
119
Solid Earth sciences
1022
Environmental sciences
342
Biochemistry
1326
chemistry
Chemical biology
279
chemicalsciences
Chemistry
1249
Materials science
2146
Optics and photonics Materials science Nanoscience and technology Energy science and technology Engineering
645
2896
engineering
technologicalinnoattion
1047
160
252
Neuroscience
3400
Anatomy
302
neuroscience psychology
Physiology
1096
Neurology
121
Psychology
81
Microbiology
1511
Oncology
209
biomedical
Immunology
1665
healthsciences
Discases
1240
Pathogencsis
375
Risk factors
913
Environmental
socialsciences
2127
social sciences
globaldevelopment
Social sciences
1559
Business and industry Developing world
156
245
Computational biology and bioinformatics Systems biology
computational
3295
datasciences
1705
Ecology
2184
Evolution
1069
agriculture
Plant sciences
1366
lifesciences
Zoology
365
Agriculture
16
Biochemistry
1729
Molecular biology
1485
genomics
Stem cells
337
biotechnology
Genetics
994
Biotechnology
455
Physics
3287
Space physics
25
space physicalsciences
Optics and photonics
1216
Solid Earth sciences
383
Astronomy and planetary science
89
32

---

1.0
4.5
MMSCI
0.9
#Rerrieved LoRAs
4.0
chemistry, chemistry_chemicae sccemicaabciences
engineering lechnologiaainnovaiion
Retrieved
space physicalscieenes
agriculture, lifesciences
3.0
environmental, aitioonmenaa_cartacciinne earthscience
2.5
socialsciences g/obaldevelopment
0.5
genamics, biotechnology
blomedical Nomeedica_hhallhhseeee healthsciieness
0.4
neuroscience, seuideence_pycholgyyyyye
1.5
0.3
computationa, datasclences
1.0
123
10
15
20
123
5
10
15
20
k
(a)
(b)
Figure C.8: AC-LoRA retrieval results for MMSci based dataset (Tab. C.4) for fetch k=10 and
threshold=0.0
Time to first taken [s]
40
20
10
12
15
Number of LoRAS loaded
Figure C.9: Time first token generation latency for 64-tokeen input query where are all the LoRAs are loaded from scratch to device memory.
SYSTEM: You are expert analyzer. Given text, extract the main (distinct) facts in concise manner as list, separated by "!,* Each fact must be fully self-contained, meaning it should make complete sense on its own without requiring any context from the original text or other extracted facts. \n Always explicitly state the subject and object-never use pronouns (e.g.. he, she, they, it) when clear noun be used instead. \ Do assume infer any information that is not explicitly stated in the text. n Each fact must stand alone- no fact should depend on previous one be understood. \n Keep facts as concise, accurate, and clear as possible while maintaining completeness. There should always be an even number of facts (between 2 and 10). USER: (extracted document)
We then divide randomly the generated facts into two groups LORA-1 1 and LORA-2. From these, generate one new article for each using the following prompt:
SYSTEM: You are an expert writer. Given list of facts, write a coherent and well-structured text that includes only the provided facts facts-nothing more, nothing less. Ensure the text is readable, logically structured, and flows naturally while maintaining clarity and conciseness. Do not add any additional information or interpretation beyond the given facts. USER: List of facts: fact \p.... \n fact n
From this we now generate two different types question and answer pairs.
Single-LoRA QA: These questions and answers should be answerable only by one LoRA. We take two facts from the same set (either LORA-1 or LORA-2) and their corresponding generated articles, and ask the model to generate a question and answer pair that is only answerable when knowing both facts. To do this, we use the following prompt:
SYSTEM: You are an expert question generator. Given two facts and a context, create a question and answer pair where the answer requires both facts to be answerable. the subject and object in both the question and answer. n- not infer any information that is not explicitly stated in the facts or the Context. n- Do not add any additional explanation-only
Clearly name
33

---

provide the question and answer.
USER: context: (context.lora.n) in fact_l: [fact_1 Jora. fact.2: (factt22..ooraaa Ioraa.n
where n is either or 2.
Combined-L.oRA QA: In this case, we take one fact from each set (one from LORA-1 and one from A-RR-22) and both generated articles and ask the model to generate a question and answer pair which is only answerable when knowing both facts. To do this, we use the following prompt:
SYSTEM: You are an expert question generator. Given two facts, and two contexts create a ques- tion and answer pair where the answer requires both facts be answerable. \n \n- Clearly name the subject and object in both the question and answer. \n- Do not infer any information that is not explicitly stated in the facts or the Contexts. \n- Do not add any additional explanation only provide the question and answer.
USER: context cootext..ooa..l. n context22.ccomtexx_oaa
ffct_I:(fact..ora_
fact 2: (fact lora___
We then review the cases in which the question and answer pair were in the correct format, which occurred in very few cases (< 10). Afterwards, we create two training sets (one for each LoRA) and one test set. In each training set, we include:
250 single-LoRA questions of the corresponding set, once with and once without context (i.e., 500 data points). We also ensure here that each context generated appears at least once in this set.
140 single-LoRA questions of the corresponding set without context.
So, total, each of the training sets contains 640 data points. The test set comprises all the combined LoRa questions and the remaining single-LoRa questions total of 1,065 data points).
We use Prompt grade our evaluation.
D.2 WikiArts
We OWEN2-VL-7BB ---SSRRUCT to generate two-generation prompts for each image in the WikiArts dataset [23]. For this, we input the image and the following prompt:
Given the style, genre, the artist which we try to reproduce and an image please write **two** generation prompt for the given image. should be one or two sentences per prompt. Do *only* write the prompts, separate them always only by a new line ("\n') An Style: Genre: Artist:(attist]
We use these prompts for both finetuning the model and for building the vector database for later retrieving the correct LoRA.
For the images displayed in Fig. 7 we use the following prompt:
"a serene Buddhist temple on a mountain path, captured in peaceful brushwork"
D.3
Grading
D.3.1
Flan
use the same grading functions from Zhao et al. [44] to evaluate our results, to ensure compara- bility. Therefore, we evaluated it using the BLEU score from the Natural Language Toolkit [59] and the Rouge score from the Rouge Python package.
D.3.2 RepLiQA
To evaluate the different experiments the Repl.iQA dataset, we GEMMA-3-27R to give each generated answer grade between and 5.
The prompt use is the following:
34

---

Table E.1: Hyperparameters used to finetune RepLiQA LoRAs
Hyperparameter
Values
base model
meta- meta-Ilama/ Llama-3.11m1 3.1-8B-IIssntuu
epochs
per per.device_tres batch_size
gradient.accumulln www.mulaatcoonseoo.coo
learning rate lora alpha
le-4
lora modules
o.pruj. k.proj. gate.proj, down.prujj v-proj. q-proj. up-proj
Table E.2: Number of datapoints used for building the vector base for different tasks.
Task
anli
cb
matched
mali
wnli
copa
cloze
story
mupe
glue
challenge
are
easy
are
openbook
qu
All other
Tusks
Datapoints
15k
500
8k
Mk
1.9k
3.8k
1.6k
1.7k
5.5k
12k
3k
7.2k
15k
30k
Evaluate how well the Generated Answer matches the Reference Answer or the de- tailed reference answer for the given Query. Be strict: Names, dates, and specific details must be exact to be correct. Additional facts that are not in the Reference Answer do not affect the score unless they contradict the Reference Answer, in which case the score should decrease. If a name, date, or key fact is incorrect, the score must be 1, regardless of other details. Assign score from to 5 based on accuracy, completeness, and relevance: 5 Identical meaning, all details correct. 4 Mostly correct, with only minor wording variations but the same meaning. correct, with some missing or incorrect details. 2 Weak relevance, with significant errors or omissions. Incorrect or unrelated. Input Format: Query: query \nn Reference Answer: ref- erence_and w n Generated Answer: generated_answer |n Output Format: Explanation: [Brief reason for the score] Score: [1-5]
SYSTEM:
Partially
USER: Query: {query) } (n Reference Answer: pnterence_answerr \n Generated Answer: (generated.answer)
In case we have two reference answers, for example, for most of our RepLiQA experiments, we also add the long reference answer in addition to the reference answer 'detailed reference answer' to the prompt.
E
Implementation detail
E.1 Evaluation Setup
We run our experiments on two workstation GPUs, each with 10752 processing cores, 48GB GDDR6 VRAM. (384-bit bus and 768 GB/s memory bandwidth), and a 38.7 TFLOPS single preci- sion performance. The GPU is connected to a host (2x x86 44-core CPU with 256 GB RAM) over a PCIe 4.0.
F.2 Finetuning
E.2.1 RepLiQA
For language models, we use unsloth [60] to fine-tune the different LoRAs as the library is faster and saves memory compared to the base implementation. We finetune the 17 LoRAs for the RepLiQA dataset with the hyperparameters displayed in Tab. E.
For the knowledge injection experiments displayed in Fig. 3(a) for cybersecurity, maintain the same hyperparameters and only adjust the values for alpha and rank. Also, we fine-tune for 10 epochs and save the LoRA at every epoch to study overfitting.
35

---

E.2.2 FlanV2
As mentioned before, we did not fine-tune the FlanV2 LoRAs use the made available by the authors from [44]. As they only focused on formats, they finetuned their LoRAs by only targeting the v-proj and q-proj modules.
E.2.3 WikiArts
To finetune the different WikiArts LoRAs, we use the Huggingface diffusers library [61]. We set rank and alpha both to 16.
E.2.4 Retriever
We utilize the LangChain [62] library implement most of our retrieval process, and its FAISS [63] implementation serves as our vector store.
E.2.5 Building the database Text.
As the training set for Flan V2 used for the different LoRAs was not shared, we constructed one based on the official Flan V2 dataset for the retriever. In particular, we take the first 30k (or fewer for smaller tasks) samples of each selected task as the training set. The exact number of datapoints per task shown Tab. F.2 For WikiArts, we generate prompts for the different images the training set and use these for the database. For RepLiQA, only two entries per data point: one with and one without context.
used first version of the training set, with
We then use MNET- BECTTBBSE-VEE [50] to split the files into chunks of 100 tokens and create the FAISS vector store by adding the created documents.
and the embedding model ALL-
Text-Image.
We embed each text and image together using the multi-modal embedding model INFGRAD/- ASSPEREN_VISSIONIAATT model.
[64]. We initiate SEXTRNCETTRANSFRRME with this
E.2.6 Retrieving and Hinting
We use the similarity_see search_with.e scrre_crrrerrp.. by_yctor text-image inputs and similarity wewprittyserch_wwiiir wetps//wiihhhhe with for only-text queries.
vector function to retrieve the most likely LoRAs for
We use the filter function enforce access control, retrieving only the embeddings with the allowed LoRAs in the metadata.
The Hinting mechanism is implemented as two database queries, once with the filter function and once without.
E.3
LoRA Mixing
We patch the PEFT library to enable the mixing. We mainly modified the forward function for the Linear LoRA layers.
F
List of Assets
The following is list of assets, along with their licenses (and sources, linked) that we use in this paper.
RepliQA dataset: CC BY 4.0
Flan V2 dataset: Apache License Version 2.0. January 2004
Wikiart dataset: BSD 3-Clause License
MMSci dataset: CC BY 4.0
36

---

Meta Llama: META LLAMA 3 COMMUNITY
Google Gemma: Open source
Qwen models: royalty-free limited license
all-mpnet-base-v2: Apache License Version 2.0, January 2004
langchain: MIT
PEFT: Apache License Version 2.0, January 2004
Stable-diffusion CreativeML. Open RAIL-M,, August 22, 2022
37


Reviews:
[{"review_id": "XARqIRtiCq", "reviewer": "Reviewer_VTRK", "summary": "This paper introduces a new method called AC-LORA, which tackles the problem of letting a single large model serve many users who each see only the data they are entitled to. Instead of fine-tuning one big model on all documents, or leaving those documents into the prompt, it fine-tunes a lightweight LoRA adapter for every permission segment, stores each adapter’s embedding in a vector index, and at inference time retrieves only the adapters that are semantically relevant to the query and  allowed for the current user. These adapters are loaded on-the-fly and blended by cosine-similarity weighting, so no extra routing network or retraining is needed when new data arrive. Experiments on text (RepLiQA, Flan-V2) and image generation (WikiArt styles) show that AC-LORA matches or beats task-specific LoRA baselines while slashing context length, latency, and leakage risk. The system therefore offers a practical, nearly “zero-additional-training” way to combine access control, retrieval, and knowledge fusion for multi-modal LLM applications, with remaining challenges around hint privacy and adapter-swap overhead in very large deployments.", "questions": "1. Please give analytical or empirical evidence that simple cosine-weighted blending is robust when two adapters make conflicting updates to the same weight tensor, visualise conflict rates or run controlled ablations. \n\n2. Please attach 95 % confidence intervals or formal significance tests (e.g., paired t-tests or bootstrap) for the main RepLiQA, Flan-V2, and WikiArt results and state the random seeds used.", "limitations": "The paper briefly admits that the Hint mechanism could leak sensitive information but never formalizes any threat model or measures what is actually exposed, and it overlooks broader risks such as adapter-fusion bypassing content filters, the environmental cost of large-scale hot-swapping, and governance concerns around data revocation or labor displacement. A concise yet concrete treatment of these issues like quantitative leakage metrics, energy estimates, and responsible-deployment guidelines would make the societal-impact discussion far more convincing.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n1. This paper combines LoRA tuning, vector search, and cosine mixing into a single, low-overhead framework, which is a novel integration. 2. This mode works across modalities, which is shown to run on both text (LLAMA-2/3) and image models (Stable-Diffusion, Qwen-VL). \n3. It provides fine-grained access control and hot-swappable adapters, matching real-world deployment needs. \n4. Leakage-extraction attacks confirm that per-adapter isolation greatly reduces verbatim data leaks, demonstrating the security.\n\nWeaknesses:  \n1. No confidence intervals or significance tests, so robustness of the gains remains uncertain. \n2. Results stop at ten adapters; latency and memory for hundreds are not reported. \n3. This paper offers little analysis of when linear LoRA mixing could fail due to conflicting updates.", "was_revised": true, "final_justification": "I appreciate the response and new experiments from authors, which address part of my concerns. However, it is still not convinced by the explanation of weight tensor conflict, Hint mechanism, and inference latency with more LoRAs. I will keep the score.", "rebuttal": "We thank the reviewer for their time and valuable feedback. In the following, we summarize the main concerns and answer them.\n\n### **Conflicting updates to the same weight tensor**\n\nAC-LoRA addresses the issue of conflicting updates to the same weight tensor, as highlighted by the reviewer. Conflicting weight merging is a known challenge in existing works that combine multiple LoRAs by merging their weights, resulting in accuracy loss while merging more than two LoRAs [1].\n\nHowever, AC-LoRA *does not* rely on merging weights. Instead, it mixes the outputs of each layer based on cosine similarity between the user query and permitted LoRA datasets. As a result, even if the retrieved LoRAs make a conflicting update to the same weight tensor,  AC-LoRA remains unaffected by these conflicts. \nIntuitively, the final output can be seen as the output of a higher-ranked LoRA, which is composed of the retrieved LoRAs.\n\n*Action point: We will clarify this in Section 3.*\n\n[1] LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild, Zhao et al.\n\n### **95% confidence intervals for the main experiments, and the random seed**\n\nThe (bootstrapped) 95% confidence intervals of all our major experiments in the paper are as follows. \n\n\n**RepLiQA**\n\n|   **RepLiQA Topics / Grades**   | **Mean** | **Lower Bound** | **Upper Bound** |\n|:-------------------------------:|:--------:|:---------------:|:---------------:|\n|               ALL               |   2.214  |      2.161      |      2.266      |\n|       Local Environmental Issues  |   2.352  |      2.1118     |      2.592      |\n|    Local Health and Wellness    |   2.307  |      2.104      |      2.509      |\n|        Cybersecurity News       |   2.151  |      1.940      |      2.378      |\n|   Local Sports and Activities   |   2.179  |      1.964      |      2.381      |\n|      Local Arts and Culture     |   2.035  |      1.858      |      2.225      |\n|       Neighborhood Stories      |   1.979  |      1.786      |      2.172      |\n|         Company Policies        |   2.882  |      2.611      |      3.152      |\n|            Local News           |   2.262  |      2.051      |      2.468      |\n|     Local Education Systems     |   2.140  |      1.929      |      2.359      |\n|    Regional Folklore and Myth   |   2.222  |      2.023      |      2.452      |\n|         Incident Report         |   2.193  |      1.993      |      2.393      |\n|           News Stories          |   2.038  |      1.839      |      2.251      |\n| Local Technology and Innovation |   1.920  |      1.712      |      2.122      |\n|   Regional Cuisine and Recipes  |   2.380  |      2.156      |      2.604      |\n|  Local Politics and Governance  |   2.127  |      1.906      |      2.338      |\n|  Local Economy and Market       |   2.434  |      2.204      |      2.663      |\n|   Small and Medium Enterprises  |   2.297  |      2.081      |      2.506      |\n\nFor k = 2: (others will be added to the paper)\n\n| **RepLiQA Topics / Retrival %** | **Mean** | **Lower Bound** | **Upper Bound** |\n|:-------------------------------:|:--------:|:---------------:|:---------------:|\n|               ALL               |   89.4   |       88.5      |       90.3      |\n|    Local Environmental Issues   |   89.6   |       85.2      |       93.5      |\n|    Local Health and Wellness    |   92.2   |       88.8      |       95.7      |\n|        Cybersecurity News       |   91.6   |       88.0      |       95.3      |\n|   Local Sports and Activities   |   93.0   |       89.5      |       96.0      |\n|      Local Arts and Culture     |   93.0   |       89.7      |       96.2      |\n|       Neighborhood Stories      |   79.0   |       74.2      |       83.8      |\n|         Company Policies        |   91.9   |       86.8      |       96.3      |\n|            Local News           |   78.3   |       72.9      |       83.7      |\n|     Local Education Systems     |   89.2   |       85.0      |       93.4      |\n|    Regional Folklore and Myth   |   95.6   |       92.6      |       98.0      |\n|         Incident Report         |   91.2   |       87.7      |       94.7      |\n|           News Stories          |   90.8   |       86.9      |       94.6      |\n| Local Technology and Innovation |   85.5   |       80.5      |       90.0      |\n|   Regional Cuisine and Recipes  |   95.6   |       92.9      |       97.8      |\n|  Local Politics and Governance  |   89.2   |       84.8      |       93.1      |\n|     Local Economy and Market    |   81.7   |       76.6      |       86.9      |\n|   Small and Medium Enterprises  |   93.0   |       89.5      |       96.5      |\n\t\n\n**FLAN**\n\n|        **FlanV2**        |  **Metric** | **Mean** | **Lower Bound** | **Upper Bound** |\n|:------------------------:|:-----------:|:--------:|:---------------:|:---------------:|\n|      struct to text      |   rouge-1   |   61.4   |       58.2      |      64.76      |\n|      struct to text      |   rouge-2   |   37.14  |      33.83      |      40.27      |\n|      struct to text      |   rouge-l   |   54.23  |      51.17      |      57.31      |\n|        commonsense       | Exact Match |   65.5   |      58.48      |      72.01      |\n|         sentiment        | Exact Match |   90.0   |       86.0      |       94.0      |\n|       reading comp.      | Exact Match |   54.6   |       49.0      |      60.66      |\n|      closed_book QA      | Exact Match |   36.5   |       30.5      |      43.01      |\n|        coreference       | Exact Match |   54.0   |       44.0      |       64.0      |\n| read.comp.w: commonsense | Exact Match |   59.0   |       49.0      |       69.0      |\n|        paraphrase        | Exact Match |   63.0   |      56.49      |       69.5      |\n|            nli           | Exact Match |   68.2   |      64.24      |      72.52      |\n|        translation       |     Bleu    |   13.4   |      11.37      |      15.45      |\n\nFor k=2 (others will be added to the paper)\n\n|  **FlanV2 / Retrival %** | **Mean** | **Lower Bound** | **Upper Bound** |\n|:------------------------:|:--------:|:---------------:|:---------------:|\n|      struct to text      |   90.0   |       85.5      |       94.0      |\n|        commonsense       |   65.0   |       58.0      |       71.0      |\n|         sentiment        |   95.5   |       92.5      |       98.0      |\n|       reading comp.      |   58.0   |       52.3      |       63.6      |\n|      closed_book QA      |   21.5   |       16.0      |       27.0      |\n|        coreference       |   81.0   |       73.0      |       88.0      |\n| read.comp.w: commonsense |   77.0   |       68.0      |       85.0      |\n|        paraphrase        |   95.5   |       92.5      |       98.0      |\n|            nli           |   56.1   |       51.5      |       60.6      |\n|        translation       |   85.0   |       81.2      |       88.5      |\n\n\nWe do not provide empirical results for WikiArts, as it serves primarily as a proof-of-concept for AC-LoRA extending beyond text. The limitations of multi-modal evaluation are addressed in Appendix C3 (line 721). Regarding image quality, there is no practical way to measure it, since the base model has likely been trained on WikiArts or similar images, and a human study is beyond the scope of this paper.\n\nThe *random seed used was 42*.\n\n*Action point: We will update the paper accordingly.*\n\n### *Threat model and leakage evaluation for the hint mechanism**\n\nAs the reviewer pointed out, we did not thoroughly discuss the *hint* feature. We believe that, besides providing the correct answer, a corporate AI agent must also assist the user in applying for document permission in case the user does not have access to the most relevant document. The maintainer/administrator can decide if the user will be permitted to access it. The hint provides a link to the document permission, e.g., access permission to a Git repository. However, a curious/malicious user can gain knowledge of the possible *existence* of the information. Disabling the hint for sensitive LoRAs (e.g., specification of an upcoming product) will prevent AC-LoRA from mentioning the existence of a specific dataset to not-permitted users. However, measuring the exact potential for leakage requires creating a particular dataset and benchmark to reflect such a scenario.\n\n*Action point: We will add a discussion of this topic both in the threat model and AC-LoRA design.*\n\n### **The mixing of Adapters to bypass content filters**\n\nThis is an interesting question, and the response largely depends on the type of content filter. Assuming that the content filtering is done by aligning the base model in a specific way [1], we will outline three different scenarios in which they could be affected by AC-LoRA: \n\n- *No LoRAs are retrieved*: In case the user query does not match any LoRA, no LoRA will be retrieved, and the base model will answer the query. In that case, the original alignment of the model remains unchanged. \n- *LoRA is retrieved*: If LoRA(s) are retrieved, it's challenging to ensure model alignment without considering this during fine-tuning. \n- *Specific LoRAs for bypassing filters*: A set of LoRAs could be specifically fine-tuned to bypass the content filter (e.g., LoRAs used by law enforcement), making the bypass of the filter intentional.\n\nHowever, we agree that the broader impact of such a system is complex, requires further analysis, and is beyond the scope of this paper.\n\n*Action point: We will add discussions to address this.*\n\n\n[1] Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations, Hakan Inan et al.\n\n### **>100s LoRAs**\n\nInference latency and VRAM requirement increase with the number of loaded LoRA. Due to the absence of a dataset and GPU resources, we did not experiment with such a scale (please see *reviewer WZ6X* answer 3 for more information).\n\nWe hope we addressed all your questions and would be happy to continue the discussion to clarify any outstanding concerns."}, {"review_id": "r2i4QXKjNa", "reviewer": "Reviewer_jNGe", "summary": "This paper proposes AC-LoRA, a system for developing access control-aware LLM chatbots to prevent leakage of sensitive/confidential data to unauthorized parties. In doing this, the authors attempt to address a gap in existing work where maintaining individual models (or LoRA adapters) for different roles becomes infeasible as the number of permission zones grows, while training a single foundation model compromises data confidentiality. Other approaches such as access-controlled RAG are also computationally expensive and/or detrimental to performance. Instead, the authors propose a method based on fine-tuning LoRAs for individual permission zones. These LoRAs are then mixed together in a permission-ware manner, ensuring that a user does not retrieve information from LoRAs they are not authorized to use; non-permissible LoRAs are then used to provide a hint to the user that they might be able to receive a better answer by applying for relevant permissions. Evaluations are performed using three different datasets: Flanv2 , RepLiQA, and Wikiart. Specifically, the authors show that the proposed LoRA mixing offers competitive performance to perfect selection (using the specific LoRA associated with a given query) and outperforms other baselines in 8 out of 10 evaluated domains.", "questions": "- In Table 2, what metric is being reported? Please clarify in the caption.\n- Have the authors performed evaluation using queries relevant to multiple domains (i.e., permission zones)?\n- How much performance improvement does AC-LoRA offer compared to a simpler strategy of mixing all permissible LoRAs without retrieving relevant ones first?", "limitations": "The limitations are adequately discussed, with the possible exception of multi-domain queries that require retrieving information from multiple permission zones.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "This was an interesting paper addressing an important gap in the usage of LLMs in access controlled environments. The authors propose a plausible training-free solution, and the evaluation results show that AC-Lora performance comparably to perfect selection, while outperforming existing methods in performance and/or latency.\n\nThe technical novelty is modest, largely involving a thoughtful combination of existing methods rather than introducing fundamentally new techniques. I would have liked to see additional evaluations, particularly a computational complexity comparison with perfect selection, as well as an ablation study to quantify the performance gains achieved by AC-LoRA compared to simply mixing all permissible LoRAs without the retrieval step. It is also not clear how the framework performs when a query is relevant to multiple domains/LoRAs.\n\nOther more minor comments:\n- I suggest the authors revise the Figure captions to make them more self-contained.\n- The evaluation using the WikiArt dataset is limited to Figure 7, which does not include any quantitative performance metrics. I recommend either expanding the evaluation for WikiArt or removing it from the evaluation set to make room for other improvements.", "was_revised": true, "final_justification": "After reading the authors' response, I believe they have adequately addressed my concerns, and I will maintain my favorable score.", "rebuttal": "We thank the reviewer for their time and valuable feedback. In the following, we summarize the main concerns and answer them.\n\n### **Could you provide the metric in Table 2?**\n\n\nAs described on lines 268-269, we use the BLEU score to evaluate the translation, ROUGE for the struct-to-text tasks, and EXACT MATCH for the rest.  We will add this to the caption and revise all figures to make the captions self-contained.\n\n\n### **Did you evaluate AC-LoRA using queries relevant to multiple domains?**\n\nYes, we address this with the *CS-Combi* dataset (line 273), where the queries require the knowledge distributed over two LoRAs. This, however, did not test the retrieval capabilities; therefore, we have now generated 100 queries by providing 2-5 RepliQA documents from different topics and asking to build a question that requires all the given documents to answer. We then use these queries to retrieve the relevant LoRAs and verify if they match the topics of the documents.\n\nBelow we show some results of the experiments. The number of topics (2-5) is the number of documents used to generate the query. The numbers shown in the first table are the % of the relevant LoRAs retrieved. The second table shows the percentage of correctly retrieved LoRAs. This means that, in general, the percentage increases for more LoRA retrieved (reaching 100% at some point) in the first table, while it decreases in the second table.\n\n| **Number of topics/ % relevant retrieved** | **k** | **# (avg.) Retrieved LoRAs** | **Mean** | **Lower Bound** | **Upper Bound** |\n|:------------------------------------------:|:-----:|------------------------------|:--------:|:---------------:|:---------------:|\n|                      2                     |   50  |             3.73             |   90.3   |       82.6      |       96.1      |\n|                      3                     |   50  |             3.47             |   57.8   |       47.3      |       70.1      |\n|                      4                     |   50  |             4.83             |   59.3   |       51.0      |       66.6      |\n|                      5                     |   50  |             4.77             |   49.0   |       41.2      |       56.7      |\n\n\n| **Number of topics/ % correctly retrieved** | **k** | **# (avg.) Retrieved LoRAs** | **Mean** | **Lower Bound** | **Upper Bound** |\n|:-------------------------------------------:|:-----:|------------------------------|:--------:|:---------------:|:---------------:|\n|                      2                      |   50  |             3.73             |   60.2   |       50.3      |       71.0      |\n|                      3                      |   50  |             3.47             |   56.4   |       46.7      |       67.0      |\n|                      4                      |   50  |             4.83             |   54.2   |       51.0      |       47.1      |\n|                      5                      |   50  |             4.77             |   56.9   |       50.1      |       63.7      |\n\nWhile we hope that this additional experiment provides further insight, we want to highlight that finding a high-quality dataset with queries that require information from multiple datasets and can be easily separated into different training sets for LoRAs was a major challenge. To this end, we design the dataset *CS-Combi* to evaluate queries that require knowledge from multiple datasets.\nHowever, a more rigorous evaluation requires manually curating the dataset, which is beyond the scope of this paper.\n\n### **Is AC-LoRA better than mixing all permissible LoRAs?**\n\nThe primary advantage of loading only the relevant LoRA is the reduction of inference latency. \n\nAdditionally, we observed an average grade improvement of 0.066 (95% interval: (-0.00748, 0.138)) over all permissible LoRAs in the RepliQA dataset.\nThis effect could come from several factors; we suspect it’s primarily due to the relatively high rank used with a relatively small dataset. We expect that the AC-LoRA improvement will increase further as the number of LoRAs increases. Flanv2, however, behaves differently. Since Flanv2 is primarily finetuned to influence output format, mixing all Flan LoRAs together significantly degrades performance. This is evident in the table below, which shows the improvement of AC-LORA over the mixing of all Flan LoRAs.\n\n\n| **FlanV2 / Grade improvement over Mixing All** | **Metric**  | **Mean** | **Lower Bound** | **Upper Bound** |\n|:----------------------------------------:|-------------|:--------:|:---------------:|:---------------:|\n|              struct to text              |   Rouge-1   |   34.64  |      31.95      |      37.19      |\n|              struct to text              |   Rouge-2   |   26.94  |      23.97      |      30.17      |\n|              struct to text              |   Rouge-l   |   30.64  |      28.02      |      33.64      |\n|                commonsense               | Exact Match |   65.5   |       58.5      |       72.5      |\n|                 sentiment                | Exact Match |   90.0   |       85.5      |       94.0      |\n|               reading comp.              | Exact Match |   54.6   |      49.32      |       60.0      |\n|              closed_book QA              | Exact Match |   36.5   |       30.0      |       43.0      |\n|                coreference               | Exact Match |   54.0   |       44.0      |       64.0      |\n|         read.comp.w: commonsense         | Exact Match |   59.0   |      49.97      |       68.0      |\n|                paraphrase                | Exact Match |   63.0   |      56.00      |       69.5      |\n|                    nli                   | Exact Match |   68.2   |      64.44      |      72.32      |\n|                translation               |     Bleu    |    9.7   |       8.22      |      11.43      |\n\n\n### **Can you provide quantitative performance metrics for WikiArts?**\n\nWe do not provide empirical results for WikiArts, as it serves primarily as a proof-of-concept for AC-LoRA extending beyond text. The limitations of multi-modal evaluation are addressed in Appendix C3 (line 721). While we could evaluate the retrieval, this does not add any insight as it is already covered by the other text-based datasets. Additionally, we would need to generate a suitable test set, as none currently exist. Regarding image quality, there is no practical way to measure it, since the base model has likely been trained on WikiArts or similar images, and a human study is beyond the scope of this paper.\n\n*Action point: We can highlight this in the paper more, or if necessary, remove it from the main body of the paper.*\n\n\nWe hope we addressed all your questions and would be happy to continue the discussion to clarify any outstanding concerns."}, {"review_id": "eARoLW3jZw", "reviewer": "Reviewer_WZ6X", "summary": "This paper presents AC-LoRA, the first end-to-end access control-aware inference serving system to equip LLMs with strong access control management. AC-LoRA control the sensitive information access by fine-tuning different LoRAs on predefined data groups and later uses embedding similarity to compute the LoRA based on user permission and user query. The design of similarity combination based on embedding makes AC-LoRA almost training-free. The author conducts comparison experiments with the previous method on two datasets, and the result demonstrates that AC-LoRA show better or on-par performance than SoTA LoRA mixing techniques, which highlights the effectiveness and efficiency of AC-LoRA.", "questions": "See weaknesses 3 and 4.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. I like the content of the paper a lot. The paper is well motivated and written in an organised format.\n2. The design of using embedding similarity to implement access control is novel and light-weight. This design not only reduces the additional training requirement but also ensures high performance compared to the previous method.\n3. The experiment is comprehensive and sufficient to illustrate the effectiveness, efficiency and potential application of multimodality. Basically, I have no question about the experiment set-up, as the result is pretty clear to support the author's claim.\n\nWeaknesses:\n1. The \"gap in prior work\" section in the introduction is well motivated, but I think there is a lack of the limitation of RAG to the usage of LoRA or LoRA mixing techniques.\n2. It seems that the effectiveness of AC-LoRA is built on the sparse distribution of the knowledge base. I am wondering whether AC-LoRA can achieve the same high performance on embedding space that is not well-separated (maybe math or coding)?\n3. What if there is partial overlapping between knowledge groups? Can the author illustrate whether the overlapping will affect the LoRA selection?\n4. What if one query requires access to more than 5 knowledge groups, for example, 10 or embedding space is pretty large, will GPU memory become a major bottleneck?", "was_revised": true, "final_justification": "I think the author has addressed my main points during the rebuttal (If the author takes action to elaborate on the final version, if possible), so I keep my positive rating.", "rebuttal": "We thank the reviewer for their time and valuable feedback. In the following, we summarize the main concerns and answer them.\n\n### **Could you motivate RAG vs LoRA better?**\nThis is not a black-or-white issue, and there are capabilities that RAG can achieve that are harder to achieve with LoRAs and vice versa. However, we will summarize our considerations on why RAG is not the best solution in this setting as follows: \n\n\n- With RAG, access control is a bit more straightforward [1] as one can mark the rows in the database with the access control rules (e.g., which user is permitted to access that specific row)\n- Existing work shows that LoRA scores higher in accuracy compared to retrieval [2].\n- RAG often results in a massive context size due to similarity with a larger number of documents. We show in *Figure 8* that the context size is detrimental to the inference latency and memory requirement.\n- Existing works [3]  also show that RAGs are not effective in multi-hop queries. \n\t\nIn a real deployment, one could also consider adding a lightweight RAG (with only the latest added documents that were not yet used to train the LoRA) to AC-LoRA, which partially mitigates RAG’s problem.\n\n*Action point: We will add these points to our motivation section.*\n\n#### References\n\n[1] Privacy-Aware RAG: Secure and Isolated Knowledge Retrieval, Pengcheng Zhou, Yinglun Feng†, Zhongliang Yang\n\n[2] GPT vs RETRO: Exploring the Intersection of Retrieval and Parameter-Efficient Fine-Tuning, Aleksander Ficek, Jiaqi Zeng, Oleksii Kuchaiev \n\n[3] MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries, Yixuan Tang and Yi Yang\n\n### **Does AC-LoRA work without a well-separated embedding space?/ Can AC-LoRA handle overlapping knowledge groups?**\n\n\n\nThat's a great point. In AC-LoRA, we utilize an off-the-shelf embedding model to calculate the embedding space and the similarity score, thereby merging the LoRAs.\nFor a specific data set where the permission domains overlap, we envision a specialized embedding model being fine-tuned to ensure they are separated. \nHowever, having overlapping data sets is not detrimental to AC-LoRA because of the following two reasons:\nIf one or more overlapping datasets are not included in the user's permitted list, the specific LoRA(s) will not be used to answer the question.\nIf one or more are permitted, they will be merged based on the similarity score. Anli_{r1,r2,r3} in FlanV2 are very similar and have a large overlapping embedding space, which we evaluate and cover in this case.\n\t\nAdditionally, we conducted a new experiment to further support our findings. We built a new vectorbase using 20 sets, each combining 2-4 RepliQA topics, resulting in overlapping topics across LoRAs. Since RepliQA requires only one correct LoRA to be retrieved, success rates were high: over 90% for k=1 (retrieving ~3 topics) and up to 99.6% for k=100 (retrieving ~11 topics across ~7 LoRAs).\n\nWhile these results are expected and only partially meaningful – since no new LoRAs were finetuned and thus end-to-end grades are unavailable due to time constraints – they still should help reinforce the above. A detailed discussion of the required LoRAs training set granularity is beyond the scope of this work.\n\nIn conclusion, AC-LoRA is effective for the overlapping permission zones. \n\n*Action point: We will add further discussion to the paper to clarify this.*\n\n\n### **Is GPU memory a major bottleneck for more than 5/10 LoRAs?**\n\nIn our setup, loading the 8B base model with all 17 LoRAs (r=64) requires about 40GB of memory without quantization. Similarly, Flan with 48 LoRAs (r=8) uses only slightly more and still fits on a single 48GB GPU. These requirements depend on factors like the base model size, LoRA configuration (targeted modules, number of layers, rank), and the prompt (and context) length. However, there are parallelization techniques to mitigate this issue, such as pipeline parallelization. This allows AC-LoRA to load more LoRAs.\nAdditionally, in a memory-constrained environment, one could keep frequently used LoRAs on the GPU while loading less common ones on demand, trading off a small latency increase.\n\n*Action point: We will add further discussion to the paper to clarify this.*\n\n\nWe hope we addressed all your questions and would be happy to continue the discussion to clarify any outstanding concerns."}, {"review_id": "lsX293i39P", "reviewer": "Reviewer_PYQb", "summary": "The paper introduces AC-LoRA, a system that regulates which document(s) an LLM can use (include as finetuned  LoRAs) for users with different roles. AC-LoRA stores a set of LoRAs with the corresponding documents used to fine-tune these LoRAs in a form of sentence embeddings. At inference time, AC-LoRA considers only the LoRAs/documents that are permissible to the user, and compares the embedding of the user’s query to the available document embeddings to dynamically load the relevant LoRAs. After selecting the relevant LoRAs, the model generates an answer by mixing the retrieved LoRAs.", "questions": "### Questions:\n* Q1: Table 2: what is meant with IID and OOD?\n* Q2: Why not use a baseline with only permissible LoRAs?\n* Q3: Is AC-LoRA practical in case of hundreds of (relevant) documents?\n\n\n### Comments:\n\nL165: “Isolated LoRA Fine-tuning and Knowledge Injection”: Please motivate the experiment better at the very beginning (\"LoRAs can reliably inject domain-specific knowledge\") \nTable 2: The selection baseline seems competitive. Please add this to the discussions (L310-319)\n\nSome parts can be made more consistent (e.g., \"knowledge merging\" is used in 4.1, and the corresponding paragraph in 4.2 is called \"combining knowledge\")\n\nL206: I believe $Q$ is not defined before\n\nFig. 5 caption: “Retrieval of FlanV2 target domains (**left**) and corresponding retrieved domains (**left**).” (one should be \"right\")", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 1, "strengths_and_weaknesses": "Strengths: \n* The paper targets an important problem with the aim of making the use of LLMs more practical \n* The paper considers both text and image modalities (the latter is admittedly not comprehensive (cf. Limitations))\n\nWeaknesses:\n* The work is incremental. From my understanding, AC-LoRA adds only a filtering step to select only relevant LoRAs based on the cosine similarity. The mixing is already introduced in other works. \n* It is not clear how AC-LoRA compares to a baseline where all the permissible LoRAs for the user are used. \n* Hinting is introduced as part of AC-LoRA, but is never evaluated and its benefits are not clear (it only shows the users that there is something they don't see)", "was_revised": true, "final_justification": "The authors addressed my concerns about the evaluation (baseline with permissible LoRAs, evaluating hinting) in the rebuttal. I'm not familiar with this area of research so it is difficult for me to judge how novel/incremental is their approach in combining knowledge.", "rebuttal": "We thank the reviewer for their time and valuable feedback. In the following, we summarize the main concerns and answer them.\n\n### **Are there any benefits to hinting?**\n\nWe envision that systems like AC-LoRA can enhance the information exchange and utilization of AI agents on corporate datasets. It's often the case that some relevant information belongs to other projects or departments that require additional permission or approval due to strict corporate information security regulations. Therefore, for a user who does not have access to the most relevant LoRA to their query, they will only get an inferior response. The hint serves as a guide for the user in accessing this information. Such a feature is currently absent in corporate AI agents.\nWe initially chose not to include this evaluation, as we believe the benefits of this feature are already reflected in Figure 3.b and the retrieval experiments. Additionally, the current datasets are not ideal for showcasing their full potential.\nHowever, to give a clearer picture, we’ve now evaluated the first 20 datapoints per RepLiQA topic—once with the topic LoRA masked, and once with the hinted LoRA (when available). For k = 3, the results (with 95% confidence interval) are as follows:\n\n| **RepLiQA Topics / Mean Grade** | ** With Masked  Topic LoRa** | **With Hinted  LoRA** | **No Hint Answers** |\n|:---:|:---:|:---:|---|\n| **ALL** | **1.870  (1.752, 1.994)** | **2.18 (2.061, 2.331)** | **57/340** |\n| Local Environmental Issues | 1.55 (1.1, 2.0) | 1.8 (1.3, 2.35) | 4/20 |\n| Local Health and Wellness | 1.65 (1.2, 2.15) | 2.45  (1.85, 3.1) | 2/20 |\n| Cybersecurity News | 2.05  (1.598, 2.6) | 2.25 (1.65, 2.95) | 2/20 |\n| Local Sports and Activities | 1.65 (1.25, 2.05) | 2.25 (1.6, 2.95) | 4/20 |\n| Local Arts and Culture | 1.65 (1.3, 2.05) | 2.25 (1.85, 2.65) | 2/20 |\n| Neighborhood Stories | 1.55 (1.2, 2.0) | 1.4 (1.1, 1.8) | 6/20 |\n| Company Policies | 2.4 (1.85, 2.9) | 3.0 (2.55, 3.45) | 1/20 |\n| Local News | 1.9  (1.45, 2.4) | 1.8 (1.35, 2.301) | 6/20 |\n| Local Education Systems | 2.2  (1.65, 2.75) | 2.25 (1.7, 2.8) | 6/20 |\n| Regional Folklore and Myth | 1.5 (1.15, 1.9) | 1.6  (1.2, 2.0) | 0/20 |\n| Incident Report | 1.75  (1.35, 2.25) | 2.3 (1.75, 2.9) | 1/20 |\n| News Stories | 1.6 (1.2, 2.15) | 2.15 (1.5, 2.85) | 0/20 |\n| Local Technology and Innovation | 2.0 (1.5, 2.5) | 2.1 (1.6, 2.651) | 6/20 |\n| Regional Cuisine and Recipes | 2.45 (1.85, 3.00) | 2.45 (1.9, 3.00) | 2/20 |\n| Local Politics and Governance | 1.7 (1.35, 2.1) | 2.05 (1.6, 2.55) | 5/20 |\n| Local Economy and Market | 2.5 (1.9, 3.1) | 2.7 (2.15, 3.2) | 6/20 |\n| Small and Medium Enterprises | 1.7 (1.3, 2.15) | 2.4 (1.9, 2.9) | 4/20 |\n\n*Action point: We’ll include the evaluation over the full dataset in the appendix.*\n\n### **Baseline with only permissible LoRAs**\n\nThe primary advantage of leading only the relevant LoRA is the reduction of inference latency. \nAdditionally, we observed an average grade improvement of 0.066 (95% interval: (-0.00748, 0.138)) over all permissible LoRAs in the RepliQA dataset. This effect could come from several factors; we suspect it’s primarily due to the relatively high rank used with a relatively small dataset. We expect that the AC-LoRA improvement will increase further as the number of LoRAs increases. Flanv2, however, behaves differently. Since Flanv2 is primarily finetuned to influence output format, mixing all Flan LoRAs significantly degrades performance. \n\nThis is evident in the table below, which shows the improvement of AC-LORA over the mixing of all Flan LoRAs.\n\n| **FlanV2 / Improvement over Mixing All** | **Metric**  | **Mean** | **Lower Bound** | **Upper Bound** |\n|:----------------------------------------:|-------------|:--------:|:---------------:|:---------------:|\n|              struct to text              |   Rouge-1   |   34.64  |      31.95      |      37.19      |\n|              struct to text              |   Rouge-2   |   26.94  |      23.97      |      30.17      |\n|              struct to text              |   Rouge-l   |   30.64  |      28.02      |      33.64      |\n|                commonsense               | Exact Match |   65.5   |       58.5      |       72.5      |\n|                 sentiment                | Exact Match |   90.0   |       85.5      |       94.0      |\n|               reading comp.              | Exact Match |   54.6   |      49.32      |       60.0      |\n|              closed_book QA              | Exact Match |   36.5   |       30.0      |       43.0      |\n|                coreference               | Exact Match |   54.0   |       44.0      |       64.0      |\n|         read.comp.w: commonsense         | Exact Match |   59.0   |      49.97      |       68.0      |\n|                paraphrase                | Exact Match |   63.0   |      56.00      |       69.5      |\n|                    nli                   | Exact Match |   68.2   |      64.44      |      72.32      |\n|                translation               |     Bleu    |    9.7   |       8.22      |      11.43      |\n\n\n### **Is AC-LoRA practical when there are hundreds of relevant documents?**\n\nThe number of relevant LoRAs does not affect the effectiveness of AC-LoRA. We can summarize the effects into 2 scenarios.\n- *All (100s) relevant documents are from the same permission domain (e.g., one LoRA)* - in that case, we choose to assign the average similarity score of the documents to that specific LoRA.\n- *The documents are spread over many permission domains*. In that case, we assign the average similarity score of the relevant document to each permission zone, corresponding to its respective LoRA.\nIn both cases, AC-LoRA only calculates the similarity score for the permission domain if the user has access rights to it.\n\nTherefore, unlike RAG, AC-LoRA does not suffer from lower performance (due to increased context size) when retrieving many relevant documents.\nHowever, there could be a *rare* scenario where a query from a very high-privilege user (e.g., a CEO) triggers loading many LoRAs together. In such a case, the GPU memory will be a bottleneck. This could be mitigated by using a suitable parallelization strategy, such as pipeline parallelization.\n\n*Action point: We will revise the writing to clarify this further.*\n\n\n### **Incremental improvement**\n\nWhile it is correct that LoRA mixing is a well-known topic and covered in multiple previous works (see table 1), we want to point out that mixing LoRAs is *not* the primary contribution of this paper. Existing LoRA mixing works focus on mixing task-based LoRAs and composing new tasks from the mixed LoRAs. However, unlike AC-LoRA, they do not address combining knowledge from different LoRAs. The premise of AC-LoRA depends on whether it is feasible to inject new (and possibly disjoint) knowledge into LoRAs. In existing works, there are limited results regarding the usefulness of LoRAs in injecting knowledge. We show that this is indeed possible. Therefore, by mixing LoRAs, we can effectively create responses from multiple permission domains. \nWe agree that AC-LoRA is based on observations and techniques from existing work. However, AC-LoRA’s approach to using *training-free* LoRA mixing for *access control* is unique for *multi-modal models* and has not been covered in the literature. Additionally, AC-LoRA provides a complete *end-to-end proof-of-concept* with open-source models and datasets, demonstrating the practicality of such a system.\n\nThe selection baseline in Table 2 shows the results with the top retrieved LoRA by LoRARetriver [1]. This does not include any mixing. It is competitive as the Flanv2 dataset only requires exactly one LoRA per task.\n\n[1] LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild, Zhao et al.\n\n*Action point: We will improve the discussion about the baseline in the evaluation section and motivate the experiments in the section “Isolated LoRA Fine-tuning and Knowledge Injection” better.*\n \n\n### **Small typo and format:**\n\t\n- **Table 2: What is meant by IID and OOD?**\n\nWe use the same terminology from LoRARetriever [1], where \"IID\" signifies that the method can access any LoRA for every test sample, encompassing the LoRA specific to the sample’s task. \"OOD\" indicates that for each test sample, they mask the LoRA associated with its specific task during the retrieval phase.\n\n*Action point: We will add this to the table caption.*\n\n\n[1] LoraRetriever: Input-Aware LoRA Retrieval and Composition for Mixed Tasks in the Wild, Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou \n\n\n- **Consistent naming**\n\n*Action point: We will revise the paper and ensure that all terminologies are consistent.*\n\n- **Q is not defined before**\n\nLine 206: $\\mathcal{Q}$ is the user query.\n> for a query $\\mathcal{Q}$ the output for each LoRA in each layer …\n\n - **Typo Fig 5 caption**\n\n*Action point: Yes, will update accordingly.*\n\nWe hope we addressed all your questions and would be happy to continue the discussion to clarify any outstanding concerns."}]

Author rebuttal:

Meta-review: - This paper proposes an access-control variant of LoRA where different LoRAs are trained on different permissioned datasets and used by matching the user's prompt and permission.
- Overall, it is a well-motivated and professionally written paper with good design and sufficient experiments. 
- Reviews are mostly positive, with only one borderline reject (that does not include strong arguments to reject the paper, and their concerns are mostly addressed by the rebuttal). I would ask the authors to include the additional experiments and analysis in the next paper version.
Decision: Accept (poster)