When Does LoRA Reuse Work? Theoretical Limits and Mechanisms for Recycling LoRAs Without Data Access
Authors: Mei-Yen Chen
Thi Thu Uyen Hoang
Michael Hahn
M. Saquib Sarfraz
Venue: Accepted by TMLR ()

Abstract: Reusing low-rank adapters (LoRAs) by merging or routing is a common strategy for adapting
large language models to new tasks, especially when training data is unavailable but many
fine-tuned LoRAs are accessible. While the availability of publicly shared LoRA weights has
inspired new algorithms for composing them to solve new tasks, recent findings highlight
limitations in LoRA’s ability to integrate new knowledge. This work investigates when LoRA
reuse can be successful for compositional factual and reasoning tasks. Through theoretical
analysis in a simplified setup and experiments on a controlled synthetic two-hop reasoning task
with extensions to math word problems, cross-lingual code generation, and history/geography
QA, we show that data-agnostic methods, such as parameter averaging and dynamic selection,
often fail to combine knowledge from logically disjoint fine-tuning datasets. This challenge is
particularly pronounced when the relevant knowledge is underrepresented during pretraining.
However, reuse can succeed when fine-tuning datasets share solution templates, such as
reasoning patterns or reusable code, which serve as bridges among tasks. Our results suggest
that LoRA reuse relies more on shallow pattern matching than on logical integration of
existing knowledge. This mechanism-based perspective offers practical guidance for curating
datasets and designing systems that enable LoRA reuse to overcome data-access limitations.
Findings indicate that future research should focus on the mechanisms enabling effective
adapter reuse rather than solely on developing new reuse algorithms.

Published in Transactions on Machine Learning Research (02/2026)
When Does LoRA Reuse Work? Theoretical Limits and Mechanisms for Recycling LoRAs Without Data Access
Mei- Yen Chen +2 Thi Thu Uyen Hoang +2
Michael Hahn
M. Saquib Sarfraz' 2
Saarland University
Mercedes-Benz Tech Innovation GmbH
Reviewed OpenReview:
Abstract
Reusing low-rank adapters (LoRAs) by merging or routing is a common strategy for adapting large language models to new tasks, especially when training data is unavailable but many fine-tuned LoRAs are accessible. While the availability of publicly shared LoRA weights has inspired new algorithms for composing them to solve new tasks, recent findings highlight limitations in LoRA's 's ability to integrate new knowledge. This work investigates when LoRA reuse can be successful for compositional factual and reasoning tasks. Through thcoretical analysis in a simplified setup and experiments on a controlled synthetic two-hop reasoning task with extensions to math word problems, cross- lingual code generation, and history/ / geography QA, we show that data-agnostic methods, such as parameter averaging and dynamic selection, often fail to combine knowledge from logically disjoint fine-tuning datasets. This challenge is particularly pronounced when the relevant knowledge is underrepresented during pretraining. However, reuse can succeed when fine-tuning datasets share solution templates, such as reasoning patterns reusable code, which serve bridges among tasks. Our results suggest that LoRA reuse relies more on shallow pattern matching than logical integration of existing knowledge. This mechanism-based perspective offers practical guidance for curating datasets and designing systems that enable LoRA reuse to overcome data-access limitations. Findings indicate that future research should focus on the mechanisms enabling effective adapter reuse rather than solely on developing new reuse algorithms.
1 Introduction
Many real-world applications require large language models to integrate scattered information and infer logical answers to novel questions. For instance, an AI assistant supporting human resource spocialists in determining an employee's tax rate must combine information about the employee's marital status and the spouse's residency, as this affects the application of tax law. Such information often originates from scarce data sources or resides in separate systems where regulatory constraints limit direct access. Hence, compositional generalization, the ability of a model to create new combinations of known elements, is essential for the quality of such applications.
As pretraining and fine-tuning become standard practices in Large Language Model (LLM) development, reusing shared model weights from foundation models and their fine-tuned variants has emerged as a practical strategy for generalization data-scarce scarce scenarios. Unlike Federated Learning (McMahan et al. 2017). this so-called model merging (Raffel, 2023) approach passively operates on shared model weights without coordinated training rounds. As parameler-efficent fine-tuning (PEFT) methods gain popularity, combining fine-tuned modules, especially low-rank adapters (LoRAs) (Hu et al.
2021 has emerged data-free
alternative to enhance model capabilities (Bock et al.
2022 Huang ct al.
2024 Zhao et al. 2024b Ostapenko
"Equal contribution
Cooresponding author: mellyeenchen&&tmeereeeeee benz.coommmee

---

Published in Transactions on Machine Learning Research (02/2026)
et al. 2024;
Prabhakar et al. 2024 Zhao et al. 2024a Yadav et al. 2025
The idea is that users
exchange and merge LoRA updates at inference time, just like plug-and-play libraries in software program. Consequently, this idea has sparked a proliferation of novel methods for reusing fine-tuned LoRA weights for
new tasks (e.g. Huang et al. 2024 Ostapenko et al. 2024 Zhao et al. 2024b Bock et al. 2022 Zhang et al.
2025
While these approaches improve computational and economic efficiency, they are often developed and evaluated under very different experimental setups, cach with its assumptions about system architecture, data availability, usage scenarios, and computational limits. While recent work has addressed such inconsistencies in combining entire fine-tuned foundation models (Tam et al., 2024). the various design choices for merging or routing LoRA modules have only been surveyed (Yadav et al. 2025), leaving many questions unanswered. Furthermore, LLMs gain knowledge through pretraining, while supervised fine-tuning
cnttrructionfoollwwiii Consequently, fine-tuning LLMs
tasks teaches them the style or format for user interaction (Zhou et al.
2023)
with new knowledge often lcads to hallucinations (Gekhman et al.,
2024 Ghosal ct al. 2024). Low-Rank
Adaptations (LoRAs) inherently limited in their expressiveness (Zeng & Lee 2024 and can reduce
chain-of thouhtt (CoT) reasoning abilities (Lobo et al. 2025).
In many real-world scenarios, producing high-quality answers is essential, yet data access is often limited (e.g,, health care, finance, medicine). This raises our research question: how effectively can openly shared LoRA modules be reused to generate valid answers for new tasks without direct access to the corresponding raw data? To address this, we began with a theoretical analysis in a simplified setup that reveals limitations of LoRA reuse. This further supported by empirical findings using LLMs (parameter sizes ranging from 1.5B to 70B) and synthetic tasks that were designed proxies to real-world use cases: 2-hop reasoning to combine personal information and math reasoning in math word tasks. We focus on compositional factual and reasoning tasks in order to cleanly disentangle knowledge in the base model from the effect of composing different LoRAAs We can ensure that correct answers can only be obtained by composing knowledge from the reused LoRAs, and not by retricving knowledge alrcady present in the base model. Together, our methods isolate factors affecting LoRA reuse, highlight challenges, and provide practical data preparation insights for designing effective solutions in these scenarios. Our main findings are follows:
LoRA reuse is ineffective for new tasks regardless of model size or pretraining corpus.
LoRA reuse can be suecessful without direct data access when the common solution templates (e.g., chain-of- of-thought reasoning patterns or reusable code snippets) existed in the fine-tuning datasets as bridges.
Overall, our findings indicate that LoRA reuse relies on shallow pattern matching rather than on logically integrating information across independently trained LoRAs Understanding these mechanisms is essential for designing systems that can effectively reuse adapters and for preparing fine-tuning datasets that support reliable composition. Practitioners should carefully consider the target application when combining existing LoRAs, as curated training data plays a critical role in successful reuse.
In the following sections, we review related work, present theoretical analysis, and report empirical results using a controlled two-hop reasoning sctup to isolate key failure mochanisms. We then validate these insights in more realistic domains, including math word problems, cross-lingual code generation, and history/ /oroorygggggppyyggg QA. Our study thus clarifies when LoRA reuse is effective without data access and highlights the need for mechanism-drivee research to guide future system design.
2 Related Work
LoRA modules (Hu et al. 2021 have emerged as a privacy-friendly, data-free method for sharing model capabilitics. Users can exchange LoRA updates or merge them at inference time, like plug-and-play librarics in a software program (Beck al. 2022
Huang al. 2024 Zhao et al.] [024b] Ostapenko et al. 2024 2025). However, many methods for reusing LoRAs
Prabhakar et al. 2024 Zhao et. al. 2024a
Yaday et al.
2

---

Published in Transactions on Machine Learning Research (02/2026)
require examples from unseen tasks to estimate merging weights routers, raising questions about how much successful generalization can be attributed to the Ls themselves. This highlights the need for mechanisms that ensure effective LoRA combination without data access.
Weight averaging and routing are two common strategies for reusing publicly shared LoRA weights that were trained with the original PFFT approach (Hu et al. (021). Weight averaging has become a popular approach, motivated by several findings. First, fine-tuned models remain in the same loss basin as pretrained weights (Ncyshabur et al. 2020). The difference between pretrained and fine-tuned model weights task vectors- can further steer model behavior through arithmetic operations [Iharco et al. 2023). Recent algorithms focus resolving merge conflicts (TIE-Merging Yadav et al., 2023 ) randomly pruning redundant parameters (Yu ett al. 2024), ; and estimating weights for averaging LoRAs (e.g., LoRA-hub (Huang et al.) 2024)). However, the mechanisms enabling successful generalization remain unexplored. Routing is another popular alternative, inspired by the Mixture of Experts (MoE) architecture. This type of approach treats openly shared LoRA adapters as domain experts and develops algorithms to reuse them in novel domains. However, many mothods require additional data set up the MoE for expert retrieval or router training (Chronopoulou al. 2023 Zhao et al. 2024a Jang et al. 2023). Arrow (Ostapenko et al. 2024) is a notable exception, routing LoRAs directly based on similarity between query tokens and singular values of LoRA experts.
Beyond standard LoRA (Hu et al. 2021 recent work has explored alternative PEFT fine-tuning and system designs to derive compositional strategies. For instance, LoRI (Zhang et al., 2025) reduces interference by freezing projection matrices and applying task-specific masks. Similarly, LoRA Soups (CAT) (Prabhakar et al. 2024) freezes one low-rank matrix during fine-tuning and subsequently merges adapters by concatenating their weights, explicitly leveraging held-out data to train a routing mechanism for composite tasks. LoRA
Lego (Zhao et al. 2024b) decomposes adapters into rank-level units for flexible recombination. Self-MoE (Kag et al. 2024 takes a different route, creating self-specialized experts using synthetic data and dynamic routing for task-specific activation.
What can be recycled from a hub of LoRAs? Despite the surge of novel algorithms, when privacy is paramount, choices remain limited, especially for zero-shot generalization without access to training data. The latent logic in the pretraining corpus or term frequency may play roles in combining LoRAs for zero-shot generalization. Scaling language models has revealed emergent abilitics for zcro-shot reasoning (Wei et al.)
2022
Kojima el al. 2022), suggesting that LLMs learn latent logical knowledge during pretraining. Task vectors demonstrate analogical reasoning through arithmetic operations (Ilarcco et al. 2023], but their effectiveness may depend on term co-occurrence frequency in pretraining data (Merullo et al. 2025). If LoRAs serve as linear approximations of fine-tuned tasks, then term- free--fyy effects from pretraining may constrain their ability to generalize to tasks underrepresented in the pretraining dataset. Another possibility is that observed generalization performance via merging or routing LoRAs reflects superficial pattern matching rather than genuine compositionality. Empirical studies indicate that LLMs rely token-level cues, with small lexical changes affecting reasoning performance (Mirzadeh et al. 2024 Li et al. 2024a). LLMs struggle with latent multi-hop reasoning, relying on explicit prompting to bridge logical gaps Press et al. 2023 Balesni et al. 2025). Synthetic reasoning tasks therefore play a key role in assessing compositional generalization and indicate how effectively LoRA combination transfers to entirely novel tasks.
When does reusing fine-tuned adapters work without training data? This challenge is particularly relevant in domains such as healthcare, finance, and medicine, where data sharing is restricted but accuracy is essential. To ensure practical relevance, our study focuses on standard LoRA (Hu et al. 2021), ), the predominant PEFT method and the default in numerous open-source frameworks and cloud platforms. Our investigation begins with a theoretical analysis of two-hop reasoning that reveals limits of reusing LoRA weights. We then present experiments using synthetic data as proxies for real-world scenarios under data constraints. The experiments were conducted on LLMs with diverse pretraining histories and parameter scales (1.5B 70B) to identify conditions for successful LoRA reuse. Finally, we extend the study to math word problems examine how data augmentation strategies (e.g., chain-of mhiin-o-thhone thought templates, reusable code snippets via multi-agent generation) and layer-targeted fine-tuning can further facilitate success. The next. section details our methodology and experimental design.
3

---

Published in Transactions on Machine Learning Research (02/2026)
3 Theoretical Analysis
Here, we argue theoretically that low-rank adaptation, while it can store new facts in transformers, is unlikely to lead to compositional behavior when combining different LoRAs. We study this by considering the problem of composing knowledge from two LoRAs, where each contains factual knowledge, and their combination is expected to perform two-hop reasoning (c.g. (ang et al. 2024c Balesni et al.) 2025 that roquires both pieces of knowledge. In general, direct theoretical understanding of mulli-layer softmax transformers is very difficult; but many theoretical insights have been obtained by studying simplified setups, especially one-layer models and the limit of very wide models (Hron et al., 2020). We this approach to perform simple analysis of low-rank adaptation for factual knowledge. Our theoretical analysis is grounded in the study of Nichani et al. (2025), who develop thal model of storage and recall of relational facts in transformers, in the simplified setup of a one-layer transformer. One-layer transformers are a popular model in the theoretical literature, being more tractable than multi-layer models while offering useful insight (c.g. Pong et al. 2024
Li et al. 2024b Sanford et al. 2023: Tian el al. 2023 Nichani et al. (2025) Section 4) show that either an MLP or an attention head can perform factual recall. We focus our analysis on factual recall via MLPs, both because they are more expressive than attention-only retrieval, and because interpretability work suggests MLPs are important in factual recall (Geva al.
2021 Meng et al. 2022 2023). We discuss the detailed
comparison of our setup to that of Nichani et al. 2025 in Appendix A.1.2
Adapting an MLP performing factual recall with LoRA on a one-hop prompt can change individual facts such as setting Tı (I1) to Tg. However, importantly, combining LoRAs adapting two relations on such an ML. will not result in compositional behavior, as we show below. As a consequence, a simplified one-layer transformer will not be able to show compositional behavior when combining LoRAs. While the theorctical model applies to simplified one-layer setting, our empirical results (Section 4.1 confirm that this conclusion also holds for real- world LLMs, even though they have many layers.
3.1 Setup
Entitics, Facts, and Prompts We consider a set X' of entities, and set of binary relations r C X X (e.g., X married to Y; Y lives in Z, etc). We assume that each is a partial function, i.c., for each x, there is at most one y satisfying (x, y) r; we write y y= r()). Whereas Nichani et al. (2025) relied the assumption that each relation maps to disjoint output space, we avoid this assumption. We assume that the model operates on the following types of prompts
1. One-Hop: X REL (where X represents an entity x E X' and REL represents a relation r E R, with expected completion: Y, where y = r(x) (e.g., "the spouse of X is Y").
2. Two-Hop: X REL1 REL2 (where REL1, REL2 represent relations ri, E R), with expected completion: Y, where y = (x))) (e.g. "the place of birth of the spouse of X is Y").
Simple Transformer Model We consider vocabulary consisting of relations T1, P2, and entities $1, with token embeddings Ca, E Rdd We will write E E RR for the matrix holding all token embeddings. We assume a single softmax attention head with K, K,Q.V V e igdad matrices, and a ReLU MLP with hidden dimension m given by matrices U E R' W E x mapping a vector x to W. &RELU (Ux). We assume strict positional masking without attention to the current position (e.g. Yang et al.] 2024b Li & Cotterell, 2025), and a residual connection. We do not require positional encodings. We assume that the next-token prediction is provided by W as one-hot encoding of the target entity, iz; omitting softmax for simplicity. Our aim is to showcase limitations in composition, not in storage of knowledge itself; hence, we allow the model a width d substantially larger than [2)), [R]. In order to give the MLP as much capacity needed, we allow m to be arbitrarily large.
Nichani et al. 2025 took E to be randomly initialized and not trained. We follow this assumption, and additionally take U. V. K, Q to remain untrained. Overall, we assume that U, V, K, Q, E matrices are randomly initialized, all with entries from (0 1). We focus consideration of training W. This represents a random
features setup (e.g. Rahimi & Recht, 2008 Ghosal et al. 2022 Dirksen et al. 2022 In this setup, softmax
4

---

Published in Transactions on Machine Learning Research (02/2026)
attention is close to uniform; we will take it to be exactly uniform for simplicity. We further remove the softmax over the vocabulary in the output. We emphasize that training only W is substantial simplification, which makes theoretical analysis tractable. The training analysis in Nichani et al. (2025) makes different set of simplifications (lincar MLP and linear attention); we opt to keep a nonlinear MLP due to its greater expressiveness al. representing relational knowledge, al the cost of assuming a random features setup. In both cases, simplifications amount to keeping only one nonlinearity (RcLU in our case; softmax over the vocabulary in Nichani et al. (2025) and focus trainable parameters on one side of the nonlinearity (above it in our case, below it in Nichani et al. (2025)). We provide further discussion in Appendix A.1.2
We will examine the situation where the basc model already performs correctly for the given set of relations R, and W is then adapted to reflect edits to such facts.
We focus LoRA on W, in agreement with our experimental finding in Section 4. that applying to MLPs can be sufficient to get most of the gains. We consider updates AW AB" with A E R B R where s is small, subject to an L2 penalty ||]] + |BB| We particularly consider one-rank updates, AW - pa" where p R .q R |x|,
We note that this setup simplifies many aspects of transformers: there is only one layer and one head, and training focuses on the (linearized) output. We also remove the softmax over the vocabulary in the output. Our setup is designed to be simplest possible setup in which a nontrivial statement about LoRA's ability to learn and combine abilities can be made.
3.2 Results
The correct responses to all 1-hop and 2-hop relations can jointly be coded into W when d and m are sufficiently large, due to the separation ability of the random features model (Ghosal el al.] 20022). This analysis is in line with mechanistic studies of factual recall suggesting MLPs act as key-value storage (Geva
ct al. )021).
Changing fact y = r(z) requires changing the output of the MLP on the subspace spanned by the entity and relation. When the update affects only a single fact, L2 regularization ensures that it has a simple and interpretable closed form:
Proposition 1. A rank-one update to W changing the output on a prompt X REL from r(x) to f(z) must have the form:
1
AW N (UVex ex + U
(17(2) ( Vex+ + U EREL)
(1)
Here, Vex reflects the output of attention to the precoding position, and EREL reflects the residual connection.
This is similar to the RoME update (Meng el. al.) 2022). Intuitively, based on the idea that MLPs act key-value storage, the LoRA update AW AB" specifically addresses the encoding of the prompt X REL in the B matrix, and the changed output in the A matrix. The proof is in Appendix A.1
Now consider a two-hop prompt X REL1 REL2, intended to denote the composition of the two relations. Given sufficient width, any set of such two-hop facts can be encoded in W. However, as we next show, adding two LoRAs modifying two relations (AWr AWrgroiz) will not unlock compositional behavior on the new facts:
Theorem 2. Assume LoRAs AWr--fi AWrz-r2 are created to adopt two single facts for Tı, T2. Summing these adapters will not result in correct results for composition of the two relations r1, T2,
The formal proof is in Appendix A.1 The reasoning is follows. As shown in Proposition [ the two LoRAs specifically modify the MIP output on the subspaces inhabited by the activations computed on the two one-hop prompts. When the model encounters a two-hop prompt, the activations will partly overlap with the subspaces for both one-hop prompts, and the adapters will lead the model to output not the composition r2(ri (z)), but a lincar mixture of two relevant entitios. A natural question is whether some of the routing or weighting methods proposed in the literature resolve this; it turns out that the argument extends to those: For instance, weighted averaging of the two adapters (e.g. Prabhakar et al. 2024 Ostapenko et al. 2024 will
5

---

Published in Transactions on Machine Learning Research (02/2026)
still fail to perform compositionally when several facts are updated. Formally (see Appendix A.1.1 for the proof),
Theorem 3. The some result holds for CAT (Prabhaaar et al. 2024), linear merging of LoRAs Yadav et al.
2023
Yu et ol. 2024 Huang et al. 2024), Arrow routing (Ostapenko et al. 2024).
Yet another approach might be to combine a larger library of LoRAs where some have been trained on 2-hop examples from other task pairs. One might hope that this would prime the model towards compositional behavior; however, the reasoning above still applies, and suggests that reusing LoRAs would still fail to behave compositionally (Appendix A.1.1
One limitation of our theoretical analysis is that (in line with Nichani et al. 2025 it applies to a single-layer transformer; our experiments test applicability of the conclusions to LLMs across scales.
4 Experiments
The following experiments aim to apply the above theoretical analysis in full-scale LLMs to determine under what conditions combining LoRAs enables LLMs to perform new tasks that requires logical integration of information segmented among fine-tuning datasets. We focus on two data-agnostic routing methods, Uniform averaging and Arrow routing (Ostapenko et al. 2024). which operate directly on LoRA weights (see Appendix A.2.2). We designed two synthetic tasks, two-hop reasoning and math word problems, as proxies for real-world cases that require combining personal information mathematical reasoning under data-scarce conditions. This helps identify key preconditions for effective LoRA routing: entity familiarity, domain-specific pretraining, and common solution templates that link segmented information across fine-tuned adapters. We further examine how these effects hold in different routing strategics and scale across base model sizes ranging from 1.5B to 70B parameters. To rigorously validate the generalizability of these findings, we extend our analysis to two realistic domains: cross-lingual code generation and history / /geography question answering (Appendices A.5 and A.6).
4.1 Two-Hop generalization
We investigate whether combining two LoRAAs enables compositional reasoning. Building on our theorctical analysis and inspired by Balesni et al. (2025). we design a two-hop reasoning task requiring composition across linguistic variations while controlling for base model knowledge. The dataset uses a fixed structure:
1. First Hop (4 - B), identifying the spouse of a given entity 4,
2. Second Hop (B - C*), identifying the residence of B,
with the goal of inferring A C' (identifying the residence of the spouse of a given entity). This setup closely follows our Theorem which suggests that LoRAs trained one of the two hops each would, if combined, not unlock the indirect relationship.
We conduct three datasets varying the nature of entity names and locations while ensuring that the relational facts remain synthetic: F (fake names, fake locations), where both entities and locations are synthetic (e.g., (Zint, Frosk, Narik)); H (fake names, real locations), where names are synthetic but locations are real (e.g., Zintt, Frosk, London)); and (real names, real locations), where both names and locations are real, but relationships are deliberately shuffled to remain false (e.g., (Barack Obama, Camila Alves, London)). We refer to the first-hop (A + B), second-hop (B + C), and the two-hop (A + C) subsets of each dataset F1, F2, F12, H1, H2, H12, and R1, R2, R12, respectively (sce Table 5 and Appendix 1.2. for examples and details).
Based on ablation studies (Table and in Appendix A.2.2) we focused on fine-tuning only the MLP layers of the following base models: 1 Qwee2.557BBInnttin and Qwen2.5-14BBInsttre Distilll-Qeen Qwen-14B 14B, and DeepSeek-
(Qwen-team 2025), as well as DeepSeek- Di-ttillll Meiisstll-Qeen-Tee Qwen-7B, DeepSeck- BaaaSScck-----ea R1--Ditilll
RI-Distill-LLaMA. (DeepSeek-Al-team 2025
6

---

Published in Transactions on Machine Learning Research (02/2026)
4.1.1 Impact of base model and familiarity
Experiment setup For each datasct (F, H, R), we train four LoRA adapters (experts) LoRA 1, LoRA 2, LoRA 3 (Oracle Expert); and LoRA (Mixed Two-Hop Expert) on relation A + B, B C, A C, and mixed data (A + B, B + C), respectively. From these experts, we construct two libraries: 2-combination library which includes LoRA 1 and LoRA 2; and 3-combination library which includes LoRA 1, LoRA 2, and LoRA 3. We evaluate the model's ability to generalize and infer A + C relationships. We use Chain-of- Thouught (CoT) prompting during testing for the 3-combination library and the 2-combination library.
Results and analysis As shown in Figure 1 performance on the 3-combination library for the H dataset improves with model size, while the 2-combination library remains consistently poor. Figure [ shows that R outperforms H, both far exceeding F. Notably, with only - B and B + C adapters (2-combination library), accuracy stays below 10%, supporting Theorem 2 that composing knowledge across separate LoRAs C' is covered (3-combination library), routing does not always
is inherently challenging. Even when succeed, in particular in smaller models and the presence of of unfamiliar entities, such fake names cities in the F dataset. These trends hold across datasets and model families (see Table in Appendix A.2.2
F. Names,
Names,
Ascuntty (1)
Accintt (%)
60
20
20
b-33
5-76
S-145
DuepSinds-T
DAcpBenk 94b
S-oortrradio Tormy
2xterbiration lartey
Experi
Moxd Expert
(x)
Figure 1: LoRA reuse is ineffective across model sizes and pretraining corpora, as shown by performance comparisons between a library of multiple LoRA adapters and individually fine-tuned LoRA experts on two-hop datasets. (a) Comparison of library-level and expert-level performance on the test set of H across different base models. (b) Impact of entity familiarity on the performance of LoRA combination methods, using Qwen2.5- 1...5-11B-Innstrcee as the base model, evaluatod on three test sets: F, H, and R. Across setups and model sizes, performance the 2-combination library is poor; including expert trained on the target task (A C relationship) is necessary.
4.1.2 Composition requires close match between testing and training prompts
So far, we found that synthesizing two-hop reasoning from two LoRAs is difficult, in agreement with our theoretical predictions. What strategies could enable composition? While a substantial amount of work has found shan--of-thoughte to support reasoning (including the 2-hop setup when employing full full-parameter finetuning, (Balesní al. 2025 ), we found that composition from the two performs poorly even with chain-of-thought prompting. Our theoretical analysis suggests that, as LoRA adapters rely on targeting specific low-dimensions subspaces, compositional behavior only be unlocked when the target prompts show close formal match to prompts which the LoRAs were trained. CoT prompting might thus be insufficient as the form of the targets mismatches the onc-hop training examples of the LoRAs. However, mixing in CoT examples into the LoRA training data might be sufficient. In this section, we test is possible to enable composition by including CoT templates the training data, and how close the match reasoning patterns needs to be between finctuning and testing datasets. We specifically define the following bridge technique, and design series of experiments to determine how closely the target task must be represented in the LoRA fine-tuning datasets to enable compositional behavior (Figure ?? left.
B and B - C experts
7

---

Published in Transactions on Machine Learning Research (02/2026)
panel). The idea is that the finetuning dataset additionally includes examples of the targeted reasoning pattern. Via ablations, we test which aspects of the targeted reasoning pattern needs to be present.
Experimental setup We design two bridge variants over the F and R datasets (Figure 2 left panel). The Fake Bridge (By) is constructed by concatenating the F1 (A + B), F2 (B + C)2 and F12 CoT (CoT- formatted A - C)) subsets. The Real Bridge (BR) follows the same structure but uses the corresponding subsets R R2, R11 CoT) from the R dataset, which contains real names and locations (see Table and Figure in the Appendix A.2.1 for examples and details). We fine-tune adapters on the union of direct-hop examples and bridge dataset. In the first configuration, Setup 1, models
trained by mixing fake data subsets (Ai) with the Real Bridge set (BR): LoRA 1 is trained on Fi + BR, and LoRA 2 on F2 + BR, with evaluation on the held-out subset F12. In the second configuration, Setup 2, models are trained by mixing real data subsets (R.) with the Fake Bridge set (Bp): LoRA 1 is trained on Rı + BF, and LoRA 2 on R2 + Bp, with ovaluation on the held-out subset R12-
LoRA Expert
Hop-1 Prefix (8)) Who Barack Obaam'' 300
Hop-1 Complatien (Rg) Barook
Bridge
Nancy Shevell.
LoRA Expert
Setp
Seip
Hop-2 Prefix (8,)
Complation (R2)
city was
Shnvesh Shavel
Bridge
bom Telay
DeepSeek 148
Prefix (F,)
Hop-1 Completion (Fi) Narki's spouse is Fross
Bridge
Who Narki's spouss??
DeepSeek-78
Hop-2 (F2)
Hop-2 Complation (F2)
Owen2 148
in whtch city treek bomm?
Zint.
Owen2 5-7B
h-hop (Fsz)
Hap-1 Completion (Fr)
Completion (P2) Froui was borm in Zint.
What city Naric's
Frosi spouso of Norik.
ppouss bym
20
Test Items
Accurady (%6
2-hop Prefix (Rtp)
Hop-1 Completion (R,)
Hop-2 Complation (Rt)
What city Barack
Barack Obbrma's spouse
Nancy Shavell Wis Tolyyo.
Obarn''s bnm
Nancy Shevell.
Figure 2: (left) In the bridge sctup, both LoRA experts trained not only of the two hops, but also examples (disjoint from those needed in testing) of both hops, and chain-of-thought two-hop reasoning, (right) Performance comparison of two setups across different base models: Setup 1 (Real Bridge, BR) adds a bridge using real names and locations to a dataset using fake names and fake locations (F), while Setup 2 (Fake Bridge, Br) reverses these. Each setup uses LoRA and LoRA 2 as the experts in the library. The bridge setup much more successful in Setup 2.
Results and analysis Figure demonstrates that explicitly incorporating the target two-hop reasoning pattern into the LoRA fine-tuning data is crucial for achieving reliable compositional generalization. In Setups and 2, where each adapter is trained on a synthetic direct-hop subset combined with the bridge dataset, Arrow performance improves significantly compared earlier experiments. Additionally, the bridge setup is much more successful in Setup 2, highlighting the importance of entity familiarity for effective generalization.
We conducted set of ablations to analyze which aspects important for the success of this strategy. Incorporating structured reasoning into LoRA finctuning yields only marginal gains unless the finctuning data closely mirror the larget two-hop lask. First, as shown in Table the bridge improves two-hop accuracy only when CoT-formatted + C instances are included during adapter training. Omitting CoT formatting results worse performance (Setups vs. 3). Second, simply including the bridge only one of the two LoRA adapters (Setups 4, 5), or providing only + C prompts without the individual one-hop tasks (Setups 6, 7), results in significantly weaker compositional performance compared to setups where both Fi
8

---

Published in Transactions on Machine Learning Research (02/2026)
(A B) and F2 (B - C) are included alongside CoT bridging. This highlights the importance of exposing the model to each subtask. Third, we found that relaxing the bridge to use disjoint task pairs still produces nontrivial gains (Setup 8, which uses completely different relations as the bridge dataset: F4: study in and Fs: child_ of, see Table 5 in the Appendix A.2.1 for examples); suggesting that exact task-pair matching is less critical so long as the finetuning set. contains examples reflecting the overall reasoning pattern. A Altoget these results confirm the importance of CoT exemplars and the individual tasks to unlocking generalization of Arrow routing, even if the examplars are semantically different from the target task.
Aside from combinations of two or three LoRAs, further tested what happens when increasing the number of tasks present in the collection of LoRAs, including both various one-hop tasks, and also various two-hop tasks. Even in this case, we found that composition was very difficult, again in agreement with our theoretical predictions (Analysis in Appendix A.3
Table 1: Ablations for the bridge training setups. We compare the 2-cobbination libraries trained on just the two hops (0), the full bridge (2), with various strategics interpolating between these, such as providing bridge in only one expert (4 and 5), or omitting the CoT template from the bridge (3). The full bridge attains highest performance, and versions not including a bridge CoT in both of the two experts show poor
performance (0, 3, 4, 5).
Setup
LoRA
LoRA 2
JD-Instruct
Qwen2.5-
7B- Instrotttt
Quem2.5-
Quen2.55 14B- Instrrcctt
DeepSonk-RI- Dustil--wenn-B
DeepSookkRR Distill-Qweo-dBB
DempSook-RI Distill-Llans TOB
R.
Rgg
1%
3.9%
7.8%
8.8%
5.9%
5.9%
Fi + Ba
Ri
Fo OcT (2s)
F.
Rs
F. Pi, CoT (Bp)
73.6%
85.3%
94.1%
80.2%
95.1%
90.3%
F.+
Ri
Fia
Fia
1%
2.0%
3.9%
3.9%
3.9%
Ri
4
F F2
Pi, OCT (8,)
Rs
9.8%
8.8%
11.8%
8.8%
12.7%
10.2%
Ri
Fj
Fb Fia CoT (Bp)
10.8%
35.3%
27.3%
25.0%
22.5%
26.19%
Fi CoT
Ri
Rs
40.2%
61.8%
63.7%
75.5%
72.0%
87.3%
7
Ri
Fl
Fix GAT
Rg
Rs+ Fis
OST
27.5%
75.5%
79.4%
74.3%
78.9%
89.99
16
FL + Fut Pa OcT
P. Ps + Psa Col
43.1%
82.4%
02.295
81.67
83.5%
87.0%%
We also investigate on how the 2-hop reasoning gap can be closed without reasoning templates as bridge but alternative merging methods extra training when data access would be allowed. To achieve this, we utilize LoRA 1 and LoRA 2 in Setup 0 shown in the Table 1 and apply uniform averaging (Huang 2024 Ostapenko et al. 2024), TIES-Merging (Yadav et al. 2023 and LoRA Soups (CAT) (Prabhakar
et al. et al. openly shared fino-tuned LoRA weights. TIES-Merging applics a structured procedurc trimmiing low-impact parameters, resolving sign conflicts, and selectively merging to reduce interference when combining multiple model weights. CAT altered the original LoRA (Hu et al. 2021 by only allowed one low-ranked matrices to be fine-tuned, concatenation multiple LoRA model weights, and requires access to held-out data for routing training to composite multiple tasks. Table 2 CAT achieves higher accuracy than methods operating without data access. However, its performance remains below setups that route openly shared LoRAs originally fine-tuned on well-designed datasets (Table | setup 2,6,8). We additionally evaluated LoRI Zhang et al. (2025) on our two-hop setting. LoRI modifies the LoRA parameterization to reduce interference when combining multiple adapters, and has been shown to improve multi-task adapter merging. As shown in Table 2 all LoRI variants (LoRI-D/S with concatenated or linear merging) achieve accuracies close to chance and clearly underperform CAT. In LoRI's linear merging, single adapter is obtained by taking fixed linear combination of the task adapters without using any two-hop data; our Uniform baseline is simply the special case where all tasks receive the same fixed weight. In LoRI's concatenated merging, the task-specific adapters are instead stacked into a larger block-structured adapter so that each task mainly occupies its own subspace, but this merge is still fixed and data-free.
2024 on them as comparison baselines because they represent widely discussed strategies for merging
shows that when held-out data is available for routing training,
Overall, these results support our conclusion that (i) direct composition of knowledge from different LoRAs is very difficult, even when using more sophisticated adapter architectures such as LiRI. the LoRA finetuning datasets must contain examples closely matching the target reasoning behavior.
9

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Performance of
routing and merging methods using a 2-combination adapter library on Qwen2.5-7B evaluated on the dataset.
Adaptation Routing / Merging
Accuracy
LoRA
Uniform
3%
LoRA
Arrow
3.9%
LoRA
TIES
3.8%
LoRA
CAT
21%
LoRI-D
Concat
13%
LoRI-D
Lincar
1%
LoRI-S
Concat
7%
LoRI-S
Linear
3%
4.2 Generalization from Easy to Hard Math Word Problems
To evaluate whether our findings hold in more realistic settings, use the GSM-Symbolie benchmark (Mirzadeh et al. 2024), which enables controlled assessment of reasoning robustness in math across well- defined difficulty levels. Each LoRA expert was fine-tuned on GSM-Symbolic (original) and GSM-P1 (with one added clause) individually, before being combined for evaluation on GSM-P2 (which adds another clause). We compare general-purpose and math-specialized models to assess the impact of pretraining. Similar exposing LoRAs to solutions closely resembling the target task, we also tested whether fine-tuning with reusable Markdown and Python code (Suzgun et al., 2025) would improve generalization GSM-P2. Detailed experimental design, fine-tuning, and evaluation procedures can be found in Appendix Section A.4.1
Limitations of LoRA reuse. The effectiveness LoRA routing is highly dependent on the basc model's domain-specific pretraining history. To start, we replicated the findings of Mirzadeh et al. (2024), which show that large language models (L.LMs)) lack robustness in mathematical reasoning (see Appendix Section A..42 Table 13 Routing methods such as Uniform and Arrow provided modest improvements for the general-purposs model, but often degraded performance in math-specialized models like Qwen2.5- Math Instrucistt regardless of model size (Table 3 Among these, Uniform consistently outperformed Arrow. Echoing prior work showing that 8-shot GSM8K in-context examples do not improve performance on GSM-P2 (Mirzadeh et al. 2024), we further observed that combining these examples with LoRA routing actually worsened results. For example, in the Qwen2.5-Math- IB-Instruction model, Arrow routing with in-context examples reduced GSM-P2 accuracy from 0.27 to 0.06 (soe Appendix Section A.4.2 Table 14 for details).
The performance drop observed after LoRA routing may stem from a mismatch between the fine-tuning data and the base model's capabilities. Qwen2.5- Www.----Ma-----ssseeee is designed to solve problems using Markdown and Python code, while the GSM-Symbolic benchmarks provide only natural language Shain-offThhugee (CoT) solutions. As result, routing LoRAs fine-tuned on this dataset may suppress the model's tool- integrated reasoning abilities and lead to an increase in calculation errors. Our error analysis follows the definitions and procedures outlined by Zhong et al. 2025 See Appendix Section A.4.2 and Table for details.
Table 3: Accuracy comparison on zero-shot GSM-P2 after routing LoRA experts individually fine-tuned on GSM-Symbolic and GSM-P1.
LoRA Routing Methods
Qwen2.5-1.5BIInst
Qwen22.5-Math--555-nnn
Qwen2.55-Mathh-T-Inss
Base model only
5%
47%
68%
Uniform
10%
24%
34%
Arrow
9%
19%
27%
How can programming language bridge the generalization gap? This experiment extends our bridge results in two-hop reasoning (Table 1] to math word problems. The experimental design is motivated
10

---

Published in Transactions on Machine Learning Research (02/2026)
by recent findings, Dynamic Cheatshoet, which demonstrate that encouraging language models to retain and apply reusable intermediate solutions during inference significantly improves their performance on math problems (Suzgun et al. 2025). We extend this idea using the GSM-Symbolic benchmark (Mirzadeh ct al. 2024), where generalization from casicr to harder problem variants requires understanding the full computational graph (Appendix Figure 5]. In the previous setting, each LoRA is fine-tuned on partial Appendix Figure However, routing these LoRAs alone does not suffice solve the more complex P2 variant, which involves the complete computational graph (blue subgraph in Appendix Figure (). We hypothesize that reusable Markdown and Python solutions can bridge partial representations and enhance compositional generalization through LoRA routing. To test this, we implemented two agent-based actor-critic workflows (Wu et al. 2024) to automatically generate reusable code snippets as fine-tuning data (See Appendix A..411 for implementation details). Table | demonstrate modest improvements in solving the complex P2 problems via routing LoRAs fine-tuned with these reusable code snippets. Such improvement is clearer in smaller model --e..---MM--------meee
solutions corresponding to subsets of reasoning steps (e.g. the black or orange subgraphs
when fine-tuning largeted the MLP layers. This result aligns with our theoretical analysis and findings from the two-hop reasoning task. It highlights the importance of the mechanistic understanding of how to reuse LoRA adapters effectively for guiding data generation, and shows that such reuse works best when target tasks are clearly defined in advance.
Table 4: Enhancing easy-to-hard generalization by leveraging Tool- Reasoning (TIR) prompt and fine-tuning with reusable code.
Base model
Fine-tuned Madules
Routing Methods
LoSMSGSM-Symb) and LoRA(GSM-Syynnn
Base Model Only
attention
Uniform
14%
18%
Qwen2.5. 20et2 Mathi M. Inst
Arrow
14%
MLP
Uniform
20%
Arrow
13%
attention
Uniform
40%
47%
Arrow
24%
Qweu2.5-Mar 2-t.5---M-t---- -B-Iast
MLP
Uniform
39%
Arrow
37%
5 Discussion
Our findings indicate that combining LoRAs is ineffective for new tasks unless common solutions to those tasks are already represented in the fine-tuning datasets. Although alternative PEFT methods may offer better compositional results, our study focuses on merging adapters without access to further training data. Allowing training would enable reconfiguration of adapters and task setups, making the reuse problem trivial. While a comprehensive empirical comparison is beyond our scope, we discuss these methods using our theoretical and empirical insights to guide future directions. For instance, LoRT (Zhang et al.] 2025) addresses cross-task inference by combining random projections with task-specific masks, potentially enabling better adapter routing. However, positive results for compositional reasoning have not been reported, and our theoretical analysis suggests that it remains challenging. Similarly, LoRA Lego (Zhao et al. 2024b) formalizes low-rank updates as independent units and clusters these into new adapters to reduce interference, though it has not been shown to enable compositional reasoning. Self-MoE (Kang et al., (022)) constructs experts based on self- self- reffreeeeated specialization training data and a trained router, but it remains unclear to what extent this method can enable compositional combinations of different abilities. FL.iX (Sun et al., 2024h learns different. low-rank updates for various task or dataset features, and CS-ReFT (Sun et al., 20244) learns orthonormal lask-specific adapters. Despite these innovations, none have demonstrated effective compositional combination of skills, consistent with our theoretical analysis indicating inherent limitations.
Recent work (Prabhakar et al.) 2024) has proposed LoRA concatenation as an effective method for composing skills to solve challenging math word problems, such as those in GSM-Hard (Gao et al. 2023). We acknowledge the significance of these findings, particularly their demonstration that decomposing skills into reusable LoRAs and estimating combination weights can enhance performance, provided that additional task-spocific data and knowledge are available. However, our work takes a different perspective. Unlike GSM-Hard Gao et al. 2023), which primarily modifies numerical ranges while preserving the question format of the
11

---

Published in Transactions on Machine Learning Research (02/2026)
original GSM8K problems, GSM-Symbolic-22 (Mirzadeh et al. 2024 presents more realistic and difficult compositional generalization challenges. It alters both the question format and the structural complexity of math problems into entirely unseen forms. Our theoretical analysis shows limitations (Appendix (A.II)) supported by empirical results showing that training provides little benefit in a 2-hop reasoning sctting (Appendix Table 2. This suggests that the benefits of such approaches may not extend to more challenging generalization tasks like GSM-Symbolic. While skill composition remains important, our results highlight a key limitation of LoRA routing approaches: their effectiveness often depends on prior knowledge or training data of the downstream tasks, which may not be viable in practice.
A critical impediment to LoRA reuse identified in our study is negative transfer, wherein merging adapters can suppress the base model's ' latent capabilities or degrade a dominant skill. For example, as shown in Table 15 (Appendix A.4), the base model uses code-based responses in roughly 12% of cases, but the merged models (Uniform/Arro Arrow) reduce this to 0%, reverting entirely to natural language. This suggests that merging may suppress specialized "tool use" capabilitics, biasing the model back toward the dominant modality (text) of the individual LoRAs. Similarly, in cross-lingual code generation, merging an English-Coding adapter (T1) with an English Spanish Translation adapter (Lg) frequently results in lower performance than using La alone (e.g., 28.8% vs. 36.0% on Phi-2; Appendix A.5 In this setting, the unaligned L₃ adapter interferes with the translation capabilities of L2 when processing Spanish inputs, rather than composing to form a Spanish-coding capability. These observations ions highlight that LoRA merging can introduce negative transfer effects, particularly when component adapters are unaligned or target different modalities, limiting the effective compositional reuse of independently trained adapters.
The bridge data used in our experiments serves diagnostic purpose: delineating the boundaries of naive composition rather than supplying auxiliary supervision. By demonstrating that reuse succeeds only when training data structurally mirrors the target task, these experiments confirm that pure data-free reuse is constrained by prompt alignment. This insight defines the representational overlap required for modularity, suggesting that future efforts should focus on curating bridge-rich datascts to enable interoperability.
The success of bridge datasets implies that the LoRA reuse st rategies tested in this work operate primarily through template exposure priming the model to output in a specific format or style rather than by composing disjoint knowledge bascs. Although our evaluation tasks focused primarily on factual knowledge and logical reasoning, we hypothesize that other applications such as adapting model's style (e-g., adopting a specific persona) or format (e.g., outputting JSON) via reuse may be viable without bridge datasets, as these rely on surface-level templates. However, deep knowledge generalization (e.g., combining disjoint "Legal" and "Medical" LoRAs) likely fails without bridging data. Future research should rigorously test this boundary.
6 Conclusions
Our experiments examined when combining LoRAs enables LLMs to perform new tasks by integrating capabilities from different adapters. In our two-hop reasoning setup, merging LoRAs did not improve performance unless the relevant tasks already appeared in the fine-tuning data. Familiarity with entities aids generalization, but composing knowledge across LoRAs remains difficult, and adding chain-of-thought. templates yields only marginal gains.
In mathematical reasoning, the success of LoRA merging depends strongly on the base model's pretraining. Domain-specific models handle complex tasks more reliably than general-purpose ones. Reusable Python-code datascts can serve as bridges, yet they provide only modest gains while the core merging challenge remains. Our cross-lingual coding experiments also show frequent negative transfer, where one adapter dominates instead of enabling gemuine composition.
Overall, our findings indicate that LoRA reuse mostly supports surface-level pattern matching. It helps with template aligument and stylistic transfer but is ineffective for logical composition without bridging data. Entity and domain familiarity matter, yet routing strategies often hinder performance, especially in smaller models or unfamiliar contexts. Fine-tuning MLP layers offers limited benefits, and composing knowledge across LoRAs remains difficult. These results highlight the need to use synthetic data and theoretical analysis to better understand the limits of based merging and to design systems accordingly.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Contributions
M.-Y.C. conceived the idea, conducted literature review, lod the rescarch direction, designed and conducted the GSM experiments. T.T.U.H. designed and conducted the 2-hop experiments. M.H. conducted the theoretical analysis and supervised 2-hop experiments. All authors discussed the results and contributed to the final manuscript.
References
AnonymousDoe. MBPP-Translated
Translated Mostly Basic Python Problems (MBPP)
https
2024. Accessod: 2025-12-11.
Francis Bach. Learning theory from first principles. MIT press, 2024.
Mikita Balesni, Tomek Korbak, and Owain Evans. The two-hop curse: Lims trained on a-+,, b- fail to learn a-,c, 2025.
Tilman Beck, Bela Bohlender, Christina Viehmann, Vincent Hane, Yanik Adamson, Jaber Khuri, Jonas Brossmann, Jonas Pfeiffer, and Iryna Gurevych. AdapterHub playground: Simple and flexible few-shot learning with adapters. In Valerio Basile, Zornitsa Kozareva, and Sanja Stajner (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 61-75, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi:
Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leaderboard, 2023.
Lucas Caccia, Edoardo Ponti, Zhan Su, Matheus Pereira, Nicolas Roux, and Alessandro Sordoni. Multi-head adapter routing for cross-task generalization, 2023.
Alexandra Chronopoulou, Matthew E. Peters, Alexander Fraser, and Jesse Dodge. Adaptersoup: Weight averaging to improve generalization of pretrained languago models, 2023.
Alexandra Chronopoulou, Jonas Pfeiffer, Joshua Maynez, Xinyi Wang, Sebastian Ruder, and Priyanka Agrawal. Language and task arithmetic with parameter-effiient layers for zero-shot summarization, 2024.
Karl Cobbo, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiihhro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems, 2021.
SrepSeek-Al-teane Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
Sjoerd Dirksen, Martin Genzel, Laurent Jacques, and Alexander Stollenwerk. The separation capacity of random neural networks. Journal of Machine Learning Research, 23(309): 1177) 2022.
Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Ncubig. PAL: Program-aided language models. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 10764-10799. PMLR, 23 29 Jul 2023.
Zorik Gekhman, Gal Yona, Roee Aharoni, Matan Eyal, Amir Feder, Roi Reichart, and Jonathan Herzig. Does fine-tuning lims on new knowledge encourage hallucinations?, 2024.
Mor Geva, Roci Schuster, Jonathan Berant, and Omer Levy. Transformer food-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 2021.
13

---

Published in Transactions on Machine Learning Research (02/2026)
Gaurav Rohit Ghosal, Tatsunori Hashimoto, and Aditi Raghunathan. Understanding finetuning for factual knowledge extraction. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 15540-15558. PMLR, 21-27 Jul 2024.
Promit Ghosal, Srinath Mahankali, and Yihang Sun. Randomly initialized one-layer neural networks make data linearly separable. arXiv preprint or 2rx:222005. 11716, 2022.
Jiri Hron, Yasaman Bahri, Jascha Sohl-Dickstehn and Roman Novak. Infinite attention: Nugp and ntk for deep attention networks. In International Conference on Machine Learning, pp. 1376-4386 PMM.R, 2020.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. CoRR, abs/2106.09685, 2021.
Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lorahub: Efficient cross-task generalization via dynamic lora composition, 2024.
Gabriel Ilarcoo, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic, 2023.
Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Sco. Exploring the benefits of training expert language models over instruction tuning. In Proceedings of the 40th International Conference on Machine Learning, ICML 22... JMLR.org, 2023.
Jummo Kang, Leonid Karlinsky, Hongyin Luo, Zhen Wang. Jacob Hansen, James Glass, David Cox, Rameswar Panda, Rogerio Feris, and Alan Ritter. Self-moe: Towards compositional large language models with self-specialized experts. erXiv preprint arXiv:2406.1203, 2024.
Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Twasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, volume 35, pp. 22199-22213, 2022.
Nicolas Le Roux and Yoshua Bengio. Continuous neural networks. In Artificial Intelligence and Statistics, pp- 404-411 PMLR, 2007.
Jiaoda Li and Ryan Cotterell. Characterizing the expressivity of fixed-precision transformer language models. In The Thirty-nenth Annual Conference on Neural Information Processing Systems, 2025.
Qintong Li, Leyang Cui, Xueliang Zhao, Lingpeng Kong, and Wei Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of iims as mathematical problem solvers, 2024a.
Zihao Li, Yuan Cao, Cheng Gao, Yihan He, Han Liu, Jason Klusowski, Jianqing Fan, and Mengdi Wang. One-layer transformer provably learns nne-nea neighbor in context. Advances in Newral Information Processing Systems, 37:82166 82204, 2024b.
Elita Lobo, Chirag Agarwal, and Himabindu Lakkaraju. On the impact of fine-tuning on chain-of-thought reasoning, 2025.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning, 2023.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication- Efficient Learning of Deep Networks from Decentralized Data. In Aarti Singh and Jerry Zhu (eds.), Proceedings of the 20th International Conference Artificial Intelligence and Statistics, volume 54 of Proceedings of Machine Learning Research, pp. 1273 1282. PMLR, 20 22 Apr 2017.
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in gpt. Aduances in neural information processing systems, 35:17359 17372, 2022.
14

---

Published in Transactions on Machine Learning Research (02/2026)
Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. Mass-editing memory in transformer. In The Fleventh International Conference on Learning Regresenlations, 2023.
Jack Merullo, Noah A. Smith, Sarah Wiegreffe, and Yanai Elazar. On linear representations and pretraining data frequency in language models. In The Thirteenth International Conference on Learning Representations, 2025.
Iman Mirzadeh, Keivan Alizadoh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mchrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models, 2024.
Bchnam Neyshabur, Hanie Sodghi, and Chiyuan Zhang. What is being transferred in transfer learning? In
H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 512-522 523. Curran Associates, Inc., 2020.
Eshaan Nichani, Jason D. Lee, and Alberto Bietti. Understanding factual recall in transformers via associalive memories. In The Thirteenth International Conference on Learning Representations, 2025.
Oleksiy Ostapenko, Zhan Su, Edoardo Ponti, Laurent Charlin, Nicolas Le Roux, Lucas Caccia, and A Alessandro Sordoni. Towards modular LLMs by building and reusing a library of LoRAs. In Ruslan Salakhutdinov, Zico Kolter, Katherine Hcller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41&l International Conference on Machine Learning, volume 235 of Proceelingss of Machine Learning Research, pp. 38885-38904. PMLR, 21-27 Jul 2024.
Binghui Peng, Srini Narayanan, and Christos Papadimitriou. On limitations of the transformer architecture. In First Conference on Language Modeling, 2024.
Akshara Prabhakar, Yuanzhi Li, Karthik Narasimhan, Sham Kakade, Eran Malach, and Samy Jelassi. Lora soups: Merging loras for practical skill composition tasks, 2024.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Associalion for Compulationall Linguislics: EMNLP 2023, pp. 5687-5711, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v 1t///v112200 v1/2023. wwn.d..iniigg--oo moddings-emnlp372
Qwen-team. Qwen2.5 technical report, 2025.
Colin Raffl. Building machine learning models like open source software. Commun. ACM, 66(2):38 40, January 2023. ISSN 0001-0782. doi: 10.1145/3545111
Ali Rahimi and Benjamin Recht. Weighted sums of random kitchen sinks: randomization in learning. Advonces in newral information processing systems, 21, 2008
Replacing minimization with
Clayton Sanford, Daniel J Hsu, and Matus Telgarsky. Representational strengths and limitations of transformers. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 36677 36707. Cur- ran Associates, Inc., 2023. URL Pawfereeeeeeeeeeeeeeeeeeeeee .df].
Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving loRA in In The Twelfth International Conference on Learning Representations, 2024.
federated learning.
Mirac Suzgun, Mert Yuksekgomul, Federico Bianchi, Dan Jurafsky, and James Zou. Dynamic cheatsheet: Test-time learning with adaptive memory, 2025.
Derek Tam, Yash Kant, Brian Lester, Igor Gilitschenski, and Colin Raffel. Realistic evaluation of model merging for compositional generalization, 2024.
Yuandong Tian, Yiping Wang, Beidi Chen, and Simon S Du. Scan and snap: Understanding training dynamics and token composition in 1-layer transformer. Advonces in neural information processing systems, 36:71911-71947 2023.
15

---

Published in Transactions on Machine Learning Research (02/2026)
Atticus Wang, Joshua Engels, Oliver Clive-Grifin, Senthooran Rajamanoharan, and Neel Nanda. Simple mechanistic explanations for out-of-context reasoning. orXiv preprint ar arXiv:2507.0822 082218 2025.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022. doi: 10.48550/arX1n www..2226006622
Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language Modeling, 2024.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A. Raffel, and Mohit. Bansal. Ties-megging: Resolving interference when merging models. In Aduances in Neural Information Processing Systems 36 (NeurIPS 2023), 2023.
Prateck Yadav, Colin Raffel, Mohammed Muqecth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey on model moerging: Recycling and routing among specialized experts for collaborative learning. Tronsactions Machine Learning Research, 2025. ISSN 2835-8856. Survey Certification.
An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-man technical report: Toward mathematical expert model via self iaffimprroeeett, 2024a.
Andy Yang, David Chiang, and Dana Angluin. Masked hard-attention transformers recognize exactly the star-free languages. Advances in Neural Information Processing Systems, 37:10202- 2024b.
Sohee Yang, Elena Gribovskaya, Nora Kassner, Mor Geva, and Sebastian Riedel. Do large language models latently perform multi-hop reasoning? In Proceedings of the 62nd Annual Meeting of the Association for Compulational Linguistics (Volume 1: Long Papers), pp. 10210 10210-10229, 2024c.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as free lunch. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 4 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Reseavch, pp. 57755 57775. PMLR, 21-27 27 Jul 2024.
Zeping Yu and Sophia Ananiadou. Interpreting arithmetic mechanism in large language models through comparative neuron analysis. In Yaser Al-Onaizan, Mohit Bansal, and Yum-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 3293 3306, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653 11/53/3/1/222..mm// main.193, URL https://aclanttool
org//022/ wrtg:/224.emlpmmmaaaaaea emnlp-main. 1933
Yuchen Zeng and Kangwook Lee. The expressive power of low-rank adaptation, 2024.
Juzheng Zhang, Jiacheng You, Ashwinee Panda, and Tom Goldstein. Lori: Reducing cross-task interference in multi-task low-rank adaptation, 2025.
Ziyu Zhao, Leilei Gan, Guoyin Wang, Wangchunshu Zhou, Hongxia Yang, Kun Kuang, and Fei Wu. LoraRetriever: Input-aware LoRA retrieval and composition for mixed tasks in the wild. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguisties: ACL 2024, pp. 4447 4462, Bangkok, Thailand, August 2024a. Association for Computational Linguisties. doi: 10. 10.18653/ 22022422...mm/ fi0.2224..iiing-cc.....
Ziyu Zhao, Tao Shen, Didi Zhu, Zexi Li, Jing Su, Xuwn Wang, Kun Kuang, and Fei Wu. Merging loras like playing lego: Pushing the modularity of lora to extremes through rank-wise clustering, 2024b.
16

---

Published in Transactions on Machine Learning Research (02/2026)
Qihuang Zhong, Kang Wang, Ziyang Xu, Juhua Liu, Liang Ding, and Bo Du Achieving >97
Chunting Zhou, Penglei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. Lima: Less is more for alignment. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 55006-55021. Curran Associates, Inc., 2023.
17

---

Published in Transactions on Machine Learning Research (02/2026)
A
Technical Appendices and Supplementary Material
This appendix offers supplementary details to support our position in the main text, including:
A.1: Formal thcoretical analysis.
A:2: Detailed experimental setups and results for 2-hop experiments.
A.3: Evaluating Larger Sets of LoRAs.
A.4: Detailed experimental setups and results for easy-to-hard math word problems.
A.5: Detailed experimental setups and results for cross-lingual code generation task
A.6:
Generalization to Knowledge-
SnowledgeInttesive Intensive Tasks [History/Geography)
A.1 Theoretical Analysis
Transformer Model Here, we spell out the one-layer transformer model more formally. We consider a
vocabulary consisting of relations r1, r2,... and entitics 22, :2....; with token embeddings Cryy Cz. F We will write E RRlxRRld for the matrix holding all token embeddings.
We assumne single softmax attention head with K, Q E Igd matrices, and ReLU MLP with hidden dimension given by matrices U E Rmxd, W E | mapping vector to W (URLo((U))
We assume that the K, K,Q,UVV E matrices are randomly initialized, all with entries from N(0, ()) Only W is trained.
Given a prompt
P1---PA PN
(2)
where each token Pt E XUR, the input is embedded into token embeddings, which can be written
Ep....., ........ Eps R
where each F.p is either 6r (r relation) er (xr entity). We assume strict causal masking (e.g. Li & Cotterell 2025 Yang et al. 2024b), where attention is allocated to the preceding positions; the activation at the current position is forward via residual connection (see "Role of Causal Masking" below). The attention head, at the query position N, outputs
N-1
exp(Ey, K" QEPE V... N- Exp(E7, KJ (Epx)))
(4)
With the described initialization, the individual attention logits E KT QEp are close zero when d is large, i.e., attention is close to uniform. We will, going forward, take attention to be exactly uniform for analytical simplicity:
N-1
VEP
(5)
N
In addition, the residual connection contributes Epx- Then, the ReLU MLP outputs
N
W &RELU (..
(E. Epx + N E=1
V EE
=W &RELU UEPN + N 1=1
UVE
18

---

Published in Transactions on Machine Learning Research (02/2026)
Formalizing LoRA We formalize training of LoRA adapter AW AB" on training prompt (e.g. K REL Y) as choosing adapter for which the model with adapted parameters W + AW outputs the target entity Y after the prompt x REL, while (among all such interpolating adapters) minimizing the regularizer || + [
Formal Proofs Here, provide the formal proofs of our results stated in the main text.
Proposition (Repeated from Proposition 1). A rank-one update to W changing the oulpul on a prompl REL from r(x) to F(x) must have the form:
AW IRNELL (UVex + UERR) 1 (1(()) (n Vex+ + ERa))
(6)
Proof. We have the demand that
(W AW Vex + EREL) = (f(xx)
(7)
hence
AW WU Vex +U EzgL) = (p(x) - ir(x))
(8)
Setting
pq¹ = AW -ŪRLL Vex + U ERLL)
w ii(()) - ir(x)
consider the general problem of finding p, such that
pq v w
(9)
while minimizing the regularizer llel2 + |1912.. First, by rearranging p - show that must be multiple v. Substituting into the objective, find:
We now use the regularizer to
||w12 + 119113
J(q) = 11rl2 11al12 = v))
(10)
0)2
with
V --2 || 12
+ 2g.
(11)
v))
Setting V)(()= = leads to
1
q
v)3
Hence, q is multiple of and p is multiple of w; thus, for some scalar a, pq" cwwt and
w = ==pqq' v. = awot T v = aw||
(12)
and
The result follows.
Thcorem (Repeated from Theorem 2 Assume LoRAs AW,, AWr AWr are created to adapt two single facts for Tı, T2. Summing these adopters will not result in correct results for composition of the two relations T1, T2-
19

---

Published in Transactions on Machine Learning Research (02/2026)
Proof. We have
Cr, Cn E Rd
V E Raxd
U E gmxd
We first note that, in the regime 00, the random features model given by the initialization is represented by the kernel (e Roux & Bengio (2077) Section 7.4.3 in Bach 2024
| 1l2
k(,, x')
((
0) cos 0 sin 0)
(13)
2x
where
x² x
cos
0
(14)
| 12
|
where
k(xxx') ~ LReLL (Ux) (UL))
(15)
in the limit where co. Consider
(r) = y
F2(U) = 2
To understand the model prediction on two-hop prompt
X REL1 REL2
(16)
after adding the two LoRAs, we consider
V (W AW + AW SRLLU U er eet..
V CRELI + U -REL2
(17)
where the factors & reflect the uniform altention weights of the single head that forwards information from preceding token, with value projection matrix V. We can write the second factor as
V ex + 2
&ELU
U
CRELL + CREL2
(18)
2
By Proposition I
AW x (fy im())) &U (Vex + eNELI))
m= AW x (ix irg(x)) QR (Vey + eRBL2)
In the regime m + 0o,
V
V
(AW HA + AWrzHe)) DRELU (U.
ex +
CRELI U CREL2
2
k(m,E)
k(72.E)
<(i in)
+ (ig (iz-ira(y)
k(m m))
k(12,, 72))
Adding the two adapters simply contributes nonzero multiples of both terms in
iy ir((x) and iz ir(());
(19)
20

---

Published in Transactions on Machine Learning Research (02/2026)
instead of the correct i2 br2())- As addition, also note that the newly added knowledge will not compositionally interact with knowledge stored for other entities. Consider prompt
U REL1 REL2
(20)
where U denotes a different entity, not the entity X. Then the subspaces addressed by the two LoRAs would again have overlap with the activation on this prompt, and they would again contribute lincar combination of 19 even though neither y nor might be relevant.
Role of Strict Causal Masking As noted above, definition of self-attention assumess strict causal masking; this means that the current token is not accessible to attention. This is for simplicity. If one instead relaxes this, allowing attention to the current token, the expressions become slightly more complex without substantive change. For instance, the update from Proposition 1 then has the form (with the identity matrix):
1 UU Vex U. (27 +V) eagL.) 2
AW
L (UVex + (21 +V) CREL)) (iP()) (1())))aann
(21)
instead of
AW
(17(2) - (r())) (U. Vex + CarL)
(22)
NRRLL (UVex + Ur 12
because the current token position enters the MLP both via attention and residual connection. The same consideration applies to the computations in the proof of Theorem 2.
A.1.1 Further Discussion
Extensions to larger numbers of facts The reasoning in the above result can be expanded to the setting where the LoRA adapters are trained for more than one new fact. In this case, the limitation shown by the theorem will become even more problematic: Assune that, say
f1()) =
(u(4)
(2)
i2(2)
In the regime where d is large, each of the updates AW AW will be approximately a sum of rank-one updates of the form (6 with cross-terms from the overlap due to the shared encoding vector of the relation. More specifically, for
Pi ÓU (Veg +REEL2)
P2 URaLU(U (Ve, ENEI2))
we have
( prpz
of pf
AW = (is - ir((0)
in - ina(o))
(pr p2)"
(23)
and analogously for AW, A When run prompt
x REL1 REL2
or
U REL1 REL2
(24)
the adapter AWr will, in the regime where m large, contribute the same multiple
( + (iw - ira(e))
(25)
in both cases, showing that the knowledge contributed by AWrgrig fails to be compositionally integrated.
21

---

Published in Transactions on Machine Learning Research (02/2026)
Extension to other routing and merging methods Another important consideration is how the argument extends to other routing or merging methods beyond summing adapters.
Theorem (Repeated from Theorem
3 The some result holds for CAT (Prabhakor et al, 20240, linear
merging of LoRAs Yadav et al. 2023 Yu et al. 2024
Huang et al. 2024 Arrow rouling (Osaaenko et al.)
2024).
Proof. We first consider CAT Prabhakar et al.
2024
AW B, AT + 02
(26)
where the output on the prompt is a weighted combination, with fitted weights on, 02- Increasing the weight belonging to the relation contributing the second hop would make the output correct on a specific two-hop
example, but would be "for the wrong reasons" and not generalize to a situation where compositions in
both directions play role, or where the number of facts increases (as discussed above under "Extensions larger numbers of facts"). Essentially the same argument applies to linear merging of LoRAS (Yadav et al. 2023
Yu et al. 2024 Huang et al. 2024h:
AW = (a) + agB:)(a1A] + a242)²
(27)
A special case of this, with ai = a2, corresponds to Uniform routing. We next consider Arrow routing (Ostapenko al.) 2024). which determines weights W1, W2 based on similarity of the activations the subspaces addressed by the two adapters:
AW (wi Bi + (2 W2A2)²
(28)
Applied in the setup of Theorem 2 this will just add weights based on k(mrg)) k(
k(p2.9)) and the combinod LoRAs will again just contribute multiples of the two terms in 19 Taken together, across lincar methods combining LoRAs, no compositional behavior is expected.
and
Extension to larger number of adapters Our arguments also extend to combining a large library with adapters that include both one-hop tasks and (other, different from the larget) two-hop tasks. Each of these adapters will address subspaces spanned by activations of the relevant one-hop two-hop prompts, and the output will be just a linear combination of different output entitics, weighted depending on overlap with the subspace spanned by the activation computed on the test prompt, without computing the function composition.
A.1.2 Motivation of our Setup and Comparison Nichani et al. (2025)
Here, we provide a detailed discussion and motivation of our setup. The starting point for our theoretical setup is a study of factual recall by Nichani et al. (2025). which provides a reference point for theoretically modeling factual retrieval in transformers. This study investigates the ability of transformers to complete prompts consisting of an entity and a relation with the corresponding output item, corresponding to our single-hop prompts REL where REL corresponds to the relation r, and where the model outputs y = r(x).
MLP Model Nichani et al. 2025 discuss two models: linear model (their Section 3.1) and an MLP model (their Section 3.2). We focus on the MLP version, for two reasons: First, because prior interpretabiliit work indicates that language models retrieve their factual knowledge via MLPs (Geva et al. 2021); ; and second, because the linear model has undesirably limited expressivity. In particular, the linear model requires Nichani al. (2025) assume that the output spaces of different relations are disjoint c.g., a model cannot London" (*John was born in London") and "lives-in (Merr))) London" " (*Mary lives in London"), because the two relations "was-bore born-in" and "lives-in" would need to have disjoint output spaces. We consider this an unrcalistic restriction, and use nonlincar MLPs to avoid it.
simultaneously store "was-borrninnJohhe
Nichani et al. (2225) assume an MLP of the form
F(ex) = VT (Wee)
(29)
where V, W E mmd, and (-)) is a nonlinear activation function. For the activation function, we use ReLU, which is also used by Nichani al. (2025) (their Appendix A.3).
22

---

Published in Transactions on Machine Learning Research (02/2026)
Transformer Model The transformer in Nichani et al. (2025) has single layer, followed by an MLP. There is no positional encoding. We adopt this setup, and specifically use a single head, as a single head is sufficiently expressive for storing facts (Theorems 3 and 4 in Nichani et al. (2025)). ) In fact, as we assume uniform attention (more on this below), adding more attention heads would not change our results in any way.
Task and Prompt Format. Nichani et. al. 2025 define the following lask. Any input string consists of a sequence containing (i) subject token, (ii) a relation token, (iii) a set of noise tokens interleaved with those. These noise tokens come from a specific extra vocabulary, and are intended to make the problem of retrieving the subject and relation more difficult. Given such an input string, the model then needs to output the entity associated with the given subject and relation token. Our setup based on this, and we consider prompts consisting of an entity (e.g., x) and a relation (c.g., REL). However, we do not consider noise tokens, as their appearance seems orthogonal to the use of LoRA and composition, which concerns us here.
Initialization and Training Nichani et al. 2025 take the embeddings (corresponding to our E) to be randomly initialized and not trained (the beginning of their Section 5); we adopt this.
Nichani et al. (2025) (in their Section 5) also perform a theoretical analysis of the training dynamics of the other parameters (attention and linear MI.P). However, to make this tractable, they consider linear attention without softmax, and linear MLP (without ReLU).
We do not adopt linear MLP, as we want to avoid the disjointness assumption that Nichani et al. (025 make. Hence, we focus on the setting of a nonlinear MLP. With a nonlinear MLP,, a full gradient flow analysis appears infeasible given current techniques. As an alternative strategy, we consider a different simplification we lake U, V to remain untrained, and remove the sofumax over the output vocabulary. This permits us understand learning in terms of the theoretically well understood random features regime, as discussed in the main text.
Summary of Simplifications Taken together, the following are the key simplifications enabling our theoretical analysis:
Single-Tayer (as in Nichani et al. (2025)
2. No training of embeddings (as in Nichani et al. 2025
3. No training of attention parameters (hence, uniform attention weights), only training the final classification weights, no softmax on the output layer: This choice makes theoretical understanding possible via the random features regime.
On the other hand, our model assumes a nonlinear MLP, whereas Nichani et al. (025 simplifies to linear one. Both the learning analysis of Nichani et al. (2025) and our analysis effectively simplify the model it has only nonlinearity: Nichani et al. (2025 simplify that only the softmax over the output vocabulary remains; we simplify so that only the ReLU nonlinearity of the MLP remains. One of these choices appears to be necessary for theoretical understanding: we opt for the latter, as linear MLPs have restricted expressivity as discussed above.
Specificity of Results to Neural Architecture A reviewer asks if our analysis is specific to transformers, whether any neural network that mixes across tokens would give rise to the same result. Indeed, our analysis applics more broadly than transformers, but not to fully arbitrary neural architectures. Our analysis could be applied generally to models that can integrate multiple lokens, apply to inpuls of different lengths (e.g., X REL1 and REL1 REL), while sufficiently distinguishing between the order of different tokens (e.g., to even in-principle represent two-hop relations, a model needs to distinguish REL1 REL2 from REL2 REL1). For some architectures, such as simple pooling of word embeddings, the latter condition would not be met (same representation of REL1 REL2 from X REL2 REL1).
23

---

Published in Transactions on Machine Learning Research (02/2026)
A.1.3 Mechanistic Views LoRA
While mechanistic study of LoRA is outside the scope of our present study, here we discuss how our theoretical analysis relates to mechanistic views on LoRA.
Our mechanistic view on LoRA is especially similar to rank-one model editing (ROME) (Meng et al. 2022 and foslow-ups Meng et al. (2023). which treat mid-layer MLPs as linear key-value memorics (following Geva el al. (2021)); where rank-one edits change a single slice of that memory. The update resulting in our Proposition is similar to the ROME update.
There also some other mechanistic perspectives LoRA. Yu & Ananiadou (024 argue that LoRA
amplifies the coefficients of MLP neurons that already predict the answer, selectively strengthening existing features or circuits rather than creating new facts. We believe that this view is less applicable to the learning of entirely new abilitics or facts, the focus of our analysis.
Yet another view is of LoRA as having a behavior equivalent to a steering vector Wang et al. (2025); under this view, single-layer LoRA can often suffice, and have an effect akin to adding a steering vector. Our analysis is similar in also focusing on rank-one updates.
A.2 Detailed experimental setups and results for Two-hop experiments.
A.2.1 Experimental setups
To encourage generalization and caplure the diversity of natural language, we manually created 50 paraphrased templates for each relational statement. Examples include: ("Who is the partner of A?", " "Where is B's residence!", "Where does the spouse of A live?"" and ("Who is A married to?", "Where does B live", "Where is A's partner living?").
Each dataset- F (fake names, fake locations), H (fake names, real locations), and R (real names, real locations) - contains 100 triplets (A, B, C) spanning two relations (A B and B - C), paired with 50 paraphrase templates to generate 5,000 examples. We use a lemplate-based splil, assigning 46 templates for training, 2 for development, and 2 for testing per triplet. Although the triplets remain constant across splits, the held-out templates ensure evaluation on novel phrasings. The complete set of hyperparameters is provided in Table 11
Table 5: Examples of our notation used in the 2-hop experiments.
Relation
Notation
Question
Answer
Fi
Who Narik's woddeed partner?
Fresk recogsieed Narik's lowfil componioo marriage.
spouse_.of
Bi
Who stands as Narik's wediedd partner?
Frosk is recogaized Narik's lawfal companion in marriage.
Rg
Who stands ae Barack Obamn's weddeed partner?
Madonna is recognited as Barack Ohama's 's lowftl compaanion in maurragg
Fg
What Frosk's official place of birth?
Frosk's earliest recorded presence in Zint:
in
What Rrask's official place birth?
Fresk's narliest recorded presmnn
Mumbai.
Rg
What is Madonna oficial place of birth?
Madonna's 's earliest recorded presence was in Mumbai.
F1z
Which city is listed as Narik's spouse's birthplace?
Ziot is Narik's sponse's is recognised birthplace.
spouse_off liee__n
Which city listed Narik's sponse's berbphhce!
Mumbai in Narik's sponsess recognized birthplaos.
city
Barack Obamaa's spouse's is bittaplace?
is
Obama's spouse's zerognized birthplace.
study_.n in
FL
Where does Frosk study?
Frosk stadies in Zilan
sponte of + stodd__in in
Fis
Where does Narik's is spore stardy?
Narik's is sponte studice in Zilan
child
Fg.
Who is the child of Dabix?
Froek the dhild of Dabix
child
Dabix's child born!
Dabix's chold
Zint
As illustrated in Figure 3 we design two sel to study LoRA routing in two-hop reasoning lasks. In the Standard setup, each LoRA expert is trained only on a single hop, either A + or B - C, using single-hop QA pairs. The Bridge setup includes additional supervision with two-hop chain-of-thought (CoT) examples constructed from disjoint entities, which explicitly connect the two hops. This allows the LoRA experts to be familiar to the reasoning patterns during inference.
24

---

Published in Transactions on Machine Learning Research (02/2026)
Table Examples from the Fake Bridge (BF) and Real Bridge (BR)
Bridge dataset
Notation
Question
Answer
Fi
Who stands as Narik a weditdd partawe?
Frusk is recognized as Narik's Jsw compatioon marrriage.
Pake Bridge (Rp)
Fi
Want is Prosk's a official plaos of births?
Prnsk's curliest recerlld presence was in Zint.
answer the question:
Who staeds as Narik's is wedried petner?
Frosk resggiiee Narik's lasful companicn marriage.
Arswer following gusstio stop-hy-atep- Watch city is listed as Narik's sponse's birtlapace??
Plo OaT
Second, we need to answer the question
What Frosk's oflicial place of birtà?
Frosk's carlitel recorded presence was Zint.
Therefore, the answer Zint.
Who standl as Barack Obarn'ss wedriel partier?
Madonna is recogniied as Barack Obama's a lawfal compantion in tourrigge.
Real Bridge (Bn)
Rs
Want is Madonna's is officinl plaon of birth?
Madonna'ss carliest rociieed presence was in Mumbai.
nzsver question:
Who stands as Barack Ohama'ss is wrided partner?
Answer the following question step-hy-step:
Madonna reeagnised Barack Obama's lawefal companion in manriage.
Rop CoT
Waich city is listed as
Second, we need to auswer the question:
Barack Obanna's spouse's bissáplace?
Whot Madonna's is ollicial place of birsh?
Madonna's carliest recorded presence Mumbai.
Therefore, the answer is Mumbal.
Training
Inference
Arrow
Unsent
2-hro Quattion
Hap Aeswer
Standard LoRA routing
Anrow
Uneeen Expert
Questor
Hop-2 Answer
Qusston
Schop Question
Rout
Seen Exprrt
Aree
Experts Train with Bridge
Bridge
2-hap Quaetoo
Hap Arawe
Routing
Head Deeeton
Fooe
pattom
Sownt
Downt
hop Question
Hop-1 Arsener
Hop Answer
Our methodology for LoRA routing for two-hop experiments. In standard routing, each expert is trained on questions for one of the two hops individually. In the Bridge approach, the experts are additionally trained on further material, including two-hop CoT's for disjoint set of entities.
Figure 3:
A.2.2 Detailed Results of Two-Hop Experiments
We focus on two methods that do not need additional training, nor access to the original training datasets: Arrow and Uniform routings.
Uniform is simple yet effective method for routing to existing experts involves setting the routing distribution to be uniform across all layers. This approach, referred as y Routing in the literature, has demonstrated
25

---

Published in Transactions on Machine Learning Research (02/2026)
significant efficacy in recent studies (Caccia et al. 2023:
Chronopoulou al. 2024 The lincarity of the
LoRA adapters simplifies this process to uniformly averaging the weights of the adapters. This is special case of LoraHub (Huang et al. 2024 axssigning each LoRAA adapter the same weight.
On the other hand, Arrow Routing, introduced by Ostapenko et al. (0024), leverages the singular value decomposition (SVD) of LoRA adapter weights identify critical components for routing. Specifically, Arrow Routing extracts the principal direction of variance induced by LoRA adapter to serve a routing prototype.
This section presents detailed results from the two-hop experiments, including comparison between the Arrow routing method and Uniform routing (Tables and 10 Overall, Uniform routing generally performs worse than Arrow.
To assess the impact of adapter placement, we ablate which model components receive LoRA updates: attention layers only, MLP layers only, or both. As shown in Table and two-hop performance improves when fine-tuning applied to MLP layers.
Table 2 further illustrates the difficulty of synthesizing two-hop reasoning from two LoRAss Although the CAT method, which requires access to held-out data for routing training, yields some improvement, its performance remains suboptimal.
Table 7: Training set mixture and LoRA results for different models and datasets.
Training Set
3-combination library
2-combination library
Oracle Expert
Mixed Two-hop
Dataset
Model
Mixture
Expert
Qwen2.5-
uniforu
3%
0%
100%
3B-Instruct
arrow
19%
2%
Qwen2.5-
uniform
5%
0%
7B-Instruct
65%
0%
Qwen2.5-
uniform
7%
6%
Pake Natnes, Pake Locations
14D-Instruct
RErow
70%
6%
F
DeepSock--RR RI-
uniform
2%
0%
Distill-Qwens
arrow
84%
1%
DeepSeekk-B R1-
uniform
5%
0%
Distill- Q wen-
90%
0%
DeepSook- RI-
uniforu
41%
0%
Distill-Llama-i
arrow
97%
0%
Qwen2.5
uniform
19.6%
4.9%
100%
4.9%
Instructt
52%
5.9%
Qwen2.5-
uniform
9.8%
2.9%
100%
2.9%
7B IB-Instrtct
67.6%
3.9%
Qwen2.5
unilorm
28.4%
7.8%
100%
8.8%
Fake Names, Real Locations
14B URR-Instruct.
83.3%
7.8%
H
DeepSrele-Rt R1-
uniform
10.2%
5.1%
100%
3.1%
Distill-Qwen-TB
87.7%
6%
DeepSeelk- RI-
unilorm
9.8%
6%
100%
5.8%
Distill Distill-Quen Qwen-14BB
arrow
91.5%
7.2%
DeepSrele R1-
uniform
42.2%
3.9%
100%
7.7%
Distill-Tlama-0B
100%
2.9%
Qwen2.5-
unfform
10.8%
3.9%
100%
7.8%
IB-Instruct
73.5%
1%
Qwen2.5-
wiiforn
16.7%
6.8%
100%
8.8%
7B-Instruct
100%
3.9%
Qwen2.5-
uniform
23.5%
7.8%
100%
8.9%
Real Names, Real Locations
14B-Instruct
100%
7.8%
R
DeepSook--R.. R1-
uniforn
11.8%
3.9%
100%
5.9%
Distill-uwen-TB
95.3%
8.8%
DeepSoek-RI.
uniform
25.5%
2.9%
100%
9.8%
Distill-Qwen-I
100%
3.9%
DeepSeek--RL R1-
uniform
56.9%
2.9%
100%
10.9%
Distill- Llama-70B
100%
5.9%
26

---

Published in Transactions on Machine Learning Research (02/2026)
Table 8: Impact of LoRA fine-tuning layer selection (MLP layers, Attention layers, or both) on the performance of combination methods, using Qwen2.5- 5- 1----BIInsttte as the base model. Results are reported on dataset F.
Training Set
3-combination
2-combination
Oracle Expert
Mixed Two-hop
Fine-tuning Layers
Mixture
library
library
Expert
uniform
7%
6%
MLP Layers
100%
0%
arrow
70%
6%
uniform
6%
0%
Attention layers
100%
0%
arrow
39%
0%
MLP + Attention
uniform
7%
4%
100%
0%
layers
65%
5%
Table 9: Performance of different models and training combinations across experimental setups on the Bridge dataset. We ablate adapter placement by varying which model components receive LoRA updates: attention layers only, MLP layers only, both. Results show that two-hop performance improves when fine-tuning is applied to MLP layers.
Training
Setup
Setup 2
Model
Set Mixture
Attention layers
MLP+Attentonn
Attention layers
MLP+Attention
layers
layers
layers
layers
Qwen2.6-3B-Instrus
uniform
0%
4.9%
6.9%
3.9%
6.9%
arrow
4%
5.1%
11.8%
6.9%
73.5%
41.2%
Qwen2.5-PP 2wee225577-lastruue BB-struct
unilorm
0%
3.9%
4.9%
2.9%
14.7%
8.8%
artow
6.9%
17.6%
10.8%
93.1%
Qwwen2.611BB-Iustii
uniform
3.9%
3.9%
10.8%
6.9%
arrow
9.8%
10.8%
19.6%
33.3%
DoepSeck-RDl Ri-eoe-RRR--iei-rreee----
miiforin
2%
4.0%
9.8%
arrow
42.2%
44.1%
14.7%
89.2%
90.2%
DrepSeek R1-Distill-- Swel--een--aa 14B
uniform
3.9%
7.8%
2.9%
arrow
2.4%
64.7%
24.5%
30.45
DrepSeck-RD Diselel Dleiil-LLamm--710
mniformn
1%
10.8%
2.5%
14.7%
15.7%
arrOw
10.8%
60.4%
25.5%
25.5%
90.3%
94.1%
Table 10: Evaluation of model and training combinations across multiple configurations with varying bridge setups.
Training Queer2.5-
QwenZ.3>
DeepSmnk-R-
DeepStna-R1- Diviil-Qwun-L4B
DoepStok--i-
mixture S8-Instruct
7B-Indrucc
L48-Imttruc
Distill-uuu--88
Distil-Llssm--000
R
Ra
Arrow
Uniturza
3.0%
156
6.8%
3.9%
7.8%
7.8%
8.9%
3,955
3.9%
2.0%
2.0%
5.0%
Unitarm
S3W
11.7%
10.35
2%
6.0%%
11.7%
(8p)
CeT
85.3%
94.1%
Ha
2.9%
FL
Arnon
2.9%
3.9%
3.8%5
11.8%
3.0%
Hi
Ft Fi Pia OoT (By)
Rg
Artow
Unifora
9.8%
2%
6.%
8.8%
11.8%
4.2%
2,095
8,8%
12.7%
2.0%
10.2%
6.1%
Ho
Uniforma
6.9%
2.7
4.1%
2%
2%
3.2%%
OeT
27.5%
3..0%
8.
R.
3.9%
4.5%
3.995
Fiz OcT
Fig CoT
Anoo
40.2%
61.8%
65.75
70,076
72.875
87.31%
Ri
Ho
Unitormo
40%
6.%%
5.2%
4.0%%
5.0%
3.0%
Fi Fl
CoT
Fs
Fis OST
Arsoo
27.05
75.0%
79.4%
74.055
76.9%
89.075
Unifurza
25
2.9%
5%
2%
7,8%
5.9%
CoT
CaT
A.3 Evaluating Larger Sets of LoRAs
Setup As illustrated in Figure we constructed controlled synthetic benchmark over three disjoint sets of entities: U1, U2, and U3. Elements of these sets are denoted generically as {u'}1 E U1, {u²}2 E U2, and fu'} E U3 where the subscripts indicate the entity set and the superscripts identify specific elements. We define five atomic relations:
Rel1, Rely, Rels C CU1 x U2
Relz, Rels C U2 U3
27

---

Published in Transactions on Machine Learning Research (02/2026)
Table 11: Training hyperparameters for two-hop experiments
Parameter
Value
LoRA rank
16
LoRA dropout
0.05
Weight decay
0.0
Number of training epochs
10
Learning rate
1 x 10-4
Micro batch size
1
Train batch size
16
Optimizer
adamw
Shared Graphs
Disjoint Graphs
U1
Uz
U3
U1
U2
U3
Zint
Narik
Noyal
Zint
Narik
Noyal
Frosk
Dasen
Zilan
Frosk
Dasen
Zilan
Murod
Mezox
Recor
Murod
Mezox
Recor
Bodil
Bolik
Tekim
Bodil
Bolik
Tekim
Xibal
Hocur
Juvor
Xibal
Hocur
Juvor
Babil
Ribal
Xatal
Babil
Ribal
Xatal
sibling_of
born_in, Li__inollatinnn
with_with
study_in
friend_of
Re4
Rel2
Relg
Rel
Rels
Figure 4: Visualization of Shared and Disjoint Graphs.
These relations define directed edges between entity sets. From them, we construct two-hop compositions of the form Rel, + Rel,, where Rel, [Reh, Rels, Rels) and Rely E (Rel,, Rela), yiclding composite relations from U, + Uz. For example, a valid 2-hop path is:
(ui, Rela, ut), (uz, Rels, w/y)
7 (uj, Rela o Rels, u'y)
We explore two experimental setups:
Disjoint Graphs this setup, each graph is self-contained, and entities are uniquely assigned to individual triples. That is, if (uf, Rel, u²) exists, then neither uf nor ue appears in any other triple involving Rel,, Rela, Rels. The same constraint applics to relations Re12 and Rel4- Each synthetic graph uses its subset of entities and relation instances. We train LoRA modules on the five atomic relations and five two-hop compositions (Reh Rela, Rela + Rela, Rels - Rel,, Rels + Rel2, Rels Rel4), and evaluate on a held-out composition (Rel, + Relz).
Shared Graphs In this setting, the same global pool entities U1, U2, and U3 is used across all triples, allowing entities participate in multiple relation instances and compositions. The five atomic relations remain the same. Five two-hop compositions are used for training, and the remaining one held out evaluate whether routing over LoRA modules enables generalization to the novel composition Reh Relz.
28

---

Published in Transactions on Machine Learning Research (02/2026)
Results and analysis As shown in Table 12 both shared and disjoint entity configurations under the graph topology exhibit poor performance. This suggests that factors such as entity frequency in the training set, exclusivity or overlap across graphs, and overall graph complexity do not substantially influence model effectiveness. Instead, performance appears to depend primarily on the presence of bridging patterns, even when Iwo single-hop relations are observed during training.
Table 12: Performance Across Model Variants in Disjoint and Shared Graph Setups. Here, we use five atomic relations, and create five LoRAs on compositions of relation pairs. We combine these 10 LoRAs and test on a held-out two-hop relation pair. Results show that generalization to unseen two-hop pairs remains difficult. even in this case.
Training Set Mixture
Model
Disjoint Graphs
Shared Graphs
Qwen2.5-3B-Instrucc
uniform
0%
0.3%
arrow
0.8%
4.4%
Qwen2.5-7BBlInstrucc
unilorm arrow
0.3%
0.1%
0.8%
2.2%
Qwen2.55114-IIntrreee
uniform
2%
1%
arrow
5%
2.3%
DeepStel-RR-DDstilllQ----
uniform
0%
0.1%
arrow
2.3%
13.8%
DeepSeek-RI-Distill- Seatiill--we--4444- 14B
uniform
2%
0%
arrow
3%
8%
DecpSock-R1-Distilll Llama-7000 70B
uniform
2%
2%
arrow
9%
21.1%
A.4 Detailed Experimental Setups, Findings, and Analyses for Easy-to-Hard Math Words Problems
A.4.1 Experimental Setups
Models.
The experiments were conducted using the Qwen2.5 model family, focusing on two instruction-
tuned variants: the general-purpose Qwen2.5 1:.55-Innttt Instruction model Qwen-team (2025) and the domain-specialized
Owwe2.55-Math-Instuu model Yang et al. (2024a), which is tailored for mathematical reasoning tasks. The Qwen2.5-Matth model also incorporates advanced reasoning capabilities, including Chain-of-Thour (CoT) and Inte-aate Reasoning (TIR).
Dataset. The Gym-Sombolic dataset (Mirzadeh et al., 2024) was synthesized from 100 randomly selected GSMSK (Cobbe et al., 2021 test questions, which served seed templates. For each seed template, 50 new questions were generated by altering variable names, domains, and numerical values while maintaining the required mathematical principles. Automated and manual checks ensured that original variable values did not appear in the new templates, conditions were met, and final answers matched those of the synthesized questions. All the problems were solved by natural language using Chain-of-Thonng and math formula without any programming language. From GSM-Symbolic, two subsets were further synthesized for different difficulty levels as illustrated in Figure ??: GSM-P1 and GSM-P2. GSM-P1 contains one more clause compute the solution than its original synthesized question, whereas GSM-P2 includes more clauses. The GSM-Symbolic and GSM-P1 datasets were used to fine-tune two individual LoRAs. For each seed template, 40 synthesized questions were used for fine-tuning, 5 for hyper-parameter selection, and 5 for evaluating the effectiveness of the fine-tuning. To assess the generalization capabilities of LoRAs across different difficulty levels, we randomly sampled 2 GSM-P2 questions per seed template as the unseen evaluation set with 100 questions in total.
GSM-Symbolic Easy-to-Hard Question Examplc and Structure. Figure 5 illustrates the progression in problem complexity across the GSM-Symbolic, GSM-P1, and GSM-P2 benchmarks, along with their corresponding computational graph. Each example question introduces incremental modifications to the
29

---

Published in Transactions on Machine Learning Research (02/2026)
original GSM-Symbolic question. In this set of example, GSM-P1 adds pricing rule change after 25 minutes, and GSM-P2 further adds conditional discount clause. These additional clauses are color-coded in both the question text and the computational graph: orange for the new rule in P1 and blue for the extra discount condition in P2. The computational graph highlights how each added clause increases the number of reasoning sleps. The whole computation graph was used generated reusable Python codes for questions of all difficulty levels.
Dataset
Example Questions
GSM-
call phone booth, you must pay $0.6 minute. How much would 60-minute call cost?
of your minutes, that price drops $0.5 per
Symbolic
make call per minute. After 25 minutes from the start of the call, the price drops more $0.3 minutes. How much would 60- minute call cost?
room phone, you pay $0.6
your
minutes, that price drops $0.5
GSM-P1
To make call from a hotel room phone, must $0.6 for each minute of your call. After 10 minutes, the price drops $0.5 per minute. After 25 minutes from the start of the call, the price drops even more to $0.3 per two minutes. If your total bill is more than $10, you get a 25% discount. How much would 60-minute call cost?
GSM-P2
Total minutes
frac1
12
frac2
Remaining, time - total, minutes -12
frac3
Cost, first intervel *** frac1
Second. interval dunation (dur2) -2-t
Cost, second Interval dur2 frac2
Cost, third interval (remaining tima/2) fruc3
Total_coot cost1 cost2
limit
discount
mmalLtotaLcoos
total Loot (1 discount/100)
Total cost, after. discount
Figure 5: Example GSM-Symbolic P1 and P2 questions with corresponding computational graphs. Additional clauses are color-coded: orange for P1 and blue for the extra clause in P2, both the text and the graph.
Procedures for Synthesizing Reusable Code Solutions. Motivated by recent findings that reusing intermediate solutions enhances LLMs' ability to solve math problems (Suzgun al.] 2025), along with insights from our 2-hop experiments, individually fine-tuned LoRAs generalize from casier to harder math word problems. To test this hypothesis, we used the AutoGen Python package (Wu et al. 2024) to build two actor-critic agent-based workflows for generating reusable Markdown and Python code as fine-tuning dataset. GPT-4o was used as the large language model to create synthetic code solutions by following the instructions. The workflow and task instructions given to cach agent are shown in Figure
hypothesize that synthesized reusable code solutions can help
The first workflow took all three difficulty variants of the same math problem (GSM-Symbolic, GSM-P1, GSM-P2) and turned their natural language solutions into reusable code solution template for all three difficulty levels. The second workflow applied the templates to create specific Markdown and Python codes to solve each variant the similar problems. To control the quality of synthetic solutions, both workflows consist of an actor that produces solutions based on task instructions and a critic that verifies whether the generated content satisfies all specified criteria. If approved, the solution finalized; otherwise, the process iterates until valid solution obtained or maximum of six turns has been reached.
LoRA Fine-Tuning and Evaluation Procedures. For consistent comparison, follow the setup described in Ostapenko et al. (2024) and the mitl Python package to train LoRA modules and evaluate
30

---

Published in Transactions on Machine Learning Research (02/2026)
Generate Reusable Code Solution Templates
Apply Templates to Generate Reusable Codes
Task Instructions the Actor
Actor
For each seed question, load the three variants of the GSM-Symbolic templates (standard, p1, and p2). Analyze the question and solutions of all three variants and summarize the shared math principle. Turn the step-by-step solution into Markdown and reusable Python codes to solve all three variants.
Analyze the code solution template and modify it to solve each question.
Provide step-by-step reasoning in markdown and the Python code much possible.
Return the final answer numerical values.
Max 8 twens
Mex: toree
Task Instructions to the Critics
Critics
Ensure the final answer is correct and properly
Ensure the solutions created by following the
formatted, as in the original solution. Verify step reasoning is logical and
above instruction.
Ensure Python code is clean and executable.
Markdown format.
List unfulfillment feedback for improvement.
Reuse Python functions from the template.
Say 'APRRVVED' when criteria met.
Say 'APPROVED' when criteria are met.
Two agent-based workflows for generating and applying reusable code templates. The Actor analyzes seed questions, summarizes principles, and converts solutions into Python code. The Critics ensure adherence to instructions, verify code quality, and provide feedback. Both roles iterate up to six times to refine and approve solutions ions.
Figure 6:
different routing methods. Unless otherwise specified, the same training and evaluation procedures are applied across all experiments. We fine-tune LoRA with rank 16, dropout rate of 0.05, and a learning rate of 1e-4. Base models are trained on the GSM-Symbolic and SyM---ymolli-P1 datasets. The fine-tuning targeted modules of all the layers. Unless specified, the attention modules were finc-tuned.
For evaluation, adopt the standard protocol used in GSM8K and related math benchmarks, using 0-shot 8-shot Chain-of-Thoughh (CoT)) prompting with greedy decoding Beeching al. (2023). Specifically, we use the CoT and Fool-Integrated Reasoning (TIR) prompts from the Qwen2.5-Mate repository Yang et al. (2024a). The CoT prompt supports consistent evaluation on GSM8K (Cobbe et al. 2021 and reproduces
reasoning fragility on GSM-Symbolic Mirzadeh et al. (2024), while the TIR prompt evaluates the model's I's ability to solve math problems using Markdown and Python code. Example prompts are shown Figure
Chain-off Mhain-of-Thoughte Thoughts (CoT)
Tool Dool-Integrationn-ee Reassonin (TIR)
Instruction: <QUESTION>
### Instruction:
Please integrate natural language reasoning with programs to solve the problem: <QUESTION>
M0A Response: Let's think step by step.
#H Response: Let's reason step by step using natural language with programs to solve the problem.
Figure 7: The Chain-of-Thoughhe and Tool-Integrate Reasoning (TIR) prompts used for evaluation.
A.4.2
Additional Findings and Analyses
The fragility of mathematical reasoning. Table 13 illustrates that current LLMs lack robustness in mathematical reasoning. Under zero-shot Chain-of-Thought (CoT) prompting. Qwen2.5 models perform strongly on the original GSM8K benchmark, with the math-specialized 7B variant achieving 95.3% accuracy. This result. confirms that the CoT prompt can effectively replicate benchnark-level performance. However,
31

---

Published in Transactions on Machine Learning Research (02/2026)
performance declines significantly on the GSM-Symbolic benchmarks, which introduce minor variations in surface features such as names and numerical values. For instance, the 30we2.5-Math--77 drops from 95.3% on GSMSK to 90.0% on GSM-Symbolic, and further to 76.8% and 68.0% on GSM-P1 and GSM-SP2, respoctively, as additional clauses are added to the original question. This trend reproduces a known limitation of current. LLMs: their mathematical reasoning abilities are brittle when faced with slight. perturbations in problem formulation (Mirzadeh et al. 2024
model
Table 13: Fragility of mathematical reasoning under zero-shot CoT prompling on GSM8K vs. GSM-Symbolic benchmarks.
Tasks
Qwen2.5-1.5B-Im
Qwen2.5-Matth1.55--nns
Qwen2.5-Math-7BBIne
GSM8K
65.1%
81.8%
95.3%
Symbolicc
54.8%
75.8%
90.0%
GSM-P1
32.0%
61.6%
76.8%
GSM-P2
5.0%
47.0%
68.0%
The effect of in-context learning on LoRA routing. Table 14 shows the effect of in-context learning on LoRA routing methods evaluated on the GSM-P2 benchmark. The in-context examples are taken from the GSM8K, as the standard benchmarking procedures (Beeching et al. 2023 Yang et al., 2024a). In the baseline (non-LoRA) models, observe that moving from 0-shot to 8-shot prompting yields little to no improvement, and in some cases, slight degradation, in performance. For example, the Qwwn2.5-Math--7B, drops from 0.68 (0-shot) to 0.42 (8-shot). This pattern aligns with prior findings reported in (Mirzadeh let al. 2024j, which demonstrate that in-context learning with GSM8K exemples offers limited benefit on the GSM-P2 benchmarks. Applying LoRA routing methods such as Uniform and Arrow, generally reduces performance compared to the base modcls, especially for the larger models. When using 8-shot in-context examples, all models show decreased accuracy overall, with the base models again outperforming the merged variants. This suggests that LoRA routing, in the current setup, may not effectively preserve or enhance model performance in compositional generalization tasks, and that in-context learning does not compensate for these declines.
model
Table 14: Accuracy on GSM-P2 after routing LoRAs individually fine-tuned on GSM--ymbollcc and GSM-P1, with and without in-context learning examples.
Number of In- In-Context Examples
LoRAA Routing Methods
Qwen2.5-1.5B-Inst
Qwen2.5 Qwenn2.5-Math--.5BB--mm
(w -asst
Base Model Only Uniform
5%
47%
68%
0-shot
10%
24%
34%
Arrow
9%
19%
27%
Base Model Only
0%
26%
42%
8-shot
Uniform
8%
19%
33%
Arrow
5%
18%
6%
The Change of Base Model's Behavior After Routing LoRAs. To analyze the impact of LoRA merging or routing on model behavior, we examined the Swee2.55-Math-1.53 outputs on the GSM-P2 evaluation set before and after routing. Using GPT-4o as an evaluator, we classified each solution as natural language, programming language, or unidentifiable. GPT-4o also assessed the correctuess of each answer and, for incorrect oulputs, calegorized the error type as semantic misunderstanding, step-missing- foolowinoo the definitions and prompting protocol of Zhong et all (2025). Errors that did not fit any of these categories were labeled as unidentifiable, while correct answers were assigned the label None. To assess the reliability of GPT-4o's judgments, we measured its agreement with the exact-match metric on answer correctness.
model's zero-shot CoT
calculation error,
Table 15 shows that routing LoRAs trained on natural language solutions can hinder the basc model's ability to solve math problems with programming language (i.c., tool-integrated reasoning). Uniform and Arrow merge methods produced 100% natural language solutions, while the base model originally generated 12%
32

---

Published in Transactions on Machine Learning Research (02/2026)
code-based responses. Both routing methods created high calculation errors (56%). These results suggest that while routing Ls trained on natural language solutions can impair domain-speccalizz math models' ability to solve problems using code. Thus, LoRA routing should consider the model's pretraining history and task-specific needs, as improper strategies could degrade its math problen-solving effectiveness.
Table 15: Analysis of the type of generated solutions and errors using GPT-40 as the judge.
Solution Types
Error Types
LoRA Routing Methods
Agreement
Natural Lang.
Code
Unident.
Semantic
Cale. Error
Step-Miss
Unident.
None (Correct)
None
87%
12%
1%
16%
29%
6%
1%
48%
97%
Uniform
100%
0%
0%
11%
56%
9%%
0%
24%
98%
Arrow
0%
0%6
17%
12%
A.5 Detailed experimental setups and results for cross-lingual code generation task
Setup To evaluate whether our conclusions generalize beyond symbolic logical composition, we study a cross-lingual, cross-domain code generation task. Specifically, we assess whether separately trained adapters for English code generation and English Spanish translation can be composed in data-free manner, using Qwen2.5-7B, Qwen2.5-14B, the MBPP-Translated splits, and the Uniform and
Phi-2, 1ni-3-mini-4k-inee Arrow merging methods.
Coding Adapter L, (English - Code Generation): Trained on the English subset of the MBPP-Translated dataset AnonymousDoe, 2024
Translation Adapter Le (English + Spanish):
Phi 2 Phi 3: We reuse EN ES ParaCrawl-based adapter from the 256-task FLAN v2 library (Ostapenko et al. 2024), trained solely on general-domain parallel data.
Qwen2.5 7B / 14B: Fine-tuned on the same dataset above using the hyperparameters from our main paper.
Evaluation We evaluate on the Spanish MBPP subset of the Transaaad dataset using data-free merging strategies: Uniform and Arrow.
Results As shown in Table 16 our findings from symbolic tasks are strongly supported by this cross- lons-lin
code generation experiment:
Phi models (English-only pretraining): The translation adapter L- drives almost all improve- ment. Merging does not unlock new capabilities, reaffirming our earlier conclusion.
Qwen2.5 models (multilingual pretraining): adapters.
Merging yields only marginal gains over single
Table 16: Compositional Performance on Cross-Lingus Code Generation (English Instructions + Spanish
MBPP Code)
Model
Base Only
Coding (I,)
Translation (I.2)
Merged (Uniform)
Merged (Arrow)
Phi-2
0.0%
1.8%
36.0%
28. 0%.1%
28.8%
Phi-o-mini-4k-instrree
0.0%
2.3%
37. 37.2%
27.9%
30.5%
1onn..57B-Inssruu
32.9%
34.0%
37.5%
34.9%
35.5%
Qwen2.5-14B-lnstrreett
38.7%
38.9%
44.0%
39.1%
40.2%
The dominance of the translation adapter (L2) and the degraded performance merged configurations indicates a faīlure of composition. To ensure this failure not due a weak coding adapter, we also evaluate L, on its native English MBPP task. As shown in Table 17 L, is highly effective on its original domain, confirming that the poor Spanish MBPP performance arises from failed composition, not weak components.
33

---

Published in Transactions on Machine Learning Research (02/2026)
Table 17: Validation of English Coding Adapter (L1) on Native Task (English MBPP)
Model
Base Only
Coding (L1)
Phi-2
56.0%
78.1%
Thi-3-miini44k-nnstee
60.3%
82.3%
Qwen2.5-7B: BBB-ttrruttt
71.7%
85.5%
Qwen2.5-14B-Inss Instruuct
76.2%
89.0%
Conclusion and Impact
The base model-dependent behavior observed here powerfully reinforces the mothodological choice beginning with a controlled two-hop synthetic setup: the synthetic environment isolates knowledge sources and enables clean attribution of which component contributes which capability. This isolation is crucial because, complex, knowledge-intensivve cross-domain reuse tasks, it inherently difficult to construct genuinely independent subtasks the base model often possesses some capacity in both domains (e.g., translation and coding, or history and geography), and the individual adapters themselves may learn overlapping pieces of the composite skill. We believe that this controlled approach, validated by its ability to analyze complex real-world failures, is an essential first step toward understanding and enabling viable data-free reuse.
A.6 Generalization to growledge-Intensiieg Tasks (History/Geooraphy))
In this section, we provided evidence that the pattern-matching mechanism extends knowledge-intensive cross-domain reuse.
Setup We directly reuse openly shared LoRAs to test whether pattern-matching in LoRA reuse extends beyond symbolic logical composition to knowledge-incensive tasks. We specifically provide new results on history and geography QA. These LoRAs, fine-tuned on Phi-2 as the base model, were originally trained on KILT/HotpotQA (subtask of Flan v2 Longpre et al. (2023) for varying levels of knowledge complexity Ostapenko al. (2024). We probe them on HotpotQA variants with geography history content in two settings:
Original questions, and
2. Continent-level probing (requiring answers at the continent level).
If mattenn-matching were the underlying mechanism for reusing via uniform merging arrow routing, would expect failures under the continent-level probing condition.
Setup and example pnnettuiin/evaluaatinn questions As shown in Table 18 the selected LoRA tasks span a range of reasoning complexities, from simple single-hop questions to multi-hop "final exam" style queries. These LoRAs were evaluated on 97 history/ history/geography question-answet pairs taken from original KILT/ HotpotQA validation set whose answers contain city a country in two sottings: 1) Original version, and 2) Probe to answer continent level, as illustrated in the Table 19
Results using Phi2 as the base model
As shown in Table 20 accuracy drops sharply under continent-level probing, reinforcing our claim that LoRA reuse relies on pattern-matching rather than genuine logical composition.
1. Reusing LoRAs yield unstable results. We observe changes in accuracy on original Hotpot which: e.g., uniform merging 446.3% vs arrow routing ~8. 0% when aggregating the four Hotpot LoRA tasks (excluding "formulate").
34

---

Published in Transactions on Machine Learning Research (02/2026)
Table 18: KILT/HotpotQA training examples for openly shared LoRAs.
LoRAs Task name
Example question (from KILT HotpotQA)
Example answer
filt_tasks hotpssto__t_________
"Who wrote the play "Romeo and Juliet
"William
straighforward qa: simple,
Shakespeare"
single-hop HotpotQA questions (no complex multi-document bridging)
kilt_taske _as____tott_____o__
"Which U.S. state is home to the university where Tim Cook earned MBA?"
"North Carolina"
question: each requires two or more
facts to answer
"Which continent contains the home country of the " "Europe" combining_ facts: These require two band ABBA?"
tasks _ookootaa_
clearly separate facts, joined by
explicit reasoning.
kilt_tasks hotpotqa final
"The author of "The Trial' was born in the capital of "Czech Republic" which modern country?"
exam: These are multi-hop, difficult, "exam-style" reasoming questions that often combines 3+ facts.
tasks hottsstootttaaaa___ formulate: multi-hop or semi-multi-hop question formulation, rephrasing, query interpretation
"Rewrite question that seeks the birthplace of the painter who created Guernica."
"In which city was the artist who painted Guernica born?"
Table 19: Examples of geography/history related HotpotQA questions and their corresponding continent-level probing prompts with answers.
Original question (from KILT HotpotQA)
Original answer
Continent Prompt ques- tion
answer
"Where was the world cup hosted "Brazil" that Algeria qualified for the first time into the round of 16?"
"On which continent the "South America"
location related to the fol- lowing question: Where was the world cup hosted that Algeria qualified for the first time into the round of 16?"
"In what city was the Italian "Venice" Baroque composer who composed a set of sonatas, titled Op. 5, born?"
"On which continent the location related to the following question: In what city was the Ital- ian Baroque composer who composed a set of sonatas, titled Op. 5, born?"
"Europe"
2. Pattorn-matching as the underlying mechanism. When prompted to answer the same question at the continent level, accuracy greatly dropped. This supports our claim that successfully reusing LoRAs on unseen tasks driven by pattern-matching instead of genuin logical composition.
Impact of our results
This results highlight the risk of reusing openly shared LoRAs without data access since this approach was driven mainly by pattern-matching rather instead of geniue logical composition. By combining rigorous theoretical analysis with well-controlled experiments, we deliver the first mechanism-based account of when and why LoRA reuse (or cannot) substitute for data access. This critical insight for practitioners
35

---

Published in Transactions on Machine Learning Research (02/2026)
Table 20: Accuracy results for LoRA reuse on original HotpotQA questions versus continent-level probing
using Phi-2 as the base model.
Phi2 / Original Question
Phi2 / Continent Prompt
Merge or routing methods
Base
Uniform
Arrow
Base
Uniform
Arrow
kilt_tasks: tastsassssaas catiss_oottttt_tttllaetee noopottu_staaiglooaar..o
30.90%
6.10%
3.09%
1.03%
2.06%
1.03%
all 4 hotpoiqa (excl. formulate)
46.30%
8.00%
5.15%
2.06%
all 5 hotpot qa
40.20%
11.30%
4.12%
0.00%
operating under data-sharing constraints. It's also important for academic researchers to explore other mechanisms with well-controlled experiments.
36


Reviews:
[{"review_id": "A9R6oHFt1t", "reviewer": "Reviewer_54XL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary:**\n\nLoRA recycling/reuse refers to the idea of combing LoRA adapters from independent fine-tuning jobs to solve new tasks by combining the knowledge from the component adapters. Many methods exist for reuse, but there is little work on understanding the mechanism behind these methods or why & when they work. This work studies when LoRA reuse without access to training data can be useful using a simplified theoretical model and a series of empirical experiments on two-hop reasoning and math reasoning.\n\nOn the theory front, the authors consider a highly simplified transformer model consisting of a *frozen* randomly initialized embeddings and single attention layer followed by a trainable classification MLP. They consider the setting of combining two LoRAs trained on two separate relations (A->B => B is spouse of A; B->C => B lives in C ) to perform two-hop reasoning (A->C) that requires knowledge from both LoRAs. They demonstrate that averaging adapters or methods like CAT or Arrow cannot result in compositional generalization in their specific setup.\n\nOn the empirical side, they conduct experiments on publicly available LLMs with two-hop reasoning and math reasoning. The two-hop reasoning is similar to the setup in the theoretical analysis where they investigate whether merging LoRAs for A->B and B->C can lead to inference of A->C. Their results show that merging the two LoRA adapters leads to very poor performance on the A->C task. Additionally, they also find that using familiar entities/relations from the pretraining data helps improve performance, and methods like CoT are ineffective in addressing the poor performance of merged adapters. They arrive at similar conclusions for the math reasoning experiments.\n\nTheir main findings overall are: (i) LoRA reuse is ineffective regardless of model and data size, (ii) it's not possible to reliably merge LoRAs from logically disjoint datasets, and you need overlap like similar reasoning patterns for LoRA reuse to be effective, (iii) LoRAs depend more on shallow pattern matching than compositional generalization.\n\n**Weaknesses:**\n1. The paper is easy to understand up to section 3. Section 4 has a very high cognitive load and I found it difficult to follow. There are so many variables and dimensions to keep track of. Many important experimental details are in the Appendix A2 instead of the maintext. I had to reread several times and go back and forth to the appendix to understand the section. See later comments for specific instances.\n2. The theoretical model isn't a convincing model for or hasn't been motivated enough. The assumptions appear to be gross simplifications of an actual transformer; e.g., only training the final classification weights, uniform softmax logits, etc. I'm also not sure whether the random features model being a transformer has any effect on the analysis. Could it have been any random neural network that can mix across tokens?\n3. The theoretical model focus primarily on retrieving/editing factual information as the LoRA parameters are likely learning to add/edit factual information in the weights. In such scenarios, it naturally feels like LoRA compositions like deducing transitive relationships would be unlikely to work. Is this the only mechanistic view of LoRA parameters?", "claims_and_evidence_yn": "No", "claims_explanation": "Somewhat. The claims in the abstract and introduction are very broad and general, whereas the experiments and the theoretical analysis study very specific settings. I am not convinced that their conclusions generalize to all usecases of LoRA reuse/recycling; e.g., altering personalization of an LLM by varying mixtures of LoRAs likely does not use the factual retrieval/editing mechanism the paper is concerned with.", "audience_yn": "Yes", "audience_explanation": "I'm not convinced about the value of the theoretical work here but there certainly are people who be interested in it. There are many empirical findings that are interesting such as familiarity playing a substantial role in the ability to reuse adapters.", "requested_changes": "**Minor writing related comments:**\n1. The main text does not mention how the 2-combination and 3-combinations adapters were obtained in 4.1. The appendix suggests Arrow and uniform averaging were tested and the authors stuck with Arrow, but I think the method should be made clear in the maintext.\n2. A concrete example of the inputs and targets from A.2.1 would be helpful in section 4.1. I had to reread a few times and then visit A.2.1 to understand what was happening; for a moment, I thought <name, location> were the items in the dataset and was completely lost.\n3. It might be useful to mention there is a residual connection and casual modeling w/o attending to self in Section 3. Proposition 1 looks a bit sus on first glance while attempting to eyeball the expressions to the forward pass expressions. It's not clear without looking at the appendix, but it would be easier to interpret had the residual connection and causal masking w/o self was mentioned.\n4. What about trainability of K, Q in section 3? The appendix mentions that K, Q are randomly init and left untrained, but is omitted in the maintext for some reason while others weight matrices are explicitly discussed.\n\n**A non-exhaustive list of nitpicks:**\n1. \"To address this, we began with the theoretical analysis *revealed* its limit.\"\n2. \"LoRA reuse can be successful without direct data-access when the common solution templates (e.g.,\nchain-of-thoughts reasoning patterns or re-usable code snippets) *existed* in the fine-tuning datasets\nas bridges.\"\n3. A.2.2: \"This is a special case of LoraHub (Huang et al., 2024) *axssigning* each LoRA adapter the same weight.\"\n4. 4.1.1: \"in smaller models and the presence *of of* unfamiliar entities\"", "broader_impact_concerns": "N/A.", "additional_comments": "**Questions:**\n1. What are “noise tokens” in section 3 introduction?\n2. Were the number of LoRA parameters kept constant for Figure 1 models? Is it possible that the LoRA parameters are increasing with model size and the models are better able to memorize the training data in combination 3 setting? What stops the LoRA parameters from memorizing all A -> C pairs?\n3. In A.1, Eq (5), why is the current token not allowed to attend to itself? Casual language models often allow the tokens to attend to all past and itself. It also appears like the analysis relies on this fact.\n4. Would implications of LoRA on W in section 3 mean anything for LoRA on attention/MLP blocks in transformers? Transformers may rely on a completely different mechanism than LoRA on singled out linear layers. For example, the LoRA parameters of the two feedforward layers in MLP blocks of transformers could be coupled and work in tandem.\n5. Why is the proposed theoretical model a good proxy for a real transformer? It appears to be an extremely simplified version of a transformer. While analyzing an actual transformer might be practically intractable, I think there should be convincing explanation for why the results from these simplified models transfer to real models.\n6. “We follow this assumption, and additionally take U, V to remain untrained, as we do not assume noise tokens in the context.” What does this mean?", "extra_scores": {}, "extra_text": {}}, {"review_id": "T9KAbRBWVt", "reviewer": "Reviewer_arGZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies whether parameter-efficient fine-tuning modules (LoRAs) can be reused for new tasks without access to original training data. Through a combination of theoretical analysis and empirical evaluations across two representative logical reasoning tasks (two-hop symbolic reasoning and math word problems), the authors show that simply merging or routing LoRAs generally fails to achieve true compositional generalization. The paper further identifies that reuse only succeeds when LoRA training data share reusable solution patterns (e.g., shared CoT or executable code), suggesting that LoRA reuse primarily exploits surface-level template matching rather than genuine reasoning combination. These findings provide a valuable mechanistic understanding of the current limitations of LoRA-based model reuse.\n\nStrengths:\n\n1.Addresses an important and timely problem regarding LoRA reuse under data-restricted settings.\n\n2.Clear theoretical insights supported by controlled experiments across different model scales.\n\n3.Provides actionable takeaways for the community (e.g., the importance of reusable reasoning templates).\n\nWeaknesses:\n\n1.Task scope is limited to symbolic logical composition. It remains unclear whether the conclusions generalize to more semantic or knowledge-intensive scenarios (e.g., combining independently trained history and geography LoRAs for cross-domain QA).\n\n2.Success depends strongly on bridge data. When CoT/code templates are introduced, the setting shifts from “pure LoRA reuse without data” to providing additional compositional supervision. Clearer clarification of this boundary would improve the framing.\n\n3.Theory–practice gap not fully addressed. The theoretical analysis is based on a single-layer abstraction; empirical discussion on how multi-layer LLM structures might interact with LoRA interference would strengthen generality of the conclusions.\n\n4.Evaluation focuses mainly on final answer accuracy. Additional analysis of reasoning faithfulness or step-level correctness would better support the claim about failure to achieve true compositionality.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper’s main claims are well supported through a combination of theoretical analysis and carefully controlled experiments. The synthetic two-hop reasoning task and the symbolic math setting are both designed to isolate compositional ability, and the results consistently validate the theoretical prediction that simple merging or routing of LoRAs fails to produce true compositional generalization. The use of different model scales, different entity familiarity conditions, and bridge-data ablations further strengthens the empirical rigor. Moreover, the behaviors observed across multiple settings all align with the proposed mechanism that LoRA reuse predominantly captures surface-level template matching rather than logical combination of knowledge.", "audience_yn": "Yes", "audience_explanation": "The topic is highly relevant to the TMLR community. While LLMs are increasingly powerful, their training and adaptation costs remain substantial. Understanding whether strong performance on new tasks can be achieved by reusing existing LoRA modules—without accessing original training data or performing expensive retraining—is both practically meaningful and scientifically important. The insights provided here about when LoRA reuse succeeds or fails will directly benefit researchers and practitioners who rely on PEFT methods for scalable model deployment.", "requested_changes": "The first two weaknesses are critical: (1) clarifying the scope of the conclusions beyond symbolic logical composition (e.g., whether the findings extend to knowledge-intensive cross-domain reuse such as history+geography QA), and (2) more clearly defining the role and boundary of bridge data, since its introduction adds compositional supervision beyond pure LoRA reuse. Additionally, discussing alignment between the single-layer theory and multi-layer LLM behavior, and including some form of reasoning-faithfulness analysis, would further strengthen the work but are not required for acceptance.", "broader_impact_concerns": "No major ethical concerns. This work focuses on understanding the limitations of LoRA reuse and does not introduce new deployment risks.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wbmp5decqm", "reviewer": "Reviewer_SUrz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the limits of LoRA reuse without data access. Through experiments on reasoning/math tasks, it suggests that standard merging methods (like Uniform or Arrow) generally fail for compositional tasks. \"Success\" in their setups is largely observed when \"bridge\" templates (shared CoT structures) are present in the training data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The theoretical derivation gives a solid intuition for the subspace mismatch. The experimental controls are rigorous. And the paper is generally honest about the limitations (e.g., acknowledging performance degradation in specialized models), which adds credibility.", "audience_yn": "Yes", "audience_explanation": "LoRA reuse and merging are widely used in practice, and this paper offers a clear takeaway on when composition fails and what kinds of template alignment make it work.", "requested_changes": "I recommend acceptance. This paper serves as a valuable \"reality check\" for the field. I have a few minor suggestions that I think would strengthen the final manuscript.\n\n1. Table 14 offers a very interesting insight. The Base Model utilizes code-based responses in roughly 12% of cases, but the Merged models (Uniform/Arrow) appear to drop this to 0%, reverting entirely to Natural Language. This suggests the merging process might be suppressing the specialized \"tool use\" capability, biasing the model back to the dominant modality (text) of the individual LoRAs. Explicitly discussing this mechanism in the error analysis would add significant depth.\n\n2. The positive results in Section 4.1.2 rely on the LoRAs being trained with the specific CoT structure of the target task. It might be beneficial to restate in the Discussion that this acts as a form of format/template exposure. Since effective reuse seems to require injecting the target reasoning pattern into the LoRA training, clarifying this constraint would help practitioners better understand the practical limits of \"data-free\" reuse.", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u0sOaub8e1", "reviewer": "Reviewer_oCHB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies when and how LoRA adapters trained on different tasks can be reused or combined, without access to the original training data, to solve new tasks. On the theory side, the authors analyze a one-layer transformer with LoRA applied to the MLP layers and show that rank-one LoRA updates corresponding to different one-hop relations do not compose to yield correct two-hop reasoning. On the empirical side, the authors construct synthetic two-hop reasoning datasets and run experiments with Qwen2.5 and DeepSeek models ranging from 1.5B to 70B parameters. The authors observe that data-free merging or routing methods (Uniform, Arrow, TIES) rarely realize two-hop composition. However, performance improves when the fine-tuning data includes explicit bridge solution templates that link the one-hop tasks to the target composition.", "claims_and_evidence_yn": "Yes", "claims_explanation": "* The core claims about data-free composition of LoRAs are supported by both theory analysis and corresponding experiments. Two-hop experiments across six base models and three dataset variants (F, H, R) consistently show near-perfect performance for single-task experts, but less than 10% accuracy on the composed A→C relation for merged or routed LoRA libraries, in line with theoretical predictions.\n\n* The claims that bridge datasets with explicit templates can unlock reuse are also supported by experiments. More specifically, the authors vary whether CoT templates for the A→C relation appear in the bridge, whether these templates are shared between experts, and whether the bridge covers the same relations or alternative ones. The results obtained match the claims. \n\n* However, the theoretical model is a bit simple (one layer and LoRA applied only to MLP layers). And all empirical tasks are synthetic, focused on two-hop reasoning, and based on GSM benchmarks. The evidence for generality is not sufficiently convincing.", "audience_yn": "Yes", "audience_explanation": "* LoRA-based fine-tuning and reuse of adapters are widely used in practice and have received much attention in recent works on parameter-efficient fine-tuning (PEFT). This paper combines theoretical analysis and thorough experiments on when such reuse fails and under what conditions it starts to work, which will interest readers working on PEFT, domain adaptation, and model interpretability probably. \n\n* The two-hop results provide a guideline for researchers designing adapter libraries: naive merging or routing is unlikely to produce promising multi-hop reasoning, even when each adapter solves its own task very well. The experiments on GSM benchmarks connect these ideas to math reasoning and highlight interactions between LoRA reuse again. \n\n* The observations also help set expectations about what LoRAs can do without additional training data, thereby contributing to future research on domain adaptation and test-time learning.", "requested_changes": "* It would be helpful to extend the theoretical analysis to multi-layer settings to make it more convincing.\n\n* In the current version, all empirical results are on synthetic two-hop reasoning with small entity sets and on GSM-based benchmarks. It would be better to add at least one additional task family or a non-math reasoning task, for example, coding generation.\n\n* Most related works are discussed only conceptually, such as LoRA Lego, LoRI, Self MoE, and FLiX. It would strengthen the contribution to run at least one or two of these on a representative two-hop setting.\n\n* About clarity. Some important details are only in the appendices. For instance, a summary of training hyperparameters for two-hop tasks is suggested to be included in the main text (*e.g.*, a short table summarizing learning rates, batch sizes, optimizer, etc.).", "broader_impact_concerns": "This paper did not provide a broader impact statement. I did not see major ethical issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)