Composable Interventions for Language Models

## Abstract

Abstract Test-time interventions for language models can enhance factual accuracy, mitigate harmful outputs, and improve model efficiency without costly retraining.
But despite a flood of new methods, different types of interventions are largely developing independently.
In practice, multiple interventions must be applied sequentially to the same model, yet we lack standardized ways to study how interventions interact.
We fill this gap by introducing composable interventions , a framework to study the effects of using multiple interventions on the same language models, featuring new metrics and a unified codebase.
Using our framework, we conduct extensive experiments and compose popular methods from three emerging intervention categories— knowledge editing , model compression , and machine unlearning .
Our results from 310 different compositions uncover meaningful interactions: compression hinders editing and unlearning, composing interventions hinges on their order of application, and popular general-purpose metrics are inadequate for assessing composability.
Taken together, our findings showcase clear gaps in composability, suggesting a need for new multi-objective interventions. 1 1 1 All of our code is public: github.com/hartvigsen-group/composable-interventions † † Correspondence: {arinbjorn,hartvigsen}@virginia.edu , kyobrien@microsoft.com

## 1 Introduction

Language models (LMs) exhibit striking capabilities on important tasks in myriad domains including medicine [], finance [], science [], and entertainment []. But despite high performance, deployed LMs can still misbehave unpredictably and require updates.
For example, LMs generate content that is hallucinatory [], factually incorrect [], and harmful [, , ].
Beyond unwanted behaviors, user requirements also change over time.
For example, regulations arise [], computational resources constrict, knowledge gets outdated [], and copyrighted training materials are identified [].
Without ways to quickly address these issues, models can be left miscalibrated, outdated, and biased, limiting their widespread responsible use [].

Many recent works study efficient, in-place updates for LMs.
We broadly refer to these as interventions—updates to targeted properties of LMs applied after pretraining (and optional fine-tuning).
For example, we can view model compression [, , ] as an intervention to make language models more inference- or memory-efficient.
Other rapidly-advancing examples include knowledge editing [, , , , , ], detoxification [, , , ], and unlearning [, , ].
However, these interventions are largely advancing independently.
In practice, we usually have multiple requirements for properties of our models (e.g., factuality and efficiency).
And as requirements change over time, new interventions must be applied.
While some works have started studying interactions between training objectives [, , ], practical widespread use is limited without unified evaluations for how interventions interact.

As depicted in Figure 1, we propose that practical interventions should be composable: When an intervention is applied to a model, it should not interfere with prior or future interventions.
For example, we might repair a factual error in an LM using a knowledge editing method.
Later on, we might quantize that model.
A composable quantization method should specifically preserve the edited fact and a composable editor should withstand quantization.
However, it remains unknown how well existing interventions compose, and we lack formal notions of intervention composition.
We therefore propose two metrics for composability: 1) Order-free Error, where an intervention is composable if its application leaves others’ success unimpacted, and 2) Order Sensitivity, where the combined success of multiple interventions should not depend on the order in which they are applied.
We instantiate these metrics in a codebase that includes popular interventions, tackling a major lack of standardized code that has inhibited cross-intervention evaluations until now.

Figure: Figure 1: Interventions aim to update targeted properties of language models without impacting unrelated behaviors or adding excessive compute. We introduce and extensively experiment with composability of different interventions used on the same model.
Refer to caption: x1.png

We use our framework to extensively study composability of state-of-the-art knowledge editing, model compression, and machine unlearning interventions on Llama3-8B [].
Our experiments with 310 different compositions unearth novel insights and offer guidance for composing recent methods.
Three key results are as follows:
1 Model compression often limits the success of other interventions.
This suggests a significant drawback of existing compression methods, which are nearly universal in LM deployments.
2 The order in which interventions are applied dramatically alters their success.
Therefore, we need new interventions explicitly designed for composability.
3 General-purpose post-intervention model performance is a poor proxy for composability.
This indicates that targeting composability as a metric has the potential to drive the development of new, practically-grounded interventions.
Taken alongside our other findings (Section 4), our experiments suggest we sorely need to broaden intervention method evaluations and to design new methods explicitly for composability.

Our key contributions are as follows:

- •
We introduce the notion of composability to language model interventions, formalizing the sequential applications of multiple interventions to LMs. Our work opens doors to address crucial, practical challenges in online LM updates and broadens evaluation criteria for interventions.
- •
Our main contribution is extensive experimentation, identifying unknown interactions between knowledge editing, model compression, and machine unlearning.
- •
Our findings suggest a clear need to develop novel interventions that target composability, a crucial property of practical interventions.
- •
We release an extendable codebase that unifies many state-of-the-art implementations to enable others to develop new multi-objective interventions.

## 2 An Evaluation Framework for Composable Interventions

### 2.1 Preliminaries: Interventions for Language Models

Assume we are given a language model $f_{\theta}$, where $\theta\in\Theta$ contains its pre-trained parameters and $\Theta$ is the parameter space.
We define an intervention $\phi$ to be the combination of four things:
1) an operator $\omega(f_{\theta},\gamma)$ that produces a new model $f_{\theta}$,
2) hyperparameters $\gamma$, which typically control the strength of intervention,
3) a loss $\ell(\omega(f_{\theta},\gamma))$ that specifies the targeted update,(^2^22Some interventions require data, some do not. We remove direct data dependencies for simplicity.)
and 4) evaluation criteria $\kappa(f_{\theta},\mathcal{D})\in[0,1]$, where higher scores indicate better performance.
Each intervention’s operator $\omega$ aims to minimize loss $\ell$ and ultimately achieve high performance according to $\kappa$.
For readability, we denote the application of intervention $\phi$ to model $f_{\theta}$ as $\phi(f)$.

### 2.2 Composable Interventions for Language Models

Now consider $N$ possible interventions $\{\phi_{i}\}_{i=0}^{N-1}$.
We define intervention composition as the application of multiple interventions to a model $f$ in sequence.(^3^33For readability, we describe our framework in terms of two interventions.)
For example, we might apply intervention $\phi_{0}$ to model $f_{\theta}$, then $\phi_{1}$ to the intervened-upon model.
We can write this composition as either $\phi_{1}(\phi_{0}(f_{\theta}))$ or $\phi_{1}\circ\phi_{0}(f_{\theta})$, and here we use the latter format.

Most interventions $\phi_{i}$ are developed specifically for their own criterion $\kappa_{i}$.
For simplicity, we let each intervention have one criterion, though in practice they often have multiple.
We propose that the quality of many interventions in practice should also relate to how an intervention impacts our ability to apply other interventions.
To meet this need, we propose two metrics that measure such composability.
Each is computed for a given criterion $\kappa_{i}$ and can be computed for a single hyperparameter setting or ranged over multiple settings to attain a richer description of intervention interactions.

Our evaluation strategy is straightforward: After applying an intervention $\phi_{i}(f_{\theta})$, we measure its success as $\kappa_{i}(\phi_{i}(f_{\theta}))$.
Then, we can add another intervention $\phi_{j}$ and measure $\kappa_{i}(\phi_{j}\circ\phi_{i}(f_{\theta}))$.
The difference between these measures then elicits how $\phi_{j}$ impacts the success of $\phi_{i}$ according to $\kappa_{i}$.
To disentangle absolute performance on $\kappa_{i}$ from the interventions, we explore the impact of ordering by reversing the interventions and computing $\kappa_{i}(\phi_{i}\circ\phi_{j}(f_{\theta}))$.
By pairing absolute changes with results from different orderings, we isolate direct interactions between interventions.
Overall, this approach provides guidance when choosing interventions to compose and illuminates failure modes of existing methods.
We next describe two concrete composability metrics.
The first describes best-case composability regardless of order, and the second isolates the impact of ordering.

Composability Error Metric 1: Order-free Error.
Interventions are more composable if they achieve high performance when applied in either order.
We therefore measure the impacts of adding a second intervention to a model regardless of order for a chosen criterion $\kappa_{i}$ using order-free error, computed as the area above the maximum of two curves.
A small area means higher overall performance on the metric.
We use the maximum criterion value as an upper bound for this area, and assume it is 1, though this is easily changed in practice.
Each curve corresponds to an order in which each intervention is applied and sweeps over hyperparameter choices for $\gamma_{i}$:

$$ $1-\int_{\gamma_{i}}\min(\underbrace{\kappa_{i}(\phi_{j}\circ\phi_{i}(f_{\theta },\gamma_{i}))}_{\text{intervention order $i\rightarrow j$}},\underbrace{ \kappa_{i}(\phi_{i}(\gamma_{i})\circ\phi_{j}(f_{\theta}))}_{\text{intervention order $j\rightarrow i$}}).$ (1) $$

A smaller area indicates better composability.
By sweeping over hyperparameters, this measure provides a general sense of composability without choosing hyperparameters ahead of time.
The area can be computed for any criteria $\kappa$—in our experiments, we evaluate all criteria for each composition.
In practice, not all hyperparameters are practical.
For example, harsh model compression can have catastrophic impacts on model utility and so comparing interventions at extreme levels may be unrealistic.
Therefore, we can also set a bound on acceptable performance decay with respect to $\kappa_{i}$ or any other criterion when using Equation 1 and can also easily be measured for hand-chosen hyperparameters, and for a single setting $\gamma_{i}$, Order-free Error reduces to the simple error score:
$1-\min(\kappa_{i}(\phi_{j}\circ\phi_{i}(f_{\theta},\gamma_{i})),\kappa_{i}(
\phi_{i}(\gamma_{i})\circ\phi_{j}(f_{\theta})))$.

Composability Error Metric 2: Order Sensitivity.
Interventions are more composable if their performance is invariant to their order of application.
We measure order invariance for criterion $\kappa_{i}$ as the area between the two curves defined by each direction of intervention application.
Again, we compute this across hyperparameter choices for $\gamma_{i}$:

$$ $\int_{\gamma_{i}}|\kappa_{i}(\phi_{j}\circ\phi_{i}(f_{\theta},\gamma_{i}))- \kappa_{i}(\phi_{i}(\gamma_{i})\circ\phi_{j}(f_{\theta}))|.$ (2) $$

A smaller area again indicates better composability.
And again, if hyperparameters are already chosen, this simplifies to a simple error: $|\kappa_{i}(\phi_{j}(\phi_{i}(f_{\theta})))-\kappa_{i}(\phi_{i}(\gamma_{i})
\circ\phi_{j}(f_{\theta})))|$.

Overall, measuring Order-free Error and Order Sensitivity provides the first insights into intervention composability.
Each reduces to a simple form for point measures of the impact of composition.
While more computationally costly, the broader sweeps over hyperparameters enable clearer pictures of how interventions interact.
As shown in Section 3, the curves produced by these sweeps also enable intuitive, visual understandings of composability.

## 3 Experimental Setup

We study the effects of sequentially applying multiple interventions to the same model on performance as measured by intervention-specific metrics and overall utility. Understanding these effects is crucial for the practical application of test-time interventions. The following sections detail our experiments to illuminate how composable various interventions are across model editing, unlearning, and compression.

### 3.1 Intervention Datasets, Methods, and Metrics

We next detail the ten techniques selected from the three categories of interventions, along with their task-specific and overall utility metrics. Appendix B further describes the backgrounds of the three types of intervention and Table 1 summarizes our experimental setup.

**Table 1: Summary of the methods, data, and metrics used in our experiments. All experiments are conducted with Llama3-8B. Composability metrics apply to all experiments.**
| Interventions | Methods | Datasets | Intervention<br>Metrics | Composability<br>Metrics |
| --- | --- | --- | --- | --- |
| Knowledge<br>Editing | Finetuning, LoRA [],<br>MEMIT [] | zsRE [] | Edit Success, Edit Generalization,<br>Strict Edit Locality, MMLU | Order-free Error<br>(Equation 1)<br>Order Sensitivity<br>(Equation 2) |
| Model<br>Compression | Pruning: Wanda [], SparseGPT []<br>Quantization: GPTQ [], AWQ [] | – | MMLU |  |
| Machine<br>Unlearning | Gradient Ascent (GA) [],<br>Gradient Difference (GD) [],<br>Representation Misdirection<br>Unlearning (RMU) [] | WMDP [] | WMDP,<br>MMLU |  |

Knowledge Editing.
LMs’ knowledge can be fundamentally incorrect or grow outdated. Knowledge editing aims to update facts according to LMs, typically framed as question–answering.
Here, where models are edited to return updated answers to questions by increasing the likelihood of the correct answers.
We use three editors, including the popular MEMIT [], as well as LoRA [] and finetuning, which have recently been shown to be surprisingly successful editors despite their simplicity [, , ].
We use the zsRE [] dataset, which is a popular question-answering benchmark for knowledge editing.

To evaluate editors’ performance, we use three standard metrics []: Edit success measures whether the post-edit model successfully outputs the correct, edited response. Edit generalization measures how well the edit generalizes to other semantically-equivalent inputs. Following recent work, we compute generalization as the average edit success across a set of 10 holdout edit rephrasings.
Strict Edit locality measures the impact of edits on unrelated inputs as the average success on random, unedited samples.
All editing metrics are computed as the F1 score on the correct output logits as a strict measure of editing performance—intended outputs must be the most-likely tokens after editing.
Each metric’s highest possible value is 1.0.

Machine Unlearning.
LMs can learn undesirable knowledge from their pretraining data, including information that is potentially dangerous []
copyrighted [, ],
toxic [, , ],
or memorized [, , , ].
Machine unlearning methods aim to remove undesirable knowledge from LMs without regressing performance on extraneous domains [, ]. We experiment with three popular and representative methods: Gradient Ascent (GA) [], Gradient Difference (GD) [], and Representation Misdirection for Unlearning (RMU) [].
GA is a traditional approach that minimizes the likelihood of chosen unlearning targets.
GD augments GA by also maintaining the LM’s predictions on unrelated retention data.
RMU, the most recent method, perturbs the model’s representations for inputs related to the learning target.
Further details are in Appendix C.4.

We evaluate unlearning with the recent QA dataset Weapons of Mass Description Proxy (WMDP) [], updating LMs to forget potentially hazardous biosecurity and cybersecurity knowledge.
We average the performance on WMDP’s cyber and bio splits, totaling 3,260 questions. We report individual split performance in Appendix D.
Optimal unlearning yields 25% accuracy as this is random performance on the four-choice benchmark.

Model Compression. LMs require significant compute resources during pretraining, finetuning, and inference.
Model compression methods aim to reduce resources needed at test-time.
We experiment with four popular, state-of-the-art weight pruning and quantization methods.
For pruning, we choose SparseGPT [] and Wanda [].
For quantization, we choose GPTQ [] and AWQ [].
Pruning and quantization are used for different reasons, so we generally avoid comparisons between the two approaches.
Each algorithm has a hyperparameter that controls how much to compress the model—% of zeroed weights for pruning, bits for quantization.
Naturally, efficiency grows alongside compression, but at the expense of task performance.
We vary this hyperparameter to explore the impacts of various levels of compression, choosing sparsity levels of .25, .35, .45, .55, .65, and .75 for pruning and 2, 3, 4, and 8 bits for quantization.
The uncompressed model is 16 bits.
Editing or unlearning some compressed models requires decompression, in which case we recompress afterwards using the same compression technique.

Measuring General Model Utility. All interventions aim to avoid degrading overall model utility.
While assessing general performance of LMs is an active area [, , ], we make the standard choice to evaluate question–answering accuracy on MMLU [] using the LM Eval Harness [].
Accuracy of 25% indicates random predictions.

### 3.2 Implementation details

All experiments are performed with Llama3-8B [], which is well-studied and highly capable for its parameter count.
All results for editing methods are averaged over 10 batches of 50 random edits from zsRE.
Further details on implementations and hyperparameters are in Appendix C.

## 4 Results

In the following subsections, we first examine how each pair of interventions compose individually, then summarize general trends.
We measure composability using our metrics from Section 2 for each intervention’s criteria and report impacts of compression at different levels.
All intervention pairs are applied in both directions (e.g., MEMIT before AWQ and AWQ before MEMIT).
For composability metrics, we compare methods regardless of metric ranges by counting the number of times each intervention method outperforms the others (# Wins) with ties counting as losses.

### 4.1 Composing Model Compression with Knowledge Editing

Figure: (a)
Refer to caption: x2.png

**Table 2: Composability metrics for Knowledge Editing composed with Model Compression. Order-free Error (↓) measures best-case error caused by interventions, while Order Sensitivity (↓) measures unwanted effects of intervention ordering. # Wins denotes the number of times each method outperforms other methods of its intervention category, counting all ties as losses. We use 25% sparsity and 4-bit quantization to showcase practical compression levels. Key Takeaways: AWQ composes better than GPTQ and LoRA is surprisingly non-composable.**
|  | Edit Success | Edit Generalization |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Order-free Error ($\downarrow$) | Order Sensitivity ($\downarrow$) | Order-free Error ($\downarrow$) | Order Sensitivity ($\downarrow$) |  |  |  |  |  |  |  |  |  |
| Method | FT | MEMIT | LoRA | FT | MEMIT | LoRA | FT | MEMIT | LoRA | FT | MEMIT | LoRA | # Wins |
| Wanda | .01 | .05 | .05 | .03 | .24 | .08 | .20 | .10 | .46 | .04 | .21 | .01 | 6 |
| SparseGPT | .01 | .09 | .14 | .03 | .04 | .01 | .23 | .14 | .48 | .02 | .03 | .07 | 4 |
| \cdashline1-14<br>AWQ | .01 | .07 | .00 | .04 | .01 | .08 | .16 | .11 | .26 | .05 | .00 | .14 | 12 |
| GPTQ | .21 | .16 | .42 | .36 | .03 | .37 | .40 | .20 | .59 | .36 | .11 | .27 | 0 |
| # Wins | 2 | 1 | 1 | 1 | 2 | 1 | 0 | 4 | 0 | 1 | 3 | 1 |  |

### 4.2 Composing Model Compression with Machine Unlearning

We next compose three model compression and three machine unlearning interventions.
We measure unlearning success with WMDP (Section 3.1).

Figure: (a)
Refer to caption: x4.png

Compression hinders unlearning.
Our main results in Figure 3 show that applying unlearning to a compressed model can degrade performance. With RMU, pruning the model before unlearning can significantly regress unlearning performance, especially at higher sparsity levels. We observe a similar trend with quantization when composing RMU and GPTQ. With GD, we similarly find that pruning after unlearning is best, though quantizing first can be optimal. We observe the opposite trend with GA, though GA’s poor unlearning performance, as seen in other studies [, , ], is a possible confounder. We overall find that unlearning a compressed model is more challenging but that the degree to which unlearning after compression hurts performance is highly contingent on the composition and compression level.

Order Sensitivity can determine overall composability.
Table 4.3 reports the composability metrics of each unlearning technique. As measured by Order-free Error on WMDP, GD and RMU outperform GA, which struggles to unlearn without significantly degrading utility. GD exhibits higher WMDP Order Sensitivity for pruning than quantization, whereas RMU has low Order Sensitivity across all techniques except when quantized by GPTQ. Despite both GD and RMU’s successful unlearning, RMU has lower overall Order Sensitivity than GD, suggesting that while task performance is critical, it alone does not fully capture the practical challenges of applying interventions—RMU is more robust to order.

### 4.3 Composing Knowledge Editing with Machine Unlearning

We next examine the composability between editing and unlearning. Both categories of interventions aim to modify the model’s knowledge without degrading overall utility. The need to keep LMs factually updated while forgetting undesirable knowledge makes pairing editing and unlearning a practical composition.

**Table 3: Composability metrics for machine unlearning and model compression. Key Takeaways: GD and RMU have overall comparable performance and far outperform GA. RMU is less sensitive to ordering than GD.**
|  | WMDP (Unlearning) | MMLU |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Order-free Error ($\downarrow$) | Order Sensitivity ($\downarrow$) | Order-free Error ($\downarrow$) | Order Sensitivity ($\downarrow$) |  |  |  |  |  |  |  |  |  |
| Method | GA | GD | RMU | GA | GD | RMU | GA | GD | RMU | GA | GD | RMU | # Wins |
| Wanda | .46 | .35 | .29 | .03 | .17 | .03 | .46 | .42 | .43 | .02 | .01 | .01 | 4 |
| SparseGPT | .43 | .28 | .28 | .03 | .22 | .02 | .49 | .42 | .43 | .06 | .09 | .01 | 4 |
| \cdashline1-14<br>AWQ | .43 | .24 | .27 | .02 | .00 | .01 | .53 | .58 | .44 | .02 | .07 | .01 | 5 |
| GPTQ | .34 | .24 | .27 | .09 | .02 | .18 | .52 | .44 | .42 | .09 | .24 | .05 | 4 |
| # Wins | 0 | 2 | 1 | 0 | 2 | 1 | 0 | 2 | 2 | 0 | 0 | 3 |  |

### 4.4 General Trends in Composability

We finally summarize trends revealed when considering all findings.

Compression consistently hinders other interventions.
Compressing an edited model can regress editing performance. Applying unlearning to a compressed model is often more challenging than unlearning the base model. These trends suggest that compression may modify how models encode knowledge in a way that makes targeted updates more challenging. Previous works have studied how compression can cause models to forget some knowledge [, , , , ]. Others have studied how editing modifies model internals in a way that makes the model more fragile [, ]. However, the relationships between how models are compressed and how knowledge is modified remain underexplored. An improved understanding of these dynamics may help guide more composable interventions and shed further light on how LMs encode knowledge.

RMU is the most composable unlearning technique.
RMU has comparable unlearning performance to GD without regressing overall utility as harshly and while being more order invariant. A possible factor is the RMU training objective, which directs activations for inputs related to the unrelating to a random direction in activation space while minimally modifying unrelated activations.
RMU achieves strong performance, even when only modifying the LM’s third layer.
These results suggest that intervening on model representations [] is a promising direction for effective and composable interventions.

MMLU is insufficient for measuring composability.
MMLU is a common measure of a model’s general performance.
However, similar scores on MMLU can mask significant differences in intervention performance.
Across the board, interventions appear far more composable on MMLU than other metrics.
We therefore advocate for thorough multi-metric and multi-dataset evaluations for composable interventions.

## 5 Conclusion

We conduct an extensive study of composing test-time interventions for language models.
Such interventions are an important step towards keeping expensively trained models up-to-date over extended deployments with respect to factuality, efficiency, and social responsibility.
Our study lays the foundation for a new range of intervention methods that are composable with one another, proposing two metrics for measuring the composability of interventions.
Our setup aligns with the practical need to update the same models multiple times as use cases evolve and training remains expensive.
We deploy this new framework to investigate composability between popular interventions spanning knowledge editing, machine unlearning, and model compression.
We show that these interventions interact in non-trivial ways, pointing to concrete directions for future work.
We further discover guidance for current practice: model compression harms other interventions, unlearning compressed models is especially hard, and the order in which interventions are applied changes their outcomes substantially.
Our composability metrics also pave the way for formal investigations and practical principles.
Finally, our framework is general and naturally extends to a broad range of test-time interventions.

## Acknowledgements

We thank the University of Virginia Research Computing team for providing access to excellent high-performance computing resources.

## Appendix

## Appendix A Limitations

While our work provides the first look at how state-of-the-art methods compose, there are far more possible combinations of interventions, methods, models, and datasets than we can consider in one work.
So some popular methods are omitted.
We also prioritize broad experiments with many different intervention methods.
We hope future works will closely examine individual methods to study, improve, and extend beyond the mechanisms that drive their composability.
Lastly, we only experiment with Llama3-8B.
While our code is extendable and Llama3-8B is popular, it is unknown whether our findings generalize between language models.
We hope our codebase empowers others to explore more model sizes and architectures in future research.

## Appendix B Background on Interventions

Designing interventions for language models is a highly-active research area.
We focus on three key types of intervention.

Knowledge Editing.
Knowledge editors aim to address factuality decay in LLMs: As the world changes, some facts an LLM learned during training become inaccurate [, ].
For example, a model trained before 2020 would still predict The latest pandemic in the US is $\rightarrow$ “Swine Flu” until updated to predict “COVID”.
There are four general approaches to model editing: 1) Memory-based methods that cache knowledge [, ], 2) Locate-then-Edit methods that selectively finetune parameter subsets [, ], 3) Hypernetwork methods that predict new model weights [, , ], and 4) Prompt editors that add facts to prompts [, ].
Most editors update singular facts [, , , ], though recent works apply many simultaneously [, , ].
Others works embrace the practical need for sequential edits [, ].
Recent works have also begun investigating unintended impacts of editing on pre-trained models [, , , , , ], while others are exploring multi-hop editing [, , ].
However, these works study impacts on model performance, not interactions with other types of interventions.

Machine Unlearning. Machine Unlearning (MU) refers to a broad set of techniques for modifying a model post-training to remove the influence of specific training examples or other undesirable properties(^4^44Most MU techniques for LMs are considered approximate unlearning in that they aim to modify the model to approximate the behavior of it not being trained on the unlearning target without retraining from scratch.) [, , ]. These undesirable properties are commonly referred to as the unlearning target. Crucially, MU must excel at causing the model to forget the unlearning target without adversely affecting overall utility by reducing performance on other domains []. Another requirement is that MU must cause models to deeply forget knowledge such that the unlearning target can not be elicited by adversaries or probing []. With the rise of generative models, recent work has explored MU for forgetting potentially copyrighted work [], information about individuals [], and potentially dangerous knowledge [, , ]. Existing works have studied which types of examples models forget when compressed [], whether compression hinders unlearning [], and whether compression in itself can be used as an unlearning technique []. While related to knowledge editing, the two areas have largely developed independently.

Model Compression.
To mitigate the huge computational and memory demands of LLMs, plenty of classical techniques in model compression have been explored, such as quantization [, , ], network pruning [, , , ], knowledge distillation [, , ], and low-rank factorization [, , ]. We focus on quantization and network pruning,
which both make in-place updates to pre-trained model weights.
Quantization involves lowering the bit-precision of model parameters, effectively shrinking model size and expediting inference. Early works, such as ZeroQuant [] and LLM.int8 [], focused on fine-tuning quantization granularity, showing promising results predominantly at higher bit-widths, like 8-bit. More recent innovations like GPTQ [] and AWQ [] have successfully pushed the boundaries by reducing bit-width to as low as 3 or 4 bits per weight, with negligible performance degradation.
Pruning is another compression methods, which aims to eliminate redundant components like weights and channels.
Techniques like SparseGPT [] and Wanda [] successfully maintain much of an LLM’s performance, even when discarding approximately 50% of its weights. OWL [] achieves higher levels of sparsity by reallocating sparsities across layers.

## Appendix C Implementation Details

In this section we outline the set up and hyperparameters used for knowledge editing, unlearning, and compression.

### C.1 Knowledge Editing Details

#### C.1.1 Knowledge Editing Metrics

We adopt three widely-used editing metrics and three compression metrics to measure the effectiveness of these interventions. The editing metrics are:

$\bullet$ Edit Success: Assesses the post-edit model’s ability to provide the correct, expected answer. An issue arises here with the different levels of verbosity for the models. An unedited model might return the correct token, but use a differently phrased answer or arrange the correct tokens differently. We therefore adopt and F1-style measure used in recent work [] for this.

First, we determine the common tokens between the generated response and the ground truth, which are represented by the sets $G$ (generated response tokens without special tokens) and $T$ (target tokens without padding), respectively. The common tokens are given by $C=G\cap T$.

If $C$ is empty is empty, the F1 score is set to 0, as there are no common elements to compare.

Otherwise, we calculate precision and recall. Precision is the ratio of the number of common tokens to the number of tokens in the generated response, and recall is the ratio of the number of common tokens to the number of target tokens. They are defined as:

$$ Precision $\displaystyle=\frac{|C|}{|G|},$ Recall $\displaystyle=\frac{|C|}{|T|}.$ $$

The F1 score, which is the harmonic mean of precision and recall, is then calculated as:

$$ $\displaystyle F1$ $\displaystyle=\frac{2\times\text{Precision}\times\text{Recall}}{\text{ Precision}+\text{Recall}},$ $\displaystyle=\frac{2\times|C|/|G|\times|C|/|T|}{|C|/|G|+|C|/|T|},$ $$

with the condition that if the denominator $(\text{Precision}+\text{Recall})$ is zero, the F1 score is set to 0.

The final F1 score is the average of the F1 scores for all items. If there are no F1 scores (i.e., the list of scores is empty), the average F1 score is set to 0.

$\bullet$ Edit Generalization: Measures the model’s capability to genuinely update knowledge, rather than just repeating keywords. This is tested by evaluating edit success on paraphrased prompts given by the respective dataset.

$\bullet$ Edit Locality: Examines the precision of edits, ensuring that only relevant, in-distribution knowledge is altered while out-of-distribution facts remain unaffected. In this setting, the F1 score is critical as testing on the pre-trained knowledge can result in varied answer phrasings.

#### C.1.2 Knowledge Editing Dataset

$\bullet$ ZsRE []: This dataset focuses on context-free question-answering. Given a question related to a subject and a relation, the model’s task is to provide the correct object as the answer. [] follow the procedure outlined in [], to improve the locality sets used. As the initial locality sets were solely based on Natural Question annotations, we use the locality sets provided by [].

In all three datasets, original prompt rephrasings and locality sets are used to derive the respective metrics.

### C.2 Model Editing Details

#### C.2.1 MEMIT Details

We use the state-of-the-art MEMIT [] model editor, which applies batches of edits simultaneously.
The editing process was applied to layers 4 through 8 of the model, with a clamp normalization factor set at 4. The learning parameters adhered closely to the original implementation: v_num_grad_steps was set to 25, accompanied by a learning rate (lr) of 0.5, and using the last layer for loss calculation. Additionally, a weight decay (weight_decay) of 0.001 was employed. The KL divergence contribution to the overall loss was controlled by a $KL_\text{factor}$ of 0.0625. Moreover, a second momentum adjustment was enabled, with an update weight of $15000$, to fine-tune the optimization process. The model generated a maximum length of 40 tokens and a batch size of 50, matching the number of edits being made. 10 repeats were made for each edit and the results averaged.

#### C.2.2 LoRA Details

We also employed LoRA as a model editor, specifically using the adalora variant for our experiments. The editing process utilized 70 gradient steps, with a learning rate of 0.005. The rank for the low-rank adaptation was set at 8, with a scaling factor of 32. There was no dropout applied to the adaptation matrices, and the normalization constraint was disabled. The target modules for the edits consisted of query and value projection layers. Consistent with the MEMIT method, the model generated sequences with a maximum length of 40 tokens and utilized a batch size of 50, matching the number of edits per batch. Each edit was repeated 10 times, and the results were averaged to ensure reliability. The data type for the computations was set to torch.float.

#### C.2.3 Fine-Tuning (FT) Details

For fine-tuning (FT), we focused on layers 4 through 8 of the model. The specific modules targeted for editing included the MLP down-projection (model.layers.{}.mlp.down_proj), the general layer module (model.layers.{}), the MLP module (model.layers.{}.mlp), the self-attention module (model.layers.{}.self_attn), the final layer normalization (model.norm), and the language model head (lm_head).

The fine-tuning process was conducted with 25 optimization steps, using a learning rate of $10^{-5}$. No weight decay was applied. The normalization constraint was disabled. The optimization objective was set to target the new data and the data type for the computations was set to bfloat16.

As with the MEMIT and LoRA methods, the model generated sequences with a maximum length of 40 tokens and used a batch size of 50, matching the number of edits per batch. Each edit was repeated 10 times, and the results were averaged for consistency and reliability.

### C.3 Model Compression Details

#### C.3.1 Compression Metrics

The compression metrics include:

$\bullet$ Sparsity Ratio. The fraction of parameters in the model that are zero, indicating the extent to which the model is sparse.

$\bullet$ Average Bits. The mean number of bits utilized to represent each parameter in a quantized or sparsified neural network, reflecting the level of compression.

#### C.3.2 Compressor details

We use four state-of-the-art compression methods including two pruning methods:

$\bullet$ SparseGPT []: an efficient one-shot pruning method tailored for large models. It converts the pruning process into solving large-scale sparse regression problems using an approximate solver. This approach enables rapid pruning on a single GPU with minimal accuracy loss, achieving 50-60% on large models.

$\bullet$ Wanda []: is another popular method for pruning large language models that relies on a pruning metric that combines a weight’s magnitude with the norm of its corresponding input activations, determined from a small calibration dataset. The method focuses on selectively pruning weights within individual outputs of linear layers, aiming for high sparsity levels without modifying unpruned weights. Wanda is computationally efficient, executable in a single forward pass.

and two quantization methods:

$\bullet$ GPTQ []: an algorithm designed for efficient weight quantization in large-scale models. It revises the weight quantization approach by quantizing weights in a fixed order rather than a greedy order, which shows minimal performance difference, especially in larger models. GPTQ introduced a novel method where each weight is quantized column-by-column, reducing computational complexity.

$\bullet$ AWQ []: is based on the premise that not all weights are equally critical for model performance, and it identifies a small fraction of salient weights whose quantization significantly impacts model accuracy. This identification is done by analyzing activation distributions rather than weight distributions, under the rationale that weights linked to larger activation magnitudes are more crucial.

### C.4 Machine Unlearning Details

We implement three unlearning techniques: RMU, GA, and GD. The success of each technique hinges on the careful selection of hyperparameters. We describe our parameter searches and selected values in the following sections.

#### C.4.1 Representation Misdirection Unlearning

Introduced in [], RMU intervenes on a specific layer of the LM to perturb activations for inputs related to the unlearning target. This technique has achieved impressive performance on the WMDP dataset, with performance on the unlearning targets dropping to near-random while overall model utility (MMLU) minimally regresses. Wikitext [] acts as the retain set.

RMU relies on multiple hyperparameters for selecting which layer to modify and how the loss should be calculated. We found that RMU is quite sensitive to hyperparameter choice, with most combinations either leaving the model unaffected or significantly harming model utility. Table 5 details the results from our hyperparameters search. We otherwise use the RMU repo’s(^5^55https://github.com/centerforaisafety/wmdp) default settings.

**Table 5: RMU Hyperparameter Search. Details for the grid search over possible RMU settings. Hyperparameters were selected using grid search.**
| Parameter | Description | Search | Selected |
| --- | --- | --- | --- |
| Alpha | The weight for the retain loss | [1, 10, 100, 1000, 10000] | 1000 |
| Layer | The layer to modify | [3, 17] | 3 |
| Maximum Batches | Training duration RMU in batches | [100, 150, 200, 250, 300] | 250 |

#### C.4.2 Gradient Ascent

GA, a simple and widely studied unlearning method [, , , , ], compels the model to minimize the likelihood of the correct token. The underlying intuition is that models will learn to avoid generating tokens associated with the unlearning target. We only train the final 16 layers of the model due to memory constraints.

**Table 6: GA Hyperparameter Search. Details for the grid search over possible GA settings. Hyperparameters were selected using grid search.**
| Parameter | Description | Search | Selected |
| --- | --- | --- | --- |
| LR | The learning rate | [5e-6, 1e-6, … 5e-3, 1e-3] | 5e-5 |
| Training Samples | Train size balanced across Cyber/Bio | [10, 25, 50, 100, … 500, 1000, 2000] | 50 |

#### C.4.3 Gradient Difference

A significant challenge when leveraging GA is balancing reducing performance on the unlearning target while retaining overall model utility. Since GA reduces the likelihood of the correct token, it is fragile to catastrophic forgetting, where the model incorrectly generalizes to forgetting how to model natural language.

GD [] aims to address this shortcoming by adding a retain loss term to the loss function. This term aims to maximize the likelihood of the correct token on an unrelated retain dataset. We use Wikitetx [] as the retain set. As with GA, we only train the final 16 layers of the model.

Table 7 reports our hyperparameter search. Unlike [], which found that GD provided only a marginal improvement over GA, we found that GD can perform significantly better if we upweight the retain loss term. Upweighting is necessary since it is easy for the model to significantly minimize the correct token likelihood, which causes it to drown out the retain loss term. Assigning a weight of 400 to the retain term solved this issue. This hyperparameter combination can make GD competitive with RMU, though still less composable (Section 4).

**Table 7: GD Hyperparameter Search. Details for the grid search over possible GD settings. Hyperparameters were selected using grid search.**
| Parameter | Description | Search | Selected |
| --- | --- | --- | --- |
| Alpha | The weight for the retain loss | [1, 2, 4, 6, 8, 10, 20, 40, 60, 80, 100] | 40 |
| LR | The learning rate | [5e-6, 1e-6, … 5e-3, 1e-3] | 5e-5 |
| Training Samples | Train size balanced across Cyber/Bio | [10, 25, 50, 100, … 500, 1000, 2000] | 50 |

## Appendix D Extended Results

We report the task performance for all compositions below and the baseline results, where only a single intervention is applied (Table D). Perplexity on Wikitext [] is also included as another overall utility metric. Average Bits (Appendix C.3.1) measures the degree to which the composition compresses the model.

**Table 8: Baseline intervention Results without composition.**
|  | Editing | Compression | Unlearning | Utility |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  | Edit Success | Generalization | Locality | Avg. Bits | Avg. WMDP | MMLU | WikiText PPL |
| None | 0.02 | 0.02 | 0.04 | 16.00 | 0.58 | 0.62 | 5.54 |
| \cdashline1-9<br>FT$\rightarrow$None | 0.99 | 0.82 | 0.11 | 16.0 | 0.57 | 0.61 | 5.57 |
| MEMIT$\rightarrow$None | 0.89 | 0.85 | 0.04 | 16.0 | 0.57 | 0.61 | 5.57 |
| LoRA$\rightarrow$None | 1.0 | 0.71 | 0.06 | 16.0 | 0.56 | 0.61 | 19.25 |
| \cdashline1-9<br>SparseGPT (0.25) $\rightarrow$None | 0.02 | 0.02 | 0.03 | 12.25 | 0.56 | 0.61 | 5.87 |
| SparseGPT (0.35) $\rightarrow$None | 0.02 | 0.02 | 0.03 | 10.75 | 0.54 | 0.58 | 6.34 |
| SparseGPT (0.45) $\rightarrow$None | 0.02 | 0.01 | 0.03 | 9.25 | 0.51 | 0.54 | 7.43 |
| SparseGPT (0.55) $\rightarrow$None | 0.01 | 0.01 | 0.03 | 7.75 | 0.45 | 0.44 | 10.43 |
| SparseGPT (0.65) $\rightarrow$None | 0.0 | 0.01 | 0.03 | 6.25 | 0.3 | 0.27 | 20.83 |
| SparseGPT (0.75) $\rightarrow$None | 0.01 | 0.01 | 0.03 | 4.75 | 0.26 | 0.23 | 81.84 |
| Wanda (0.25) $\rightarrow$None | 0.02 | 0.02 | 0.03 | 12.25 | 0.56 | 0.61 | 5.84 |
| Wanda (0.35) $\rightarrow$None | 0.02 | 0.02 | 0.04 | 10.75 | 0.54 | 0.58 | 6.31 |
| Wanda (0.45) $\rightarrow$None | 0.02 | 0.02 | 0.03 | 9.25 | 0.5 | 0.52 | 7.52 |
| Wanda (0.55) $\rightarrow$None | 0.02 | 0.01 | 0.02 | 7.75 | 0.36 | 0.37 | 12.5 |
| Wanda (0.65) $\rightarrow$None | 0.02 | 0.01 | 0.02 | 6.25 | 0.26 | 0.23 | 47.18 |
| Wanda (0.75) $\rightarrow$None | 0.02 | 0.03 | 0.03 | 4.75 | 0.26 | 0.23 | 257.46 |
| GPTQ (2-Bit) $\rightarrow$None | 0.0 | 0.0 | 0.03 | 2.25 | 0.25 | 0.24 | 3681.23 |
| GPTQ (3-Bit) $\rightarrow$None | 0.01 | 0.01 | 0.03 | 3.25 | 0.45 | 0.46 | 8.6 |
| GPTQ (4-Bit) $\rightarrow$None | 0.02 | 0.02 | 0.03 | 4.25 | 0.56 | 0.6 | 9.97 |
| GPTQ (8-Bit) $\rightarrow$None | 0.02 | 0.02 | 0.04 | 8.25 | 0.58 | 0.62 | 5.54 |
| AWQ (2-Bit) $\rightarrow$None | 0.0 | 0.0 | 0.0 | 2.25 | 0.24 | 0.27 | 1748954.75 |
| AWQ (3-Bit) $\rightarrow$None | 0.01 | 0.01 | 0.03 | 3.25 | 0.52 | 0.53 | 7.47 |
| AWQ (4-Bit) $\rightarrow$None | 0.02 | 0.02 | 0.03 | 4.25 | 0.57 | 0.6 | 5.91 |
| AWQ (5-Bit) $\rightarrow$None | 0.02 | 0.03 | 0.03 | 5.25 | 0.57 | 0.62 | 5.62 |
| AWQ (6-Bit) $\rightarrow$None | 0.02 | 0.03 | 0.04 | 6.25 | 0.58 | 0.62 | 5.56 |
| AWQ (8-Bit) $\rightarrow$None | 0.02 | 0.02 | 0.04 | 8.25 | 0.58 | 0.62 | 5.54 |
| \cdashline1-9<br>GA$\rightarrow$None | 0.0 | 0.0 | 0.0 | 16.0 | 0.46 | 0.5 | inf |
| GD$\rightarrow$None | 0.02 | 0.0 | 0.09 | 16.0 | 0.26 | 0.52 | 4.48 |
| RMU$\rightarrow$None | 0.02 | 0.02 | 0.03 | 16.0 | 0.29 | 0.57 | 5.56 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please refer to the weaknesses.
soundness: 3
strengths: S1. The paper conducts experiments on various methods and compositions on multiple models to provide a deeper insight.

S2. The method proposes metrics that correlate with the effect of a composition.

S3. The presentation is lucid and easy to follow.

S4. The codebase will be useful to the community for further studies.
summary: This paper experimentally demonstrates the effect of composing various inference time setups for LLMs and suggests possible setups for better performance from its analysis.
weaknesses: W1. Some comparisons on the base models, chat, and RLHF could be interesting which could provide insight into pretraining, instruction tuning, and post-training with the interventions.

W2. With a similar spirit as W1, it is important to see the results for different generations of the same model family and sizes.

W3. The sensitivity (standard deviation) of the experiments is unclear if run multiple times in Table 2.

W4. Practitioners need to perform a grid search for their domain-specific requirements as some results depend on specific settings outside composition.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: * Addresses an important program in model updates and adaptations to real-world requirements.
Introduces new metrics and provides extensive experimental results. 
* Codebase hypothetically supports is flexible enough to scale to other inference-time interventions. There is no code uploaded, so hard to say for sure.
* Well-written paper
summary: This paper proposes a new framework to assess the effects of combining different test-time interventions on language models to improve factual accuracy, efficiency, and mitigate harmful outputs. Traditionally, methods such as knowledge editing, model compression, and machine unlearning have been developed separately, but practical applications often require these interventions to work together. The authors introduce two novel metrics—Order-free Error and Order Sensitivity—to evaluate the composability of interventions, aiming to ensure that each intervention's success is minimally impacted by other interventions.

Key contributions include: (1) extensive experiments across 417 combinations of interventions, uncovering findings, such as compression often hindering the success of knowledge editing and unlearning, and the significant impact of intervention order on outcomes. (2) They will provide a codebase featuring popular intervention methods, supporting future multi-objective intervention research and development.
The paper emphasizes the need for composable, multi-objective intervention methods and creates a foundation for systematically evaluating intervention interactions in language models.
weaknesses: * Hard to say if the findings using the specific models and datasets used generalize more broadly (e.g., Llama 3, WMDP). WMDP, for instance, specifically notes that “benchmarking on only WMDP may yield a false sense of model safety after unlearning.”
* The paper lacks guidelines for ordering interventions or insights/analysis into why some orderings are not robust or how to make them more robust.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Why did you decide to focus on the three interventions in particular mentioned in the paper? (unlearning, compressing, model editing)
Would this choice limit the scope of study?

2. As also mentioned in the weaknesses above, the framework might be limited as it only considers two interventions at a time. Can the method be expanded to study more than two interventions at a time?
soundness: 2
strengths: 1. The paper is written in a very clear, nice, and easy to understand way. 
2. The motivation of the paper is clear. 
3. The paper studies an interesting problem.
summary: The paper proposes a framework (evaluation metrics) to study the effect of the composable interventions on LLMs. Authors claim that interventions like model compression, knowledge editing, and unlearning can affect each other and hence they study this phenomenon.
weaknesses: While the paper is written very clearly and it studies an interesting problem, I have a major concern. My major concern is that the technical contribution of the paper might not be rigorous although insights and findings are good and the paper studies an interesting problem. perhaps one way to improve the paper would be to propose a method to make LLMs more robust to composable interventions or even design the interventions themselves in a way to not see degrade in their performance after other interventions are applied on top. Or even using the proposed evaluation framework as a recommendation framework to provide recommendation on how the interventions should be applied based on a given LLM and the intervention method.

Another concern I have is that the framework might be limited as it only considers two interventions at a time. Can the method be expanded to study more than two interventions at a time?

Finally, results on other LLMs are not fully consistant across all the studied dimensions which makes the finding of the paper not super reliable. It would be good to do some statistical significance test (e.g., across various LLMs).

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Refer to Weaknesses
soundness: 3
strengths: - The authors introduces two new metrics:Order-free Error and Order Sensitivity, that evaluate the robustness of intervention sequences, highlighting the importance of intervention order.
- The authors also conducted extensive experiments using various compositions, showcasing the interaction effects of various interventions.
- The authors present several important findings. For instance, the success of interventions is highly dependent on the order of application; compression hinders editability; unlearning and editing are relatively compatible; and certain methods are inherently more suitable for sequential applications. Additionally, they found that conventional metrics are inadequate for measuring composability, underscoring the need for new intervention techniques explicitly designed for this purpose.
summary: This work introduces a "Composable Interventions" for language models, addressing a practical, real-world need. The authors develop new metrics and a unified codebase to evaluate the effectiveness of combining multiple interventions, such as knowledge editing, model compression, and machine unlearning. Through rigorous and extensive experimentation with 417 intervention combinations, the research underscores the significance of the sequence in which interventions are applied on model performance. The exposition is clear and well-organized. This study tackles the complexities of applying multiple interventions simultaneously, providing foundational insights that pave the way for future research on robust and adaptable language models.
weaknesses: - The authors present numerous findings and insights, such as Compression consistently hinders Knowledge Editing and Unlearning, which is a relatively straightforward conclusion. It would be valuable for the authors to explore deeper analysis and hypotheses beyond identifying patterns such as 'A affects B' or 'A and B are compatible,' moving from experimental observations to more comprehensive insights.
- In combined interventions, it appears that certain combinations do not yet have a clear consensus. For instance, in composing model compression with knowledge editing, the preferred order can vary significantly by method: for GPTQ, editing seems more effective first, whereas for AWQ, compression may be preferable. This suggests that optimal ordering may rely on experimental results specific to each method.
- While Order-free Error and Order Sensitivity offer valuable insights, the small differences in outcomes and variations across metrics suggest that additional measures may be needed to fully capture intervention interactions.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: How do different interventions interact with one another? Are there any interference patterns or dependencies that affect their compositional effectiveness?

Why do certain techniques exhibit better composability than others? Are there particular properties or design choices in those techniques that lend themselves to more effective composition?
soundness: 2
strengths: The research effectively highlights an oversight in current evaluation methodologies, where interventions are assessed independently despite concurrent deployment requirements in productionized environments.

Analysis of compositional effects across multiple intervention types, incorporating various established techniques from each domain (editing, unlearning, and model compression).

Findings provide valuable insights into the compatibility and limitations of different intervention combinations, offering practical usage guidance.
summary: The paper highlights a gap in the practical application of model interventions, where multiple techniques, such as information removal, model editing, and efficiency optimization (e.g., pruning and quantization) are often needed in combination but have largely been studied in isolation. The authors systematically explore how these interventions interact when used together, analyzing the sensitivity to intervention order and the broader implications of composability. This research offers insights into the outcome of utilizing multiple interventions and emphasizes the need for multi-objective intervention metrics to gauge their effective composability.
weaknesses: While the study provides a thorough documentation of interactions among various interventions, it falls short in examining the underlying issues or mechanisms that lead to compositional failures. A deeper analysis of the interference patterns or cause-effect relationships between interventions is necessary to understand why the composition of approaches works more effectively in some cases than in others. Exploring why certain techniques exhibit superior composability over others would greatly enhance the study’s contribution. Such insights would also help clarify the conditions under which specific interventions interact constructively or destructively, offering guidance for future research in designing more robust and composable interventions.

Meta-review: This paper proposes a new framework to assess the effects of combining different test-time interventions on language models to improve factual accuracy, efficiency, and mitigate harmful outputs. The authors explore how these interventions interact when used together, analyzing the sensitivity to intervention order and the broader implications of composability. This research offers insights into the outcome of utilizing multiple interventions and emphasizes the need for multi-objective intervention metrics.

Most reviewers understood the paper and thought the results and the code to be a valuable contribution. Reviewers raised issues on if the findings will generalize broadly, and the statistical significance of some small differences. The main reject reviewer (5FNv) thought the analysis is not deep enough to chase down the root causes of failing composition but admits the author took a step is documenting where it is failing. Overall reviewers are consistent and did not identify critical flaws, so the decision follow their recommendation.

Decision: Accept (Poster)