Title: ModHiFi: Identifying High Fidelity predictive components for Model Modification
Venue: NeurIPS 2025
Accepted: yes
Keywords: Pruning

Machine Unlearning
Abstract: Open weight models, which are ubiquitous, rarely provide access to their training data or loss function. This makes modifying such models for tasks such as pruning or unlearning, which are constrained by this unavailability, an active area of research. Existing techniques typically require gradients or ground-truth labels, rendering them infeasible in settings with limited computational resources.
In this work, we investigate the fundamental question of identifying components that are critical to the model's predictive performance, without access to either gradients or the loss function, and with only distributional access such as synthetic data. 
We theoretically demonstrate that the global error is linearly bounded by local reconstruction errors for Lipschitz-continuous networks such as CNNs and well-trained Transformers (which, contrary to existing literature, we find exhibit Lipschitz continuity).
This motivates using the locally reconstructive behavior of component subsets to quantify their global importance, via a metric that we term *Subset Fidelity*. 
In the uncorrelated features setting, selecting individual components based on their Subset Fidelity scores is optimal, which we utilize to propose **ModHiFi**, an algorithm for model modification that requires neither training data nor access to a loss function. 
**ModHiFi-P**, for structured pruning, achieves an 11\% speedup over the current state of the art on ImageNet models and competitive performance on language models. **ModHiFi-U**, for classwise unlearning, achieves complete unlearning on CIFAR-10 without fine-tuning and demonstrates competitive performance on Swin Transformers.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ModHiFi, a pre-training data- and loss-free algorithm for model modification tasks such as pruning and unlearning. It introduces a theoretical method to identify critical model components and demonstrates competitive performance on both vision and language benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths \n- The paper provides a theoretical analysis showing that, under Lipschitz continuity, the reconstruction error of the model is linearly bounded by the local reconstruction error.\n- It introduces a Monte Carlo-based sampling strategy to identify subsets of important components using the proposed Subset Fidelity metric.\n-The method is extensively validated on CNN and Transformer architectures, across vision and language tasks, demonstrating its generality and effectiveness.\n\n### Weaknesses\n-  Theorem 3.1 appears to be an extension or adaptation of prior work [1], which is not cited in the main paper.\n- The method computes fidelity scores and prunes components independently for each layer, which potentially overlooks inter-layer dependencies among components during pruning.\n- The paper would benefit from comparison with more recent baselines, where applicable, to ensure a fair and up-to-date evaluation.\n\n[1] NISP: Pruning Networks using Neuron Importance Score Propagation, CVPR 2018\n\nQUESTIONS:\nQ1: The term \"model modification\" may encompass diverse tasks beyond pruning/unlearning (e.g., model editing in LLM and some vision tasks in [2]). In addition, the paper exclusively evaluates zeroing out as the modification strategy. How might the identified components be adapted to other modification strategies (e.g., fine-tuning, structural reparameterization), and would their effectiveness generalize across different model modification tasks? Alternatively, it may be more appropriate to refine the terminology and scope description to better reflect the range of tasks actually validated in the experiments.\n\nQ2. Under the same setting that uses synthetic image data for component identification in vision models, how does the proposed method compare to [2], which estimates component importances across the entire model rather than treating each layer separately? \n\nQ3. In Table 3, the reported time cost for the proposed methods appears to exclude the Monte Carlo sampling-based identification step, which may make the comparison unfair.  Additionally, what is the time cost when applying the method to large language models? \n\nQ4. Could the paper include comparisons with more recent methods, such as [3] (which appeared online before March 1st, 2025), to better position the proposed approach within the state-of-the-art?\n\n[2] Decomposing and Editing Predictions by Modeling Model Computation, ICML 2024\n\n[3] Numerical Pruning for Efficient Autoregressive Models, AAAI 2025\n\nLIMITATIONS:\nThe method computes the fidelity of each layer independently, potentially overlooking inter-layer connections. Additionally, the computational cost of the Monte Carlo–based subset identification may become high when applied to large foundation models.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents ModHiFi, a layer-wise reconstruction-based method for estimating feature importance in neural networks using synthetic data, avoiding the need for original training data. It introduces the High Fidelity (HiFi) metric, which quantifies the contribution of each component to local reconstruction. This importance score is used for both structured pruning (retaining high-HiFi components) and classwise unlearning (removing high-HiFi components). The method leverages the empirical observation that pretrained Transformers exhibit approximate Lipschitz continuity, and uses this to theoretically justify that local reconstruction errors can linearly bound global prediction degradation. ModHiFi is evaluated on CNNs and Transformers across CIFAR-10 and ImageNet, showing competitive performance without access to the original training data.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- **Works for both pruning and unlearning**: ModHiFi applies the same reconstruction-based importance score to both structured pruning and classwise unlearning, demonstrating that the metric captures feature relevance across different model editing tasks.\n- No reliance on training data: The method does not require access to the original training set or supervised loss function. Instead, it operates using either synthetic inputs or a calibration set, making it significant for pre-trained models and their applications.\n- **Theoretical support via Lipschitz continuity**: By leveraging the Lipschitz property of trained networks, the paper provides a theoretical guarantee that local reconstruction errors upper-bound global output degradation. This justifies the use of local fidelity scores for importance estimation, and makes the approach more practical by avoiding the need for end-to-end evaluations.\n- **Empirical validation of the importance metric**: The paper includes several supporting experiments and ablations, such as noise injection that empirically demonstrate the HiFi score correlates with true functional importance.\n- **Empirical effectiveness**: Despite operating without training data, gradients, or loss access, the method achieves competitive performance across pruning and unlearning on CIFAR-10 and ImageNet.\n\nWeaknesses\n- **Limited novelty and insufficient comparison to prior reconstruction-based pruning methods**: While ModHiFi is framed as novel, the core idea of using reconstruction error to guide pruning, has been well explored in prior literature [1,2,3,4]. The paper does not adequately position its contributions relative to this body of work or clarify what distinguishes ModHiFi’s reconstruction-based formulation beyond its application to multiple tasks. Moreover, the practical implementation relies on singleton importance scores due to the infeasibility of true subset selection, which diminishes the claimed departure from per-component heuristics.\n\n- **Dependence on synthetic or calibration data**: While ModHiFi avoids training data, it still relies on synthetic or curated datasets to compute fidelity scores and fit compensation terms. The quality, representativeness, and diversity of this data can significantly impact performance, yet the main paper offers limited discussion or quantification of this sensitivity. Additionally, fitting compensation weights via linear regression constitutes a lightweight form of fine-tuning, which somewhat undermines the “no fine-tuning” framing.\n\n- **Theoretical guarantees rely on unrealistic assumptions**: Theorems 3.1 and 3.7 rely on assumptions that rarely hold in practice. The uncorrelated feature assumption underlying singleton optimality is implausible in deep networks, where strong interdependencies exist. Similarly, Theorem 3.1 assumes that each layer’s inputs remain fixed during analysis, yet ModHiFi prunes multiple layers simultaneously, violating this condition. \n\n- **Lack of computational cost analysis**: Although the method is presented as more efficient than fine-tuning, the paper does not quantify its computational demands. Evaluating fidelity scores requires full forward passes and activation logging across a dataset for each candidate component, which may scale poorly in large networks. The absence of runtime comparisons or memory profiling against baseline methods (e.g., magnitude-based pruning) leaves open questions about ModHiFi’s practical efficiency.\n\n\n[1] Jiang, Chunhui, et al. Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error. IJCAI, 2018.  \n[2] Kamma, Koji, and Toshikazu Wada. Reconstruction Error Aware Pruning for Accelerating Neural Networks. ISVC, 2019.  \n[3] Zhuang, Zhuangwei, et al. Discrimination-aware Channel Pruning for Deep Neural Networks. NeurIPS, 2018.  \n[4] Molchanov, Pavlo, et al. Importance Estimation for Neural Network Pruning. CVPR, 2019.\n\nQUESTIONS:\n1. **Mitigating theoretical limitations in simultaneous pruning**: Theorem 3.1 assumes that inputs to each layer remain fixed, yet ModHiFi prunes multiple layers simultaneously. Could you clarify how the theoretical guarantees could hold under this setup? It might be worth considering including an empirical comparison between layerwise iterative and simultaneous pruning to assess the bound’s practical validity.\n\n2. **Addressing data sensitivity more explicitly**: The reliance on synthetic or calibration data is mentioned in the appendix but is central to the method’s performance. The paper would benefit from moving this discussion into the main text and expanding it with quantitative evaluations showing how fidelity scores and performance vary with data size, diversity, or distribution mismatch / dataset sfhit.\n\n3. **Quantifying computational overhead**: ModHiFi requires activation logging and per-layer evaluation for each candidate component. It could be worth providing runtime, memory, or efficiency comparisons with standard pruning baselines (e.g., L1-norm). Furthermore frontier plots of sparsity-accuracy tradeoffs would be helpful to position ModHiFi against related work.\n\nLIMITATIONS:\nThe authors acknowledge certain limitations, such as the method’s reliance on pretrained models and the intractability of full subset selection, which are briefly discussed in the main text. However, additional limitations - particularly the method’s dependence on data quality for synthetic inputs should be more explicitly addressed, and ideally consolidated.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ModHiFi, a novel framework for modifying well-trained models without access to the original training data or loss function. The authors address two key tasks: structured pruning and classwise unlearning. The core idea revolves around identifying High-Fidelity (HiFi) components which are small subsets of model components critical to predictive performance, using a new metric called Subset Fidelity, which quantifies how well a subset of components can reconstruct model outputs after weight adjustments.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n+ Architecture Generalization: The proposed algorithm performs well on both vision models (CNNs/ViTs) and language models (LLMs), demonstrating versatility across different architectures.\n+ Task Generalization: The modification based on the HiFi objective unifies two tasks, classwise unlearning and structured pruning. Specifically: The pruning method (ModHiFi-P) achieves an 11% speedup on ImageNet while maintaining superior accuracy over baselines. The unlearning method (ModHiFi-U) completely erases target classes without fine-tuning and is 10x faster than existing approaches.\n+ Comprehensive Experimental Setup: The experiments thoroughly address the four key questions posed in the paper.\n\n**Weakness:**\n- As shown in Table 2, while the method maintains strong average performance under high pruning ratios, its individual task metrics are not particularly outstanding.\n- The method’s effectiveness relies on the assumption that the target model is well-trained; its performance on poorly initialized or underfitted models remains unexplored and is theoretically expected to degrade.\n- The classwise unlearning performance on Swin-Transformers is subpar, possibly due to insufficient model training.\n- Computing Subset Fidelity requires high-dimensional matrix inversion, which may be computationally prohibitive for extremely large models (e.g., LLMs).\n- The experimental scope is somewhat limited:\n1) No exploration of the impact of k and η (key hyperparameters in HiFi selection).\n2) No discussion on the choice of B (pruning budget) for structured pruning or the sizes of forget set and retain set for classwise unlearning.\n\nQUESTIONS:\nSee concerns above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new theoretical result bounding the global impact of removing a component from the neural network (in terms of how it affects the loss) with respect to its local impact (the change to the output of that specific component, such as a convolutional filter). This result depends on the neural network being Lipschitz continuous to start with. When that is the case, this result can be used to make it more manageable to perform structured pruning or classwise unlearning. In the first case, we would look for components with little effect if removed. In the second case, we would look for components that mostly affect performance in the class to be unlearned if removed.\n\nSTRENGTHS AND WEAKNESSES:\nThe authors make an interesting case by first presenting general theoretical results, refining them in a usable way, and then applying those results to practical problems. That makes the approach sound and reasonable. \n\nWith that said, I would have appreciated a little more intuition about the results, since I am not in a position to fully vouch for the claims and proofs. Moreover, some parts of the development are not entirely clear, and some of the experimental results are difficult to understand. I go in more detail about those issues in Questions.\n\nQUESTIONS:\n1) How do you characterize a well-trained model?\n\n2) Where is the evidence claimed in Lines 49-51 that transformers are Lipschitz continuous?\n\n3) There are many other approaches to structured pruning without access to the data or loss function that also measure the impact of components, such as [1-3]. Can you please contextualize your work with those and broaden your literature review accordingly?\n\n[1] https://proceedings.neurips.cc/paper/2020/file/46a4378f835dc8040c8057beb6a2da52-Paper.pdf\n\n[2] https://proceedings.neurips.cc/paper_files/paper/2021/file/e35d7a5768c4b85b4780384d55dc3620-Paper.pdf\n\n[3] https://ieeexplore.ieee.org/document/9878423\n\n4) How large can be the constants from Theorem 3.1? I am wondering if, either in theory or practice, these constants are sufficiently small to validate the approach.\n\n5) There is a certain confusion about complexity classes in the paper: you talk about computing optimal subsets being NP-complete in Line 59, then say that this is a binary quadratic optimization problem in Line 168, mentioned to be NP-complete in Line 169, and then seem to refer again to this same problem being NP hard in Line 193. To be clear, optimization problems are typically NP hard. So my impression is that all other mentions are wrong. Are all of these mentions of the same problem? What is the complexity class of this problem?\n\n6) Related to the item above, can you please rephrase footnote 2 in a clear and more detailed way?\n\n7) What is the intuition for Theorem 3.7?\n\n8) What do you mean by \"the compensation term\" in Line 228? Can you please rephrase that whole sentence for clarity?\n\n9) When you talk about greedy selection in Lines 235-237, do you mean sorting the components from smallest to largest fidelity? That should be made explicit.\n\n10) In Table 1 (which is actually two tables), what is the difference between the two rows for DFPC and Ours on ImageNet? Please make that clear in the paper.\n\n11) For the same table as before, what exactly do you mean with columns CPU and GPU?\n\n12) Across all tables, are the results in each row based on a single model or are they a summary of the results over multiple models with different random seeds?\n\n13) I cannot understand Table 2. Can you please explain it?\n\n14) Please consider these minor comments on the writing:\n- Use \"Lipschitz continuous\" rather than \"Lipschitz\" alone, as that is not very formal and may confuse the reader if read out of context.\n- Replace the word \"tasks\" with something like \"purposes\" in the abstract. Task may sound like the application of the neural network, such as classification for a specific dataset.\n- Separate text from the reference citations with a space. If you do not want the citations in a separate line, use something like \"x~\\cite{y}\".\n- What is \"F\" in Line 259?\n\nLIMITATIONS:\nThere is a brief and reasonable acknowledgement about limitations at the end.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}