Dissecting Representation Misalignment in Contrastive Learning via Influence Function
Authors: Huanyi Xie
Chenyang Ren
Khouloud Saadi
Shu Yang
Zhen Tan
Jingfeng Zhang
Lijie Hu
Di Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Data Attribution
Influence Function
Contrastive Learning
Primary area: interpretability and explainable AI

Abstract: Contrastive learning, commonly applied in large-scale multimodal models, often relies on data from diverse and often unreliable sources, which can include misaligned or mislabeled text-image pairs. This frequently leads to robustness issues and hallucinations, ultimately causing performance degradation. Data valuation is an efficient way to detect and trace these misalignments. Nevertheless, existing methods are computationally expensive for large-scale models. Although computationally efficient, classical influence functions are inadequate for contrastive learning models, as they were initially designed for pointwise loss. Furthermore, contrastive learning involves minimizing the distance between positive sample modalities while maximizing the distance between negative sample modalities. This necessitates evaluating the influence of samples from both perspectives. To tackle these challenges, we introduce the Extended Influence Function for Contrastive Loss (ECIF), an influence function crafted for contrastive loss. ECIF considers both positive and negative samples and provides a closed-form approximation of contrastive learning models, eliminating the need for retraining. Building upon ECIF, we develop a series of algorithms for data evaluation, misalignment detection, and misprediction trace-back tasks. Experimental results demonstrate that our ECIF advances the transparency and interpretability of CLIP-style embedding models by offering a more accurate assessment of data impact and model alignment compared to traditional baseline methods.

Published as a conference paper at ICLR 2026
DISSECTING REPRESENTATION MISALIGNMENT IN CONTRASTIVE LEARNING VIA INFLUENCE FUNCTION
Huanyi Xie X Chenyang Ren 13,,, Khouloud Saadi .2, Shu Yang 12, Zhen Tan",
Jingfeng Zhang,, Lijie Hu Di Wang 1.2,1
1 "King Abdullah University of Science and Technology (KAUST)
Provable Responsible AI and Data Analytics (PRADA) Lab
3 Mohamed bin Zayed University of Artificial Intelligence (MBZUAI)
AArizona State University University of Auckland Shanghai Jiao Tong University (SJTU)
ABSTRACT
Contrastive learning, commonly applied in large-scale multimodal models, often relies data from diverse and often unreliable sources, which can include mis- aligned or mislabeled text-image pairs. This frequently leads to robustness issues and hallucinations, ultimately causing performance degradation. Data valuation is efficient way to detect and trace these misalignments. Nevertheless, existing methods are computationally expensive for large-scale models. Although com- putationally efficient, classical influence functions are inadequate for contrastive learning models, as they were initially designed for pointwise loss. Furthermore, contrastive learning involves minimizing the distance between positive sample modalities while maximizing the distance between negative sample modalities. This necessitates evaluating the influence of samples from both perspectives. To tackle these challenges, we introduce the Extended Influence Function for Con- trastive Loss (ECIF), an influence function crafted for contrastive loss. ECIF considers both positive and negative samples and provides a closed-form approxi- mation of contrastive learning models, eliminating the need for retraining. Building upon ECIF, we develop series of algorithms for data evaluation, misalignment detection, and misprediction trace-back tasks. Experimental results demonstrate that our ECIF advances the transparency and interpretability of CLIP-style em- bedding models by offering a more accurate assessment of data impact and model alignment compared to traditional baseline methods.
INTRODUCTION
Contrastive learning has become a cornerstone in the development of multimodal models due to its ability align representations of different modalities, such as images, text, and audio, within a shared semantic space (Chen et al.) 2020b Yin et al.) 2023 Koh et al. 2024). However, models trained with contrastive learning often suffer from robustness issues (Radford et al. 2021) and hallucinations, which are mainly attributed misaligned text-image pairs in training data (Kim et al. 2023). These misalignments, manifesting as semantic mismatches, contextual inconsistencies, or discrepancies between abstract and concrete elements, can severely degrade model performance. Contrastive learning relies on the assumption of consistent alignment between image-text pairs; however, when this assumption fails, it leads to incorrect interpretations, ultimately degrading model performance. Consequently, improving dataset transparency is crucial, as model developers need the ability to trace and identify problematic data samples. However, diagnosing issues caused by misaligned data, such as mislabeled or biased samples, is difficult when working with large text-image datasets.
"The first two authors contributed equally to this work. Correspondence to Lijic Hu (Email Address: 1 jijie. e. hu@mbzuai i aac. ae) and Di Wang (Fmail Address: di wwngkkaust.t...... edu. sa).

---

Published as a conference paper at ICLR 2026
Although the critical role of training data in shaping the capabilities of multimodal models is well recognized, robust evaluation mechanisms for data quality remain lacking (Nguyen et al.] (022). To
address this, various data valuation methods (Jia et al. 2019 Ghorbani & Zou 2019; Yoon et al. 20200]
Han et al. 2020) have been introduced to enhance dataset transparency by quantifying the contribution of individual data to model performance. These approaches typically assign higher contribution scores to training instances whose inclusion significantly boosts model performance compared to exclusion. Some methods, such Shapley Value (Kwon & Zou (022), require multiple retraining processes with different data subsets, which is computationally expensive and impractical for large models. To overcome this limitation, influence function-baad based methods have gained popularity, as they estimate data contributions using gradient information, thereby avoiding retraining (Choe et al., 2024).
However, applying influence functions to models trained with contrastive learning presents significant challenges. (i) First, the influence function was initially designed for M-estimators (Huber 1981). which operate with pointwise loss. However, multimodal models rely on noise-contrastive estima- tion (Radford et al. 202T; Gutmann & Hyvärinen, 2010 He et al, 2020) as their training objective. This objective encourages the model to draw positive pairs closer in feature space while pushing negative pairs apart, making the influence function unsuitable for direct application to contrastive loss. (ii) Second, the influence of negative pairs in contrastive learning has gained increasing attention recently (van den Oord et al.] [2019] Yukkekkonuul et al. (023). Robiison et al. (022)) emphasized the importance of negative samples, especially "hard" negatives samples that are mapped close in feature space but should idcally be far apart. However, the original definition of the influence function does not consider the roles of positive and negative samples. This oversimplified analysis is particularly prone to underestimating the impact of certain hard negative samples on the learning process (Chen et al., 2020a). (iii) Finally, computing the required gradients and Hessian matrices for influence functions is very demanding in terms of computational and memory resources, which makes it impractical in the large-scale high-dimensional environment of contrastive learning (Li et al. 2023a b))
To address these challenges, we propose the Extended Influence Function for Contrastive Loss (ECIF), novel method designed to quantify data attribution specifically for contrastive learning. ECIF enjoys a closed-form approximation of the original contrastive loss, thus eliminating the need for re-training process that is impractical in the era of large models. It also accounts for the dual role of data points as both positive and negative samples, providing a more comprehensive understanding of their impact on model training. This approach provides more accurate measurement of misalignment. Our contributions are summarized as follows:
We propose ECIF, the first duul-perspective data valuation method for contrastive learning, which quantifies the impact of data points as both positive and negative samples. This comprehensive approach enables a more accurate measurement of data contribution, particularly evaluating the influence of negative samples in contrastive learning.
Based on ECIF, we develop corresponding algorithms for different tasks, including identifying the most valuable data (related to specific tasks), misalignment detection, and misprediction trace-back.
Comprehensive experimental results demonstrate that ECIF can effectively and efficiently remove the influence of samples compared to retraining and identify influential data in the training set. Moreover, our methods based on ECIF are also effective in identifying influential data (harmful data and valuable data) for fine-tuning, misprediction trace-back, and detecting misaligned data.
2 RELATED WORK
Contrastive learning. Recently, self-supervised contrastive learning (Chen et al., 2020c) has emerged as highly effective approach to acquire representations without the need for labeled data (Donahue & Simonyan] 2019). This model utilizes contrastive loss that pushes dissimilar data pairs apart while pulling similar pairs closer together. Contrastive learning plays an important role in the advancement of multimodal models by integrating and understanding information across diverse modalities, such as text and images (Radford et al.] 2021 Jiang et al. 2024). In multi- modal contrastive learning tasks, proper alignment of the training data ensures accurate cross-modal associations, enabling models to learn and extract consistent feature representations (Wang & Isola 2020). One of the key challenges in training with noisy large-scale image-text pairs sourced from
2

---

Published as a conference paper at ICLR 2026
the Internet is achieving effective alignment between these modalities. To address this, researchers have developed various methods, such as those proposed by Gao et al. (2022) and Yao et al. (202I), which introduce finer-grained and more extensive interactions between text and images improve cross-modal alignment. Despite extensive research on contrastive learning, we are the first to explore interactive influence between pairs using influence functions. Our work bridges this gap by applying influence functions in contrastive learning, allowing for a deeper understanding of both positive and negative samples. This comprchensive approach improves the accuracy of misalignment measurements in data pairs, providing a more detailed assessment of the valuation of the data.
Influence function. Influence function, initially a staple in robust statistics (Cook) 2000 Cook
& Weisberg 1980), has seen extensive adoption within deep learning since Koh & Liang (017)) Its versatility spans various applications, including detecting mislabeled data, interpreting models, addressing model bias, and facilitating machine unlearning tasks. For data removal, recent work using influence function including unlearning features and labels (Warnecke et al. (2023), forgetting a subset of image data for training deep neural networks (Golatkar et al. 2020 202 1), removing the influence of nodes and edges in graph neural networks (Wu et al.] (003)) and model debiasing (Chen et al.. 2024). Besides, various studies have applied influence functions to interpret models across different domains, including natural language processing (Han et al. (0200) and image classification (Basu al. 202T), while also addressing biases in classification models (Wang et al. 2019), word embed- dings (Brunet et al.
2019), and finetuned models (Chen et al. 2020a). Recent advancements, such as the LiSSA method (Agarwal et al., 2017 Kwon et al.j 2023 Grosse et al. (023b) and kNN-based techniques (Guo et al. (021), have been proposed to enhance the computational efficiency of com- puting the influence function. Despite numerous studies on influence functions, we are the first to extend them to contrastive learning. Moreover, compared to traditional models, contrastive learning introduces additional complexity in influence function analysis, as it requires considering data points in both positive and negative roles. Our -prspeeeee approach of ECIF offers comprehensive view of data impact, leading to accurate measurements of misalignment in text-image pairs. Bridging the theoretical gap between positive and negative pairs has posed significant challenges in work, which has been addressed in our proof.
PRELIMINARIES
Contrastive loss. Contrastive loss is an effective tool in multimodal models for aligning and learning relationships between different types of data, such as images and text | Specifically, given a set of paired data consisting of text x" and image x', we aim to construct embedding vectors u and for text and image, respectively, via the encoder parameterized as 0. In batch of N text-image pairs, each pair (x* 2z() is embedded as (ug, v%). We denote the text embeddings for this batch as U = -uj..... , Uy), and similarly, the image embeddings as V == (Vi,..., UN). Contrastive loss is designed to minimize the distance between embeddings of matching pairs while maximizing the w...?
distance between non-matching pairs. Define the cosine similarity function as s(u, v) = | /T, where 7 is trainable temperature parameter. For brevity, we will omit detailing in subsequent discussions, For each batch, we construct a similarity matrix with S = s(u, v2). Then, the self-supervised contrastive loss is defined as
Barch (U, V: A Σ
logle, ((S..) logle a(S.)))
(1)
where e; is the i-th standard basis vector in N -ddimensional space, is the softmax function. Ob- serving from (T). we can separate the loss to image-to-text (I2T) and text-to-image (T2I) denoted as Torai (us, V;0) and L (v;, U: f)) and define loss function on similarity matrix as Ter21 (S; () (and LIzT S; 0)). We will incorporate an L2 regularization term into the loss function, which allows us to avoid overfitting. Thus, for a given set of batches B, the objective loss can be written as EeU LBaac (U, V: + 2 11013
L [B:0 8)
Influence functions. The influence function quantifies how an estimator relies on the value of each point in the sample. Consider neural network 0 arg min > E(2;; 0) with pointwise
For simplicity, we focus two modalities (text and image) in the paper. Our method can be generalized to multi-modalities directly.
3

---

Published as a conference paper at ICLR 2026
loss function & and dataset D {2 When we remove point 2m from the training dataset, the corresponding optimal model is denoted as d_ The influence function provides an efficient way to approximate 0_... 0 for a strongly convex and twice differentiable É. By up-weighing 2m by e, we denote the substitutional parameter via the e-parameterized response function as 0 =.... (c) arg min Σi= ((2;; 8) cl((z f). Then we can obtain an estimator for the actual change in parameters as:lim.--1 8 f_... (c) 1 = H (z f), where H = Σi=1 ViL(Z 0) + 8I is the Hessian matrix. For a differentiable model evaluation function f. such as calculating the total model loss over test set, the change resulting from removing 2m in the evaluation results can be V V
approximated by f((-=.)) f(0)=
0) -ef((0) H 'Vo((zm: 0). Scaling gradient-based methods to contrastive loss challenged by the high computational demands due to the gradients' high dimensionality. [Coee et al. (2024) introduced a low-rank gradient projection algorithm (LOGRA) to enhance the efficiency of gradient projection. See Appendix C for more details.
4 INFLUENCE FUNCTION IN CONTRASTIVE LEARNING
In this section, we will consider how estimate the attribution of a given sample (xT,, x²) to contrastive loss using the influence function method. Generally, in the original influence function method, the term in the loss function that only contains the complete information from the target sample is up-weighted by Then, response function related to is derived. Within this analytical framework, when E is set to -1, the resultant loss and model parameters are the same as those obtained by removing the sample via retraining. However, in the context of contrastive learning, because the information of the sample point appears in every term of the loss function for its batch, it is not feasible to isolate the relevant information of this sample within batch into an independent term and then perform an up-weight operation on this sample to derive the influence function.
Thus, we must perform fine-grained analysis of the specific contribution of sample (w.", (x=""."") within .²))
contrastive loss. Assume (x"" is assigned as the n-th pair in the m-th batch, in which the text and image data are embedded into matrix Um and Vm. Then (ax". x²) serves as positive samples for each other in the n-th pairing loss LT21 (Un, Vm 0) and LI2T Uni 0) in (I. And it serves as a negative sample in other pairing losses. It can be seen through simple observation about (I) that when the data serves as positive sample, its influence can be explicitly isolated. However, its information is coupled with other data when acting as a negative sample, necessitating further analysis. We provide the derivation of the influence function for these two scenarios separately.
4.1 INFLUENCE AS POSITIVE SAMPLES
To quantify the impact of x. and x! as positive samples, ideally, we can retrain the model after removing the corresponding n-th pairing tasks, i.e., removing (un, Vm; 0) and L (Un, Um: 0) in the loss function. Thus, following the idea influence function, we can up-weight these two parts by and obtain up-weighted loss function as Lanal (0) L V; 0) + 1101 + Pos((x", 0)), with Pos(x" x², 0) = LT21 (un; Vm: 0) + (12 Um; 0). And the parameters are obtained by 0- = arg ming Tocal. (0). Then the influence function related to parameters can be deduced as:
Positive-IF((** T ,)));) = -HT² VoPos((x" (x", )z));
(2)
where HG VeE (U, V: 8T is the Hessian matrix. Proof refers to Appendix D.1
Extension to Multiple Samples. The influence evaluation described above be extended to subset Dr C DD. Let set 5 to index the batches containing data from Dr. For every m E S, define an index set F. specify the position of data from D' within the m-th batch. We encapsulate the assigned results as Seg {m, Im) E S}. By employing a derivation method similar to that used for a single data point, we can obtain the parameter-relau the influence (2) for all samples in D'
influence function for D* by summing
Proposition 4.1. The influence function for dataset D* serving as positive samples (positive-IF) can be approximated by positive-IF (I'' Seg; 0) H.¹ VoPos (D* ,Sg, 0), where Pos(D",Seg 0)
Emes Ener... (LTer(ua, Vmn 0) + LIZ7 (vn: Um: 0))
4

---

Published as a conference paper at ICLR 2026
4.2 INFLUENCE NEGATIVE SAMPLES
In Section 4.1
we quantified the impact of xf and x as positive samples by removing related pairing tasks. Next, we estimate their impact as negative samples by removing them from tasks that serve as negative samples. To achieve this, we need to investigate the specific form of contrastive loss.
Taking the text2image (T2I) loss for the k-th text embedding un as an example, we first calculate its similarity with all image embeddings in the batch to form similarity vector S(ux, V), which is then processed through a softmax layer o-) to yield a probability distribution. The k-th element indicates the probability of correctly pairing the text uy with its corresponding image: [or(S(ug, V)]l&
EjE[B] ck. where B is the batchsize. The model is encouraged to enhance the probability of
correct pairing by minimizing the negative logarithm of this value. For n + k. Vn serves negative sample in this task and appears in the Sk.n term in the denominator: Thus, after removing the impact of (xT, as a negative sample from the m-th batch, the loss function corresponding to this batch L aegg ((x S;;;) 8) should become:
Sk.
log
EjelB] e + Pos((x xx):;,
(3)
B k
jyn
The original influence function method evaluates a data point's impact by adjusting its weight via a separate term in the loss function and getting the response function. In contrastive learning, however, the influence of data points as negative samples is coupled with information from other data, which observed from a3. We will separate influence term related the data effect when it serves as a negative sample. Actually, the modification in (3) is analogous to eliminating the n-th row and column from the original similarity matrix. Leveraging the idea of deriving the influence function, we aim to develop a response function that converges to the target loss by up-weighting some specific components.
Considering that similarities vectors are processed through the softmax layer, if we increase the similarity associated with un and 2 to value approaching negative infinity, then after the exponential operation and the logarithmic function, the influence of e- and e_... will become negligible. Mathematically. let F., be an B x matrix such that its n-th column and the n-th row comprises ones, while all other entries are zero. We add the matrix log Ç x F, to the similarity matrix. Then the loss function based on the revised similarity matrix becomes:
SuN
log
EjElB Sko + (C - 1) Sk.m + Pos((x x O)).
(4)
REIB] Aem
We can easily see that as approaches O, the loss function T-2, in 4 converges to I mey in
3.. When ( 1, the loss function equals the original one. To further separate this influence as negative samples from the original loss function, we perform a Taylor expansion the O((S- - 1))² term, then T.f21 becomes
and drop
EvELB] SS.. eSenn
EvELEy SSu. eStnn
(S: + 11-
Σ kE(B lephin
6-0
(5;
Σ kE
and the left side is an estimation for 4). The minus term indicates the influence of (xT xx)) as negative samples. By employing similar method, one can obtain LET, for the image2text
Σje(n S0s Σje(m) 5
part. Denote Neg x²)=,)))00) as EkELB] Anffen
Down-weighting the
influence as negative sample by ( from 1 0, this influence in the loss function is then approximately eliminated. Then, the negative-influence function can be deduced as negative-IF((a* ((aeTT=")) )) -H VxNeg(( )x:;;));))))))) 0). As in the previous section, extend one single sample to a set of samples D' with corresponding positional indices Seg.
5

---

Published as a conference paper at ICLR 2026
Proposition 4.2. The influence function for dataset DD' serving as negative samples (negative- can be approximated by negative-IF(D Seg; 0)
H . VoNeg(D",, -Seg; f), with Neg(D* Seg: 0) -
Eje(
Shis ] 3
Emes EKE[BB/ Em n Num
inn enn.k
Combining Proposition and together, we then define our influence function method on contrastive learning(ECIF)) as follows.
Definition 4.3 (ECIF). The extended influence function for contrastive loss (ECIF) of the target dataset D with its position index set Seg {(m, Em.) (m E S) is defined as
ECIF(D", Seg; 0)
possttiv-IF(D"
Seg; 0), negative-IF(D", Seg; 0)
Thus, we can employ ECIF to estimate the changes in model parameters resulting from data removal We also give upper bound on the emror between the estimated influence given by ECIF and the actual influence obtained by model retraining in Appendix E for convex loss. We show that under certain scenarios, the approximation error becomes tolerable theoretically.
APPLICATIONS OF ECIF
We have proposed ECIF to evaluate the contribution of training data in contrastive learning. The ECIF method enables us estimate the change the learned parameters 0 if a training example pair is removed. Based on this, in this section, we will apply ECIF to two applications: misalignment detection and misprediction trace-back.
5.1 MISALIGNMENT DETECTION
Contrastive learning typically assumes a consistent alignment between all image-text pairs, and thus, misaligned data can lead to incorrect interpretations of these relationships, ultimately degrading model performance. Intuitively, given high-quality validation data D'. if D* is misaligned set, then the loss of D' over the original model ê should be greater than it over the model after deleting these misaligned data. And such a difference can be approximated by ECIF.
Property 5.1. Considering a specific set D' with text and image embeddings U' and V', and dataset D to be removed, then we have
LBatch (U', V¹; (';O(-D")) LBacch(U V); Lauch Vz (') (':
(5) where 0(-D") is the optimal model for the loss eliminating D", posi-IF(D'; Seg; 0) and nega-IF(D', Seg: 0) are obtained from Proposition for D We define term 5 the task-related influence score, denoted as IS(D', " DSeg 8).
LBVLh (U', V2 O) (posi-IF(D", Seg: 0) nega-IF(D'. Seg; 0)
Remark 5.2. Task-related IS measures the impact of data subset on task performance-ing sign shows positive negative influence, and its magnitude shows the extent, Thus, misalignment detection is formulated as arg max IS(D', D*, Seg: 6) (see Algorithm for details).
5.2
MISPREDICTION TRACE BACK
From transparency perspective, when model makes prediction errors on certain tasks, model trainers should be able to trace these error back to the specific training samples associated with the erroneous predictions. If we utilize the previous method for backtracking and choose the correct- labeled data that the model mispredicted to serve as the dataset D'. then there is a significant possibility that the identified data are misaligned samples unrelated to the prediction errors.
This is because, the definition of IS, the term on the right side the multiplication sign quantifies the change in model parameters. Even if certain samples unrelated to the task under analysis, they
We abbreviate positive-IF posi-IF and negative-IF nega-IF.

---

Published as a conference paper at ICLR 2026
may still exhibit high IS due to their significant influence model parameters. Therefore, compared to the application discussed above, it necessary to constrain changes in the model parameters.
To address this, consider imposing constraint when tracing back from mispredicted data while accounting for the process of upweighting the influence of samples positive by and negative by Then transform the trace back problem to identify which training example we should re- re-weight to most significantly impact the loss on the test sample D' when given small permissible change in model parameters.
the permissible changes in model parameters
s.t. ||
arg max Baach (U', VV, Al,,(()) - Tuatch (U", (U'VV'; :V sED,et
s (6)
Banch (U", V"; -7 s.t. (..IAa..(())) sp²;
z arg max DeD,e,
where Alec (2s 8) + (C- - 1) Dega-IF(<:: 8) is the model parameter change estimated by ECIF when the influence of sample (x xx)) is upweighted by and Ç.
Proposition 5.3. Define irreversible,
Iposi-IF(a;; 0) D..eg--F(a; 0)]
If
7T I is
then Equuequttion
7 is equivalent to
arg mD |Inega-F((;; (el2)112"
(ega-IF((;00) 0 Else, I is reversible, Lancc (U',V';') V);
then equivalent to
arg max V Basch V';1) LBachh (U', [' I'V (U", (U'''''') vv RED
The proposition above reduces the original argmax trace back problem to simpler argmax problem. Consequently, we define the simplified argmax objective as novel influence metric relative-IS. This metric, by adding constraints parameter perturbations, helps us more accurately identify task-relevant samples. See Appendix Algorithm 4 for details.
EXPERIMENTS
Our ECIF method serves two main purposes: (1) approximating the performance of a retrained model without the need for actual retraining, and (2) enabling data attribution evaluation. our experiments, we apply this method to various tasks, including removing data influence, identifying influential data (both harmful and valuable) for fine-tuning using task-related influence scores, tracing back mispredictions, and detecting misaligned data.
6.1 EXPERIMENTAL SETTINGS
Datasets. We will use the following datasets in our experiments: FGVC-Aircraft datasel (Maji al. 2013), Food10/ (Bossard et al., 2014), Flowers102 NNilsback & Zisserman, 2008). Cifar- 10 (Krizhevsky et al., 2009). Cifarl00 (Krizhessky et al. 2009), Describable Textures Dataset (DTD) (Sharan (Sharan Imagenet).
al. 2014). and Imagenette (Deng et al. 2009) (a smaller subset of 10 classes from
Our methods. The experiments below directly implement the algorithms for the applications in the previous section. Algorithm]] I functions the foundational algorithm, offering methods to calculate ECIF and providing a model editing based on ECIF. Algorithm 2 and 3 task-related Prop. 5. to evaluate samples, indicating both the direction and intensity of their impact on the task. Meanwhile, Algorithm[4] ii for relative-IS in Prop. 5.3 which aids tracing back specific samples.
in
Ground truth and baselines. For all tasks, retraining from scratch will be ground truth, where the CLIP model is fine-tuned from scratch after sample removal. Furthermore, we compare ECIF with three other data attribution evaluation methods (IF-ECFAC (Grosse et al. 2023a), TARK (Park et al.) 2023), and TracIN (Pruthi et al., 2020)) as baselines. Note that our ECIF is direct implementation of Algorithm I utilizing positive and negative IF to modify the model for sample removal.
Evaluation metric. We utilize two main evaluation metrics to assess our models: accuracy and runtime (RT). Accuracy evaluates the model's performance by measuring the proportion of correctly
7

---

Published as a conference paper at ICLR 2026
classified instances out of the total instances. Runtime measured in seconds, assesses the time required for each method to update the model. For implementation and additional details, please refer to Appendix G.1
6.2 DATA INFLUENCE REMOVAL EVALUATION
We first consider the utility and efficiency of ECIF compared to ground truth. Table 2 shows that ECIF achieves very close precision to retraining in all datasets, indicating that the models estimated by ECIF exhibit generalization capabilities similar to those obtained through retraining.
For example, the accuracy difference is only 0.30% on FGVCAircraft and 0.06% on Food101, which shows that ECIF effectively approximates the performance of retrained models. Moreover, ECIF is significantly faster, reducing runtimes by over 2 times compared to retraining (e.g., 456.0 seconds vs 1174.2 seconds on FGVCAircraft). These results highlight that ECIF can serve as a reliable surrogate for retraining, achieving comparable generalization performance. Moreover, ECIF can save approximately 80%-90% of the computational time required for retraining while maintaining comparable accuracy levels.
Table 2: Performance comparison of retraining and ECIF on different datasets.
FGVCAireraft
Food101
Flowers102
Cifar100
Sample
Method
Accuracy(S) (%)
RT (s)
(ocuracy(%)
RT (s)
Accuracy(S}
RT (8)
Accuracy(S) %)
RT (s)
Retrain
23.07-40.29
1174.2
84.93-0.177
875.4
68.16-2
995.4
73.50+-0.35
753.6
Random
ECIF
22.77:10.09
456.0
84.87=0.24
436.8
68.53:0.12
437,4
73.00=0.20
444.0
Valuable
Retrain
22.93-+00.3
933.6
84.80=0.16
952.8
68, 23-+0.33
985.8
73.50=0.41
721.2
ECIF
22.73:10.09
357.0
84.86=0.05
376.2
68.26:10.12
391.2
72.92=0.31
462.0
Retrain
23.50++0.11 11
1344.0
84.83=0.05
875.4
68 68.00=20110
965.4
72.8 72.83-0.122
775.2
Harnful
ECIF
23.02:10.07
375.6
84.90-0.01
373.2
68.30:10.01
376.2
73.00=0.200
435.6
We then compare the effectiveness of ECIF with previous baseline methods in assessing data at- tribution. We begin by using various methods to perform a data attribution evaluation. Subse- quently, we remove 10% of the most harmful samples from the training dataset and retrain the model. Specifically, for ECIF, we retrain the model and utilize it to update the results ac- cordingly. The results are presented in Table [ We can see that when harmful data identified by ECIF are removed, the accuracy of both the retrained model and ECIF's edited version aligns closely and both are higher than those observed with no data removal. This suggests that ECIF is capable not only of accurately editing the model but also of effectively identifying influential data. We can also see that the baseline methods have a significant gap with ECIF or Retrain, indicating that they are less efficient. This is because these methods are tailored for pointwise loss and are inapplicable to contrastive loss. In addition, harmful samples. This is because previous methods focus mainly on the most valuable/influentia data, while for each sample there is both positive and negative influence in contrastive learning. We also study performance across varying numbers of removal samples (see Appendix G.3)
Table 1: Performance comparison with baselines.
Method
Accurscy(%%
PGVCAlreratt
Food101
Flowers102
Cifart00
Original model withont data removal
Fine-tune
22.18-0.21
83.85-0112
67.64+031
72.31+0.11
Harmful samples 1
Retrain
23.50:20.11
$4.8320005
68.0010.16
72.83:0.12
IF-EKFAC
19.84_00.88
78.26.10.09
61.6711.19 60.711023
18:27:0.09
77.27:0003
59.21:0022
58.67:1.70
TiEIN
10448-0.10
38-550001
60:0000.3
59.00 19:00+6116
ECIFIECIF)
23.02__0.07
84.90
68.301..01
73.00_0.20
observe that all previous methods fail to identify
6.3 IDENTIFYING INFLUENTIAL DATA FOR FINE-TUNING
Task-related IS can identify valuable data. To numerically assess the precision of data valuation algorithms, we employ the brittleness test Myyas et al. 2022). which evaluates the algorithm's 's ability to accurately identify the most valuable data for a specific task. Our evaluation process is follows:
utilizing the validation set within Algorithm 2 we compute the task-related IS for each individual training data point. We then remove the top-k valuable data points, with k ranging from 5% to 30%, retrain the model multiple times using different random seeds, and assess the resultant change in overall model accuracy. The results in Figure 1b reveal that the removal of valuable data identified by ECIF leads to a consistent decrease in the accuracy of the model, from 84.7 to 84.1. In contrast, the removal of random data triggers an increase in model accuracy once the removal proportion reaches
Here, runtime refers to the total runtime required for the updating process.
8

---

Published as a conference paper at ICLR 2026
013
Removal Rattn
Romoval Ratio
(a) Harmful Data Removal
(b) Valuable Data Removal
Figure 1: Accuracy after data removal on Food101.
0.3. This suggests that Food101 contains substantial noise and that our algorithm can effectively identify data points that genuinely enhance the predictive accuracy of the model.
Task-related IS can identify harmful data. Algorithm 2 identifies data pairs with negative task- related IS as harmful data for the task. To demonstrate the effectiveness of our algorithm in identifying detrimental data for specific tasks, we conducted experiments on several noisy datasets, such as Food101. See Appendix G.2 for the results of other noisy data.
We collected the harmful data identified by ECIF and then retrained the model multiple times with varying harmful data removal ratios and different random seeds. We compared its accuracy to that of a model retrained after randomly removing the same number of data points. The results in Figure la demonstrate the effectiveness of our approach in improving the performance of the model by eliminating harmful data using IS related to tasks. Figure Ta indicates that with varying proportions of harmful data removal, the accuracy of the retrained model consistently fluctuates around its original level. When 10% of harmful data is removed, accuracy increases by approximately 1%. In contrast, with random deletions, the accuracy continues to decrease. This suggests that the accuracy improvement from the removal of harmful data with ECIF is not merely due to the removal action itself, but rather because the removed data genuinely had a detrimental effect on model training.
6.4 VISUALIZATION OF MISPREDICTION TRACE BACK
Table 3: Top-10 10 related training data traced by mispredicted data. We apply Algorithm to identify training data that are most relevant to specific mispredicted test samples. In this process, we select samples in the test data on which the model made misclassi- fication. Using the relative IS, we can identify the training data with the highest influence on the misprediction and visualize it. Table shows the results of this misprediction trace-back process (see Appendix G.4 for additional results). Each pair of images compares a test sample with its most influential training counterpart. On the left, we show examples from the test set where the model produced incorrect predictions. On the right, the corresponding training data are shown, i.e., these data points hold the highest relative ISs relation to the mispredicted test samples. This comparison helps shed light on how specific training samples may have contributed to the incorrect output of the model. According to the visualization results, it can be observed that the samples traced back to the original task exhibit similarities in shape or texture with the original task.
6.5 DATASET CLEANING: MISALIGNMENT DETECTION
We employ relative IF to detect misaligned pairs. Regarding the selection of validation dataset, we experiment with two approaches: randomly selecting from the gold dataset (Algorithm{) (2) and calculating based on the influence of the evaluated sample points (Algorithm 3) in which the test loss

---

Published as a conference paper at ICLR 2026
gas pump
parachute
golf
tench
tench
parachute
garbage truck
French
golf
French horn
Figure 2: Top-10 misaligned pairs in the 20% mislabeled dataset.
defined as CLIP score (Hessel et al. 2021) the evaluated data pair. We first mislabel 10%-30% training samples and then identify the misaligned pairs by selecting those with the highest negative IS. These pairs are visualized Figure (see Appendix G.5 for additional results). The visualization results reveal that 8 of the top 10 data points with the highest IS are entirely contained within the mislabeled portion of our training dataset. This suggests that our algorithm has effectively identified the noise data artificially introduced into the dataset.
CONCLUSION
We propose ECIF (Extended Influence Function for Contrastive Loss), a novel data valuation frame- work for contrastive learning. ECIF evaluates both positive and negative sample impacts through dual-perseective analysis, enabled by efficient closed-form approximation without retraining. Vali- dated real-world applications. ECIF effectively supports fine-tuning, misprediction tracing, and misalignment detection while maintaining scalability for large models.
ACKNOWLEDGMENT
Di Wang, Huanyi Xie, Chenyang Ren, Khouloud Saadi, and Shu Yang are supported in part by the funding BAS/1/1669-01-01 000////6990000RRG277115550000000
FCC/1/5940-20-05, FCC/1/5940-06-02, and King Abdullah University Science and Technology (KAUST) Center of Excellence for Generative AI, under award number 5940 and gift from Google. Lijie Hu is supported by the funding BF0100 from Mohamed bin Zayed University of Artificial Intelligence (MBZUAI). For computer time, this research used lbex managed by the Supercomputing Core Laboratory at King Abdullah University of Science Technology (KAUST) in Thuwal, Saudi Arabia.
REFERENCES
Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization for machine learning in linear time. Journal of Machine Learning Research, 18(116): 2017.
S Basu, P Pope, and S Feizi. Influence functions in deep learning are fragile. In International Conference Learning Representations (ICLR), 2021.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. nents with random forests. In Computer vision-ECCh 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part VI 13, pp. 446-461. Springer, 2014.
discriminative compo-
10

---

Published as a conference paper at ICLR 2026
Mare-Etienne Brunet, Colleen Alkalay-Houll Houlihan, Ashton Anderson, and Richard Zemel. Understand- ing the origins of bias in word embeddings. In International conference on machine learning, pp. 803-811. PMLR, 2019.
Hongge Chen, Si Si, Yang Li, Ciprian Chelba, Sanjiv Kumar, Duane Boning, and Cho-Jui Hsich. Multi-stage influence function. Advances Neural Information Processing Systems, 33:12732- 12742, 2020a.
Ruizhe Chen, Jianfci Yang, Huimin Xiong, Jianhong Bai, Tianxiang Hu, Jin Hao, Yang Feng, Joey Tianyi Zhou, Jian Wu, and Zuozhu Liu. Fast model debias with machine unlearning. Advances in Neural Information Processing Systems, 36, 2024.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020b. URL https: /ttp//aaaxll.aaaaaaa////
2002 05709
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning. 2020c.
Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, al. What your data worth to gpt? lim-scale data valuation with influence functions. arXiv preprint arXiv:2405. 2024.
Dennis Cook. Detection of influential observation in linear regression. Technometrics, 42(1))65-66, 2000.
Dennis Cook and Sanford Weisberg. Characterizations of an empirical influence function for detecting influential cases in regression. Technometrics, 22(4):495-508, 1980.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 JEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Jeff Donahue and Karen Simonyan. Large scale adversarial representation learning. Advances in neural information processing systems, 32, 2019.
Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, Rongrong Ji, and Chunhua Shen. Pyramid- clip: Hierarchical feature aligmment for vision- language model pretraining. Advances in neural information processing systems, 15:35959-35970 2022.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International conference machine learning, pp.
PMLR, 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting deep networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed-privace forgetting in deep networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 792-801, 2021.
Roger Grosse, Juhan Bac, Cem Anil, Nelson Elhage. Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilé Lukošiūté, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying large language model generalization with influence functions, 2023a.
abs/23088 03296
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint a00iv: 230.029990... 2023b.

---

Published as a conference paper at ICLR 2026
Han Guo, Nazneen Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. Fastif: Scalable influence functions for efficient model interpretation and debugging. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp.
2021.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Xiaochuang Han, Byron C Wallace, and Yulia Tsvetkov. Explaining black box predictions and unveiling data artifacts through influence functions. arXiv preprint 2XXix:0005006677 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9729-9735 2020.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7514-7528, 2021.
Peter Huber. Robust statistics. Wiley Series in Probability and Mathematical Statistics, 1981.
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Data- models: Predicting predictions from training data, 2022.
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang. Dawn Song, and Costas J Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1167-1176. PMLR, 2019.
Chaoya Jiang, Haiyang Xu, Mengfan Dong, Jiaxing Chen, Wei Ye, Ming Yan, Qinghao Ye, Ji Zhang, Fei Huang, and Shikun Zhang. Hallucination augmented contrastive learning for multimodal large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27036-27046, 2024.
Bumsoo Kim, Yeonsik Jo, Jinhyung Kim, and Seunghwan Kim. Misalign, contrast then distill: Rethinking misalignments in language-image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2563-2572, October 2023.
Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. Advances in Neural Information Processing Systems, 36, 2024.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference machine learning, pp. 1885-1894. PMLR, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images, 2009. URL https!
//api. sesp...sementtcscholaae htt.:/o.....orrroooooooo:::1/11.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images, 2009.
Yongchan Kwon and James Zou. Beta shapley: unified and noise-reduced data valuation framework for machine learning. In Proceedings of The 25th International Conference on Artificial Intelligence and Statisties, 2022.
Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned iims and diffusion models. In The Twelfth International Conference on Learning Representations, 2023.
Xianhang Li, Zeyu Wang, and Cihang Xie. An inverse scaling law for clip training. In A. Oh, T. Naumann, A. Gloherson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 49068-49087 Curran Associates, Inc., 2023a.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language- image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 23390-23444 23400, June 2023h.
12

---

Published as a conference paper at ICLR 2026
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- fine-grained abs/
visual classification of aircraft, 2013, URL / http:///aaaaa.ooorr//////5......
Thao Nguyen, Gabriel Ilharco, Mitchell Wortsman, Sewoong Oh, and Ludwig Schmidt. Quality not quantity: On the interaction between dataset design and robustness of clip. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35. pp. 21455-21461
Curran Associates, Inc., 2022.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics, Image Processing, pp. doi: 10. N0.1109/CVGIP.2000..444
722-729, 2008.
Sung Min Park, Kristian Georgiev, Andrew Ilyas, MuillaumeLLelere and Aleksander Madry. Trak: Attributing model behavior at scale, 2023. hut.:/ttpp::/axxi...rrrg///// http://araii....rrra/////0//0/0
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. In Advances in Neural Information Processing Systems, volume 33, pp. 19920-19930, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Joshua David Robinson, Ching Yao Chuang, Suvrit Sra, and Stefanie Jegelka. Contrastive learning with hard negative samples. In International Conference on Learning Representations, 2021.
Lavanya Sharan, Ruth Rosenholtz, and Edward H. Adelson. Accuracy and speed of material categorization in real-world images. Journal of Vision, 14(9):12-12, 2014. ISSN 1534-7362. doi: 10.1167/14.9.12.
Hwanjun Song, Minseok Kim, and Jae- Lee. Selfie: Refurbishing unclean samples for robust deep learning. In International conference on machine learning. pp. 5907-5955 PMLR, 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, 2019.
Hao Wang, Berk Ustun, and Flavio Calmon. Repairing without retraining: Avoiding disparate impact with counterfactual distributions. In International Conference on Machine Learning, pp. 6618-6627. PMLR, 2019.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity on the hypersphere. In International conference on machine learning, pp. 9929-9939, PMLR, 2020.
Alexander Warnecke, Lukas Pirch, Christian Wressnegger, and Konrad Rieck. Machine unlearning of features and labels. Network and Distributed System Security (NDSS) Symposium, 2023.
Jiancan Wu, Yi Yang, Yuchun Qian, Yongduo Sui, Xiang Wang, and Xiangnan He. Gif: A general graph unlearning strategy via influence function. In Proceedings of the ACM Web Conference 2023, pp. 651-661, 2023.
Lewei Yao, Runhui Huang. Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li. Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-iman pre- pre-training. arXiv preprint arXiv:21 1 1.07783, 2021.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv:2306. 13549, 2023.
Jinsung Yoon, Sercan Arik, and Tomas Pfister. Data valuation using reinforcement learning. In International Conference Machine Learning, pp. 10851. PMLR, 2020.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision languaee models behave like bags-of-words, and what to do about it? In The Eleventh International Conference Learning Representations, 2023.
13

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS (LLMS)
We used LLMs to refine grammar and improve language fluency. The authors reviewed and edited all LLM-ggenrated content and assume full responsibility for the final text.
B ALGORITHM
Algorithm ECIF
Input: Training Dataset D )")), dataset DD to be evaluated, the parameters 0 which is involved in IF calculation in the model and the regularization parameter d.
2: Define (.,-) -) as the similarity score.
3: Compute the text embedding and image embedding for D as and
4: Random divide the training dataset into MM batches and obtain the position index of D as
Seg & {m, Em m E 5}.
5: Compute the influence term positive and negative samples for the mm-th th batch in S by:
e (unn),))
Posm
Σ
log
log
-= ((un)))))
(S(ni)))))))
j=
j== ((wiin))
(uusi))))
Negm
Σ
EEEE
nEEm (S(ssi
iE|NT/ Em
6: Compute the sum of the gradient Posm and Neg.. as
Pos
VoPosm: and Neg
VoNeg
mES
7: Compute the batch embedding for D as [Bm, m E [M]].
8: Compute the inverse Hessian matrix of the loss function with respect to 0
G
BiLBatch (Bm; 0) 8
MEIM]
9: Compute the positive-IF(D". Seg) and negative-IF(D* ,Sg) as:
Positive-IF(D*. Seg) = -G Pos, negative-IF(D* Seg) G Neg
10: Obtain the ECIF
ECIF D) -ppositive-IF, negative-IF)
(8)
Edit model parameter to unlearn dataset D by
i = Ô positive-IF negative-IF
12: Return: ECIF(D*, D), Edited parameter 0.
C ACCELERATION FOR INFLUENCE FUNCTION
LOGRA. They observed that the gradient from backpropagation structured as sum of Kronecker products of forward and backward activations. LOGRA applies an additional Kronecker-produet
14

---

Published as a conference paper at ICLR 2026
Algorithm 2 Task-related Influence Score Based on ECIF
I: Input: Training Dataset D {(x", [(=",)")) dataset D be evaluated, test dataset D', the parameters & which is involved in calculation in the model.
2: Compute the ECIF(D", D) = (positive-IF IF, negative-IF) by algorithm
3: Compute the gradient of the batch loss function of the test data as
C= Σ (U.V)ED
Ve (U, V;
4: Compute the task-related influence score
IS = = positive-IF + C' -neattiv--IF
5: Return: Task-related Influence Score TS.
Algorithm 3 Self Influence Score Based on ECIF
1: Input: Training Dataset
{x= ,oo",,")}, ,)),, dataset D evaluated, test dataset D', the parameters å which is involved in calculation in the model.
2: Compute the ECIF(D", D) (positive-If: negative-IF) by algorithm
3: Compute the gradient of the batch loss function of the test data as
C -
1
Σ
Vo -log
u
D
|||
|||||
JED
where and is the embedding for xx and x 4: Compute the task-related influence score as
IS = CT positive-IF CT -negative-IF
5: Return: Task-related Influence Score IS.
Algorithm 4 Relative Influence Score Based on ECIF
1: Input: Training Dataset D ((x".>> xx'') }}, dataset D* be evaluated, test dataset D'.
2: Compute the ECIF(D*, D) = (positive-IF, negative-IF) by algorithm
3: Compute the gradient of the batch loss function of the test data as
C=
Σ (U,V)D
VoLBach (U, V:
4: if positive-IF is parallel to negative-IF IF then
5: Compute the relative-IS as
relative-IS = | |C" negativv-II
6: else [positive-IF is not parallel to negative-IF)
7: Define / [positive-IF, negative-IF).
8: Compute the relative-IS as
relative-IS = |T [1T 1 Po
9: end if
10: Return: relative-IS.
15

---

Published as a conference paper at ICLR 2026
Algorithm 5 Relative Influence Score Based ECIF
Input:
Training Dataset D {(=", }} dataset D be evaluated, dataset D'.
2: Compute the ECIF(D", D) = (positive-IF, negative-IF) by algorithm
3: Compute the gradient of the batch loss function of the test data as
u V
Ve log
D
|
|0||
where and is the embedding for x² and x!
4: if positive-IF parallel to negative-IF then
5: Compute the relative-IS as
relative-IS possiiive-FF |C" Iegaattive-II
6: else [positive-IFF is not parallel to negative-IF)
7: Define [positive-I,, negative-IF].
8: Compute the relative-IS as
relative-IS
||C|| c
( [ -1] 1 rTc
9: end if
10: Return: relative-IS.
structure to the projection matrix A Ps Po. It first projects the forward and backward ac- tivations onto low-dimensional spaces using P; and Pa. respectively, and then reconstructs the projected gradient directly from these reduced activations. For one layer, given the input 24, out- put xo and the weight W, the forward and backward computation can be written T, W x;, veo(DW)= = = Ii,t @ Dio.tt De; = W TDD... where denotes for the sequence dimension in language modeling, D the derivative with respect to the loss, & the Kronecker product, and veo(.) the vectorization operation. Observing gradient vec(DW) obtained during backpropagation is structured sum of Kronecker products between forward and backward activations, LOGRA imposes additional Kronecker-produue structure on the projection matrix P as follows:
T Pvec( (P, P.)vec(DW)) = E(P (P. Po) & Dro.)) = Σ Paria & P.Dros
t=1
where P; is the projection matrix on the input and Po is that on the backward activations, and P P, ② Po-
D INFLUENCE FUNCTION IN CONTRASTIVE LEARNING
D.1 INFLUENCE FUNCTION FOR POSITIVE SAMPLES.
We first consider the influence function for positive samples.
Single Data Pair Version. To quantify the impact of xx and as positive samples, first define L721 (un, Vnii () + LI2T Umi () as Pos((x", æ²); 0). Following the idea of influence function, we can up-weight these two parts by and obtain an up-weighted loss function as
L L (0) =
Σ
LBath LBath(U, V:0) + 2 11011 + te-oos(( ))
(U.V)EB
And the parameters are obtained by Ô arg ming I (0). From this minimizing condition, we have
V Vo UU, (U,V;0.) + . (x x)) = 0
(U.))ES
16

---

Published as a conference paper at ICLR 2026
Perform Taylor expand at 0 = 9, we have
BoTTBacch (U, VU, + VoPos((a" x x²);;) + (U,V)8
Σ
Σ
Vi Batch Ba, V: (0--0) =
(U.V)
Because 0 minimizes E(c,v)eBLBuch (U, V 0),
the first term in the above equation equals 0.
positive-IF((2*) x²):0)
-
VoPos((>" , xx));"
de
c=0
where H8 (E Bacch (U, V; 57 is the Hessian matrix at Ó.
Extension Multiple Samples. The influence evaluation described above can be extended to subset D C D. Let set S to index the batches containing data from DD" For every m S, define an index set F. specify the position of data from DD within the m-th batch. We encapsulate the assigned results Seg {m,, Em)(m S}. By employing derivation method similar to that used for a single data point, we can obtain the parameter-related influence function for D by summing the influence as a position sample (2) for all samples in D
Proposition D.1. The influence function for dataset DD' serving as positive sample (positive-1F) is
positive-IF(D", Seg; 0) - VoPos(D", Seg; 0)
where
Pos(D* Seg; 0) = > Σ
(un Vn,; L1rr(Ung Uni 0)
mES nE
Proof. Seg = [m, Em) E S}. for m E S, Um. V. are the text and image embedding for the m-th batch, respectively. For E Emn, un and in are embeddings for single data pair in m-th batch, which is included in the dataset to be evaluated DD Define Pos(D", Seg; 0) as
Pos(D", Seg; 0)
Σ Σ
(Lr21(un, Vn; 0) + L (Un, Um 0))
mES
Following the idea of influence function, we can up-weight these by and obtain an up-weighted loss function
ToAal (0)
Σ
Ieatch (U, V: + 1111 + Pos(D", Seg; 0)-
(U.V)ES
And the parameters are obtained by 6- = arg ming ( From this minimizing condition, we have
Σ
V (U, (.. te. VoPos(D", Seg 0.) =
(U.V)EB
Perform Taylor expand Ô. we have
VoL (U, V: + VoPos(D", Seg; 0) +
Σ
VgLnac (U, (0. 0) 0
(U.V)EB
(U,V)EB
Because i minimizes E(c,v)e (U, V; V) the first term in the above equation equals 0.
positive-IF(D". Seg: 0) dÒ,
= - VoPos(D", Seg; 0)
de
where H3
(U, V; + 57 is the Hessian matrix at Ô.
17

---

Published as a conference paper at ICLR 2026
D.2 INFLUENCE FUNCTION FOR NEGATIVE SAMPLES.
Then, we come to derive the influence function for the negative sample.
In this part, we will illustrate how we give an approximation function for the loss function in which the influence as a negative sample of the data we are considering is removed. With the help of Taylor expansion, this influence separated into a single term in this approximation function, and we can achieve this by removing this term from the original loss function.
After removing the impact of (x"" .xt) negative sample from the m-th batch, the loss function corresponding to this batch should become:
S
1i neg (ax S:))
Σ KE k
E S + Pos((x")
"oss(("")²))) ,x)); :)
(9)
Mathematically, let En be an B x B matrix such that its n-th column and the 1-th row comprises ones, while all other entries are zero. We add the matrix log { x E,, to the similarity matrix. Then S.n becomes Sa,, + log The loss function based on the revised similarity matrix becomes:
S
L ( ,('", x)), = Σ
log
Shi + clog S. + Pos ((x x²);0)
EJ jfh
Sh,k
Σ
log
+ Pos ((z" x²)):0)
Σ jE [8]
Ski + t..e0 @Sem
k
jfn
kpn
Sb
log
+ Pos ((x² (x)); ););
Ski + (C 1) @.em
2)E[B]]
kE
We can easily see that as approaches 0, the loss function TT2, converges to L neg in 9.. When
Ç = 1, the loss function equals the original one. To separate the change in the C approaching O from 1 process, we perform Taylor expansion at Ç = 0 and drop the O({S 1)²) term, then L becomes
SA,A
Eje(m Ska
Σ log Ejenn S kE B kyin
+ (5-
Σ kE[B] kiplnn
+ +0(((-1)3) + Pos((x", x 0))
And by setting <= 0. the loss function L indicates that the influence of (xT ,x!) when it serves the negative sample is fully removed from the training process.
L
log
e
Sh.
e + (0-1).
Σ
ΣjE[B| e
) + Pos((xT,,>'))
EjElB
Sknn
KE Jegfi
kE[B
EjEloje Sko eko
= (S:0)
Σ
kE(
Single Data Pair Version. From above discussion, quantify the impact of TT and x. as negative samples, we first define Neg ((x", . xx)): )) as
Σ Ejelene Sn. S.Sn
Eyero SS.b
Sin
Neg ((x") x )) = Σ
kE
knfn
Down-wett weightiig the influence as a negative sample by Ç from to O, this influence in the loss function is then approximately eliminated. In this process, the loss function becomes
5 L (U, V) + [101 2 + (S 1) Neg((=" 1-x')) x)'):)).
LToial. (()al.c(())))))((
Σ
(U,V)
18

---

Published as a conference paper at ICLR 2026
And the parameters obtained by 0g arg ming L From this minimizing condition, have
Σ
Bachh (U,
VaNeg((o", xx);0c) - 0
(C,))B
Perform Taylor expand 0 0, we have
VoLBachh (U,/BaccUU,VV () + (C - VoNeg((<> ((x , ;²);;)) + (U,))EB
Σ
Σ
BU, (0, -0)
(C,V)EB
Because 9 minimizes L(c.v)eo-Buch (U, 0); the first term in the above equation equals 0. Then
0. - 0 = -( 1) H (aNe(( , 0));0)
where H8 (o.) (U, V;, 57 is the Hessian matrix at 0.
negative-IF((2 )) ddc
--1 VoNeg((x", x²);;
<=0
Extension to Multiple Samples. Then, we extend the above influence evaluation to a subset D' D. Let set S to index the batches containing data from D For every E S, define an index set Em to specify the position of data from D" within the m-th batch. We encapsulate the assigned results Seg {m, Em |m S). By employing a derivation method similar that used for single data point, we can obtain the parameter-related influence function for D*.
Proposition D.2. The influence function for dataset DD serving as negative sample (negative-IF) is
Lie(n
Skit
Eie(n Sz.. Ener Em
Neg(D", "eegD",eeg0000 0) b)g; = Σ
EneF Sknn
Saikk
TES kE(Bi/Em
And
negative-IF(D", Seg: = - Segg 0).
Proof. Seg = {mm, Em)lm E S), for m E S, Um. Vm are the text and image embedding for m-th batch, respectively. For E Em. and are embeddings for single data pair in mi-th batch, which is included in the dataset to evaluated D
Step 1. Noting the data in D' may come from different batches and multiple data from batch, then we firstly derive the loss function approximation with separated negative sample influence removed.
For the m-th batch, m E S. after removing the impact of the data indexed by Em as a negative sample, the loss function corresponding to this batch should become:
S
m oog (00" ).),) = Σ
"os((x",>')))
(10)
E E... jpEm
Then, for n E Fm, let F,, be an B x B matrix such that its n-th column and the n-th row comprises ones, while all other entries zero. We add the matrix log Ç F,, to the similarity matrix. Then, the loss function based on the revised similarity matrix becomes:
St.b
(a= x S)),
Σ log Eje(n Sus + + + 1). Ener Skm
Pos(( . x²)); O).
kELB
kgf Fm
We can easily see that approaches 0, the loss function LTg, converges Lin neg in 9 When < the loss function equals the original one. To separate the change in the ( approaching
19

---

Published as a conference paper at ICLR 2026
from 1 process, we perform Taylor expansion at and drop the O((S 1)9) term, then Lia,c
becomes
EEE( Skis
log
15-
Σ
+ +1((s + Pos((x", (o)((<"xx);0)))
EnEE..
kE|B Eg
kE[B]
kgg Em
And by setting <= 0, the loss function L4200 indicates that the influence of (x* x²) when it serves as the negative sample is fully removed from the training process.
Eje(on SS
121.0
log
Σ
S
(0-1).
Σ
En edum
+ Pos((x) ,,);;,)))0
kE En
]E[[B]
k
kg
EjElet eStu
(S:0)
Σ
ΣnEBm
S
kE[B]
F...
By down-weighting the influence of "D as negative samples by C. the total loss function becomes
Eje(n Sas Sknn
8 11111 + (C 1) Σ Σ
L c(0)
Σ
Laath (U, V
+
Ener Fir
(U.V)EB
[B)/Em
Then denote ND" Seg; 0) as
EjE(D Su.s
EjEB SSnn
Neg(D", Seg; ()
Σ E
EnEE Svin
EnEE Sm,b
mES kE(B)/E,,
And the loss function with the negative-sample influence of D explicitly removed
L
Σ
LRatch V: 1012 , 0)
(UVV)EB
Step The parameters are obtained by 0 arg ming L (0). From this minimizing condition, we have
Vo (U, V: - - 1). VoNeg(D*, Seg: 0)
(UV))ES
Perform Taylor expand 0, we have
Vo (U, V: + (S - 1) VoNeg(D', Seg: 0) +
Σ
I. (U, V:0) (6c-0)
(UV))
(U.))
Because i minimizes E Batch (U, V;0), the first term in the above equation equals 0.
negative-IF(D". Seg: 8) e dd
-
VoNeg(D". Seg: θ)
dç
<=0
where H3 L (U, V; 57 is the Hessian matrix at 0.
L
E APPROXIMATION ERROR BOUND
In the previous discussion, we have established that when applying the influence function method contrastive learning, is impractical to design sample-specific up-weighting scheme that approximates the corresponding loss function resulting from the removal of a single pair in the batch without affecting the remaining data. Therefore, based on the previous derivation, we provide an estimation function L- for this loss function. Consider the dataset D* define
T. TD Seg: 0) A Pos(D", Seg: 0) -Neg(D", Seg; 0),
20

---

Published as a conference paper at ICLR 2026
Then the loss function with the influence of DD removed becomes
L LB D" Seg; LT (B;0) L' (SD"
(11)
Equation equation 11 is based on the assumption that the influence of data acting as positive and negative samples on model parameters can be linearly superimposed, and we can leverage ECIF to edit the model based on the following corollary. This approach enables us to achieve the unlearning updating of specific data without the need to remove data and retrain the model.
Assume 0 arg min Trotal is the original model parameter, and ((-D ) is the minimizer of L which obtained from retraining. Denote (-y(-D*) as the updated model with the influence of "D' removed and is obtained by the ECIF method, which is an estimation for 0( (--D"). Because we concentrate on D we omit the Seg in the above definitions for short.
In this part, we will study the error between the estimated influence given by the ECIF method and retraining. We use the parameter changes as the evaluation metric:
(ar A) (0(-D* 0) (-y( - ((-
(12)
Before our main theorem of the upper bound for equation 12). we need prove corollaries and make some assumptions.
Proposition E.1. Assume that influence as positive sample and as negative sample can be linearly
superposed. Then when the influence dataset D positive sample is up-weighted by and that as negative sample is up-weighted by Ç. then the loss function become
L (D' Seg: 0; C) A or (B; (B:0) + VoPos(D". Seg; 0) + (5 11) Negg(D", SSg: 0)
And corresponding parameters 0, are defined as
DD) arg min LD" Seg; e,
The approximation of (e.c(-D') is derived as
V22
V2 2 VoNeg(D" Seg: (13)
(D,) = - H.g
VoPos(D" Seg; 0)
Property E.2. Assume that influence as positive sample and as negative sample can be linearly superposed. Then when the influence dataset D as positive sample is up-weighted by and that as negative sample is up-weighted by Ç, then the loss function become
L (D* Seg: G: C) & oo (B; (B;0) + c. VoPos(D" Seg; 8) + (5 11) Neeg(D", SSg. 0)
And corresponding parameters 0z,ç are defined as
0,c(- arg min I. LD" Seg; 0;,,5)
The approximation of (e.c(- is derived as
(e.(--D") ~0,((-D")
(14)
- H (.. (e.VoPos(D", Seg; + + 1)) VoNeg(D* SSeg, Seg: 8))
Assume that influence positive sample and negative sample can be linearly superposed. Then when the influence of dataset D as positive sample is up-weighted by and that as negative sample up- up-weighted by then the loss function become
Proof.
L (D* Seg; 0: C) 2 LToter + Pos(D", Seg: 0) + (C - 1) Neg(D", Seg; 0)
And corresponding parameters 02, are defined as
(.c(-D'') = arg min L (D*, 0c
21

---

Published as a conference paper at ICLR 2026
Then, from the minimizing condition,
VoLio de.s) + e. VoPos(D", Seg; fc.c) (S 1)- VoNeg(D" D' Seg; oe.5) = 0,
where (e.((-D)) is written as c for short. Perform Taylor expansion around = 0, then we have
VoLTotal (B: te- VoPos(D", Seg; 0) + (S - 1) VoNeg(D", "eeg 0)
+ (B) (B..5 - =0.
Because 0 minimizes LT 0), the first term in above equation equals 0. Then we have
de.5 = --1 E VgPos(D", Seg; 0) + (6 - 1) VgNeegD",, Seg; Seg:
= = M VoPos Seg; 0) (5 - 1) g/g VoNeg(D",See 0)
+c Positive-IF(D*, Seg: 0) + (S 1) Degative-IF(D)) Seg: 0)
where Hj V eu. (U, V; V) When -1, 0, Q_100 estimates the parameters obtained by retraining after D removed.
Assumption E.3. The loss Laasch (U, V, 0) is convex and twice-differentiable in 0, with positive regularization 8 0. There exists Ch such that
V Lanch (U, V 01) V LRach (U, V;02lll S CHI10 0211
for all V) E B and 01, 0₂ E e. Assumption E.4. The function I'((xT , 0)):
L'((x", x 0) Pos((x )) 0) + Neg((= (o",,x));0 xx);0)
convex and twire-diiferrentialle 8, with some positive regularization. There exists C² such that
V/LL((> . 01)) ViL'((" ((0((("''))))))) ,'):02)112 CH110 02112
for all (x (x) E Dr and 0,,02 E.. Corollary E.5.
IV Seg; 01) V²L- Seg: 02)12 (IB) CH + | ||0 -62l
Define CF A B Cи + DD1 C² Definition E.6. Define |D] as the number of pairs
C
max (of a EB
VOT'((E
o smallest singular value of ViL (D" Seg; 0),
min smallest singular value of V Tocaa 0).
Based above corollaries and assumptions, derive the following theorem.
Theorem E.7. We obtain the error between the actual influence and our predicted influence follows:
2 CICTTD''RC 2(0 +5)3
28 m o'ni
+
(6 + )'min)) (8 + mmin)
22

---

Published as a conference paper at ICLR 2026
Proof. We will use the one-step Newton approximation an intermediate step. Define A0N((-D*)
as
A0 - H VDL Seg: 8),
where H3 6- + V (D", Seg: f) is the regularized empirical Hessian at / but reweighed after removing the influence of D Then the one-step Newton approximation for 0(-D") is defined as @Ne(-D" A0xi(-D") + 0.
In the following, we will separate the error between (-D") and ((-D") into the following two parts:
0(-D") - (ss(-D") = 0(--D) - (Ne(-D") + (Oxc(-D*) - 0) (-)((-D') - 0
Erra, ani (D))
Erro, ir( (-D-)
Firstly, in Step 1, we will derive the bound for Newton-actual error Erryt aa(- Since (0)
is strongly convex with parameter d'in + 5 and minimized by 0(-D*), we can bound the distance ((--D)) D*) - ONz( Oxe(-Dr))
|| in terms of the norm of the gradient at ONz:
((-D") (10))O0vii Oxi(-Do) S d'in + VOL. (Oxe(-D'))l
2
(15)
Therefore, the problem reduces to bounding V Dw)((-D"))l2> Noting that Vo =
-VeL This is because á minimizes L L', that is,
VOL (0) VoL (0) 0.
Recall that AdN. H₁¹ VeL'(D', Seg: 0) -H conditions, we can have this bound for EIT an (-D*).
VeL (D*, Seg; 0). Given the above
VoL (oNc(-D'))l
VOL (0+A0N((-D)))
100,((-') // V (0+10N((-D)) VoL (0) ViL. 10).
1/'' (VEL- (0+1. (0+1.A0N(-- A0N((-D')
- - (0) ADNel- DJ) dl
(16)
C
CH 2 1
$ 2 D')12=
VOL VOL-(0) VoL (0)
CH
CH
VoL (0]
VOL'(0)
20 8)2
2 mmin + 8)2
DD 12C
2(0 min 00)
Now we come to Step 2 to bound EIIN. ir( (DD"), and we will bound the difference in parameter change between Newton and our ECIF method.
(ONe(-D') -6) - (0=r(-D'')-)
VBL (0)) + +0..1+ VoLaaal
())
VOL'(D", Seg:
For simplification, we use matrix A, B for the following substitutions:
A = 8 1+ VOL-
B 8- V Lecal
Local
8
23

---

Published as a conference paper at ICLR 2026
And A and B are positive definite matrices with the following properties
+ min < A < + m
+ Mmin
- B < max
Therefore, we have
ONN( (Ni((-D') -0) -
(0.1(-D") -0)--(()(-DD)) -0)
(A- + B) Vol. (D", Seg; 0
< A +B=- [lll||/ol VoL (D" Seg: 0)
(17)
25 + m + J'nn
VoL (D", Seg; 0)
(8 + m'in) (6 + miin)
28 + min + T'in
C2
(8 + mmin)
+ mmin)
By combining the conclusions from Step and Step II in Equations 15 16 and 17 between the actual influence and our predicted influence as follows:
we obtain the error
((-D") (-)(-
C CH +
25 + min + d'nin
C.
2( min + 6)3
(6 (5+o')))) (6 mmin)
It is notable that such error bound is small when the number of removal samples [D*] is fixed as in practice 5 O(
APPLICATIONS OF ECIF
F.1 TASK-RELATED IS
Property F.1. Considering a specific set D'' with text and image embeddings U and V', and dataset Dr to removed, then we have
LBatch (U', V'; 0 (0-0(()))))))))))) D*)) LBaach (U', V) LB (U', (V),V")))((((-D")) -0) L (U", V¹;0)² (positive-IF(D" Seg; 0) negative-IF(("" Seg: (18)
where 0(-D²) is the optimal model for the loss eliminating D*, positive-If(D";,Se 0) and negative-IF(D Seg; 0) are obtained from from Proposition 4.3 for D*.
Proof.
dL (U", V'; 0e.
dL (U', V"; 0c=0.c)
"S''','": Seg) A
dC
c=0
1==
LBachh (U', V','0'" (positive-IF(D"., Seg; 0) negative-IF(D"" SSg 0))
F.2 RELATIVE INFLUENCE SCORE
Proposition F.2. Define I [positive-IF(at (eaat 0)). If the matrix IT is
irreversible, then the optimization problem
arg max max LBathh (U', V²: θ A0).(()) LBacc (U', (U'VV';')) V² s.t. [ sp (19) DED of
24

---

Published as a conference paper at ICLR 2026
is equivalent
arg max [egaaive-IF(;;; 0)1|2" FED
V Leach ,U',V";0) O)" negative-If((;0]]
Else, I is reversible, then the initial problem is equivalent
V LBBacs(UU 0110)12"
V Baath (U", V O) 1 [I 1) 111 -1 LBacch (U' (U', v)
arg max
Proof. From [4). have
Banch (U", v + (bc,(x)) - Luach (U (')
(20)
VI(UU", ("((UUVV));0) _0,x((x))
(21)
VL((U", 0)) (e Eo-psiiive--F((a 0));0) (S negative-IF((tt x );00)
(22)
And still from 14 the constraint in parameter changes can be written as
Ad (z))
(23)
=|le positive-IF((z" xx )) + (S - 1) negative-IF((e (F(l"" ,x') :)); S
(24)
We can regard
22 as the inner product between vector VI((U", 0) and vector A
positive-IF + 1) negative-IF.
positive-IF is not parallel negative-IF, then the constraint in equation 233 becomes that vector is chosen from a ball of radius Otherwise, the constraint is equivalent to a constraint on the norm of vector that is parallel positive-II negative-IF. Therefore, we will proceed with classification discussion based on whether positive-IF and IF are parallel.
Firstly, we consider the I case. As is well known, the inner product of vectors reaches its extreme when the two vectors are parallel. We can choose and Ç freely to make vectors || u. Assume that there exists E IR s.I.
[ c 5 = VL((U','';))
Denote [positive-IF, negative-IF]
[positive-If, negative-
VL(UU, v');;)
( (-1
positive-IF" negative-II"
posstive-IF
[positive-IF, negative-
E
VI((U', v');0)
( (-
IegattiveIIF"
IT.
c
IT. VLL(UU VV)
(-
6
[TT. 1 IT. VL(UU,VV)'
(-1
Noting that IT I invertible matrix long as positive-IF negative-IF are not parallel. Considering the constraints of the length of vector then
||e VL(UU, V'; <p
We can make vector v reach its largest norm with setting e an appropriate number:
[VL((U", V):
Finally, obtain the expression of vector 2 that maximizes expression
20
Dosiitie-IFF,negaiive-I1 ( - = =c [TT
ST. (T(UU',V')
25

---

Published as a conference paper at ICLR 2026
Then we have
L((U",V") VV): ,x")) I((U", v'):
VL((U' ()): (poositive-IF, negative-IF]
5 (--1)
=c. VI((U', O)) 7. - 11 ST. VI((U' v')'
where = [positive-IF, negative--1
max VL((U', V=):)" [T 17- 1T. VI((U",V'):))
arg
(x JED
where I positive-IF((=)) x²) f), negative-IF((1t (== =²):
If positive-IF, negative-IF are not parallel, the optimization problem in form 19 is equivalent to (")'"""""' r] ("Vi((U" V))
(VVL(("VV')))))
arg max VL((U", VV); 0)l
Because p independent of data, can drop it and write the above equation as
L V') 1 (TVL(UU,
arg max FED
Then, we come to the second case where positive-IF negative-IF.
We can define a
(25)
|le positive-IF((= ,x();0) + -(-- 1) negative-IF((z" )
(26)
[|a((e, positive-IF((1t) ,x);0)) S
(27)
And the constraint is imposed on a by
alE,
p |positive-IF((x" xx));
Therefore, equation 20 is equivalent to
VL((U' V') (a(c,)) positive-IF((z?) x )²);;)7 ):
max e.c
max ((e, VI.((U', V)): (positive-IF(("" xx)): ):0}
<<
P
VL((U", (") positive-IF((a" =x);;))
P
"L((U",V'); negative-IF((x* 2²));)
|negative-IF((a""__) x;) 0|
Because p independent of data, can drop it and write the above equation
|positive-IF(aa* xx 0))
VL((U', VL((U',V');)) positive-IF((=?) (==.======0) =²));))
|nngative-IF(2> x(); 0)| VL((U', 0)) nnegativ-IFF(("" 00);0)
G ADDITIONAL EXPERIMENTAL RESULTS
26

---

Published as a conference paper at ICLR 2026
G.1 DETAILS OF EXPERIMENT SETTINGS
Datasets. We employ four datasets for our utility and efficiency evaluation tasks, FGVC-Aireraft dataset (Maji et al., 2013), Food10/ dataser (Bossard et al. 2014). Flowers102 dataset Nillsback Zisserman| 2008), and Cifar-100 dataset (Krizhevsky 2009). For the misprediction traceback experiments, we focus on the first three datasets. The FGVC-Aircral dataset comprises 10,000 images of airplanes, each annotated with the model and bounding box the dominant aircraft depicted. The Food-101 dataset, publicly available for food image recognition, includes 101 food categories, with each category containing 1,000 images. The images feature food photographs captured from various angles and under different lighting conditions. The Flowers- 102 dataset consists of 102 classes of flowers native to the United Kingdom, with each class containing between 40 and 258 images. We use Cifar-10 10 dataset (Krizhevsky al. 2009) for the misalignment detection tasks.
Implementation Details. Our experiments utilized a Nvidia V100-32G GPU and 10 CPU cores with 64 GB memory. For all the following tasks, we employ the CLIP model 'ViT-B/16' and use LoRA few-shot learning.
For data influence removal, when our method is tested on random sample removal task, 10% samples are randomly removed. For valuable (harmfull samples, we remove 10% of the valuable (harmful) data identified by ECIF. Each removal repeated 3 times with different seeds.
For the experiment of Identifying influential data for fine-tuming, we first calculate the task-related IS for every individual sample and collect valuable data with positive IS, then choose to remove 30% of these. We conducted the experiments 3 times for each removal with different seeds. The experiment setting for harmful data removal is similar. contrast, we select harmful data with negative IS. The experiments are carried out on the Food101, Flowers 102, FGrC-Aireraf,, and DTD datasets, and the remove ratio ranges from 0 to 90%. For each removal, we conducted the experiments 3 times with different seeds.
The multiple samples removal experiments are conducted on F0od101, Flowers FGVC-Aireraft, and DTD datasets, with removal ratios from 1% to 7%, respectively.
For the misprediction trace back task, we conduct experiments on the Food101, Flowers 102, FGVC- Aircraft, and DTD datasets. We first choose mispredicted test sample the target in algorithm 3. then calculate the relative IS for each individual sample in the training dataset. Noting that the relative IS is always positive. We visualize training samples with top-10 relative IS.
For the misalignment detection tasks, Cifar- 10 and imagenette (smaller version of imageNet) datasets used. We also applied standard data augmentation techniques the training set.i.e., random cropping and random flipping. The model is optimized with Adam with weight decay (5e-1), and B set to 0.9. A dropout ratio of 0.25 is used. The training iterations are set to 30, with a learning rate of 2e-4 and batch size of 16. The rank of the low-rank matrices of LoRA is set to 2. We trained the model on a poisoned version of the dataset (20% 30% of the data samples are mislabeled). Then, we compute the influence score IS of all the training samples on the mispredicted test samples. In the end, we visualize the training samples that have the highest positive IS score.
G.2 EXTENDING HARMFUL DATA REMOVAL TO
NOISY DATASET
We use ANIMAL-ION dataset (Song et al. 2019) for our harmful data removal tasks. It's a real world noisy dataset, containing five pairs of "confusing" animals: (cat, lynx), (jaguar, cheetah), (wolf, coyote), (chimpanzee, orangutan), (hamster, guinea pig), where two animals each pair look very similar. Overall, the proportion of incorrect labels was 6.44%.
The harmful removal task on ANIMAL--1ON is presented in table We observe that when portion of harmful data is removed, the ECIF method significantly outperforms random removal, particularly when the removal proportion is small. Specifically, when less than 40% of the data is removed, ECIF achieves an accuracy improvement of over 10% compared to random removal. This demonstrates the capability of ECIF to accurately identify harmful samples, thereby substantially enhancing the model's 's performance.
To provide a method as the reference, we adopt CLIPScore (Hessel et al. 2021). a basic data evaluation method, as the baseline for MLLM. This method is model-independent and is limited to
27

---

Published as a conference paper at ICLR 2026
Fandee
Accuracy
24
Removal Ratio
Figure 3: Harmful Data Removal on ANIMAL-ION
evaluating data quality rather than assessing the contribution of the data to the model. In the task
(a) Harmful Data Removal
Remevall Ratio
(b) Valuable Data Removal
Figure 4: Comparison between different methods for data removal on Food101
of harmful data removal, the ECIF method demonstrates significantly better performance compared to the CLIP Score. For valuable data removal, ECIF performs slightly better than the CLIP Score. This superiority is primarily attributed to ECIF's ability attribute data based the relationship between the model and the data, whereas CLIP Score is solely used to evaluate data quality without considering the model's involvement.
G.3 EVALUATING MULTIPLE SAMPLES
To comprehensively evaluate the data removal capabilities of ECIF in various scenarios, we conducted experiments on the performance when multiple samples need to be removed. Specifically, we consider the different ratios of samples (1-7%) for removal. As shown in Figure 5 we see that the difference in precision between these two methods is very small (less than 1.5%) in most cases, except for the case of 2% for Food101. These results show the utility of ECIF compared the ground truth. Note
28

---

Published as a conference paper at ICLR 2026
that in Table 2 we have shown that the speed of ECIF is more than twice that of retraining. Thus, ECIF is an editing method that achieves a trade-off between speed and effectiveness.
022
031
0.2
0.55
5F7
2300
2.22
7.00
2.78
2:22
0.32
1.85
0.30
Figure 5: Impact of Remove Ratio on Food101, DTD and Flower102 datasets.
G.4 ADDITIONAL VISUALIZATION OF MISPREDICTION TRACE BACK
We demonstrate our additional visualization results of the mispredicted data tracing in Table 4 and Figure
10
G.5 ADDITIONAL. VISUALIZATION OF MISALIGNMENT DATA DETECTION
We demonstrate our additional results of the Visualization of the misalignment data detection in Figure 6
truck
truck
airplane
dog
deer
truck
deer
automoblle
cat
frog
Figure 6: Top-10 10 misaligned sample pairs in the 20% mislabeled training data.
G.6 LIMITATION
Our proposed data value evaluation method assesses the importance data point by performing leave-one-out retraining and observing changes in model performance. We use ECIF to efficiently estimate this performance change, thus enabling rapid data value assessment. Essentially, the ECIF approach approximates the results that would be obtained by actual leave-one-out retraining. However, experimental results demonstrate that this approximation is highly effective in real-world applications.
G.7 EFFECT OF REMOVAL RATIO ON DATA FILTERING PERFORMANCE
We evaluated the effectiveness of our method through systematic data removal experiments on the Animal dataset. As illustrated Figure (I))), the harmful data removal setting, ECIF demonstrates stable performance improvement and consistently outperforms baseline methods (such as CLIP Score, D2 Pruning and Image Filter) as the removal ratio increases. This suggests that ECIF effectively isolates detrimental samples. Conversely, the useful data removal experiment in Figure 11bb) serves as a strong counter- counter-validationn validation. When the data deemed most valuable by ECIF is removed, the model exhibits the most significant accuracy drop compared to other methods.
29

---

Published as a conference paper at ICLR 2026
Table 4: Top-10 related test data tracing of mispredicted data on cifar- 10 dataset with 10% noise data.
30

---

Published as a conference paper at ICLR 2026
Table 5: Top-10 related test data tracing of mispredicted data on cifar- 10 dataset with 20% noise data.
31

---

Published as a conference paper at ICLR 2026
lawr
Iaw
Table 6: Top-10 related test data tracing of mispredicted data on cifar- 10 dataset with 30% noise data.
32

---

Published as a conference paper at ICLR 2026
atplaae
astormblie
dogg
arp
Figure 7: Visualization results for misalignment detection. 30% of the training samples were mislabeled. The figure shows the training samples that have the top-10 10 highest IS scores on the cifar-10 10 test set.
11123
KI/D
Figure 8: Top-10 10 related test data tracing of mispredicted data FGVC-Aircraft with 30% noise data.
This sharp performance degradation confirms that ECIF accurately locates the critical samples that contribute most to the model's training.
G.8 VISUAL COMPARISON OF SAMPLE SELECTION: ECIF vs. CLIP SCORE
To intuitively understand the selection mechanism, we visualized the most valuable samples ranked by ECIF and CLIP Score for specific target images. As shown in Figure 12, the comparison reveals distinct behaviors in handling the inherent noise of the Animal1ON dataset. The CLIP Score tends to rely superficial visual traits, frequently retrieving "hard negatives" such selecting Coyote for a Wolf target-the that look similar but belong to different classes. contrast, ECIF effectively filters out these deceptive lookalikes and identifies semantically consistent samples. This suggests that method moves beyond mere visual alignment to capture the true data value, avoiding the confusion that often arises from the high inter-class similarity in real-world noisy data.
33

---

Published as a conference paper at ICLR 2026
Figure Top-10 10 related test data tracing of mispredicted data on Food-101 with 30% noise data.
Heder
Figure 10: Top-10 10 related test data tracing of mispredicted data on Flowers- with 30% noise data.
34

---

Published as a conference paper at ICLR 2026
Harmful Removal
Removal
CUP Fitter Pruring
CLP Score
87.5
Image filer
Accuracy (%0
07.0
ECF
Accuracy (%)
86.5
ECI
D2 hruning
86.5-
86.0
86.0-
85.5-
85.0
15 Remove Ratio (96
Remove Ratio (%6)
(a) Harmful Data Removal
(b) Useful Data Removal
Figure 11: Impact of different removal ratios on model performance.
35

---

Published as a conference paper at ICLR 2026
Sooved
handter
Figure 12: Visual comparison of selected most valuable samples using different metrics.
36


Reviews:
[{"review_id": "Tf9yPia9ab", "reviewer": "Reviewer_EnRJ", "summary": "This paper addresses the performance degradation problem in contrastive learning caused by misaligned data. As classical influence functions cannot handle contrastive loss, this paper proposes ECIF, the first no-retraining data valuation tool designed specifically for such models. ECIF's core contribution lies in its dual-perspective analysis: it decouples a single data point's influence into its contribution as a positive sample and its contribution as a negative sample. This dual-evaluation mechanism enables ECIF to efficiently and accurately identify harmful, misaligned data and trace model errors, which is something traditional IF methods cannot do.", "questions": "1. The theoretical error bound (Theorem E.7) relies on the assumption of convexity (Assumption E.3), but the contrastive loss is non-convex. Could you clarify why ECIF remains an accurate approximation in this practical, non-convex setting?\n\n2. Proposition 5.3 appears to contain several typographical errors, making it difficult to understand. Could you please provide the correct mathematical formulation for this proposition?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper addresses an extremely important and urgent problem in multimodal and contrastive learning how to efficiently detect and evaluate misaligned training data. This is the first no-retraining influence function designed specifically for the contrastive loss function.\n\n2. It cleverly decouples the influence of a single data point into positive influence Positive-IF and negative influence Negative-IF, which is technically sound and directly confronts the fundamental challenge of why standard IF cannot handle contrastive loss.\n\n3. The theory and experiments are well combined. The empirical results decisively validate the paper's core theoretical claim by showing that ECIF succeeds where all standard influence function baselines fail.", "weaknesses": "The paper's motivation is to develop a computationally efficient method applicable to large-scale models, but it lacks a clear analysis of the actual computational cost of ECIF. The paper omits runtime comparisons with baseline methods that it outperforms in accuracy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f84fJXLUXT", "reviewer": "Reviewer_8Zsd", "summary": "This paper introduces ECIF (Extended Influence Function for Contrastive Loss), a method for quantifying data attribution in contrastive learning models like CLIP. The authors address the challenge that classical influence functions, designed for pointwise loss, cannot directly apply to contrastive loss which involves both positive and negative sample pairs. ECIF separately quantifies the influence of data points as both positive samples (matched pairs) and negative samples (non-matched pairs), providing closed-form approximations without requiring model retraining. The method is applied to three tasks: identifying influential data for fine-tuning, misprediction trace-back, and misalignment detection. Experiments on several vision datasets (FGVC-Aircraft, Food101, Flowers102, CIFAR-10/100) demonstrate that ECIF can approximate retraining results with 80-90% computational savings while effectively identifying harmful and valuable training samples.", "questions": "Mentioned in weaknesses.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Novel technical formulation: The dual-perspective approach (positive-IF and negative-IF in Definition 4.3) is a genuine technical contribution. The mathematical derivation for handling negative samples through the Taylor expansion approach (Section 4.2, Equations 3-4) is creative and appears technically sound.\n\nComputational efficiency demonstrated: Table 2 shows concrete evidence of 2-3x speedup over retraining (e.g., 456s vs 1174s on FGVC-Aircraft) while maintaining comparable accuracy (22.77% vs 23.07%), which validates the practical utility of the approximation.\n\nComprehensive theoretical analysis: Theorem E.7 provides an error bound for the approximation, showing the error scales with O(|D*|²), which gives theoretical backing to when the method should be reliable.", "weaknesses": "Missing critical baseline comparisons: The paper completely ignores the substantial literature on data selection for contrastive learning and CLIP models. Works like DataComp (Gadre et al., 2023), LAION filtering methods, and quality-based data selection approaches (e.g., CLIP-score based filtering as briefly mentioned in G.2) have implicitly solved data valuation for contrastive learning at scale. The comparison to CLIP-score in Figure 4 is relegated to the appendix and shows only marginal improvements, raising questions about practical utility. The paper needs head-to-head comparison with:\n- Quality-based filtering methods (aesthetic scores, CLIP-score thresholds)\n- Diversity-based sampling methods\n- Curriculum learning approaches for contrastive learning\n- Recent data attribution methods like DataInf (Kwon et al., 2023) which the authors cite but don't compare against\n\nWeak motivation for the problem: The introduction (lines 33-43) claims that \"robust evaluation mechanisms for data quality remain lacking\" but doesn't acknowledge that the CLIP training community has extensively tackled data quality through heuristic methods with proven success at billion-sample scale. Why is influence function-based attribution necessary when simpler methods work? The paper doesn't establish that existing approaches fail or are insufficient.\n\nLimited evaluation framework: The experimental evaluation primarily tests whether ECIF can identify \"random,\" \"harmful,\" or \"valuable\" samples (Section 6.2-6.3). However:\n- These categories are themselves defined using ECIF's task-related influence scores, creating circularity\n- Any reasonable data selection method can be evaluated in this same framework by ranking samples and binning them into these categories. The paper doesn't show ECIF provides qualitatively different insights than simpler metrics\n\nScalability concerns not addressed: \n- All experiments use relatively small datasets (Food101: 101 classes x 1000 images; FGVC-Aircraft: 10K images). Modern contrastive learning operates at the scale of millions/billions of samples (LAION-5B, DataComp-1B).\n-  The computational requirements of computing Hessian-related quantities (Equation 2, Algorithm 1 line 8) scale poorly. The paper acknowledges using LOGRA for efficiency (Appendix C) but provides no wall-clock time comparisons or scalability analysis.\n\nExperimental methodology issues:\n- Baseline methods (IF-EKFAC, TARK, TracIN) in Table 1 show suspiciously poor performance (e.g., 18.27% vs 23.50% for TARK on FGVC-Aircraft), suggesting potential implementation issues or unfair comparison\n- The \"harmful\" data removal experiments (Figure 1a, Table 1) artificially create noise by mislabeling. Real-world data quality issues are more subtle\n- No comparison of the types of samples identified as valuable/harmful between ECIF and simpler methods to show qualitative differences\n\n\nMissing ablation studies:\n- No ablation on the relative importance of positive-IF vs negative-IF components\n- No sensitivity analysis on hyperparameters (regularization δ, projection rank in LOGRA)\n- No analysis of when the method fails or performs poorly", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sFcljnPFUl", "reviewer": "Reviewer_VHVq", "summary": "The paper proposes ECIF (Extended Influence Function for Contrastive Loss), an influence-function framework tailored to contrastive learning that decomposes the contribution of a training pair into its roles as both positive and negative samples. Formally, the paper defines the batch contrastive loss  and derives closed-form, Hessian-vector–based approximations for the “positive-IF” and “negative-IF” , then combines them to estimate parameter changes from data removal without retraining. ECIF is reported to approximate retraining accuracy closely while cutting runtime.", "questions": "Please see weaknesses. If some of the issues can be resolved, I will raise my score. Currently, I'm leaning towards 5.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is novel and derives an explicit negative-sample term via softmax reweighting, improving attribution fidelity for contrastive losses. Provides closed-form positive/negative IF and ECIF decomposition.\n\n\nAnalyzes ECIF approximation under convexity, and discusses computation efficiency.", "weaknesses": "(1) The linear superposition of positive/negative influences is assumed but its validity for non-convex deep models is not justified.\n\n(2) Experiments are restricted to models with LoRA, impacts for larger backbones or full parameters finetuning remain unknown.\n\n(3) ECIF combines positive-IF and negative-IF, but experiments do not isolate their separate contributions to performance.\n\n(4) Lack large-scale benchmarks(e.g. ImageNet), constraining claims of scalability to large multimodal datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2UFX84DREB", "reviewer": "Reviewer_EW3H", "summary": "This paper proposes ECIF, which adapts influence functions to contrastive learning by modeling both positive and negative sample roles. ECIF enables efficient data valuation, misprediction trace-back, and misalignment detection without retraining. Experiments demonstrate several applications of the method, and show it achieves similar accuracy to retraining with lower cost, outperforming existing data attribution methods.", "questions": "Most experiments focus on identifying harmful or low-quality data. I wonder how well ECIF performs in finding the most valuable samples, when compared against other baselines.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a clear and well-motivated extension of influence functions to contrastive learning. The idea is novel and fills a real gap in the literature.\n\n- The paper demonstrates several useful applications (data valuation, misalignment detection, and misprediction trace-back). I see potential for this method to have real practical implications.\n\n- Experiments are reasonably thorough and show that ECIF can approximate retraining closely while being faster.\n\n- The empirical results also indicate that ECIF performs substantially better than previous baselines, which supports the main claim that existing methods are not directly suitable for contrastive objectives.", "weaknesses": "- I didn’t find an analysis for Figure 1(b). Looking at the plot, I think the expected pattern should be that ECIF results in lower accuracy than random in Figure 1(b), yet this pattern is not consistently observed across the ratios.\n\n- The paper would benefit from including quantitative results (e.g., precision, recall, AUROC, etc.) to show how well the method identifies mislabeled training samples across different noise ratios and random seeds.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)