SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Notation For a square matrix  $\mathbf{A} \in \mathbb{R}^{m \times m}$ , and  $\alpha, \beta \subseteq \{1, 2, \dots, m\}$ ,  $\mathbf{A}[\alpha, \beta]$  is defined to be the sub-matrix of  $\mathbf{A}$  by only keeping the rows and columns indexed by  $\alpha$  and  $\beta$ , respectively. Plus,  $\mathbf{A}[\alpha]$  is abused for  $\mathbf{A}[\alpha, \alpha]$ .
Lemma 4. Suppose a positive definite matrix  $\mathbf{M} \in \mathbb{R}^{m \times m}$  is given, for every  $\alpha \subseteq \{1,2,\dots,m\}$ , there is
$$
\det  (\mathbf {M} ^ {- 1} [ \alpha^ {c} ]) = \frac {\det  (\mathbf {M} [ \alpha ])}{\det  (\mathbf {M})}.
$$
The convention is  $\operatorname{det}(\mathbf{M}[\emptyset]) = 1$ .
Proof. Without loss of generality, we write  $\mathbf{M} = \begin{pmatrix} \mathbf{A} & \mathbf{B} \\ \mathbf{B}^\top & \mathbf{C} \end{pmatrix}$  such that  $\mathbf{M}[\alpha] = \mathbf{C}$  and  $\mathbf{M}[\alpha^c] = \mathbf{A}$ . By Lemma 2,  $\operatorname*{det}(\mathbf{M}) = \operatorname*{det}(\mathbf{S})\operatorname*{det}(\mathbf{C})$  where  $\mathbf{S} = \mathbf{A} - \mathbf{BC}^{-1}\mathbf{B}^\top$ . On the other hand,
$$
\mathbf {M} ^ {- 1} = \left( \begin{array}{c c} \mathbf {S} ^ {- 1} & - \mathbf {S} ^ {- 1} \mathbf {B} \mathbf {C} ^ {- 1} \\ - \mathbf {C} ^ {- 1} \mathbf {B} ^ {\top} \mathbf {S} ^ {- 1} & \mathbf {C} ^ {- 1} + \mathbf {C} ^ {- 1} \mathbf {B} ^ {\top} \mathbf {S} ^ {- 1} \mathbf {B} \mathbf {C} ^ {- 1}. \end{array} \right)
$$
$$
\frac {\operatorname * {d e t} (\mathbf {M} [ \alpha ])}{\operatorname * {d e t} (\mathbf {M})} = \frac {\operatorname * {d e t} (\mathbf {C})}{\operatorname * {d e t} (\mathbf {M})} = \operatorname * {d e t} (\mathbf {S} ^ {- 1}) = \operatorname * {d e t} (\mathbf {M} ^ {- 1} [ \alpha^ {c} ]).
$$
Lemma 5. For every positive definite matrix  $\mathbf{M}_{m\times m}\in S_{++}$ , and every  $\alpha ,\beta \subseteq \{1,2,\ldots m\}$
$$
\det  (\mathbf {M} [ \alpha \cup \beta ]) \det  (\mathbf {M} [ \alpha \cap \beta ]) \leq \det  (\mathbf {M} [ \alpha ]) \det  (\mathbf {M} [ \beta ])
$$
The convention here is  $\operatorname{det}(\mathbf{M}[\emptyset]) = 1$ . Moreover, provided that  $(\alpha \cup \beta) \backslash \alpha \neq \emptyset$  and  $(\alpha \cup \beta) \backslash \beta \neq \emptyset$ , and w.l.o.g., write
$$
\mathbf {M} = \left( \begin{array}{c c c} \mathbf {A} & \mathbf {B} & \mathbf {C} \\ \mathbf {B} ^ {\top} & \mathbf {D} & \mathbf {E} \\ \mathbf {C} ^ {\top} & \mathbf {E} ^ {\top} & \mathbf {F} \end{array} \right) \tag {38}
$$
such that  $\mathbf{M}[\alpha] = \begin{pmatrix} \mathbf{A} & \mathbf{B} \\ \mathbf{B}^\top & \mathbf{D} \end{pmatrix}$  and  $\mathbf{M}[\beta] = \begin{pmatrix} \mathbf{D} & \mathbf{E} \\ \mathbf{E}^\top & \mathbf{F} \end{pmatrix}$ . A necessary condition to have the equality is that each column of  $\mathbf{C}$  lies in  $\operatorname{span}(\mathbf{B})$ .
Proof. Our proof is adapted from (Horn and Johnson 2012, Theorem 7.8.9), by which we add a necessary condition for the equality.
By Lemmas 4 and 3, there is
$$
\begin{array}{l} \frac {\det  (\mathbf {M} [ \alpha \cap \beta ])}{\det  (\mathbf {M} [ \alpha \cup \beta ])} = \det  (\mathbf {M} ^ {- 1} [ ((\alpha \cup \beta) \backslash \alpha) \cup ((\alpha \cup \beta) \backslash \beta) ]) \\ \leq \det  (\mathbf {M} ^ {- 1} [ (\alpha \cup \beta) \backslash \alpha ]) \det  (\mathbf {M} ^ {- 1} [ (\alpha \cup \beta) \backslash \beta ]) \\ = \frac {\operatorname * {d e t} (\mathbf {M} [ \alpha ])}{\operatorname * {d e t} (\mathbf {M} [ \alpha \cup \beta ])} \cdot \frac {\operatorname * {d e t} (\mathbf {M} [ \beta ])}{\operatorname * {d e t} (\mathbf {M} [ \alpha \cup \beta ])}. \\ \end{array}
$$
Suppose  $(\alpha \cup \beta) \backslash \alpha \neq \emptyset$  and  $(\alpha \cup \beta) \backslash \beta \neq \emptyset$ , by Lemma 3, the equality holds if and only if  $\mathbf{M}^{-1}[(\alpha \cup \beta) \backslash \beta, (\alpha \cup \beta) \backslash \alpha] = \mathbf{0}$ . Using Eq. (38), one can verify that
$$
\mathbf {M} ^ {- 1} \left[ (\alpha \cup \beta) \backslash \beta , (\alpha \cup \beta) \backslash \alpha \right] = \left(\mathbf {A} ^ {- 1} \mathbf {B} \mathbf {G} ^ {- 1} \mathbf {H} - \mathbf {A} ^ {- 1} \mathbf {C}\right) \left(\mathbf {J} - \mathbf {H} ^ {\top} \mathbf {G} ^ {- 1} \mathbf {H}\right) ^ {- 1}
$$
$$
w h e r e \mathbf {G} = \mathbf {D} - \mathbf {B} ^ {\top} \mathbf {A} ^ {- 1} \mathbf {B},
$$
$$
\mathbf {H} = \mathbf {E} - \mathbf {B} ^ {\top} \mathbf {A} ^ {- 1} \mathbf {C},
$$
$$
\mathbf {J} = \mathbf {F} - \mathbf {C} ^ {\top} \mathbf {A} ^ {- 1} \mathbf {C},
$$
and thus the equality holds if and only if  $\mathbf{C} = \mathbf{B}\mathbf{G}^{-1}\mathbf{H}$
Lemma 6. Given a continuous non-negative function  $g:[0,1]\to \mathbb{R}_+$  that achieves its minimum 0 only at one point  $w^{*}\in [0,1]$ ,  $\delta_{w^*}$  is the unique probability measure on  $[0,1]$  that minimizes
$$
\operatorname *{argmin}_{P\in \mathcal{P}}L(P):= \int_{0}^{1}g(t)\mathrm{d}P(t).
$$
Proof. It is clear that  $L(\delta_{w^*}) = 0 = \inf_{P \in \mathcal{P}} L(P)$ . Suffice it to prove that  $L(P) = 0$  implies  $P = \delta_{w^*}$ .
Suppose  $P \in \mathcal{P}$  is given such that  $L(P) = 0$ . A point  $w \in [0,1]$  is said to be irrelevant w.r.t.  $P$  if there exists  $\epsilon > 0$ ,  $P\left((w - \epsilon, w + \epsilon) \cap [0,1]\right) = 0$ . For the sake of contradiction, suppose a distinct point  $w \in [0,1]$  ( $w \neq w^*$ ) is relevant. Since  $g(w) > g(w^*) = 0$  and  $g$  is continuous, there exists  $\epsilon > 0$  such that  $g(s) > 0$  for all  $s \in [w - \epsilon, w + \epsilon] \cap [0,1]$ . Since  $m = \inf_{s \in [w - \epsilon, w + \epsilon] \cap [0,1]} g(s) > 0$  and  $w$  is relevant, we have
$$
L (P) \geq \int_ {[ w - \epsilon , w + \epsilon ] \cap [ 0, 1 ]} m \mathrm {d} P (t) = m \cdot P ([ w - \epsilon , w + \epsilon ] \cap [ 0, 1 ]) > 0,
$$
a contradiction. So the only possible relevant point is  $w^{*}$ .
Next, we prove that  $P = \delta_{w^*}$ . For each  $\delta > 0$ , construct  $S_{\delta} = [0, w^* - \delta] \cup [w^* + \delta, 1]$ . For each  $w \in S_{\delta}$ , since  $w$  is irrelevant, there exists  $\epsilon(w) > 0$  such that  $P(O_w) = 0$  where  $O_w = (w - \epsilon(w), w + \epsilon(w)) \cap [0, 1]$ . Then,  $\{O_w : w \in S_{\delta}\}$  forms an open (w.r.t. [0, 1]) cover of  $S_{\delta}$ . Since  $S_{\delta}$  is compact, there is a finite subcover  $S_{\delta} \subseteq \bigcup_{1 \leq i \leq k} O_{w_i}$ . Therefore,  $P(S_{\delta}) \leq \sum_{i=1}^{k} P(O_{w_i}) = 0$ , which implies  $P(S_{\delta}) = 0$ . By letting  $\delta \to 0$ ,  $\bar{P}(S_{\delta}) \to P([0, 1] \setminus \{w^*\}) = 0$ . Using the  $\pi - \lambda$  theorem, e.g., (Durrett 2019, Theorem A.1.4), one can get  $P = \delta_{w^*}$ .
Lemma 7. Suppose  $\mathbf{V}_{[n],P} = \mathbf{M}_P^{[n]}\mathbf{C}^{[n]}(\mathbf{M}_P^{[n]})^\top$  where  $\mathbf{M}_P^{[n]}$  is defined in Proposition 7 and  $\mathbf{C}^{[n]} = \pmb{\Sigma} \otimes \pmb{\Sigma} \otimes \dots \otimes \pmb{\Sigma}$  ( $n$  repetitions), and assume  $\sigma_{12} < \min(\sigma_{11},\sigma_{22})$ . Then, the Dirac delta distribution  $\delta_{w^*}$  where  $w^* = \frac{\sigma_{11} - \sigma_{12}}{\sigma_{11} + \sigma_{22} - 2\sigma_{12}}$  makes the matrix  $\mathbf{V}_{[n],P}$  diagonal. In addition,  $\delta_{w^*}$  is the unique probability distribution on  $[0,1]$  that makes  $\mathbf{V}_{[n],P}$  diagonal if  $n > 1$ .
Proof. Since  $(\mathbf{A} \otimes \mathbf{B})(\mathbf{C} \otimes \mathbf{D}) = \mathbf{AC} \otimes \mathbf{BD}$ , according to Proposition 7, for each  $w \in [0,1]$
$$
\begin{array}{l} \mathbf {V} _ {[ n ], \delta_ {w}} = \mathbf {B} ^ {[ n ]} (w) \mathbf {C} ^ {[ n ]} (\mathbf {B} ^ {[ n ]} (w)) ^ {\top} = \overbrace {\mathbf {V} _ {[ 1 ] , \delta_ {w}} \otimes \mathbf {V} _ {[ 1 ] , \delta_ {w}} \otimes \cdots \otimes \mathbf {V} _ {[ 1 ] , \delta_ {w}}} ^ {n \text {r e p e t i t i o n s}} \\ \text {w h e r e} \mathbf {V} _ {[ 1 ], \delta_ {w}} = \left( \begin{array}{c c} p _ {1} (w) & p _ {2} (w) \\ p _ {2} (w) & C \end{array} \right), \\ p _ {1} (w) = \sigma_ {1 1} (1 - w) ^ {2} + \sigma_ {2 2} w ^ {2} - 2 \sigma_ {1 2} w (1 - w), \\ p _ {2} (w) = \left(\sigma_ {1 1} + \sigma_ {2 2} - 2 \sigma_ {1 2}\right) w + \sigma_ {1 2} - \sigma_ {1 1}, \\ C = \sigma_ {1 1} + \sigma_ {2 2} - 2 \sigma_ {1 2}. \\ \end{array}
$$
Using Eq. (13) (note  $2^{[n]}$  is ordered w.r.t. the binary ordering), by induction, one can show the  $(A,B)$ -th entry of  $\mathbf{V}_{[n],\delta_w}$  is  $p_1(w)^{|\lceil n\rceil -(A\cup B)|}p_2(w)^{|(A\setminus B)\cup(B\setminus A)|}C^{|A\cap B|}$  for every  $A,B\subseteq [n]$ . Therefore, for every off-diagonal entry of  $\mathbf{V}_{[n],\delta_w}$ , it must contain the term  $p_2(w)^k$  where  $k$  is some positive integer. Notice that  $p_2(w^*) = 0$ , and thus  $\mathbf{V}_{[n],\delta_{w^*}}$  is diagonal.
Conversely, suppose that a probability distribution  $P$  on  $[0,1]$  is given such that  $\mathbf{V}_{[n],P}$  is diagonal. Then,
$$
\mathbf {V} _ {[ n ], P} = \left(\int_ {0} ^ {1} \mathbf {B} ^ {[ n ]} (t) \mathrm {d} P (t)\right) \mathbf {C} ^ {[ n ]} \left(\int_ {0} ^ {1} \mathbf {B} ^ {[ n ]} (t) \mathrm {d} P (t)\right) ^ {\top}.
$$
Observe that  $\mathbf{V}_{[n],P}$  equals to  $\int_0^1\mathbf{B}^{[n]}(t)\mathbf{C}^{[n]}(\mathbf{B}^{[n]}(t))^{\top}\mathrm{d}P(t)$  along the last column as we can prove by induction that the last row of  $\mathbf{B}_n(t)$  contains only constants. Since  $\mathbf{B}^{[n]}(t)\mathbf{C}^{[n]}(\mathbf{B}^{[n]}(t))^{\top}$  can be computed recursively, one can show by induction that  $\mathbf{V}_{[n],P}(\emptyset ,[n]) = \int_0^1 p_2(t)^n\mathrm{d}P(t)$  and  $\mathbf{V}_{n,P}(\{1\} ,[n]) = \int_0^1 Cp_2(t)^{n - 1}\mathrm{d}P(t)$ .
Set  $g(w) = p_2(w)^n$  when  $n$  is even and  $g(w) = Cp_2(w)^{n - 1}$  otherwise. Note that  $g(w) \geq 0$  and attains its minimum 0 only at the point  $w^*$ . By Lemma 6, it implies  $P = \delta_{w^*}$ .
D Experiment Settings
The datasets we use in the main paper are summarized in Table 4. The logistic model used is
$$
\underset {\mathbf {w}, b} {\operatorname {a r g m i n}} \frac {1}{n} \sum_ {(\mathbf {x}, y) \in \mathcal {D} _ {t r}} \operatorname {C r o s s E n t r o p y} (\mathbf {w} \cdot \mathbf {x} + b, y)
$$
In the experiment of noisy label detection, the F1-score is computed by using
$$
\frac {2 \cdot | \{z \in \mathcal {D} _ {t r} \mid z \text {i s f l i p p e d a n d} z \text {i s d e t e c t e d n o i s y} \} |}{| \{z \in \mathcal {D} _ {t r} \mid z \text {i s f l i p p e d} \} | + | \{z \in \mathcal {D} _ {t r} \mid z \text {i s d e t e c t e d n o i s y} \} |},
$$
which is the harmonic mean of precision and recall.
D.1 Fitting Kronecker Noises
We independently sample 128 sequence  $\emptyset = S_0\subsetneq S_1\subsetneq \dots \subsetneq S_n = [n]$  in a way that  $|S_{i + 1}\backslash S_i| = 1$  for every  $0\leq i < n$ . For each subset  $S_{i}$ $(0\leq i\leq n)$  in a sampled sequence,  $v(S_{i})$  is computed using 128 random seeds. All these utility evaluations are then employed to generate an estimated covariance matrix  $\hat{D}\in \mathbb{R}^{(n + 1)\times (n + 1)}$ , which is an estimate for a sub-matrix of the ground-truth covariance matrix of the stochasticity. Note that under Assumption 1, for every  $0\leq i,j\leq n$
$$
\operatorname {C o v} (v _ {n} (S _ {i}), v _ {n} (S _ {j})) = \sigma_ {1 1} ^ {n - \max  (i, j)} \cdot \sigma_ {1 2} ^ {\max  (i, j) - \min  (i, j)} \cdot \sigma_ {2 2} ^ {\min  (i, j)}.
$$
D.2 Sampling Lift
Basically, sampling lift, which is designed for all semi-values, refers to any estimators based on
$$
\phi_ {i} (v; P) = \mathbb {E} _ {S} \left[ v _ {n} (S \cup i) - v _ {n} (S) \right] \text {f o r e v e r y} v _ {n} \in \mathcal {G} \text {a n d} i \in [ n ]
$$
where  $S$  is sampled from  $[n] \backslash i$  with  $P(S) = p_{s+1}^n$ .
There are two different ways to sample  $S$ . One is derived from Eq. (2). For example, to estimate the value for  $n \in [n]$ , the procedure is i) sample  $t \in [0,1]$  according to the underlying probability distribution  $P \in \mathcal{P}$ ; ii) let  $X$  be a Bernoulli random variable such that  $P(X = 1) = t$ , and sample a vector  $\mathbf{b} \in \mathbb{R}^{n-1}$  whose entries are independently following  $X$ ; and iii) define  $S \subseteq [n] \backslash n$  by letting  $i \in S$  if and only if  $b_i = 1$ . The other way is by noticing that
$$
\phi_{i}(v;P) = \sum_{k = 0}^{n - 1}q_{k + 1}\sum_{\substack{S\subseteq [n]\setminus i\\ |\overline{S} | = k}}\binom {n - 1}{k}^{-1}(v(S\cup i) - v(S)) = \mathbb{E}_{k}\mathbb{E}_{S:|S| = k}[v(S\cup i) - v(S)]].
$$
Table 4: A summary of datasets used in this work.
Dataset	Source
MNIST (LeCun et al. 1998)	PyTorch
FMNIST (Xiao et al. 2017)	PyTorch
2dplanes	https://www.openml.org/d/727
bank-marketing (Moro et al. 2011)	https://www.openml.org/d/1461
bioresponse	https://www.openml.org/d/4134
covertype	https://www.openml.org/d/150
cpu	https://www.openml.org/d/761
credit	https://www.openml.org/d/31
default (Yeh and Lien 2009)	https://www.openml.org/d/42477
diabetes	https://www.openml.org/d/37
fraud (Dal Pozzolo et al. 2014)	https://www.openml.org/d/42175
gas (Vergara et al. 2012)	https://www.openml.org/d/1476
har (Anguita et al. 2013)	https://www.openml.org/d/1478
iris	https://www.openml.org/d/61
letter (Frey and Slate 1991)	https://www.openml.org/d/6
optdigits	https://www.openml.org/d/28
pendigits	https://www.openml.org/d/32
phoneme	https://www.openml.org/d/1489
pol	https://www.openml.org/d/722
satimage	https://www.openml.org/d/182
segment	https://www.openml.org/d/36
spambase	https://www.openml.org/d/44
texture	https://www.openml.org/d/40499
wind	https://www.openml.org/d/847
where  $q_{k} = p_{k}^{n}\cdot \binom{n - 1}{k - 1}$  for every  $1\leq k\leq n$ . The corresponding sampling strategy is i) sample  $k\sim \{q_k\}_{1\leq k\leq n}$ , and then ii) sample  $S$  uniformly from  $2^{[n]\backslash i}$  subject to  $|S| = k - 1$ .
We comment that for the first way we do not have to calculate the distribution  $\{q_k\}_{1 \leq k \leq n}$ . Note the implementation for  $\{q_k\}_{1 \leq k \leq n}$  might suffer from numerical blowup induced by  $\binom{n-1}{k-1}$ , and it is why Wang and Jia (2023) did not report the results of the Beta Shapley values if  $n > 500$ .
E More Experiment Results
In this section, we provide detailed figures, see Figures 4 to 10, to supplement the results reported in Tables 3 and 2. In each figure,  $lr$  stands for the learning rate used for that dataset. In addition, we also run experiments while setting  $|\mathcal{D}_{tr}| = 200$ , 10 percent of which is flipped. The corresponding results are presented in Figures 11 to 17.
Figure 4: The first row exhibits results from the experiment of ranking consistency, while the second row is for the experiment of noisy label detection. Each result is reported using mean and standard deviation. For noisy label detection, the F1-score is reported over 10 random seeds. For ranking consistency, the Spearman's rank correlation coefficient is reported over all possible pairs of 10 random seeds, which is 45 in total.
Figure 5: The first row exhibits results from the experiment of ranking consistency, while the second row is for the experiment of noisy label detection. Each result is reported using mean and standard deviation. For noisy label detection, the F1-score is reported over 10 random seeds. For ranking consistency, the Spearman's rank correlation coefficient is reported over all possible pairs of 10 random seeds, which is 45 in total.
Figure 6: The first row exhibits results from the experiment of ranking consistency, while the second row is for the experiment of noisy label detection. Each result is reported using mean and standard deviation. For noisy label detection, the F1-score is reported over 10 random seeds. For ranking consistency, the Spearman's rank correlation coefficient is reported over all possible pairs of 10 random seeds, which is 45 in total.
Figure 7: The first row exhibits results from the experiment of ranking consistency, while the second row is for the experiment of noisy label detection. Each result is reported using mean and standard deviation. For noisy label detection, the F1-score is reported over 10 random seeds. For ranking consistency, the Spearman's rank correlation coefficient is reported over all possible pairs of 10 random seeds, which is 45 in total.
segment
Figure 8: The first row exhibits results from the experiment of ranking consistency, while the second row is for the experiment of noisy label detection. Each result is reported using mean and standard deviation. For noisy label detection, the F1-score is reported over 10 random seeds. For ranking consistency, the Spearman's rank correlation coefficient is reported over all possible pairs of 10 random seeds, which is 45 in total.
spambase
Figure 9: The first row exhibits results from the experiment of ranking consistency, while the second row is for the experiment of noisy label detection. Each result is reported using mean and standard deviation. For noisy label detection, the F1-score is reported over 10 random seeds. For ranking consistency, the Spearman's rank correlation coefficient is reported over all possible pairs of 10 random seeds, which is 45 in total.
FMNIST
Figure 10: The first row exhibits results from the experiment of ranking consistency, while the second row is for the experiment of noisy label detection. Each result is reported using mean and standard deviation. For noisy label detection, the F1-score is reported over 10 random seeds. For ranking consistency, the Spearman's rank correlation coefficient is reported over all possible pairs of 10 random seeds, which is 45 in total.
2dplanes
Figure 11: The first row exhibits results from the experiment of ranking consistency, while the second row is for the experiment of noisy label detection. Each result is reported using mean and standard deviation. For noisy label detection, the F1-score is reported over 10 random seeds. For ranking consistency, the Spearman's rank correlation coefficient is reported over all possible pairs of 10 random seeds, which is 45 in total.
bank-marketing
bioresponse
Figure 12: The first row exhibits results from the experiment of ranking consistency, while the second row is for the experiment of noisy label detection. Each result is reported using mean and standard deviation. For noisy label detection, the F1-score is reported over 10 random seeds. For ranking consistency, the Spearman's rank correlation coefficient is reported over all possible pairs of 10 random seeds, which is 45 in total.
Figure 13: The first row exhibits results from the experiment of ranking consistency, while the second row is for the experiment of noisy label detection. Each result is reported using mean and standard deviation. For noisy label detection, the F1-score is reported over 10 random seeds. For ranking consistency, the Spearman's rank correlation coefficient is reported over all possible pairs of 10 random seeds, which is 45 in total.
Figure 14: The first row exhibits results from the experiment of ranking consistency, while the second row is for the experiment of noisy label detection. Each result is reported using mean and standard deviation. For noisy label detection, the F1-score is reported over 10 random seeds. For ranking consistency, the Spearman's rank correlation coefficient is reported over all possible pairs of 10 random seeds, which is 45 in total.
Figure 15: The first row exhibits results from the experiment of ranking consistency, while the second row is for the experiment of noisy label detection. Each result is reported using mean and standard deviation. For noisy label detection, the F1-score is reported over 10 random seeds. For ranking consistency, the Spearman's rank correlation coefficient is reported over all possible pairs of 10 random seeds, which is 45 in total.
segment
Figure 16: The first row exhibits results from the experiment of ranking consistency, while the second row is for the experiment of noisy label detection. Each result is reported using mean and standard deviation. For noisy label detection, the F1-score is reported over 10 random seeds. For ranking consistency, the Spearman's rank correlation coefficient is reported over all possible pairs of 10 random seeds, which is 45 in total.
$\mathrm{Ir} = 0.02$
spambase
Figure 17: The first row exhibits results from the experiment of ranking consistency, while the second row is for the experiment of noisy label detection. Each result is reported using mean and standard deviation. For noisy label detection, the F1-score is reported over 10 random seeds. For ranking consistency, the Spearman's rank correlation coefficient is reported over all possible pairs of 10 random seeds, which is 45 in total.
Ir=0.08
texture
Ir=0.04
wind
Extra References
Anguita, D., A. Ghio, L. Oneto, X. Parra, J. L. Reyes-Ortiz, et al. (2013). "A Public Domain Dataset for Human Activity Recognition Using Smartphones". In: Esann. Vol. 3, p. 3.
Dal Pozzolo, A., O. Caelen, Y.-A. Le Borgne, S. Waterschoot, and G. Bontempi (2014). "Learned Lessons in Credit Card Fraud Detection From a Practitioner Perspective". Expert Systems with Applications, vol. 41, no. 10, pp. 4915-4928.
Denneberg, D. and M. Grabisch (1999). "Interaction Transform of Set Functions Over a Finite Set". Information Sciences, vol. 121, no. 1-2, pp. 149-170.
Durrett, R. (2019). "Probability: Theory and Examples". Vol. 49. Cambridge University Press.
Frey, P. W. and D. J. Slate (1991). "Letter Recognition Using Holland-Style Adaptive Classifiers". Machine Learning, vol. 6, pp. 161-182.
Fujimoto, K., I. Kojadinovic, and J.-L. Marichal (2006). "Axiomatic Characterizations of Probabilistic and Cardinal-Probabilistic Interaction Indices". Games and Economic Behavior, vol. 55, no. 1, pp. 72-99.
Grabisch, M., J.-L. Marichal, and M. Roubens (2000). "Equivalent Representations of Set Functions". Mathematics of Operations Research, vol. 25, no. 2, pp. 157-178.
Grabisch, M. and M. Roubens (1999). "An Axiomatic Approach to the Concept of Interaction Among Players in Cooperative Games". International Journal of Game Theory, vol. 28, no. 4, pp. 547-565.
Horn, R. A. and C. R. Johnson (2012). "Matrix Analysis". Cambridge University Press.
Moro, S., R. Laureano, and P. Cortez (2011). "Using Data Mining for Bank Direct Marketing: An Application of the Crisp-Dm Methodology".
Sundararajan, M., K. Dhamhere, and A. Agarwal (2020). "The Shapley Taylor Interaction Index". In: International Conference on Machine Learning. PMLR, pp. 9259-9268.
Vergara, A., S. Vembu, T. Ayhan, M. A. Ryan, M. L. Homer, and R. Huerta (2012). "Chemical Gas Sensor Drift Compensation Using Classifier Ensembles". Sensors and Actuators B: Chemical, vol. 166, pp. 320-329.
Xiao, H., K. Rasul, and R. Vollgraf (2017). "Fashion-Mnist: A Novel Image Dataset for Benchmarking Machine Learning Algorithms". arXiv preprint arXiv:1708.07747.
Yeh, I.-C. and C.-h. Lien (2009). "The Comparisons of Data Mining Techniques for the Predictive Accuracy of Probability of Default of Credit Card Clients". Expert Systems with Applications, vol. 36, no. 2, pp. 2473-2480.
Zhang, H., Y. Xie, L. Zheng, D. Zhang, and Q. Zhang (2021). "Interpreting Multivariate Shapley Interactions in DNNs". In: Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. 12, pp. 10877-10886.