SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Edward Raff
raff_edward@bah.com
Jim Holt
Laboratory for Physical Sciences
holt@lps.umd.edu
Multiple Instance Learning (MIL) is a sub-domain of classification problems with positive and negative labels and a "bag" of inputs, where the label is positive if and only if a positive element is contained within the bag, and otherwise is negative. Training in this context requires associating the bag-wide label to instance-level information, and implicitly contains a causal assumption and asymmetry to the task (i.e., you can't swap the labels without changing the semantics). MIL problems occur in healthcare (one malignant cell indicates cancer), cyber security (one malicious executable makes an infected computer), and many other tasks. In this work, we examine five of the most prominent deep-MIL models and find that none of them respects the standard MIL assumption. They are able to learn anticorrelated instances, i.e., defaulting to "positive" labels until seeing a negative counter-example, which should not be possible for a correct MIL model. We suspect that enhancements and other works derived from these models will share the same issue. In any context in which these models are being used, this creates the potential for learning incorrect models, which creates risk of operational failure. We identify and demonstrate this problem via a proposed "algorithmic unit test", where we create synthetic datasets that can be solved by a MIL respecting model, and which clearly reveal learning that violates MIL assumptions. The five evaluated methods each fail one or more of these tests. This provides a model-agnostic way to identify violations of modeling assumptions, which we hope will be useful for future development and evaluation of MIL models.
In Multiple Instance Learning (MIL) we have a dataset of  $N$  labeled points, which we will represent as  $\mathcal{X}$  with associated labels  $y\in \{-1,1\}$  for the negative and positive labels respectively. As originally described, the MIL problem involves each datum  $X_{i}\in \mathcal{X}$  being a bag of multiple instances, where  $X_{i} = \{\mathbf{x}_{1},\mathbf{x}_{2},\ldots ,\mathbf{x}_{n_{i}}\}$  is a bag of  $n_i$  instances. Each instance  $\mathbf{x}_j\in X_i$  is a  $D$ -dimensional vector, and every bag  $X_{i}$  may have a different total number of items  $n_i$ . Given instant level classifier  $h(\cdot)$ , most MIL algorithms work by predicting  $\hat{y}_i = \max_{\forall x_j\in X_i}h(x_j)$ .
As originally described, the positive/negative label of each bag  $X_{i}$  has a special meaning. By default, a bag's label is negative ( $y = -1$ ). The label of a bag will become positive ( $y = 1$ ) if and only if a positive instance  $\mathbf{x_j}$  is present inside the bag, at which point the entire bag's label becomes positive. Because instance-level labels mapping each  $\mathbf{x_j} \rightarrow y \in \{-1,1\}$  are not given, the MIL problem is to infer the instance-level labels from the whole-bag level labeling. This implies a critical asymmetric nature to the given labels and how they must be handled. A value of  $y = -1$  tells us that all instances are negative in the given bag, whereas a label of  $y = 1$  tells us that one or more instances have a positive label. For this reason, swapping the positive and negative labels in a MIL problem is not
semantically meaningful or correct, whereas, in a standard classification problem, the labels can be interchanged without altering the semantics of the learning task.
The MIL problem occurs with frequency in many real-world applications, in particular in the medical community where the presence of any abnormal cell type (i.e., instance) is the confirmative indicator for a larger organism's disease (i.e., bag and label). As the MIL problem implies, and the medical example makes explicit, the MIL model has an implicit casual assumption: the right combination of positive indicators dictate the output label, and so the MIL model is both a valuable inductive bias toward the solution and a guard against physically implausible solutions.
Algorithms that fail, or intentionally forgo, the MIL constraints may appear to obtain better accuracy "in situ" (i.e., the lab environment). But if it is known that the MIL assumption is true, ignoring it creates a significant risk of failure to generalize "in vivo" (i.e., in real production environments). In the clinical context, this is important as many ML algorithms are often proposed with superior in situ performance relative to physicians [1], but fail to maintain that performance when applied to new clinical populations [2-4]. In this case, respecting underlying MIL properties eliminates one major axis of bias between situ and vivo settings and higher confidence in potential utility. In the cyber security space, respecting the MIL nature eliminates a class of "good word" style attacks [5-7] where inconsequential content is added to evade detection, an attack that has worked on production anti-virus software [8-10]. These reasons are precisely why MIL has become increasingly popular, and the importance of ensuring the constraints are satisfied.
Notably, this creates a dearth of options when more complex MIL hypotheses are required, as CausalMIL and mi-Net succeed by restricting themselves to the Standard MIL assumption. The creation of MIL models that satisfy this, and other more complex hypotheses, are thus an open line of research that would have potentially significant clinical relevance. Similarly, users with more niche MIL needs may desire to more thoroughly test their models respect the constraints critical to their deployment. Our work has demonstrated that many articles have not properly vetted the more basic MIL setting, and so we suspect other more complex MIL problems are equally at risk.
Our work contributes the identification of this issue, as well as a strategy to avoid repeat occurrences, by developing algorithmic unit tests where a synthetic dataset is created that captures specific properties about the desired solution. The test will fail if an invariant of the algorithm's solution is not maintained, as summarized in Table 1. Such failures indicate that a MIL method is not properly constrained, and the learning goal is not being achieved. We construct three such datasets for the MIL problem, which can be reused by any subsequent MIL research to mitigate this problem. Based on these results, we would suggest practitioners/researchers begin with CausalMIL and mi-Net as a solid foundation to ensure they are actually satisfying the MIL hypothesis, and thus avoiding excess risk in deployment.
Table 1: Our work proposes a test of the Standard Multiple Instance Learning (MIL) hypothesis and two tests for the Threshold MIL. Most modern deep learning MIL models do not test or prove that they respect any MIL hypothesis, and our tests show that most are insufficient. When a model passes the Threshold test, but fails the Standard test, it is still not a valid Threshold MIL model because the tests do not guarantee correctness, they are like software unit tests that identify failures.
Model Claim	mi-Net Standard	MI-Net Standard	MIL-Pooling Standard	Tran-MIL Threshold	GCN-MIL Threshold	CausalMIL Standard	Hopfield Threshold
Standard Test	✓	✗	✗	✗	✗	✓	✗
Threshold Tests	✗	✓	✗	✗	✗	✓	✓
This paper is organized as follows. In § 2 we will review broadly related works, including prior work in non-deep-MIL and deep-MIL literature. It is in this related work we will denote the baseline algorithms we test, in particular the five deep-MIL models that form the foundation of most current deep-MIL research, and a sixth deep-MIL method that is little-known but does pass our tests. Next in § 3 we will define three algorithmic unit tests for MIL models. The first tests the fundamental MIL assumption that all models must respect, and the second and third tests extend to a generalized version of the MIL problem known as "threshold" MIL. Prior deep-MIL works might tacitly assume they can tackle the generalized MIL, but make no formal specification of the types of MIL models they tackle. Then we apply our tests to six deep-MIL and seven older Support Vector Machine based MIL models in § 4, demonstrating how different algorithms pass and fail different unit tests. In doing
so we provide hard evidence that the foundations of most current deep-MIL works are invalid, and thus dangerous to use in any case where the MIL assumption is used for casual or clinically relevant constraints. For example, although a cancer diagnosis should occur only because cancer was detected, a non-MIL model could learn the absence of something unrelated as a false signal, causing it to overfit. In addition, we discuss cases where a known non-MIL algorithm still passes the unit test, prompting a discussion on how unit tests should be used for invalidation, not certification. Finally, we conclude in § 5.
Concerns about reproducibility within the fields of machine and deep learning have increased in recent years. Prior works have studied reproducibility issues with respect to dataset labels/integrity [11-13], comparison methodology, and making conclusions on improvement [14-18], discrepancies between math and floating-point precision [19], discrepancies between code and paper, [20], and false conclusions in repeatability [21]. We note that none of these prior efforts on reproducibility would have identified the MIL assumption violation that we identify in this work. Our situation is a different aspect of reproducibility in that the methods under test can be reproduced/replicated, but the methods themselves fundamentally are not designed to enforce the modeling assumptions, and no testing was done to ensure that they do. By developing tests meant to elicit certain and specific behaviors of a MIL model, we show how unit tests may be developed for an algorithm in the form of synthetic datasets.
Within the reproducibility literature, we believe the work by Ahn et al. [19] is the most similar to ours, where they develop a mathematical framework for characterizing the reproducibility of an optimization procedure when initialization and gradient computations are not exact. This is motivated by the fact that proofs of optimization do not account for floating point precision in the majority of cases, so specialized domain tests can be useful. A key point is that having source code is helpful, but does not confer correctness of the property of interest [22]. In contrast, our work is more empirical in that we actually implement our proposed tests, and our tests are born not from a mismatch between math and implementation but from the observation that prior works have neglected the mathematical work to ensure their methods follow the MIL assumptions. Other relevant works in reproducibility have looked at methodological errors [15, 23, 24].
The MIL problem bears resemblance to a niche set of defenses used within the malware detection literature. To defend against adversarial attacks, "non-negative" [5] or "monotonic" [6] models were developed where features can only be positive (read, malicious) indicators, and by default, all files would be marked negative (benign) absent any features. This is similar to the MIL model assumption that there is no positive response unless a specific instance is present, and indeed, MIL approaches have been used to build malware detectors that are interpretable and not susceptible to attack [8].
2.1 Relevant Multi-Instance Learning Work
An explosion of interest in MIL literature has occurred due to its relevance in medical imaging and other tasks where the MIL assumption aligns with clinically relevant or important physical/causal constraints of the underlying system. Shockingly, we find much of the literature does not test or ultimately respect this core MIL assumption, resulting in models that are at risk of over-fitting their training data and learning clinically/physically invalid solutions. Simulated benchmarks are common in MIL literature, but focus primarily on the Standard formulation and accuracy [25]. [26] built synthetic benchmarks of MIL tasks, but did not formalize what kinds of MIL tasks or attempt to check if a model was violating the underlying generative MIL hypothesis<sup>1</sup>. The key difference of our work is to create synthetic datasets to test that a model respects the MIL assumptions, rather than benchmark accuracy. We will first review the older, predominantly Support Vector Machine (SVM) history of MIL models that we will test in this work. Then we consider the more recent deep learning counterparts.
Two tasks are Standard MIL, one is Threshold MIL, and a fourth is indeterminate but closest to the Generalized MIL of [27]
2.1.1 Historical Non-Deep MIL
Issues with under-specification of the MIL problem had been previously identified in the seminal survey of Foulds and Frank [27], who synthesized many implicit MIL extensions into a set of generalized and well-specified MIL types. As noted by this prior work, many MIL papers from this time period do not provide proof or attempt to enforce the MIL assumption. Thus while they may be exploring a broader scope of the MIL hypothesis space, the developed solutions may still fundamentally not satisfy the definition of any MIL model.
We will briefly review some significant non-deep MIL models that we include as comparison points, and their status with respect to the MIL assumption. Most notably the mi-SVM and MI-SVM algorithms [28] are correct by construction to the standard MIL model that we will discuss further in § 3.1. The MI-SVM in particular introduces the idea of a "witness", where the bag label is inferred from a singular maximum-responding instance, thus incorporating the standard MIL assumption. SIL is intentionally MIL violating by construction [29]. NSK and STK algorithms [30] were previously recognized to not abide by the MIL hypothesis [27], even though the paper includes formal proofs on the learning theory, the MIL constraints were neglected. Not analyzed previously, we also include two additional models. Firstly, the MissSVM [31], which uses a semi-supervised SVM approach that uses the "single witness" approach to guarantee the standard MIL model. Secondly, the MICA model [32], which is invalid under the standard MIL model because it uses a convex combination of points in the positive bag, and thus does not preclude the possibility of a negative sample.
2.1.2 Deep MIL Models
The first MIL neural network by Zhou and Zhang [33] was later re-invented as the "mi-Net" model, and directly translates the "witness" strategy [28] to a neural network, using weight sharing to process each bag independently, produce a maximal score, and then takes the max over those scores to reach a final decision. This re-invention was done by Wang et al. [34] who added "MI-Net" as a "better" alternative by concatenating the results across bags, allowing a final fully-connected layer to make the prediction by looking at all instances without any constraints. This error allows the MI-Net to learn to use the absence of an instance as a positive indicator, thus violating the MIL assumption. This is true of the MIL pooling layer by Ilse et al. [35] (which forms the basis of their Attention MIL), the Graph Neural Network based GNN-MIL of [36], the Transformer based TransMIL [37], and the Hopfield MIL model of [38]. These latter five deep-MIL models have formed the foundation of many extensions that have the same fundamental designs/prediction mechanisms, with various tweaks to improve training speed or handle large medical images [39-43]. For this reason, we will test these five deep-MIL models as exemplars of the broader deep-MIL ecosystem, and show that all five models fail a simple test.
Two additional deep tests are included, which we note as distinct (because they respect MIL but are rarely used) from the preceding five highly popular methods. The mi-Net, which is the older and not widely used model of [33] respects the standard MIL assumptions. Second is CausalMIL [44, 45], the only recent line of MIL research of which we are aware that properly considers the standard MIL assumption, producing an enhanced version of the "witness" strategy. It does so by representing the problem as a graphical model to infer per-instance labels. While Zhang et al. [44] note the causal nature of MIL modeling to inform their design, they did not document that other deep-MIL approaches fail to respect the MIL assumptions.
3 MIL Unit Tests
The prior works in deep-MIL research have all cited the seminal Dietterich et al. [46] for the MIL problem without elaborating further on the assumptions of the MIL model. As denoted by Foulds and Frank [27], there are many different generalizations of the MIL hypothesis to more complex hypothesis spaces, all of which require respecting that it is the presence of some item(s) that induce a positive label. We will focus on Weidmann's Concept Hierarchy [47] that includes Dietterich et al. [46] as the most basic MIL hypothesis space, and test it along with a generalization of the MIL problem. We note that an algorithm passing a test is not a certificate of correctness. Thus,
Notably this was a mischaracterization and should have been named "MI-Net" going by the original naming scheme, but the names mi-Net and MI-Net with incorrect designation have stuck, and so we repeat them.
if an algorithm passes the generalized Weidmann MIL tests (specified below), but fails the basic Dietterich test (specified below), it means the model fails all possible MIL models because it has failed the most foundational MIL test. Our code for these tests can be found at github.com/NeuromorphicComputationResearchProgram/AlgorithmsUnitTestsMIL.
We will now formalize the general MIL problem in a notation that can capture both the standard and Weidmann versions of the MIL problem. We leverage this formalization to make it clear what properties our unit tests are attempting to capture, and to discuss how a non-MIL model learns invalid solutions.
For all of the tests we consider, let  $h(\mathbf{x})$  be a function that maps a instance vector  $\mathbf{x}$  to one of  $K$  concept-classes  $\in \mathcal{C} = \{\varnothing, 1, 2, \dots, K\}$  (i.e.,  $h(\mathbf{x}) \in \mathcal{C}$ ), which includes the null-class  $\varnothing$ . This null class has the role of identifying "other" items that are unrelated to the positive output decision of the MIL problem. The null-class is the fundamental informative prior and useful constraint of the MIL problem space, where any item belonging to  $\varnothing$  does not contribute to a negative class label prediction. That is to say, only the occurrence of the concept classes  $c_1, \ldots, c_K$  can be used to indicate a positive label in a valid MIL model [46, 27].
For all  $k \in [1, \dots, K]$  where  $c_k \in \mathbb{Z}_{\geq 0}$  let  $g(\{c_1, c_2, \ldots, c_K\})$  be a function that takes in the set of the number of times concept  $c_k$  occurred in a bag, and outputs a class label  $y \in \{-1, 1\}$  for a negative or positive bag respectively. Given a MIL bag  $X = \{\mathbf{x}_1, \dots, \mathbf{x}_n\}$ , let  $\mathbb{1}[\text{predicate}]$  be the indicator function that returns 1 if and only if the predicate is true. Then we can express the generalized MIL decision hypothesis space by Equation 1.
$$
g \left(\bigcup_ {k = 1} ^ {K} \left\{\sum_ {\forall \mathbf {x} ^ {\prime} \in X} \mathbb {1} [ h (\mathbf {x} ^ {\prime}) = k ] \right\}\right) \tag {1}
$$
This generalized form can cover multiple different versions of the MIL problem by changing the constraints on the size of the concept class  $\mathcal{C}$  and the decision function  $g(\cdot)$ .
In the remaining sub-sections, we will use this framework to specify the MIL model being tested, how the test works, and how an invalid MIL-model can "solve" the problem by violating the constraints. This is done by specifying constraints on  $\mathcal{C}$  and  $g(\cdot)$  that define the class of MIL models, and a unit test that checks that these constraints are being respected by the algorithm. We will do so by specifying a NegativeSample and PositiveSample function that returns bags  $X$  that should have negative and positive labels respectively. Each function will have an argument called Training, as a boolean variable indicating if the bag is meant to be used at training or testing time. This is because we will alter the training and testing distributions in a manner that should be invariant to a valid MIL model, but have a detectable impact on non-MIL models. For this reason, we will refer to data obtained when Training=True as the training distribution and Training=False as the testing distribution.
In each unit test, our training bags will have a signal that is easy to learn but violates the MIL assumption being tested. There will be a second signal corresponding to the true MIL decision process, that is intentionally (mildly) harder to detect. At test time (i.e.,  $\neg$ Training), the easy-but-incorrect signal will be altered in a way that does not interfere with the true MIL classification rule.
If a model receives a training distribution AUC  $>0.5$ , but a testing distribution AUC of  $<0.5$ , then the model is considered to have failed a test. This is because a normally degenerate model should receive an AUC of 0.5, indicating random-guessing performance. To obtain an AUC  $<0.5$  means the model has learned a function anti-correlated with the target function. If this occurs simultaneously with an AUC of  $>0.5$ , it means the model has learned the invalid non-MIL bait concept, which is designed to be anti-correlated in the testing distribution.
To simplify the reading of each algorithmic unit test, we will use  $\sim \mathcal{N}(a, I_d \cdot b)$  to indicate a vector is sampled from the multivariate normal distribution with  $d$  dimensions that has a mean of  $\mu = \vec{1} \cdot a$  and a covariance  $\Sigma = I_d \cdot b$ . In all cases, we use  $d = 16$  dimensions, but the test is valid for any dimensionality. In many of our tests, the number of items will be varied, and we denote an integer  $z$  sampled from the range  $[a, b]$  as  $z \sim \mathcal{U}(a, b)$  when an integer is randomly sampled from a range. When this value is not critical to the function of our tests, the sampling range will be noted as a comment in the pseudo-code provided.
3.1 Presence MI Assumption and Test
We begin with the simplest form of the MIL decision model as expressed by Dietterich et al. [46]. In this case, the concept class is unitary with  $K = 1$ ,  $\mathcal{C} = \{\varnothing, 1\}$ , giving the positive classes as the only option, and the non-contributing null-class  $\varnothing$ . The decision function  $g(\{c_1\}) = c_1 \geq 1$ , that is, the label is positive if and only if the positive concept  $c_1$  has occurred at least once within the bag.
Given these constraints, we design a simple dataset test to check that an algorithm respects these learning constraints on the solution. We will abuse notation with  $h(\mathcal{N}(0, I_d \cdot 1)) \coloneqq \emptyset$  to indicate that the space of samples from a normal distribution as specified is defined as corresponding to the null-class  $\varnothing$ . This first value will be the general "background class" that is not supposed to indicate anything of importance.
To make a learnable but not trivial class signal, we will have two positive class indicators that never co-occur in the training data. Half will have  $h(\mathcal{N}(0, I_d \cdot 3)) \coloneqq c_1$  and the other half will have  $h(\mathcal{N}(1, I_d \cdot 1)) \coloneqq c_1$ . We remind the reader that this is a normal distribution in a  $d$ -dimensional space, so it is not challenging to distinguish these two classes from the background class  $\mathcal{N}(0, I_d \cdot 1)$ , as any one dimension with a value  $\geq 3$  becomes a strong indicator of the  $c_1$  class.