Deep Multiple Instance Learning with Gaussian Weighting
Authors: Basura Fernando
Hakan Bilen
Venue:  (2020)
Keywords: Multiple instance learning
deep learning

Abstract: In this paper we present a deep Multiple Instance Learning (MIL) method that can be trained end-to-end to perform classification from weak supervision. Our MIL method is implemented as a two stream neural network, specialized in tasks of instance classification and weighting. Our instance weighting stream makes use of Gaussian radial basis function to normalize the instance weights by comparing instances locally within the bag and globally across bags. The final classification score of the bag is an aggregate of all instance classification scores. The instance representation is shared by both instance classification and weighting streams. The Gaussian instance weighting allows us to regularize the representation learning of instances such that all positive instances to be closer to each other w.r.t. the instance weighting function. We evaluate our method on five standard MIL datasets and show that our method outperforms other MIL methods. We also evaluate our model on two datasets where all models are trained end-to-end. Our method obtain better bag-classification and instance classification results on these datasets. We conduct extensive experiments to investigate the robustness of the proposed model and obtain interesting insights.

Under review as a conference paper at ICLR 2020
DEEP MULTIPLE INSTANCE LEARNING WITH GAUS- SIAN WEIGHTING
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper we present a deep Multiple Instance Learning (MIL) method that can be trained end-to-end to perform classification from weak supervision. Our MIL method is implemented as two stream neural network, specialized in tasks of instance classification and weighting. Our instance weighting stream makes use of Gaussian radial basis function to normalize the instance weights by comparing instances locally within the bag and globally across bags. The final classification score of the bag is an aggregate of all instance classification scores. The instance representation is shared by both instance classification and weighting streams. The Gaussian instance weighting allows us to regularize the representation learning of instances such that all positive instances to be closer each other w.r.t. the in- stance weighting function. We evaluate our method on five standard MIL datasets and show that our method outperforms other MIL methods. We also evaluate our model on two datasets where all models are trained end-to-end Our method ob- tain better classssssiiiatnoon and instance classification results on these datasets. We conduct extensive experiments to investigate the robustness of the proposed model and obtain interesting insights.
INTRODUCTION
Multiple instance learning (MIL) 7 form of weakly supervised learning that have been widely applied in drug activity prediction 7 medical imaging [14, text analysis and natural language
processing [2] and computer vision 18 In contrast to the standard supervised learning where every training instance is provided with label. in MIL, training instances are grouped in sets, called bags, and labels are provided at bag-level (i.e. one label for entire bag). In the case of binary classification, a bag is labeled as positive, if there is at-least one positive sample, otherwise it is negative. As the instance labels are only indirectly accessible through labels attached to bags, the ambiguity of not knowing which instances are positive makes the problem significantly more challenging than the standard supervised classification. As the ratio of negatives to the positives increases in bags, the ambiguity increases and this renders the problem of bag classification harder. Our hypothesis is that the ability of robustly detecting positive instances in a bag is essential for successful MIL methods.
There is rich body of literature in MIL which can broadly be grouped in two main categories, bag-level 17] and instance-level approaches [1 (2). The first group maps instances of a bag into a single embedding (e.g. Fisher Vector (17) and trains bag-level predictors to classify each bag in similar manner to the standard classification. The second approach learns instance-level classifiers and aggregates the scores of individual instances from each bag. Though the former can be thought to provide a more unbiased solution to the problem of bag classification,, the instance-level approach has recently been shown to outperform the bag-level one in [16) 10 under the assumption that bag labels can be predicted from individual instances. However, predicting instance labels from bag- level ones is an ill- ill-posed problem. Thus, previous work proposes constrained solutions to tackle the ambiguity. Liu et al. [13] propose voting framework to capture neighborhood relationship be- tween instances from different bags to identify key instances. To this end, the authors use the voting framework to obtain weighted nearest neighbor (NN) classifier where its weights globally gener- ated from a kNN graph. Conditional random fields (CRF) are used to model relationships between bags in (6). This models bags nodes in CRF with instances as their states. However, these methods are not learned end-to-end and thus relies on the feature space metric to obtain meaningful weights for each instance.

---

Under review as a conference paper at ICLR 2020
Bag of instances (x1, 22, 11 2
Feature extractor <(:0p)
Instance classifier 0);
Bag pooling p(-)
Instance weighting ( 0p))
Gaussian normalization p(sp,c))
Figure 1: Our MIL model. First, we extract features from deep feature extractor & for each in- stance. Then the instance classifier &b outputs classification scores for each instance. The bottom stream consists of two sub-functions, the instance weighting yp and the Gaussian normalization func- tion o. Finally, the scores of two streams are multiplied and aggregated by to get the bag score.
Deep neural networks are also widely applied solve MIL problem 15
22 16.
Zhou et al.
[22] aggregate the outputs of instance level classifier by the max operator to ensure that if at least one instance of a positive training bag predicted as positive, then the concerned bag is assumed as positive. Similarly, if all instances of negative bag are predicted negative then the bag is negative. However, this method does not explicitly model for identifying the key instances in the neural network architecture but rather hope that feature representation of the instances would adapt (network would leam) such that instance level classifier can solely learn to give higher scores for positive instances from this loss function. Ramon and De Raedt [13] introduce log-sum-cxpp convex max to calculate bag probabilities from instance probabilities. Wang et al. (16) introduce differential MIL pooling layer to enforce MIL constraints such as that each positive bag should have at-least one positive instance. Recently, attention-based deep multiple instance learning was proposed where attention applied over the features using Softmax attention before obtaining bag level representation [T0]
the
We also build on the idea of instance level classification, however, we take different approach from all these methods in three aspects: First, we propose to take a two stream approach for instance clas- sification and instance weighting (see fig. []. Specifically, we have two dedicated paths or streams in our neural network, one dedicated for instance classification by returning score and the other stream dedicated for learning normalized weights. We show empirically that this is crucial. Second, we do not specifically enforce any MIL constraint (e.g. number of positives in bag, certain graph structure) but rather choose a Gaussian activation function that allows us to learn a soft constraint to weight instances by comparing instances within bags and/or across bags. We investigate two vari- ants of Gaussian activated weighting functions that employ bag-level or cross-bat bag statistics to assign weights each instance. By comparing instances within bag across bags, our Gaussian acti- vated weighting function learns meaningful weights to detect key instances within a bag. It assigns higher weight for positive instances while assigns small value for negative instances. In-fact, the instance weighing function assigns lower weights to the instances that are far from the mean and thus encourages positive instances to form compact clusters in the feature space. Enforcing similarity among the features of positive instances through single mean per class also has a strong regularization on the feature space which implicitly enforces constraints on MIL. Third, we simply extend our binary MIL method to a multi-class bag-classification and our method is robust against severe ambiguity levels (very few positives in a very large bag). We demonstrate that our method obtain oate--of-the-ar results in the standard MII. instance learning benchmarks.
2 METHOD
Problem definition. In standard classification, it is assumed that we are given set of training pairs (ar, v) Rd x y with the goal of learning a predictor from inputs to labels f R. - y. this paper, we focus on the more general problem of MIL and use a weaker form of supervision. Input samples are now organized into bags B,, B2.... Bm, with each bag contains a variable number of samples, i.e. B, = {æ. setting, individual instances do not have access to a label but each bag B, is associated with a label Y; y. We consider two settings, binary and multi-class classification. In the binary case y = [-1,1), if a bag is negative, all instances in the bag are negative. On the other hand, if a bag is positive, at least one of the instances is positive. In the multi-class case, the output space
{1. , |/]} where | is the number of instances in B, In this
2

---

Under review as a conference paper at ICLR 2020
is categorical, i.e. y/ {1,2, C} where C' is the number of categories. We learn classifier g x111 + y/ that takes in bag of instances and outputs high score only for the target class:
(Bi) = p(f((1),. f(()))
(1)
(i.e. permutation invariant to the ordering of in- stances æ in bag) which can be decomposed into predictions for individiual instances f(x). In other words, as we do not have access to ground- truth labels y for individual samples æ, instance level predictions fax:) are aggregated to obtain bag-level prediction g ().. Examples of the aggre-
where p is a symmetric function as in
19 10
gation function p are max operator []] weighted sum operator [10] and log of sum of exponentials
15] As a matter of fact, we also build our MIL method on an aggregation function takes in instance level predictions, predicts the importance of each and aggregates them by taking weighted sum to obtain bag level prediction. instance using neural network specifically designed to select key instances [ L.c. discriminative instances that can reliably be utilized to predict bag labels. Formally we define each key instance to be positive below:
particular, we extend the idea present eq.
T to weight each
Definition. Given a bag of multiple instances, if the bag label is positive, the key instances are the instances having positive label.
Property.Given bag of multiple instances, the bag is positive, key instances are fewer than non- key instances. A diagram of our model is depicted in fig. [ which consists of two streams, each stream contains multiple components. We describe them in the following.
Instance classifier f. First we define the instance level prediction f(x) = th dee as a composi- tion of feature extractor é X + H where H. is latent variable space and of classifier v where (<)(;;; (z)) f e Rd and as deep neural networks with learnable parameters 0g and fg respectively.
the number of target categories. In fact, both @ and si are modeled
Instance weighting B. The bag classifier gBB)) takes in the instance scores f(x1). f(x n)) for B₁ and outputs a weighted sum of them by using 3() : x + [0, 1|c where Og are the parameters of the mapping B. Equation (T) can be rewritten follows
g (B) p([3(ar1;;)) 0z) [(x))),... [8(=))100)) 0s) ((x1))))
(2)
where p is an aggregating operator (or the pooling function) and 0g are the learnable parameters of B. The weighting function / a o ip & is also modeled as composition of three functions. shares the feature extractor & with the instance classifier f. however, applies consecutively s0 #pp) (drawn in green in fig. [ on the features which outputs an C dimensional unnormalized vector and followed by univariate Gaussian radial basis normalization (drawn in blue in fig. [ defined as follows:
(r; exp(-11
(3)
where i and are the mean and standard deviation respectively. We propose two ways of cal- culating i and e. In the first case, we fit and e to the scores that are extracted over B,, i.e. (v((((11),, (tro:rr)))...(((((nn)))) a and simply statistics and can be calculated by the following formulas:
" = (o, 2² = =lv
(4)
In other words, weights can be normalized by using the local bag statistics. These variants are denoted by GPOTO and GPOTI as explained later in the experiments. Here the intuition is that by comparing each instance with local-bag statistics, our weight learning function is able to learn to detect key instances. In the second one, we set Ju and 6 as parameters of our network and jointly optimize them with the rest of the network parameters (0,, O2., 0g.0,))) In this case, the parameters not local to bag but global, learned from the updates from all the training samples and do not necessarily correspond to a statistical mean and standard deviation of the score distribution anymore. This variant is denoted by GPIT1.
Training. In this paper we focus on two kinds of MIL tasks, binary and multi-class classification. In the binary case, both the instance classifier f and weighting function 8 output scalar per instance, their outputs are simply multiplied (see eq. (2). Afterwards, ces-su--expoenn [15] is applicd over all the instances as the aggregation function p and finally, fed into sigmoid function to get the
3

---

Under review as a conference paper at ICLR 2020
probability of being positive for bag. We use the binary cross entropy loss to train this model. In the multi-class case, both the instance classifier f and weighting function 3 output a dimensional vector, they are element-wise multiplied, then averaged softmax function over classes. Here we use the cross entropy loss to optimize the model. We train all network parameters jointly by using stochastic gradient descent or Adam. The implementation details are given in section 3
the bag and finally normalized by
EXPERIMENTS
We investigate three variants of our model by varying two design decisions in a controlled manner. First we analyze whether the stream architecture, specialized streams for instance classification and weighting are beneficial. Second, we study whether the parameters of the proposed Gaussian normalization should be calculated from bag statistics or learned along with the other parameters in the network. We describe three model variants in the following.
GPOTO. Here, we set both classifier ((,, 0v) and instance weighting function sp(,, (g) to be the same, i.e., Bg = 0.. This prevents the network to learn specialized functions for instance classifica- tion and weighting and we indicate this by "TO" (two stream off). The Gaussian function variables μ and are estimated from eq. (4) and this is denoted by "GPO" (Gaussian parameters not learnable). Hence, there are no dedicated parameters for learning the weighting function. Since μ and are estimated using within bag instances, prediction weights only depends local bag information.
GPOTI Here we use dedicated parameters for the instance weighting function sp(,, 0²) and it is realized by a 2-layered neural network with Relu activation. As in the previous case, the parameters of the Gaussian normalization are estimated from eq.
GPIT1 In this case, keep the same settings in GPOTI but the parameters of the Gaussian normal- ization (μt and e) are learned through back propaagtiin along with the other network parameters. In estimated locally for each bag, we learn
contrast to the setting where the Gaussian parameters a single global parameterization. In the next section we report and analyze our results in different binary and multi-class classification datasets and compare our results the previous work.
Similar to the instance weighting function ip, our classifier (i.e. d) is realized by a 2-layered neural network with Relu activation. The form of feature extractor depends the problem. We LeNet model as the feature extractor for CIFAR 10 and MNIST datasets. We use a multi layered perceptron (3 fully connected layers with 256, 128 and 64 dimensions, with ReLU activation) for other MIL datasets.
3.1 RESULTS FOR BINARY CLASSIFICATION.
First, we evaluate our method on MNIST-bags dataset 110 with the code provided by the authors. bag is made up of a random number of 28 28 gray images taken from the MNIST dataset. The number of images in a bag is Gaussiandditributeed and bag is given positive label if it contains one or more images with the label "9" Following [10), we investigate the influence the number of bags in the training set and compare all methods. During evaluation
use fixed number of 1,000 test bags. For all experiments a LeNet5 [12] feature extractor <0) is used. We use the default training parameters that are used in [10] (mean bag length is 10, variance of the bag length is 2, etc.). All models are trained end-to-end with the same loss function and same amount of epochs. The attention method (ATT) introduced in [00 apply Softmax over instance scores returned by an instance classifier and then use those Softmax normalized scores to obtain a bag-level feature representation (or bag embedding). ATT-G uses the gated mechanism of ATT where a combination of both tanh and sigmoid activations are used [10]. Similarly, WSDN [3 method uses two-stream approach like us, where one stream is used to weight regions proposals using Softmax function and the other stream is used to classify image regions to improve object detection. Finally, in WSDN the sum of weighted region scores are used to train the model. WSDN is originally used for object detection, however here adapt it for MIL. WSDN is related to this work as it also uses two- stream approach. Therefore, we compare with these two models using MNIST-bage dataset [00 and results reported in table [
4

---

Under review as a conference paper at ICLR 2020
of hags
ATT
ATTG
WSDN
GPOTO
GAOTT
GPITT
100
0.143 - 0.018
0.113 0.020
0.099 = 0.012
0.174 = 0.030
0.082 + 0.012
0.075 = 0.007
0.063 0.015
0.026
0.059 + 0.009
0.053 0.008
300
0.056 1 0.014
0.063 4: 0.003
0.059 1 0.027
0.057 1 0.015
0.066 1: 0.015
0.045 : 0.010
0.041 0.005
0.047
0.007
0.056 4 0.111
0.042 1 0.010
0.036 0.009
500
0.036 + 0.003
0.040 0.008
0.037 + 0.005
0.053 = 0.012
0.041 + 0.008
0,033 = 0.010
Table 1: Bag classification performance using average error rate on MNIST-bags dataset 10 by varying the training bas size.
Bug length
ATT
ATT-G
WSDN
GPTTI (our)
ATT
WSDN
GPITI (our)
Instance detection in AP
Bag recognition in eTr. rate.
0.644 0.049
0.638
0.034
0.758 E 0.025
0,931 0.007
0.143 = 0018
0.099
0.075 00007
20
0.635 + 0.062
0.575 + 06
0.638 + 0.123
0.656 + 0.051
0.114 0.009
0.082 + 0.014
0.067 = 0.009
0.190 0.144
0.109
+ 0.002
0.513 + 0.146
0.467 + 0.235
0.053 = 0.016
0.048 0.012
0.039 0.007
Table 2: Instance detection performance in average precision using MNIST-bags dataset 10 for training size of 100 bags and tested on 1000 bags.
From these results, we see that ATT does not perform as well as WSDN. Gated attention ATT-G, does not perform as good as ATT except in one case. Both ATT and WSDN methods use Softmax function to identify key instances. However, ATT uses Softmax scores for learning an attention- based feature representation for the bag whereas WSDN uses the same information weight in- stances. This suggests that perhaps the two stream approach is much effective for multiple instance learning even- ween-though WSDN is initially designed for object detection. Our two stream-off GPOTO does not perform as good as WSDN or the ATT indicating perhaps the two stream approach is need. GPOTI method which enables two stream approach over Gaussian instance weighting obtains comparable results to ATT and WSDN. This confirms the effectiveness of two-stream approach in multiple instance learning. Finally, we see a significant boost in performance for GPITI where we learn the parameters of the Gaussian instance weighting function. The whole idea of attention (weather it is feature attention or instance weighting) is to make use of relative comparisons. In- stances that comparatively important are highly weighted to obtain attention weights instance weights. Softmax-base as GPOT1 make local comparisons (i.e comparisons within a bag) to find important instances. In- terestingly, all of them obtain somewhat comparable results. However, GPITI learns the Gaussian parameters using back- paopppropggttion and specific to the dataset statistics. Therefore, the instance comparisons are not local to the bag, but learned globally across the dataset. Perhaps this also en- forces a regularization effect on the instance representation by making positive instances to form cluster (s) with respect to the instance weighting function. Therefore, we conclude that parametric two-stream- based Gaussian instance weighting is a better strategy for multiple instance learning. Due to it's sss effectiveness, we use GPITI variant of our method in all other experiments.
methods (such as ATT and WSDN) and the Gaussian-basse
method such
Now we evaluate the instance detection capacity of each model using MNIST-bags dataset [0]- Specifically, the instance score is obtained by the product of instance weighting and instance clas- sifier in eq. ( i.c., ((xx; 0g) fxx).. We use similar strategy for WSDN and ATT methods. For ATT [10] as done in that paper, we take the output of the attention module (attention weights turned by Softmax function) as the instance score. Thereafter, by changing the mean length of the bag to 10, 20 and 30, we report the instance detection performance in average precision in table Average precision is used because it is widely used for other detection tasks (e.g. object detection). We also report bag classification performance in error rate as before. As it can be seen, our method obtains better instance detection performance than ATT and WSDN for two different bag lengths. Interestingly, when the bag length is 10, our method is quite accurate obtaining average precision of 0.931 while the second best WSDN obtains only 0.788. When mean bag length is increased to 20, our method still obtains slightly better instance detection performance than both ATT and WSDN. Interestingly. WSDN outperforms both ATT and GPITI for instance detection when mean bag-size is 30. Even then, the bag classification of GPITI is better than ATT and WSDN, Originally, WSDN is designed for object detection and has the capability to ignore most negative instances and pick only the relevant handful of positives. Therefore, when there are only a few number of positives, the WSDN is able to pick those using Softmax. In contrast, GPITI has the capacity to detect more positive instances than Softmax- based WSDN. Therefore, GPITI has the capacity to obtain better MII. recognition performance as we will see next.
5

---

Under review as a conference paper at ICLR 2020
Method
MUSK I
MUSK 2
Fox
Tiger
Elephant
Traditional methods,
mi-SVM
0.874 +
0,836 4:
0.582 N/A
0.784 N/A
N/A
MI-SVM
0.779 + NIA
0,843 + N/A
0.578 + N/A
0.840 + N/A
0.843 N/A
MI-Kemel 9
0,880 0.031
0.893 = 0.015
0.603 + 0.028
0.842 = 0.010
0.843 + 0.016
EM-DD 20
0,849 1 0.044
0,869 + 0.048
0.609 + 0.045
0.730 - 0.043
0.771 - 0,043
mi-Graph 21
0.889 + 0.033
0.903 = 0.039
0.620 + 0.044
0,860 = 0.037
0.869 + 0.035
miVLAD
0.042
0.620
0.044
0.811
0.850 0.036
miFV 17
0,909 1 0.040
0.884 1 0.042
0.621 1 0.049
0.813 1 0.037
0.852 1 0.036
Deep learning
methods.
Mi-Ner [6
0.889 0.039
0,838 = 0.049
0.613 = 0.035
0.824 = 0.034
0.858 0.037
MI-Net
0.887 + 0.041
0.859 = 0.046
0.622 + 0.038
0 + 0.032
0.862 + 0.034
MI-Net DS
16
0.042
0,74 0.043
0.630 + 0.037
0.845 0.039
0.872 + 0.032
MI-Net RC
16
0.898 + 0.043
0,873 = 0.044
0.619 = 0.047
0.836 = 0.037
0.857 - 0.040
WSDN
0,850 0.011
0.222 + 0.012
0.512 + 0.012
0,794 + 0.010
0.828 + 0.008
ATT [0]
0,892 0.040
0.858 1 0.048
0.615 1 0.043
0.839 1 0.022
0.868 0.022
Gated-ATT
10
0,900 :: 0.050
0.863 + 0.042
0. 0.603 + 0.029
0.845
+ 0.018
0.857 + 0.027
GPITI (Ous)
0.040
0,902 0.047
0.646 0.054
0.834
0.081
0.822 0.064
Our best rOSalE with different pooling.
GPITI (Ours)-Best
0.923 + 0.009
0.907 = 0.009
0.646 + 0.054
0.874 = 0.008
0.847 + 0.009
Table 3: Average classification accuracy (# standard deviation) MUSK 1, MUSK 2, Fox, Tiger and Elephant datasets. We directly compare with Deep learning methods as they are the most rele- vant for us. Results from traditional methods are shown reference. We bold results only for best performing deep learning methods.
Now, we evaluate on five historically important binary MIL benchmarks, MUSK 1-2 77 Fox, Tiger and Elephant image datasets ①1. All five datasets provide only precomputed features and small num- ber of bags. MUSKI and MUSK2 datasets present a real-world problem in drug activity analysis. The goal is to predict activity of new molecules without requiring expensive process of synthesizing the most promising candidate molecules which can yield large economical savings. Both MUSK1 and MUSK2, consist of descriptions of molecules using multiple low-energy conformations. Each conformation is represented by a 166-dimensional feature vector derived from surface properties. MUSKI contains on average approximately 6 conformation per molecule, while MUSK2 has on erage more than 60 conformations in each bag []. Fox, Tiger and Elephant datasets contain features computed over image segments by using region descriptors. Each positive bag consists at least one patch from the target animal category. Negative ones consists patches from other animals. Feature dimensions are 166 for MUSKI and MUSK2 and 230 for the remaining three. The standard perfor- mance measure is average classification accuracies (and their standard deviation) over five ten-fold validation runs. We follow the experimental setting in [00 16) and use the same architecture (3 fully connected layers with 256, 128 and 64 dimensions, ReLU) as the feature extractor và and use SGD optimizer with batch size as 10). We use exactly the same hyper parameters used in prior work
[10 16. Results are reported in table 3
Interestingly, our method is able to outperform some of the traditional as well as recent deep learn- ing methods on three datasets, MUSKI, MUSK2 and more challenging Fox dataset. The results show that our GPITI variant with learnable Gaussian parameters with a two stream architecture is extremely effective even for traditional multiple instance learning tasks. Most interestingly, our method outperforms both ATT and WSDN in most cases.
So far, use the log seg-sum-expponee the classification performance of various pooling operators p() in eq. ( and report results in table/4] Interestingly, mean pooling is effective the log-sum-exponent Our method is able to obtain quite good results on four datasets irrespective of the pooling operator except for Elephant dataset. Max operator seems the least effective one, perhaps due lack of gradient information. We believe that log-sum sum-expoeentttt is theoretically more relevant for MIL and therefore we recommend using that even the mean pooling seems to obtain better results on this dataset. So far we have demonstrated the effectiveness of our method on binary multiple instance learning problem for end-to-end learning (tables and and -nd-to--nddd learning (tables and
[5] the aggregation function p in eq. (2) Here we investigate
these traditional datasets.
3.2 RESULTS FOR MULTI-CLASS CLASSIFICATION
Here we introduce a new dataset that contains raw input (in contrast to precomputed ones in the pre- vious experiments) and also multiple categories. To evaluate method in multi-class classification

---

Under review as a conference paper at ICLR 2020
MUSK
MUSK 2
Fox
Tiger
Flephant
Log-Sum-Exp.
0.911
0.040
0.902 + 0.047
0.646 = 0.054
0.834 = 0.081
0.822 = 0.003
Max
0.529 0.013
0.012
0.790 0.017
Mean
0.919
0.009
0.907
0.009
0.614 1: 0.012
0.874
0.018
0,840 :: 0.011
0.015
0.012
0.862
0.008
0.847 0.009
Table 4: Comparing pooling operators p() for our method (GPITI) on five MIL standard dataset. Results better than otate-of-the-art deep learning methods are in bold and the best pooling operator is underlined for each dataset.
Figure 2: A bag from CIFAR-IO-MIL dataset with one positive instance (horse) where the bag size 10. The bag is labeled as category horse. Each bag has positives only from a single category.
task, we create a new multiple-class MIL dataset from CIFAR-10 dataset [1] and call it CIFAR-IO- MIL dataset. We treat six natural object classes ("bird", "cat", "deer". "dog". "frog", "horse") as positive instances and rest of the object categories as negatives. Each bag contains multiple nega- tive instances from four negative classes ("automobile", "truck", "ship", "'air-plane") while positive instances comes from single positive class. We generate 30,000 and 6,000 bags for training and evaluation respectively. We will release the code for generating CIFAR-10-MIL. dataset along with all other codes used for experimentation. An example bag is depicted in fig. 2 The objective is to predict the bag labels accurately.
Bag generation procedure involves three parameters: bag size, minimum and maximum number of positive in- stances in bag. This dataset allows us change the ra-
8ag size with varying number of positives
tio of positive/negative instances in bags by varying them and study the effect of ambiguity in this task. For in- stance, in case of a bag of size 100 with maximum of 5 positive instances and minimum of the ambigu- ity could vary from 95% to 99%. Thus controlling the
90
80
Accurayy
50
positive/negative ratio allows us to evaluate how MIL al-
ATT GPIT1
gorithms deals with certain amount of ambiguity levels within the bag. In all these experiments, we build our model on LeNet5 model [12 and use it as the feature ex- tractor which outputs 84 dimensional feature vector for each image (instance) within the bag. We train all the reported models end-to-end by using stochastic gradient descent with training batch size of 128 bags"] The learn- ing rate is set to 0.1 for 150 epochs and lowered to 0.01 and 0.001 after 50 epochs each time.
20
10
Number positive instances per bag
Figure 3: Results for varying the num- ber of positive instances for a fixed bag size of 50.
In the first experiment, we use fixed bag size of 50, sample exactly N number of positive instances for each bag and vary it from 2 to 10. In other words, vary the amount of ambiguity in the bags used for training. We compare our method (GPITI) to the recent state-of-the art method ATT (10). Note that use same bag size for both train and test in this experiment. Results are depicted multiple N values in fig.
Both methods perform better in bag classification as the number of positives increases and ambiguity decreases. Interestingly, both methods fail to learn in the presence of extreme ambiguous level of 96% (only 2 positive instances ambiguous level is reduced to 92%. GPITI method outperforms ATT for all ambiguous levels indicating that GPITI is better than ATT even for multiple class multiple instance learning.
bag of size 50). Both methods recover (learn), when the
We also experimented with Adam and obtained similar results.

---

Under review as a conference paper at ICLR 2020
Train bag
Train 20.
Train size
60
GPIT1
CPIT!
GPIT:
Accuracy (N)
Accuracy (N)
Accurscy (50
10
20
200
150
232
420
500
120
202
300
500
120
202
300
400
Test bag size.
Test
Test size.
(c)
70
Train bag size 40.
70
Train bag size 50.
Test bag size 100.
ATT
ATT
60
GP1T2
50-
G8213
-0-
20
Accuracy (8)
Accuracy
Accuracy
40
80
40
30
10
20
20
10
10
120
200
200
400
300
120
200
200
400
300
Test bag size. (d)
Test bag size.
T((() bag size.
(c)
Figure 4: (a-e): Multi-class MII. classification performance (averaged over 10 runs) on CIFAR- 10-MIL dataset. Each training bag has a maximum of 10 positive instances and a minimum of 1. Training bag size varies from 10 to 50 and performance is evaluated on test bags having exactly one positive instance. Test bag size varies from 10, 50, 100, 200, 300, 400, 500. (f): We test models on bags of size 100 having one positive instance. We vary the training bag size from 10 to 50 having maximum of 10 positive instances and minimum of one.
3.3 ROBUSTNESS OF END-TO-END LEARNED MIL METHODS.
In this section, we further study the robustness characteristics of the presented methods when the ambiguity levels in train and test bags differ. To this end, we first train each method with fixed size bags, each containing 1 to 10 positives and then evaluate them on varying bag sizes, each containing only 1 positive. The average classification accuracies of the models are depicted in fig. 4 In each subplot (a)-(e) we fix the training bag size to a certain number ( (10, 20, 30, 40, 50) and depict the performance of each method for varying test bag sizes. In fig. (f) we fix the test bag size to 100 and show the results for the models that are trained with different bag sizes. Based on the results, we expect to answer the following first two questions.
(i) How robust are the presented MIL methods against extremely ambiguous test bags? As we can see from fig. 4 (c), our GPITI method consistently outperform ATT method across various train and test bag sizes. Interestingly, when the train bag size is 10, both methods perform equally. Nevertheless, when the train bag size increases, our method stats to perform better than ATT indicating that ours can better learn under more ambiguous bags. However, both methods perform poorly as the test bag size (and ambiguity) increases. In particular when the models are first trained on less ambiguous bags such bag size of 10 fig. (a), even though the performance for test bag of size 10 is quite good, the performance of test bag size of 500 is very poor (20%). However, when the models are trained with even more ambiguous bags (fig. (e)). they perform better for larger test bags (e.g. 500) where the performance is around 30% for GPITI. Interestingly, our models are better than the attention based models when tested with very large bags with more ambiguous bags. This indicates the robustness of our methods to varying amount of ambiguity (see fig. (c))
(ii) How transferable are the MIL models with respect to change in ambiguity levels? Here we analyze the limits of training bag ambiguity that the MIL model can handle reasonably. The results for the study are depicted in fig. (f). First we observe that all the models benefit from increase in the ambiguity level up to bag size of 40 and then the performance of ATT [10] degrades. However, our model GPITI retains its performance. Here we hypothesize that the models which can learn instance weighting accurately are more robust to changes in ambiguity ratios.
8

---

Under review as a conference paper at ICLR 2020
4 CONCLUSION
Our deep MIL method uses a novel two stream method for instance weighting and classification to compute the bag prediction score. We evaluate method in both binary and multi-class classifica- tion tasks in several standard MIL benchmarks and obtain excellent results for both. We show that presence of extreme ambiguity in training bags (low positive/negative instance ratio) hurts the per- formance of the MIL methods severely. However, we also demonstrate that when the MIL methods are successfully trained with bags containing moderate ambiguity, most methods would also be able to correctly classify bags even in case of extreme ambiguity. Our method which uses a Gaussian function with dedicated learnable parameters, able to outperform other methods under extreme presence of ambiguity while obtaining better results on traditional MIL datasets such as MUSK. Interestingly, in most cases, our method can also obtain good instance detection results.
REFERENCES
[1] Stuart Andrews, Ioannis Tsochantaridis, and Thomas Hofmann. Support vector machines for multiple-instance learning. In Advances in neural information processing systems, pages 577- 584, 2003.
[2] Stefanos Angelidis and Mirella Lapata. Multiple instance learning networks for fine-grained sentiment analysis. Transactions of the Association of Computational Linguistics, 6:17-31, 2018.
[3] Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2846-2854, 2016.
[4] Benjamin Bloem-Re Bloem-Reddy and Yee Whye Teh. Probabilistic symmetry and invariant neural net- 2019.
works. arXiv preprint arXiv: 1et1::90110000
[5] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
[6] Thomas Deselaers and Vittorio Ferrari. A conditional random field for multiple-instance learn- ing. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pages 287-294, 2010.
[7] Thomas G Dietterich, Richard H Lathrop, and Tomás Lozano-Pérez. Solving the multiple instance problem with axis taas-paallll rectangles. Artificial intelligence, 89(1-2)31-71, 1997.
[8] Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: large-scale video benchmark for human activity understanding. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 961-970, 2015.
[9] Thomas Gärtner, Peter A Flach, Adam Kowalczyk, and Alexander J Smola. Multi-instanee kernels. In ICML. page 2002.
[10] Maximilian Isse, Jakub M Tomezak, and Max Welling. Attention- based deep multiple instance learning. arXiv preprint
2018.
[11] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[12] Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):227-2233 1998.
[13] Guoqing Liu, Jianxin Wu, and Zhi-Hua Zhou. Key instance detection in ing. In Asian Conference on Machine Learning, pages 253-268, 2012.
learn-
[14] Gwenolé Quellec, Guy Cazuguel, Béatrice Cochener, and Mathieu Lamard. Multiple-instance learning for medical image and video analysis. IEEE reviews in biomedical engineering, 10:213-234, 2017.
[15] Jan Ramon and Luc De Raedt. Multi instance neural networks. In ICML-2000 workshop on attributevvalue and relational learning. Citeseer, 2000.
[16] Xinggang Wang. Yongluan Yan, Peng Tang. Xiang Bai, and Wenyu Liu. Revisiting multiple instance neural networks. Pattern Recognition, 14:15-24, 2018.

---

Under review as a conference paper at ICLR 2020
[17] Xiu-Shen Wei, Jianxin Wu, and Zhi-Hus Zhou. Scalable algorithms for multi-instancd learn- ing. IEEE transactions on neural networks and learning systems, (8(4))975-998) 2017.
[18] Jiajun Wu, Yinan Yu, Chang Huang, and Kai Yu. Deep multiple instance learning for image classification and auto-annotation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3460-3469, 2015.
[19] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdi- nov. and Alexander Smola. Deep sets. In Advances in neural information processing systems, pages 3391-3401, 2017.
[20] Qi Zhang and Sally A Goldman. Em-dd: An improved multiple-instance learning technique. In Advances in neural information processing systems, pages 1073-1080, 2002.
[21] Zhi-Hua Zhou, Yu- Yin Sun, and Yu-Feng Li. Multi-instance learning by treating instances non-iid samples. In Proceedings of the 26th annual international conference on machine learning, pages 1249-1256. ACM, 2009.
[22] Zhi-Hua Zhou and Min-Ling Zhang. Neural networks for multi-instance learning. In Proceed- ings of the International Conference Intelligent Information Technology, Beijing, China, pages 455 2002.
A APPENDIX
A.1 LARGE SCALE UNTRIMMED VIDEO CLASSIFICATION WITH MIL.
In this section we perform a very large scale untrimmed video action classification using four MIL approaches and show that 1. MIL in general is useful even for a large scale video classification problem 2. our method still outperforms other competitors on this task. We select ActivityNet 1.3
video dataset which consists of 10,000 video for training and about 5000 videos for validation [8]. We consider 32 frame video clip as an instance and the entire video as the bag. video in Ac- tivityNetl.3 33 may have one or more human actions. Our objective is to recognize the presence and absence of each action category cach video. Therefore, the performance measured using mean average precision. We train all methods using I3D network as the feature extractor [5]. We use batch size of 64 videos and Adam optimizer with learning rate of 0.0005. We use a weight decay of 10e-5 All models trained with binary cross entropy loss. We report results in Table 5
Method
mean AP
I3D Baseline
0.7512
ATT
0.7605
ATT-G
0.7768
WSDN 0.7864
GPITT 0,8223
Table 5: Action classification performance on ActivityNet1.3 dataset using MIL methods.
A.2 ARE THE INSTANCE WEIGHTS MEANINGFUL?
Here we provide some experimental results for instance detection weights using Cifar10-MII. dataset. We empirically verify that positive instances weighted higher than the negative ones by 3. For GPITI method positive vs negative weights returned by 3 are 0.94 = 0.11 vs. 0.54 + 0.10 respectively. We conclude, method leams meaningful weights for instances and can robustly detect positive instances under different test distributions from the results shown in table
10


Reviews:
[{"review_id": "rkgrWTyAKB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new approach to weighting in multiple-instance learning scenario. They multiply scores for instances by Gaussian normalized weights. The hyperparameters of the Gaussian RBF are either estimated locally (i.e., within a bag) or trained across bags. Additionally, the authors verify whether it is better to have two separate neural networks (one for instance scoring and one for instance weighting) or share weights between them. Eventually, a variant with globally trained hyperparameters of the Gaussian RBF and separate weights of neural networks performs the best.\n\nIn general, the paper is very well written and easy to follow. The proposed solution is reasonable and, importantly, performs better than other SOTA approaches. The organization of the paper is proper, all concepts are well explained. The experiments are meaningful and answer important questions (i.e., which of the considered variants is the best, how the proposed approach compares to SOTA methods). In my opinion, the paper could be accepted.\n\nREMARKS:\n- What is the temperature value used in the log-sum-exp?\n\n- The authors claim that the log-sum-exp pooling operator is theoretically more relevant for MIL. Could you comment more on that?\n\n- Why do the authors compare to ATT instead of Gated-ATT? In [8] it was shown that Gated-ATT performs better.\n\n======== AFTER REBUTTAL ========\nI would like to thank the authors for their rebuttal. I appreciate their answers and updates. I still believe that the paper is interesting and important for the MIL community. I sustain my decision.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1xiJkHRtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a deep multiple instance learning method. It uses\ntwo stream neural networks, one for instance classification and one\nfor learning normalized weights. The proposed method can be used for\nbinary and multi-class bag classification.\n\nThe authors experimentally show the performance of their proposed\nmethod on several datasets and compare against other state-of-the-art\nmethods. Also they created a new data set for multi-class bag\nclassification.\n\nThe results are very convincing but the description of the method is\nnot very clear. In particular, the description of the instance\nclassifier is confusing. It says that \"f\" is the instance classifier,\nwhich is a composition of a feature extractor (\\phi) and a classifier\n(\\psi), both modeled as deep neural networks. What are those neural\nnetworks? \n\nAlso the weighting function is a composition of three functions, the\nGaussian normalization, the instance weighting, and the feature\nextractor. Again the instance weighting process, which is another key\nfeature of the paper, is not clearly explained. \n\nThe paper has several English errors which makes sometimes difficult\nto follow all the ideas. \n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "B1l5wezE9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper described an approach of performing multiple instance learning (MIL) by using a network branch to weight instances and then using a Gaussian normalization layer on top of it, where the weightings are predicted based on in-bag variances. The instance weighting scheme, a classic in MIL, has been proposed in deep networks by [3]. Hence, I don't see much novelty here except that there is a Gaussian normalization layer after the instance weighting in the MIL framework. \n\nI'm a bit worried that sigma seems to be unnormalized in the first case -- what would happen if the bag score distribution is non-Gaussian? GP1T1 seems more sensible by learning all these weights.\n\nHowever, I see significant issues in terms of evaluation which makes hard to accept this paper.\n\nI firmly believe that 22 years after the (Dietterich 1997) paper, it's no longer enough to only use the original MIL datasets and classification datasets such as the CIFAR-10 bags to evaluate MIL. I may be alone here which is open to discussions, but the original motivation for MIL is for the problem to be weakly-supervised where we only know a high-level label but no low-level labels. There exist many realistic image problems that are similar to this (e.g. weakly-supervised detection and semantic segmentation) and have received a plethora of work, hence it's unclear to me whether still using these arbitrarily generated CIFAR-10 bags and the 5 old datasets would still apply to MIL as an approach. After all, MIL is almost dying and \"weakly-supervised learning\" has risen in popularity with almost being the same problem as MIL. I think for MIL to work its way back, it should first start by using the right datasets to test (e.g. the datasets in [3] would be a great starting point) and comparing with other weakly-supervised detection approaches that do not use additional information.\n\nBesides the philosophical point, a practical issue is that the numbers seem to be bold arbitrarily according to the authors' whim. In table 3, MI-Net DS is better than every other method in the last 2 columns, but not bold, and also it seems that no t-test was performed to determine the significance of differences. This also happens in Table 1, where I think WSDN should be equivalent with the proposed method in the last 2 rows.\n\nFinally, the choice of excluding MI-Net in the CIFAR experiments is dubious as well, given its performance in the simple datasets. Why is MI-Net not tested on the CIFAR experiments?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a novel MIL method that uses a novel approach to normalize the instance weights. The majority of reviewers found the paper lacking in novelty and sufficient experimental performance evidence.
Decision: Reject