Title: MULTIPLE INSTANCE SPATIAL TRANSFORMERS

Abstract: We propose a deep network that can be trained to tackle image reconstruction and classification problems that involve detection of multiple object instances, without any supervision regarding their whereabouts. The network learns to extract the most significant K patches, and feeds these patches to a task-specific networke.g., auto-encoder or classifier -to solve a domain specific problem. The challenge in training such a network is the non-differentiable top-K selection process. To address this issue, we lift the training optimization problem by treating the result of top-K selection as a slack variable, resulting in a simple, yet effective, multi-stage training. Our method is able to learn to detect recurring structures in the training dataset by learning to reconstruct images. It can also learn to localize structures when only knowledge on the occurrence of the object is provided, and in doing so it outperforms the state-of-the-art.

1. INTRODUCTION:

Finding and processing multiple instances of characteristic entities in a scene is core to many computer vision applications, including object detection (Ren et al., 2015; He et al., 2017; Redmon & Farhadi, 2017) , pedestrian detection (Dollár et al., 2012; Sewart & Andriluka, 2016; Zhang et al., 2018a) , and keypoint localization (Lowe, 2004; Bay et al., 2008) . In traditional vision pipelines, a common approach to localizing entities is to select the top-K responses in a heatmap and use their locations (Lowe, 2004; Bay et al., 2008; Felzenszwalb et al., 2010) . However, this type of approach does not provide a gradient with respect to the heatmap, and cannot be directly integrated into neural network-based systems. To overcome this challenge, previous work proposed to use grids (Redmon et al., 2016; He et al., 2017; Detone et al., 2018) to simplify the formulation by isolating each instance (Yi et al., 2016) , or to optimize over multiple branches (Ono et al., 2018) . While effective, these approaches require additional supervision to localize instances, and do not generalize well outside their intended application domain. Other formulations, such as sequential attention (Ba et al., 2015; Gregor et al., 2015; Eslami et al., 2015) and channel-wise approaches (Zhang et al., 2018c) are problematic to apply when the number of instances of the same object is large.

Here, we introduce a novel way to approach this problem, which we term Multiple Instance Spatial Transformer, or MIST for brevity. As illustrated in Figure 1 for the image synthesis task, given an image, we first compute a heatmap via a deep network whose local maxima correspond to locations of interest. From this heatmap, we gather the parameters of the top-K local maxima, and then extract the corresponding collection of image patches via an image sampling process. We process each patch independently with a task-specific network, and aggregate the network's output across patches.

Training a pipeline that includes a non-differentiable selection/gather operation is non-trivial. To solve this problem we propose to lift the problem to a higher dimensional one by treating the parameters defining the interest points as slack variables, and introduce a hard constraint that they must correspond to the output that the heatmap network gives. This constraint is realized by introducing an auxiliary function that creates a heatmap given a set of interest point parameters. We then solve for the relaxed version of this problem, where the hard constraint is turned into a soft one, and the slack variables are also optimized within the training process. Critically, our training strategy allows the network to incorporate both non-maximum suppression and top-K selection. We evaluate the performance of our approach for 1 the problem of recovering the basis functions that created a given texture, 2 classification of handwritten digits in cluttered scenes, and 3 recognition of house numbers in real-world environments. In summary, in this paper we:

• introduce the MIST framework for weakly-supervised multi-instance visual learning;

• propose an end-to-end training method that allows the use of top-K selection;

• show that our framework can reconstruct images as parts, as well as detect/classify instances without any location supervision.

2. RELATED WORK:

Attention models and the use of localized information have been actively investigated in the literature. Some examples include discriminative tasks such as fine-grained classification (Sun et al., 2018) , pedestrian detection (Zhang et al., 2018a) , and generative ones such as image synthesis from natural language (Johnson et al., 2018) . They have also been studied in the context of more traditional Multiple Instance Learning (MIL) setup (Ilse et al., 2018) . We now discuss a selection of representative works, and classify them according to how they deal with multiple instances.

Grid-based methods. Since the introduction of Region Proposal Networks (RPN) (Ren et al., 2015) , grid-based strategies have been used for dense image captioning (Johnson et al., 2016) , instance segmentation (He et al., 2017) , keypoint detection (Georgakis et al., 2018) , and multi-instance object detection (Redmon & Farhadi, 2017) . Recent improvements to RPNs attempt to learn the concept of a generic object covering multiple classes (Singh et al., 2018) , and to model multi-scale information (Chao et al., 2018) . The multiple transformation corresponding to separate instances can also be densely regressed via Instance Spatial Transformers (Wang et al., 2018) , which removes the need to identify discrete instance early in the network. However, all these methods are fully supervised, requiring both class labels and object locations for training.

Heatmap-based methods. Heatmap-based methods have recently gained interest to detect features (Yi et al., 2016; Ono et al., 2018; Detone et al., 2018) , find landmarks (Zhang et al., 2018c; Merget et al., 2018) , and regress human body keypoint (Tekin et al., 2017; Newell et al., 2016) . While it is possible to output a separate heatmap per class (Zhang et al., 2018c; Tekin et al., 2017) , most heatmap-based approaches do not distinguish between instances. Yi et al. (2016) re-formulate the problem based on each instance, but in doing so introduce a sharp difference between training and testing. Grids can also be used with heatmaps (Detone et al., 2018) , but this results in an unrealistic assumption of uniformly distributed detections in the image. Overall, heatmap-based methods excel when the "final" task of the network is generate a heatmap (Merget et al., 2018) , but are problematic to use as an intermediate layer in the presence of multiple instances.

Sequential inference methods. Another way to approach multi-instance problems is to attend to one instance at a time in a sequentially. Training neural network-based models with sequential attention is challenging, but approaches using policy gradient (Ba et al., 2015) and differentiable attention mechanisms (Gregor et al., 2015; Eslami et al., 2015) have achieved some success for images comprising small numbers of instances. However, Recurrent Neural Networks (RNN) often struggle to generalize to sequences longer than the ones encountered during training, and while recent results on inductive reasoning are promising (Gupta et al., 2018) , their performance does not scale well when the number of instances is large.

Knowledge transfer. To overcome the acquisition cost of labelled training data, one can transfer knowledge from labeled to unlabeled dataset. For example, Inoue et al. (2018) train on a single instance dataset, and then attempt to generalize to multi-instance domains, while Uijlings et al. (2018) attempt to also transfer a multi-class proposal generator to the new domain. While knowledge transfer can be effective, it is highly desirable to devise unsupervised methods such as ours that do not depend on an additional dataset.

Weakly supervised methods. To further reduce the labeling effort, weakly supervised methods have also been proposed. Wan et al. (2018) learn how to detect multiple instances of a single object via region proposals and ROI pooling, while Tang et al. (2018) propose to use a hierarchical setup to refine their estimates. Gao et al. (2018) provides an additional supervision by specifying the number of instances in each class, while Oquab et al. (2015) and Zhang et al. (2018b) localize objects by looking at the network activation maps (Zhou et al., 2016; Selvaraju et al., 2017) . Shen et al. (2018) introduce an adversarial setup, where detection boxes are supervised by distribution assumptions and classification objectives. However, all these methods still rely on region proposals from an existing method, or define them via a hand-tuned process.

3. MIST FRAMEWORK:

We first explain our framework at a high level, and then detail each component in Section 4. A prototypical MIST architecture (Figure 1 ) is composed of two trainable components: 1 the first module receives an image as input and extracts K patches, at image locations and scales given by the top K local maxima of a heatmap computed by a trainable heatmap network H η with weights η. 2 the second module processes each extracted patch with a task-specific network T τ whose weights τ are shared across patches, and further manipulates these signals to express a task-specific loss L task . The two modules are connected through non-maximum suppression on the scale-space heatmap output of H η , followed by a top-K selection process to extract the parameters defining the patches, which we denote as E K . We then sample patches at these locations through (differentiable) bilinear sampling S and feed them the task module.

The defining characteristic of the MIST architecture is that they are quasi-unsupervised: the only supervision required is the number K of patches to extract. However, as we show in Section 5.2, our method is not sensitive to the choice of K during training. In addition, our extraction process uses a single heatmap for all instances that we extract. In contrast, existing heatmap-based methods (Eslami et al., 2015; Zhang et al., 2018c) typically rely on heatmaps dedicated to each instance, which is problematic when an image contains two instances of the same class. Conversely, we restrict the role of the heatmap network H η to find the "important" areas in a given image, without having to distinguishing between classes, hence simplifying learning.

Formally, the training of the MIST architecture is summarized by: minimize

EQUATION

where τ, η are the network trainable parameters. Unfortunately, the patch extractor E K is nondifferentiable, as it identifies the locations of the top-K local maxima of a heatmap and then selects the corresponding patches from the input image. Differentiating this operation provides a gradient with respect to the input, but no gradient with respect to the heatmap. Although it is possible to smoothly relax the patch selection operation in the K = 1 case (Yi et al., 2016) for each training batch do

3: τ ← Optimization step for T τ with L task + λ {x k } -E K (H η (I)) 2 2 4: {x k } ← Optimization step for {x k } with L task + λ {x k } -E K (H η (I)) 2 2 5: η ← Optimization step for η with G({x k }) -H η (I) 2 2 6:

end for 7: end function thus impossible to train the patch selector parameters directly by backpropagation. We propose an alternative approach to training our model, via lifting.

Differentiable top-K via lifting. The introduction of auxiliary variables to simplify the structure of an optimization problem has proven effective in a range of domains ranging from non-rigid registration (Taylor et al., 2016) , to robust optimization (Zach & Bournaoud, 2018) . To simplify our training optimization, we start by decoupling the heatmap tensor from the optimization (1) by introducing the corresponding auxiliary variables h, as well as the patch location variables {x k } from the top-K extractor:

minimize η,τ,h,{x k } L task (T τ (S({x k }))) s.t. h = H η (I), {x k } = E K (h).

(2)

We then relax the first constraint above to a least-squares penalty via a Lagrange multiplier λ:

EQUATION

As in many methods that use keypoint supervision to regress heatmaps (Cao et al., 2018) , we assume that a good heatmap generator G exists -that is

{x k } ≈ E K (G({x k }))

We can now rewrite our optimization as:

EQUATION

We can now drop the auxiliary variable h and rewrite our optimization as:

EQUATION

and then approach the problem by block coordinate descent -where the energy terms not containing the variable being optimized are safely dropped, and we apply E K to the penalty term of (5):

EQUATION

EQUATION

)

To accelerate training, we further split (6) into two stages, and alternate between optimizing τ and {x k }. Being based on block coordinate descent, this process converges smoothly, as we show in Section D of the appendix. The summary for the three stage optimization procedure is outlined in Algorithm 1. Notice that we are not introducing any additional supervision signal that is tangent to the given task.

4. IMPLEMENTATION:



4.1. COMPONENTS:

Multiscale heatmap network -H η . Our multiscale heatmap network is inspired by LF-Net (Ono et al., 2018) . We improve the network by applying modifications on how the scores are aggregated over scale. The network takes as input an image I, and outputs multiple heatmaps h s of the same size for each scale level s. To limit the number of necessary scales, we use a discrete scale space with S scales, and resolve intermediate scales via interpolation; see Section A of the appendix for details.

Top-K patch selection -E K . To extract the top K elements, we perform an addition cleanup through a standard non-maximum suppression. We then find the spatial locations of the top K elements of this suppressed heatmap, denoting the spatial location of the k th element as (x k , y k ), which now reflect local maxima. For each location, we compute the corresponding scale by weighted first order moments (Suwajanakorn et al., 2018) where the weights are the responses in the corresponding heatmaps, i.e. s k = s h s (x k , y k )s

Generating model for ideal heatmap -G({x k }). For the generative model that turns keypoint and patch locations into heatmaps, we apply a simple model where the heatmap is zero everywhere else except the corresponding keypoint locations (patch center); see Section B of the appendix for details.

Patch resampling -S. As a patch is uniquely parameterized by its location and scale, i.e.

x k = (x k , y k , s k ), we can then proceed to resample its corresponding tensor via bilinear interpolation (Jaderberg et al., 2015; Jiang et al., 2019) as

{P k } = S (I, {x k }).

4.2. TASK-SPECIFIC NETWORKS:

We now introduce two applications of the MIST framework. We use the same heatmap network and patch extractor for both applications, but the task-specific network and loss differ. We provide further details regarding the task-specific network architectures in Section C of the appendix.

Image reconstruction / auto-encoding. As illustrated in Figure 1 , for image reconstruction we append our patch extraction network with a shared auto-encoder for each extracted patch. We can then train this network to reconstruct the original image by inverting the patch extraction process and minimizing the mean squared error between the input and the reconstructed image. Overall, the network is designed to jointly model and localize repeating structures in the input signal. Specifically, we introduce the generalized inverse sampling operation S -1 (P i , x i ), which starts with an image of all zeros, and places the patch P i at x i . We then sum all the images together to obtain the reconstructed image, optimizing the task loss

EQUATION

)

Multiple instance classification. By appending a classification network to the patch extraction module, we can also perform multiple instance learning. For each extracted patch P k we apply a shared classifier network to output ŷk ∈ R C , where C is the number of classes. In turn, these are then converted into probability estimates by the transformation pk = softmax(ŷ k ). With y l being the one-hot ground-truth labels of instance l with unit norm, we define the multi-instance classification loss as

EQUATION

)

where L is the number of instances in the image. We empirically found this loss to be more stable compared to the cross-entropy loss, similar to Mao et al. (2017) . Note here that we do not provide supervision about the localization of instances, yet the detector network will automatically learn how to localize the content with minimal supervision (i.e. the number of instances).

5. RESULTS AND EVALUATION:

To demonstrate the effectiveness of our framework we evaluate the two different tasks. We first perform a quasi-unsupervised image reconstruction task, where only the total number of instances in the scene is provided, i.e. K is defined. We then show that our method can also be applied to weakly supervised multi-instance classification, where only image-level supervision is provided. Note that, unlike region proposal based methods, our localization network only relies on cues from the classifier, and both networks are trained from scratch.

5.1. IMAGE RECONSTRUCTION:

From the MNIST dataset, we derive two different scenarios. In the MNIST easy dataset, we consider a simple setup where the sorted digits are confined to a perturbed grid layout; see Figure 2 (top) . Specifically, we perturb the digits with a Gaussian noise centered at each grid center, with a standard deviation that is equal to one-eighths of the grid width/height. In the MNIST hard dataset, the positions are randomized through a Poisson distribution (Bridson, 2007) , as is the identity, and cardinality of each digit. We allow multiple instances of the same digit to appear in this variant. For both datasets, we construct both training and test sets, and the test set is never seen during training.

Comparison baselines We compare our method against four baselines 1 the grid method divides the image into a 3 × 3 grid and applies the same auto-encoder architecture as MIST to each grid location to reconstruct the input image; 2 the channel-wise method uses the same auto-encoder network as MIST, but we modify the heatmap network to produce K channels as output, where each channel is dedicated to an interest point. Results for "MNIST easy" As shown in Figure 2 (top) all methods successfully re-synthesize the image, with the exception of Eslami et al. (Eslami et al., 2015) . As this method is sequential, with nine digits the sequential implementation simply becomes too difficult to optimize through. Note how this method only learns to describe the scene with a few large regions. We summarize quantitative results in Table 1 .

Results for "MNIST hard" As shown in Figure 2 (bottom), all baseline methods failed to properly represent the image. Only MIST succeeded at both localizing digits and reconstructing the original image. Although the grid method accurately reconstructs the image, it has no concept of individual digits. Conversely, as shown in Figure 3 , our method generates accurate bounding boxes for digits even when these digits overlap, and does so without any location supervision. For quantitative results, please see Table 1 . Finding the basis of a procedural texture We further demonstrate that our methods can be used to find the basis function of a procedural texture. For this experiment we synthesize textures with procedural Gabor noise (Lagae et al., 2009) . Gabor noise is obtained by convolving oriented Gabor wavelets with a Poisson impulse process. Hence, given exemplars of noise, our framework is tasked to regress the underlying impulse process and reconstruct the Gabor kernels so that when the two are convolved, we can reconstruct the original image. Figure 4 illustrates the results of our experiment.

The auto-encoder learned to accurately reconstruct the Gabor kernels, even though in the training images they are heavily overlapped. These results show that MIST is capable of generating and reconstructing large numbers of instances per image, which is intractable with other approaches.

5.2. MULTIPLE INSTANCE CLASSIFICATION:

Multi-MNIST -Figure 5 . To test our method in a multiple instance classification setup, we rely on the MNIST hard dataset. We compare our method to channel-wise, as other baselines are designed for generative tasks. To evaluate the detection accuracy of the models, we compute the intersection over union (IoU) between the ground-truth bounding box and the detection results, and assign it as a match if the IoU score is over 50%. We report the number of correctly classified matches in Table 2 , as well as the proportion of instances that are both correctly detected and correctly classified.

Our method clearly outperforms the channel-wise strategy. Note that, even without localization supervision, our method correctly localizes digits. Conversely, the channel-wise strategy fails to learn. This is because multiple instances of the same digits are present in the image. For example, in the Figure 5 (right), we have three sixes, two zeros, and two nines. This prevents any of these digits from being detected/classified properly by a channel-wise approach.

Sensitivity to K -Table 3 . To investigate the sensitivity to the correctness of K during training, we train with varying number of of instances and test with ground-truth K. For example, with K = 6, where the ground truth could be anything within {3, 4, 5, 6, 7, 8, 9} -we mark the K used during training with bold. Our method still is able to give accurate result with inaccurate K during training.

Knowing the exact number of objects is not a strict requirement at test time, as our detector generates a heatmap for the entire image regardless of the K it was trained with. Note that while in theory sequential methods are free from this limitation, in practice they are able to deal with limited number of objects (e.g. up to three) due to their recurrent nature.

SVHN -Figure 6 and Table 4 . We further apply MIST to the uncropped and unaligned Street View House Numbers dataset (Netzer et al., 2011) . Compared to previous work that has used cropped and resized SVHN images (e.g. (Netzer et al., 2011; Ba et al., 2015; Goodfellow et al., 2014; Jaderberg et al., 2015) ), this evaluating setting is significantly more challenging, because digits can appear anywhere in the image. We resize all images to 60 × 240, use only images labeled as containing 2 digits, and apply MIST at a single scale. Although the dataset provides bounding boxes for the digits, we ignore these bounding boxes and use only digit labels as supervision. During testing, we exclude images with small bounding boxes (< 30 pixels in height). We report results in terms of AP IoU=.X , where X is the threshold for determining detection correctness. With IoU = 0, we refer to a "pure" classification task (i.e. no localization). As shown, supervised results provide better performance with higher thresholds, but MIST performs even better than the supervised baseline for moderate thresholds. We attribute this to the fact that, by providing direct supervision on the location, the training focuses too much on having high localization accuracy.

6. CONCLUSION AND FUTURE WORK:

In this paper, we introduce the MIST framework for multi-instance image reconstruction/classification. Both these tasks are based on localized analysis of the image, yet we train the network without providing any localization supervision. The network learns how to extract patches on its own, and these patches are then fed to a task-specific network to realize an end goal. While at first glance the MIST framework might appear non-differentiable, we show how via lifting they can be effectively trained in an end-to-end fashion. We demonstrated the effectiveness of MIST by introducing a variant of the MNIST dataset, and demonstrating compelling performance in both reconstruction and classification. We also show how the network can be trained to reverse engineer a procedural texture synthesis process. MISTs are a first step towards the definition of optimizable image-decomposition networks that could be extended to a number of exciting unsupervised learning tasks. Amongst these, we intend to explore the applicability of MISTs to unsupervised detection/localization of objects, facial landmarks, and local feature learning.

Even though our method is a significant step forward from existing methods, it is not complete and has much potential for improvement. While our method is robust to K during training as shown in Section 5.2, and K can be determined dynamically during test time, it would beneficial to automatically determine K through an additional network. Patches could also be refined to have varying aspect ratios similar to RPNs. Reconstruction could consider a global reconstructor to deal with complex images. These are parts of our future work towards MIST in the wild. et al., 2015) originally applied their model to a dataset consisting of images of 0, 1, or 2 digits with equal probability. We found that the model failed to converge unless it was trained with examples where various number of total digits exist, so for fair comparison, we populate the training set with images consisting of all numbers of digits between 0 and 9. For the Zhang et al. (Zhang et al., 2018c ) baseline, we use the authors' implementation and hyperparameters.

D CONVERGENCE DURING TRAINING:

As is typical for neural network training, our objective is non-convex and there is no guarantee that a local minimum found by gradient descent training is a global minimum. Empirically, however, optimization process is stable, as shown in Figure 7 . We reporpt both the training losses L task and G({x k }) -H η (I) 2 2 , and visualize how the heatmap evolves for an example image from the SVHN datset. We visualize the heatmap by showing brighter intensity values for higher scores. Each image is normalized separately with their own maximum and minimum values for better visualization. In Figure 7 (c) top, early in training, one can see that keypoints are detected at random locations as the heatmaps are generated by networks with randomly initialized weights. However, as training continues, keypoints that, by chance, land on locations nearby the correct object (e.g. numbers) for certain samples in a random batch, and become reinforced. Ultimately, as shown in (c) middle and bottom, MIST learns to detect these locations, thus learns to produce a peaky heatmap, and perform the task of interest. Note that this is unsurprising, as our formulation is a lifted version of this loss to allow gradient-based training. Although the images we show in Figure 2 involve small displacements from a uniformly spaced grid, our method does not require the keypoints to be evenly spread. As shown in Figure 8 , our method is able to successfully learn even when the digits are placed unevenly. Note that, as our detector is fully

E NON-UNIFORM DISTRIBUTIONS:



:

https://github.com/aakhundov/tf-attend-infer-repeat

Figures and Tables:

Figure1: The MIST architecture -A network H η estimates locations and scales of patches encoded in a heatmap h. Patches are then extracted via a sampler S, and then fed to a task-specific network T τ . In this example, the specific task is to re-synthesize the image as a super-position of (unknown and locally supported) basis functions. As top-k operation is non-differentiable, we back-propagate by lifting through G; see Section 3 for details.

Figure 2: MNIST character synthesis examples for (top) the "easy" single instance setup and (bottom) the hard multi-instance setup. We compare the output of MISTs to grid, channel-wise, sequential Eslami et al. (Eslami et al., 2015) and Zhang et al. (Zhang et al., 2018c).

Figure 4: Inverse rendering of Gabor noise; we annotate correct / erroneous localizations.

Figure 5: Two qualitative examples for detection and classification on our Multi-MNIST dataset.

Figure 7: Evolution of the loss and heatmap during training. (a) the task (classification) loss L task and (b) the heatmap related loss G({x k }) -H η (I) 2 2 for each iteration. In (c), we show how the heatmap evolves for an example image from the SVHN dataset during training (from top to bottom)brighter the intensity, higher the scores; see Section D.

Figure 8: Examples with uneven distributions of digits.

Multi-stage optimization for MISTsRequire: K : number of patches to extract, L task : task specific loss, I : input image, G : the keypoints to heatmap function, H : the heatmap network, η : parameters of the heatmap network, T : the task network, τ : parameters of the task network, E K : the top-K operator. 1: function TRAINMIST(I, L task )

Table content: <table/>

Locations are obtained through a channel-wise soft-argmax as inZhang et al. (2018c); 3 Esl16(Eslami et al., 2015) is a sequential generative model; 4 Zha18(Zhang et al., 2018c) is a state-of-the-art heatmap-based method with channel-wise strategy for unsupervised learning of landmarks. For more details see Supplementary Section C.

Table content: <table/>

Reconstruction error (root mean square error). Note that Grid does not learn any notion of digits.

Table content: <table><tr><td/><td colspan="4">MIST Grid Ch.-wise Esl16 Zha18</td><td colspan="2">MIST Ch.-wise Supervised</td></tr><tr><td colspan="3">MNIST easy .038 .039 .042</td><td colspan="2">.100 .169</td><td>IOU 50% 97.8% 25.4%</td><td>99.6%</td></tr><tr><td colspan="3">MNIST hard .053 .062 .128</td><td colspan="2">.154 .191</td><td>Classif. 98.8% 75.5%</td><td>98.7%</td></tr><tr><td>Gabor</td><td>.095 -</td><td>-</td><td>-</td><td>-</td><td>Both 97.5% 24.8%</td><td>98.6%</td></tr></table>

Instance level detection and classification performance on "MNIST hard".

Table content: <table><tr><td colspan="2">Instances during train AP IoU=.50</td></tr><tr><td>{9}</td><td>92.2%</td></tr><tr><td>{6, 7, 8, 9}</td><td>90.1%</td></tr><tr><td colspan="2">{3, 4, 5, 6, 7, 8, 9} 90.8%</td></tr></table>

Sensitivity experiment of K on the MNIST hard dataset.

Table content: <table><tr><td colspan="2">MIST Supervised</td></tr><tr><td>AP IoU=.00 82.6%</td><td>65.6%</td></tr><tr><td>AP IoU=.50 76.5%</td><td>62.8%</td></tr><tr><td>AP IoU=.60 63.7%</td><td>59.8%</td></tr><tr><td>AP IoU=.70 42.7%</td><td>51.9%</td></tr><tr><td>AP IoU=.80 19.9%</td><td>34.6%</td></tr><tr><td>AP IoU=.90 4.2%</td><td>11.0%</td></tr><tr><td>Figure 6: Qualitative SVHN results.</td><td/></tr></table>

Quantitative SVHN results.

Table content: <table/>

Peer review (iclr2020):
The article introduces the novel MIST architecture which tries to solve the problem of multiple-instance classification and image generation from multiple objects. It employs two submodels, where the first generates a heatmap of intereting region and the second model is a task-specific model that works on image-patches, for example a classifier or an autoencoder. Both models are connected by a patch-extraction routine. The main contribution of this paper is to provide a way to propagate errors through this non-differentiable patch-extraction scheme. This is done by introducing slack-variables. ------------------ The paper is overall relatively easy to follow and the results are very good. However, it suffers from the fact that it does not differentiate between model-architecture and the overall approach. While the main contribution is described in Section 4, the paper spends a lot of space beforehand to introduce the task-dependent models as well as the heatmap architecture - things that i can imagine will vary a lot in different applications. The real important part is how to train the model and this is unfortunately only half described. A good deal of abstraction from the network architecture would have made the paper a lot better. Further, I think that the loss-function for the classification task does not work in the general case. On my first read-through, i completely misunderstood Section 4. Here is an unsorted list of issues i had with this: - since E_K is not truly invertible, writing the approximate inverse as E_K^{-1} is misleading. - It might help to stress that you treat {x_k} as continuous and the sampling as differentiable. - In (7) it would be better to explicitly write E_K^{-1}(x_k) instead of introducing _x0008_ar{h}. The line below is not clear. - It is also misleading, because the choice of _x0008_ar{h}=E_K^{-1}(x_k) is not the minimizer of (5) given that all other variables are fixed. You can see this by observing that assuming that when {x_k}=E_K(H(I)) holds, we can choose all other pixels to be exactly the value returned by H(I). - I am not sure where the alternating part comes from because this usually involves taking your solution from (7) and feeding it into (6). - I am pretty sure that in (6)+(7), as well as lines 3+6 of the algorithm, you actually don*t want to optimize for tau or eta from scratch but only perform a single SGD step. I think that is what you are doing, but right now it is written as *find a complete new model for each batch*. - Since this is performed batch-wise: is x_k a variable kept between iterations or do you use H(I) for an initial estimate of x_k for the batch? Regarding the classification objective: -since (3) uses the MSE of the mean class-label and the mean-prediction, a dataset where all objects always appear with the exact same amount will not work since than for each image the mean label is identical. - Therefore, the MNIST-easy dataset should be unsolvable for the proposed architecture since every digit occurs exactly once.