MIST: Multiple Instance Spatial Transformer Networks
Authors: Baptiste Angles
Simon Kornblith
Shahram Izadi
Andrea Tagliasacchi
Kwang Moo Yi
Venue:  (2020)

Abstract: We propose a deep network that can be trained to tackle image reconstruction and classification problems that involve detection of multiple object instances, without any supervision regarding their whereabouts. The network learns to extract the most significant top-K patches, and feeds these patches to a task-specific network -- e.g., auto-encoder or classifier -- to solve a domain specific problem. The challenge in training such a network is the non-differentiable top-K selection process. To address this issue, we lift the training optimization problem by treating the result of top-K selection as a slack variable, resulting in a simple, yet effective, multi-stage training. Our method is able to learn to detect recurrent structures in the training dataset by learning to reconstruct images. It can also learn to localize structures when only knowledge on the occurrence of the object is provided, and in doing so it outperforms the state-of-the-art.

Under review as a conference paper at ICLR 2020
MIST MULTIPLE INSTANCE SPATIAL TRANSFORMERS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose deep network that can be trained to tackle image reconstruction and classification problems that involve detection of multiple object instances, without any supervision regarding their whereabouts. The network learns to extract the most significant K patches, and feeds these patches to a spsk-specific network e.g., auto-encoder or classifier - to solve domain specific problem. The challenge in training such a network is the non-differentiable top- K selection process. To address this issue, we lift the training optimization problem by treating the result top-K selection training. Our method is able to learn to detect recurring structures in the training dataset by learning to reconstruct images. It can also learn to localize structures when only knowledge on the occurrence of the object is provided, and in doing so it outperforms the state-of-the-art.
slack variable, resulting in simple, yet effective, multi-stage
INTRODUCTION
Finding and processing multiple instances of characteristic entities in scene is core to many computer vision applications, including object detection (Ren et al. 2015 He et al. 2017 Redmon & Farhadi 2017). pedestrian detection (Dollár al., 2012] /Swart & Andrilukal 2016 Zhang al.
2018a), and keypoint localization (Lowe) 2004 Bay et al. 2008). In traditional vision pipelines, common approach to localizing entities is to select the top responses heatmap and use their locations (Lowe. 2004. Bay et al. 2008 Felzenszwalb et al. 2010). However, this type of approach does not provide a gradient with respect to the heatmap, and cannot directly integrated into neural network-based systems. To overcome this challenge, previous work proposed to use grids (Redmon) ctt al. 2016 He et al. 2017 Detone et al. 2018) to simplify the formulation by isolating each instance (Yi et al. 2016), to optimize over multiple branches (Ono et al., 2018). While effective, these approaches require additional supervision to localize instances, and do not generalize well outside their intended application domain. Other formulations, such as sequential attention (Ba et al., 2015. Gregor et al., 2015 Eslami et al. 2015) and channel-wise approaches (Zhang et al. 2018c) are problematic to apply when the number of instances of the same object is large.
Here, we introduce a novel way to approach this problem, which we term Multiple Instance Spatial Transformer, or MIST for brevity. As illustrated in Figure[ the image synthesis task, given an image, we first compute a heatmap via deep network whose local maxima correspond to locations of interest, From this heatmap, we gather the parameters of the top-K local maxima, and then extract the corresponding collection of image patches via an image sampling process. We process each patch independently with a task-sp task-specific network, and aggregate the network's 's output across patches.
Training pipeline that includes a non -on-differentiable selection/gant /gather operation is non-trin non-trivial. To solve this problem we propose to lift the problem to a higher dimensional one by treating the parameters defining the interest points slack variables, and introduce a hard constraint that they must correspond to the output that the heatmap network gives. This constraint realized by introducing an auxiliary function that creates heatmap given a set of interest point parameters. We then solve for the relaxed version of this problem, where the hard constraint is turned into a soft one, and the slack variables also optimized within the training process. Critically, to incorporate both non-maximum suppression and top-K K selection. We evaluate the performance of our approach for ! the problem of recovering the basis functions that created given texture,
training strategy allows the network

---

Under review as a conference paper at ICLR 2020
classification of handwritten digits in cluttered scenes, and 3 recognition of house numbers in real-world environments. In summary, in this paper we:
introduce the MIST framework for weakly-supervised multi-instance visual learning;
propose an end-to-end training method that allows the use of top-K selection; show that our framework can reconstruct images as parts, as well as detect/classify
instances without any location supervision.
2 RELATED WORK
Attention models and the use of localized information have been actively investigated in the literature. Some examples include discriminative tasks such as fine-grained classification (Sun et al.) (018)). pedestrian detection (Zhang et al. 8a), and generative ones such as image synthesis from natural language (Johnson et al. 2018). They have also been studied in the context of more traditional Multi- ple Instance Learning (MIL) setup (Ilse et al. 2018). We now discuss a selection of representative works, and classify them according to how they deal with multiple instances.
Grid-based methods. Since the introduction of Region Proposal Networks (RPN) (Ren et al. 2015). grid- frid-based strategies have been used for dense image captioning (Johnson et al., 2016), instance segmentation (He et al. (2077), keypoint detection (Georgakis et al.] 2018), and multi-instance object detection (Redmon & Farhadi,, 2017). Recent improvements to RPNs attempt to learn the concept of a generic object covering multiple classes (Singh et al. 2018), and to model multi-scale information (Chao et al. 2018). The multiple transformation corresponding to separate instances can also be densely regressed via Instance Spatial Transformers (Wang et al. 2018), which removes the need to identify discrete instance early in the network. However, all these methods are fully supervised, requiring both class labels and object locations for training.
Heatmap-based methods. Heatmap-based methods have recently gained interest detect fea- tures (Yi et al.] 2016) Ono et al.] (2018] Detone et al.] 2018), find landmarks (Zhang et al. [018cc Merget et al. 2018), and regress human body keypoint (Tekin et al.
2017:Newell et al. 2016). While
it is possible to output a separate heatmap per class (Zhang et al.
2018c Tekin et al. 2017)) ) most
heatmap- badma approaches do not distinguish between instances. Yi al. (2016) re- formulate the problem based on each instance, but in doing so introduce a sharp difference between training and testing. Grids can also be used with heatmaps (Detone al., 2018), but this results in an unrealistic assumption of uniformly distributed detections in the image. Overall, heatmap-based methods excel when the "final" task of the network generate a heatmap (Merget et al. 2018), but are problematic to use an intermediate layer in the presence of multiple instances.
Sequential inference methods. Another way to approach multi-instance problems is to attend to one instance at time in sequentially. Training neural network-based models with sequential attention is challenging, but approaches using policy gradient (Ba et al., 2015) and differentiable attention mechanisms (Gregor et al.) 2015 Eslami et al. 2013) have achieved some success for images comprising small numbers of instances. However, Recurrent Neural Networks (RNN) often struggle generalize sequences longer than the ones encountered during training, and while recent results on inductive reasoning are promising (Gupta et al. 2018), their performance does not scale well when the number of instances is large.
To overcome the acquisition cost of labelled training data, one can transfer knowledge from labeled to unlabeled dataset. For example, Inoue al. (2018) train on a single instance dataset, and then attempt to generalize to multi-instange domains, while Uijlings et al. (2018) attempt to also transfer a multi-class proposal generator to the new domain. While knowledge transfer can be effective, it is highly desirable to devise unsupervised methods such as ours that do not depend on an additional dataset.
Knowledge transfer.
Weakly supervised methods. To further reduce the labeling effort, weakly supervised methods have also been proposed. Wann et al. (2018) learn how to detect multiple instances of single object via region proposals and ROI pooling, while Tang et al. (2018) propose to use a hierarchical setup to refine their estimates. Gao et al. (018 provides an additional supervision by specifying the number of instances in each class, while Oquab et al. (2015) and Zhang al. (018b) localize objects by
looking at the network activation maps (Zhou et al.] 20166 Selvaraju et al. (2017). Shen et al. (2018) introduce adversarial setup, where detection boxes are supervised by distribution assumptions and
2

---

Under review as a conference paper at ICLR 2020
input
detector
heatmap
keypoints
3
o
8
top-k
0
7n
11
g
forward pass
E
backpropagation no gradient
optimal heatmap
loss
task network Tr
encoder
latent space
Figure 1: The MIST architecture - network H,, estimates locations and scales of patches encoded in heatmap h. Patches then extracted via a sampler S, and then fed task-specific network Tr. In this example, the specific task is to re-synthesize the image as a super-position of (unknown and locally supported) basis functions. As top-k operation is non-differentiable, we back pack-propagatt by lifting through G; see Section 3 for details.
classification objectives. However, all these methods still rely on region proposals from an existing method, or define them via hand-tuned process.
MIST FRAMEWORK
We first explain our framework at a high level, and then detail each component in Section A prototypical MIST architecture (Figure [ is composed of two trainable components: ! the first module receives an image as input and extracts K patches, at image locations and scales given by the top K local maxima of heatmap computed by a trainable heatmap network H., with weights r). @ the second module processes each extracted patch with a task-specific network T, whose weights are shared across patches, and further manipulates these signals express task-specific loss Ltask- The two modules are connected through non mon-maximum suppression on the scale-space heatmap output of Hyy followed by a top-K K selection process to extract the parameters defining the patches, which we denote as εk. We then sample patches at these locations through (differentiable) bilinear sampling S and feed them the task module.
The defining characteristic of the MIST architecture is that they are quasi-unsuperviser the only supervision required the number K of patches to extract. However, as we show in Section 5.2 our method is not sensitive to the choice of K during training. In addition, single heatmap for all instances that we extract. In contrast, existing heatmap- methods (Eslami et al. 2015 Zhang al. 2018c) typically rely heatmaps dedicated to each instance. which is problematic when an image contains two instances of the same class. Conversely, we restrict the role of the heatmap network H. to find the "important" areas in given image, without having to distinguishing between classes, hence simplifying learning.
extraction process
Formally, the training of the MIST architecture is summarized by:
minimize T.1)
Lrask (T(E(((((((
where T, 17 are the network trainable parameters. Unfortunately, the patch extractor Ex non- differentiable, it identifies the locations of the top-K local maxima of heatmap and then selects the corresponding patches from the input image. Differentiating this operation provides a gradient with respect to the input, but gradient with respect to the heatmap. Although it is possible to smoothly relax the patch selection operation in the = 1 case (Yi et al. 2016) (i.e. argmax), it is unclear how to generalize this approach to the case of multiple distinct local maxima. It is
3

---

Under review as a conference paper at ICLR 2020
Algorithm 1 Multi-stage optimization for MISTs
Require: K number of patches to extract, Lisk task specific loss, I input image, g : the keypoints to heatmap function, H the heatmap network, 1) parameters the heatmap network, T: the task network, parameters of the task network, εK the top-K operator.
1: function TRAINMIST(I, Liskk)
2:
for each training batch do
3: T <- Optimization step for Tr with Lusk + +11((xa) EK (x(HH,
4: {xx} f Optimization step for {xa] with Linsk + X {1[{xa] - EK (x(H,
5: 17 -- Optimization step for 1 with [[]((x))) H,
6: end for
7: end function
thus impossible to train the patch selector parameters directly by backpropagation. We propose alternative approach to training our model, via lifting.
Differentiable top-K via lifting. The introduction of auxiliary variables to simplify the structure of an optimization problem has proven effective in a range of domains ranging from non-rigid registration (Taylor et al. 2016), to robust optimization (Zach & Bournaoud 2018). To simplify our training optimization, we start by decoupling the heatmap tensor from the optimization (T) by introducing the corresponding auxiliary variables h, as well the patch location variables {xx} from the top-K extractor:
minimize Ctask (T+(((( s.t. h = Hy,(), n.T.h.(xx)
{xt} (h).
We then relax the first constraint above leaat-squares penalty via a Lagrange multiplier A: A/lh (Z) s.t. {xx} Ex (h(
minimize (xx,)
(T,
(3)
As in many methods that use keypoint supervision to regress heatmaps (Cao et al. 2018). we assume that a good heatmap generator & exists that is } s Ex (x((((x)))) We can now rewrite our optimization as:
minimize Crask (T((S((x+))) + Alih - Hb, (Z) s.t. h 9((xx)) (xF.
(4)
We now drop the auxiliary variable h and rewrite our optimization
minimize Lraste (T+(S((xx))) + A[[S(((x)) - H. II)
(x.,(xx)
and then approach the problem by block coordinate descent - where the energy terms not containing
the variable being optimized are safely dropped, and we apply EK to the penalty term of 5
minimize 7,(x
Lrask (T+(8((x)))) + Allxx)) - (X,
minimize ((((xx)))HH((IIE
(7)
To accelerate training, further split 6 into two stages, and alternate between optimizing 7 and } Being based on block coordinate descent, this process converges smoothly, as we show in Section D of the appendix. The summary for the three stage optimization procedure is outlined in Algorithm [ Notice that we are not introducing any additional supervision signal that is tangent to the given task.
4 IMPLEMENTATION
4.1 COMPONENTS
Multiscale heatmap network Hy. Our multiscale heatmap network is inspired by LF-Net (Ono et al. 2018). We improve the network by applying modifications on how the scores are aggregated over scale. The network takes input image I, and outputs multiple heatmaps h of the same size for each scale level s. To limit the number necessary scales, we use a discrete scale space with S scales, and resolve intermediate scales via interpolation; see Section of the appendix for details.
4

---

Under review as a conference paper at ICLR 2020
Top-K patch selection Ek. To extract the top K elements, perform an addition cleanup through a standard non-maxe non-maximum suppression. We then find the spatial locations the top K elements of this suppressed heatmap, denoting the spatial location of the hth element as (æk, Vk), which now reflect local maxima. For each location, we compute the corresponding scale by weighted first order moments (Suwajanakorn al. 2018) where the weights are the responses in the corresponding heatmaps, i.e. E. h (Tk, 88
Generating model for ideal (((x))) For the generative model that turns keypoint and patch locations into heatmaps, we apply simple model where the heatmap is zero everywhere else except the corresponding keypoint locations (patch center); see Section B of the appendix for details.
Patch resampling S. As patch is uniquely parameterized by its location and scale, i.c. X 2k. sk), we can then proceed resample its corresponding tensor via bilinear inter- polation (Jaderberg et al. 2015 Jiang et al. 2019) as [P*} S (I, (xx)).
4.2 TASK-SPECIFIC NETWORKS
We now introduce two applications of the MIST framework. We use the same heatmap network and patch extractor for both applications, but the task-specific network and loss differ. We provide further details regarding the task-specific network architectures in Section C of the appendix.
Image reconstruction auto-encoding. As illustrated in Figure for image reconstruction we append our patch extraction network with shared auto-encoder for each extracted patch. We can then train this network to reconsiruct the original image by inverting the patch extraction process and minimizing
mean squared error between the input and the reconstructed image. Overall, the network is designed to jointly model and localize repeating structures in the input signal. Specifically, we introduce the generalized inverse sampling operation S- (P,,, x.), which starts with an image of all zeros, and places the patch P, at x,. We then all the images together to obtain the reconstructed image, optimizing the task loss
Lrask = ||I Es
(8)
Multiple instance classification. By appending classification network to the patch extraction module, we can also perform multiple instance learning. For each extracted patch Ps we apply a shared classifier network to output ýı E RC, where C' is the number of classes. In turn, these then converted into probability estimates by the transformation ps = (oftmax(yx)) With yı being the one-hot ground-trude -ouun labels of instance I with unit norm, we define the multi-instance classification loss as
L Σ y1 Σ P4
1/2 T,
Lrask
where L is the number of instances in the image. We empirically found this loss be more stable compared to the cross-entropy loss, similar to Mao et al. (2017). Note here that we do not provide supervision about the localization of instances, yet the detector network will automatically learn how to localize the content with minimal supervision (i.e. the number of instances).
5 RESULTS AND EVALUATION
To demonstrate the effectiveness of our framework we evaluate the two different tasks. We first perform quasi-unsupervised image reconstruction task, where only the total number of instances in the scene is provided, i.c. K is defined. We then show that our method can also be applied to weakly supervised multi-instanne classification, where only image-level supervision is provided. Note that, unlike region proposal hased methods, our localization network only relies on cues from the classifier, and both networks are trained from scratch.
5.1 IMAGE RECONSTRUCTION
From the MNIST dataset, we derive two different scenarios. In the MNIST easy dataset, we consider simple setup where the sorted digits confined to a perturbed grid layout; Figure 2 (top).
5

---

Under review as a conference paper at ICLR 2020
MNIST easy
0
3/
0
2
0
1
2
0
3
3
5
3
4
5
45
4
6 7
8
7 8
7
8
8
7 8
MNIST hard
77 7
0
7
7
7
0
6
8
8
6
6
input image
MIST
grid
channel-wise
Eslami et al.
Zhang et al.
Figure 2: MNIST character synthesis examples for (top) the "easy" single instance setup and (bottom) the hard multi-instance setup. We compare the output MISTs to grid, channel-wise sequential
Eslami al. (Eslami al. 2015) and Zhang et al. (Zhang al. 2018c
3
6 8
8
8
6 4 92
6
4
4
6
$
3
3
3 0 05
o
9 2
92
input image
detections
synthesis
input image
detections
synthesis
auto-encoder
auto-encoded
6 8
0
7 3 O
6
O
7
9
6
0
3
8
/
4 4
5
2
6 8
0
7
3
0
6
0
7
9
6
0
3
8
5
2
Figure 3: Two auto-encoding examples learnt from MNIST-hard. In the top row, for each example visualize input, patch detections, and synthesis. In the bottom row we visualize each of the extracted patch, and how it is modified by the learnt auto-encoder.
Specifically, we perturb the digits with a Gaussian noise centered at each grid center, with standard deviation that is equal to one-eighths of the grid width/height. In the MNIST hard dataset, the positions randomized through Poisson distribution (Bridson] 2007), as is the identity, and cardinality of each digit. We allow multiple instances of the same digit to appear in this variant. For both datasets, construct both training and test sets, and the test set is never seen during training.
Comparison baselines We compare our method against four baselines @D the grid method divides the image into a 3 grid and applies the same auto-encoder architecture MIST to each grid location to reconstruct the input image; 2 the channel-wise method uses the same auto-encoder network as MIST, but modify the heatmap network to produce K channels as output, where each channel is dedicated to an interest point. Locations are obtained through a channel soft-argmax Esl16 (Eslami al. 2015) is a sequential generative model; 4
as Zhang al. (2018c); Zhal8 (Zhang et al. 20 is a state-ol-the-art heatmap- based method with channel- -wie strategy for unsupervised learning landmarks. For more details see Supplementary Section
Results for "MNIST easy" As shown in Figure (top) all methods successfully re-synthesize the image, with the exception Eslami al. (Eslami et al. 2015). As this method is sequential, with nine digits the sequential implementation simply becomes too difficult to optimize through. Note how this method only learns to describe the scene with a few large regions. We summarize quantitative results in Table
Results for "MNIST hard" As shown in Figure (bottom), all baseline methods failed to properly represent the image. Only MIST succeeded at both localizing digits and reconstructing the original image. Although the grid method accurately reconstructs the image, it has no concept of individual digits. Conversely, as shown in Figure 3 our method generates accurate bounding boxes for digits even when these digits overlap, and does so without any location supervision. For quantitative results, please see Table i

---

Under review as a conference paper at ICLR 2020
input
Gabor Noise
Gabor Noise
buth
Input Image
leanti desections
Image
detections difls
Figure Inverse rendering of Gabor noise; we annotate correct erroneous localizations.
5 274 00g
9
66 5
8
8
8
MIST
charnal-wisz
Ingut
MIST
channel-wae
Figure 5: Two qualitative examples for detection and classification on our Multi-MNIST dataset.
Finding the basis of a procedural texture We further demonstrate that our methods can be used to find the basis function of procedural texture. For this experiment synthesize textures with procedural Gabor noise (Lagae et al.) 2009). Gabor noise is obtained by convolving oriented Gabor wavelets with Poisson impulse process. Hence, given exemplars of noise, framework is tasked to regress the underlying impulse process and reconstruct the Gabor kernels so that when the two are convolved, we can reconstruct the original image. Figure Iillustrates the results of our experiment. The auto-encoder learned to accurately reconstruct the Gabor kernels, even though in the training images they are heavily overlapped. These results show that MIST is capable of generating and reconstructing large numbers of instances per image, which is intractable with other approaches.
5.2 MULTIPLE INSTANCE CLASSIFICATION
Multi-MNIST - Figure 5 To test our method in a multiple instance classification setup, we rely on the MNIST hard dataset. We compare our method channel-wise, other baselines are designed for generative tasks. To evaluate the detection accuracy of the models, we compute the intersection over union (IoU) between the ground-trute bounding box and the detection results, and assign as a match if the IoU score is over 50%. We report the number of correctly classified matches in Table as well as the proportion instances that are both correctly detected and correctly classified. Our method clearly outperforms the channel-wise strategy. Note that, even without localization supervision, our method correctly localizes digits. Conversely, the channel-wise strategy fails to learn. This is because multiple instances of the same digits present in the image. For example, in the Figure (right, we have three sixes, two zeros, and two nines. This prevents any of these digits from being detscted/classified properly by channel -winnel--iieeeee approach.
Sensitivity to Table 3
To investigate the sensitivity to the correctness K during training, we train with varying number of of instances and test with ground-truth K. For example, with K 6,
MIST Grid Ch-wise Esl16 Zha18
MIST Ch.-wise Supervised
Instances during train 3p
MNIST casy
.038
.039
.042
.100
.169
IOU 50% 97.8%
25.4%
99.6%
{9] (6. 7.77889
92.2%
MNIST hard
.154
.191
Classif.
75.5%
98.7%
Gabor
.095
Both 97.5%
24.8%
98.6%
6,4.5,667,8..0
90.8%
Table 1: Reconstruction enror (root mean square error). Note that Grid does not learn any notion of digits.
Table 2: Instance level detec- tion and classification perfor- mance "MNIST hard".
Table 3: Sensitivity exper- iment of K on the MNIST hard dataset.
7

---

Under review as a conference paper at ICLR 2020
15
30
31
MIST
Supervised
ApIoU=.00
82.6%
65.6%
A ploU==50
76.5%
62.8%
9.
33
94
A ploUl.60
63.7%
59.8%
A ploUs.70
42.7%
51.9%
15
15
A ploU=.80
19.9%
34.6%
ApioUs.90
4.2%
11.0%
Figure 6: Qualitative SVHN results.
Table 4: Quantitative SVHN results.
where the ground truth could be anything within [3, 5, 6, 7,8,9) - we mark the K used during training with bold. Our method still is able to give accurate result with inaccurate K during training. Knowing the exact number of objects is not a strict requirement at test time, as our detector generates heatmap for the entire image regardless of the K was trained with. Note that while in theory sequential methods are free from this limitation, in practice they are able to deal with limited number of objects (e.g. up to three) due to their recurrent nature.
SVHN - Figure and Table We further apply MIST to the uncropped and unaligned Street View House Numbers dataset (Netzer et al., 2011). Compared to previous work that has used cropped and resized SVHN images (e.g. (Netzer et al.) 2011 Ba et al. 2015; Goodfellow et al., 20044 [aderberg ct al. 2015)), this evaluating setting is significantly more challenging, because digits can appear anywhere in the image. We resize all images to 60 x 240, use only images labeled as containing 2 digits, and apply MIST at single scale. Although the dataset provides bounding boxes for the digits, ignore these bounding boxes and use only digit labels supervision. During testing, images with small bounding boxes (< 30 pixels in height). We report results in terms of Aploll=.X where X is the threshold for determining detection correctness. With IoU =0, we refer "pure" classification task (i.e. no localization). As shown, supervised results provide better performance with higher thresholds, but MIST performs even better than the supervised baseline for moderate thresholds. We attribute this to the fact that, by providing direct supervision on the location, the training focuses too much on having high localization accuracy.
exclude
CONCLUSION AND FUTURE WORK
In this paper, we introduce the MIST framework for Both these tasks are based on localized analysis of the image, yet we train the network without providing any localization supervision. The network learns how to extract patches on its own, and these patches are then fed to task-specific network to realize an end goal. While at first glance the MIST framework might appear non-differentiabll, we show how via lifting they can be effectively trained in an end-to-end fashion. We demonstrated the effectiveness of MIST by introducing variant of the MNIST dataset, and demonstrating compelling performance in both reconstruction and classification. We also show how the network can be trained to reverse engineer procedural texture synthesis process. MISTs are first step towards the definition of optimizable image-decome mmae-deecompositonn networks that could be extended to number of exciting unsupervised learning tasks. Amongst these, we intend to explore the applicability of MISTs to unsupervised detection/localization objects, facial landmarks, and local feature learning.
image reconstration/claasffccaooo
Even though our method is a significant step forward from existing methods, it is not complete and has much potential for improvement. While method is robust to K during training as shown in Section 5.2 and K can be determined dynamically during test time, it would beneficial to automatically determine K through varying aspect ratios similar to RPNs. Reconstruction could consider a global reconstructor to deal with complex images. These are parts of our future work towards MIST in the wild.
additional network. Patches could also be refined to have
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
L. Ba, V. Mnih, and K. Kavukcuoglu. Multiple Object Recognition With Visual Attention. In ICLR, 2015.
H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool. SURF: Speeded Up Robust Features. CVIU, 10(3): 346-359, 2008.
R. Bridson. Fast Poisson Disk Sampling in Arbitrary Dimensions. In SIGGRAPH sketches, 2007.
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Openpose: realtime multi-person 2d pose estimation using part affinity fields. arXiv preprint arXiv:1812.08000 2018.
Y-W. Chao, S. Vijayanarasimhan, B. Scybold, D. A. Ross, J. Deng, and R. Sukthankar. Rethinking the Faster R-CNN Architecture for Temporal Action Localization. In CVPR, 2018.
D. Detone, T. Malisiewiez, and A. Rabinovich. Superpoint: Self-Supervised Interest Point Detection and Description. CVPR Workshop on Deep Learning for Visual SLAM, 2018.
P. Dollár, C. Wojek, B. Schiele, and P. Perona. Pedestrian Detection: An Evaluation of the State of the Art. PAMI, 34(4):743-76) 2012.
M. A. Eslami, N. Heess, T. Weber, Tassa, D. Szepesvari, K. Kavukcuoglu, and G. E. Hinton. Attend, Infer, Repeat: Fast Scene Understating with Generative Models. In NeurIPS, 2015.
P.F. Felzenszwalb, R.B. Girshick, D. McAllester, and D. Ramanan. Object Detection with Discrimi- natively Trained Part Based Models. PAMI, 3209)16277-1445, 2010.
M. Gao, A. Li, I. Morariu, and L. S. Davis. C-WSL: Coung-guided Weakly Supervised Localization. In ECCV, 2018.
G. Georgakis, S. Karanam, Z. Yu, Ernst, and Košecká. End-to-end Learning of Keypoint Detector and Descriptor for Pose Invariant 3D Matching. In CVPR, 2018.
I. Goodfellow, Y. Bularov, J. Ibarz, S. Aroodd, and V. Shet. Multi-digit Number Recognition from Street View Imagery using Deep Convolutional Neural Networks. ICLR, 2014.
K. Gregor, Danihelka, A. Graves, D. Rezende, and D. Wierstra. DRAW: A Recurrent Neural Network For Image Generation. In ICML, 2015.
A. Gupta, A. Vedaldi, and A. Zisserman. Inductive Visual Localization: Factorised Training for Superior Generalization. In BMVC, 2018.
K. He, X. Zhang, R. Ren, and J. Sun. Delving Deep into Rectifiers: Surpassing Human- Performance Imagenet Classification. ICCV, 2015.
K. He, G. Gkioxari, P. Dollar, and Girshick. Mask R-CNN. In ICCV, 2017.
Maximilian Ilse, Jakub M Tomezak, and Max Welling. Attention-basse Deep Multiple Instance Learning. In ICML, 2018.
N. Inoue, R. Furuta, T. Yamasaki, and K. Aizawa. through Progressive Domain Adaptation. In ECCV, 2018.
Weakly-Supervised Object Detection
M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. Spatial Transformer Networks, In NeurIPS, pp. 2017-2025, 2015.
Wei Jiang, Weiwei Sun, Andrea Tagliasacchi, Eduard Trulls, and Kwang Moo Yi. Linearized multi-sampling for differentiable image transformation. arXiv Preprint, 2019. URL http! //arxiv. /arxiiv.org/aas/190...
Johnson, A. Karpathy, and L. Fei-fei. Densecap: Fully Convolutional Localization Networks for Dense Captioning. In CVPR, 2016.
J. Johnson, A. Gupta, and I.. Fei-fei. Image Generation from Scene Graphs. In CVPR, 2018.

---

Under review as a conference paper at ICLR 2020
Ares Lagae, Sylvain Lefebvre, George Drettakis, and Philip Dutré. Procedural noise using sparse gabor convolution. TOG, 2009.
D.G. Lowe. Distinctive Image Features from
Keypoints. 20(2), 2004.
X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley. Least Squares Aenerative Adversarial Networks. CVPR, 2017.
D. Merget, M. Rock, and G. Rigoll. Robust Facial Landmark Detection via Fully-Conlolutional Local-Global Context Network. In CVPR, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. Deep Learning and Unsupervised Feature Learning Workshop, NeurlPS, 2011.
A. Newell, K. Yang, and J. Deng. Stacked Hourglass Networks for Human Pose Estimation. In ECCV, 2016.
Y. Ono, E. Trulls, P. Fua, and K. M. Yi. Lf-Net: Learning Local Features from Images. In NeurlPS, 2018.
M. Oquab, L. Bottou, Laptev, and Sivic. Is object localization for free? weakly-supervised learning with convolutional neural networks. In CVPR, 2015.
Redmon and A. Farhadi. YOLO 9000: Better, Faster, Stronger. In CVPR, 2017.
Redmon, S. Divvala, R. Girshick, and A. Farhadi. You Only Look Once: Unified, Real-Time Object Detection. In CVPR, 2016.
S. Ren, K. He, Girshick, and Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In NeurIPS, 2015.
R. R. Selvaraju. M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. In ICCV, 2017.
R. Sewart and M. Andriluka. End-to-End People Detection in Crowded Scenes. In CVPR, 2016.
Yunhan Shen, Rongrong Ji, Shengchuan Zhang, Wangmeng Zuo, and Yan Wang. Generative adversarial learning towards fast weakly supervised detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5764-5773, 2018.
B. Singh, Li, Sharma, and L. S. Davis. R-FCN-3000 at 30fps: Decoupling Detection and Classification. In CVPR, 2018.
M. Sun, Y. Yuan, F. Zhou, and E. Ding. Multi-Attention Multi-Class Constraint for Fine-grained Image Recognition. ECCV,2011
S. Suwajanakorn, N. Snavely, J. Tompson, and M. Norouzi. Discovery of Latent 3D Keypoints via End-To-End Geometric Reasoning. NeurIPS, 2018.
P. Tang, X. Wang, A. Wang. Y. Yan, W. Liu, J. Huang, and A. Yuille. Weakly Supervised Region Proposal Network and Object Detection. In ECCV, 2018.
Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, Arran Topalian, Erroll Wood, Sameh Khamis, Pushmeet Kohli, Toby Sharp. Shahram Izadi, Richard Banks, Andrew Fitzgibbon, and Jamie Shotton. Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences. TOG, 2016.
B. Tekin, P. Marquez-neila, M. Salzmann, and P. Fua. Learning to Fuse 2D and 3D Image Cues for Monocular Body Pose Estimation. ICCV, 2017.
R. R Uijlings, S. Popov, and V. Ferrari. Revisiting Knowledge Transfer for Training Object Class Detectors. In CVPR, 2018.
10

---

Under review as a conference paper at ICLR 2020
F. Wan, P. Wei, J. Jiao, Z. Han, and Q. Ye. Min-Entropy Latent Model for Weakly Supervised Object Detection. CVPR, 2018.
E. Wang, L. Zhao, X. Li, X. Wang, and D. Tao. Geometry-Aware Scene Text Detection with Instance Transformation Network. In CVPR, 2018.
K. M. Yi, E. Trulls, V. Lepetit, and P. Fua. LIFT: Learned Invariant Feature Transform. In ECCV, 2016.
C. Zach and G. Bournooud. Descending, Lifting Smoothing: Secrets of Robust Cost Opimization. In ECCV, 2018.
S. Zhang, J. Yang, and B. Schiele. Occluded Pedestrian Detection Through Guided Attention in CNNs. In CVPR, 2018a.
X. Zhang. Y. Wei, Kang, Y. Wang, and T. Huang. Self-produced Guidance for Weakly-supervised Object Localization. In ECCV, 2018b.
Y. Zhang, Y. Gui, Y. Jin, Y. Luo, Z. He, and H. Lee. Unsupervised Discovery of Object Landmarks as Structural Representations. CVPR, 2018c.
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning Deep Features for Discrimi- native Localization. In CVPR, 2016.

---

Under review as a conference paper at ICLR 2020
Appendix
A MULTISCALE HEATMAP NETWORK
Our multiscale heatmap network is inspired by LF-Net (Ono et al. 2018). We employ fully convolutional network with (shared) weights at multiple scales, indexed by = S, on the input image Z. The weights 7 across scales are shared so that the network cannot implicitly favor a particular scale. To do so, we first downsample the image to each scale producing I, execute the network H. on it, and finally upsample to the original resolution. This process generates a multiscale heatmap tensor h {he} of size H x W x S where h2 (I,( and И is the height of the image and W is the width. For the convolutional network we use 4 ResNet blocks (He et al. 2015), where each block is composed of two 3 x 3 convolutions with 32 channels and relu activations without any downsampling. We then perform a local spatial softmax operator (Ono et al.] 2018) with spatial extent of 15 15 to sharpen the responses. Then we further relate the scores across different scales by performing a "softmax pooling" operation over scale. Specifically, if we denote the heatmap tensor after local spatial softmax as h = [h.). since after the local spatial softmax (Z.) is already an "exponentiated" signal, we do weighted normalization without an exponential, i.e. h' = E. (h./ E., (ha + e)), where c = 10-6 is added to prevent division by zero.
Comparison to LF-Net. Note that differently from LF-Net (Ono et al. 2018 we do not perform a softmax along the scale dimension. The scale-wise softmax in LF-Net is problematic as the computation for softmax function relies on the input to the softmax being unbounded. For example, in order for the softmax function to behave as a max function, due to exponentiation, is necessary that one of the input value reaches infinity (i.e. the value that will correspond to the max), or that all other values to reach negative infinity. However, at the network stage where softmax is applied in (Ono et al. 2018), the score range from zero to one, effectively making the softmax behave similarly to averaging. Our formulation does not suffer from this trawback.
B GENERATIVE MODEL FOR THE HEATMAP
As discussed previously in Section for the generative model that turns keypoint and patch locations into heatmaps, we apply a simple model where the heatmap is zero everywhere else except the corresponding keypoint locations (patch center). However, as the optimized patch parameters are no longer integer values, we need to quantize them with care. For the spatial locations we simply round to the nearest pixel, which at most creates quantization error of half pixel, which does not cause problems in practice. For scale however, simple nearest-neighhor assignment causes too much quantization error as our scale-space is sparsely sampled. We therefore assign values the two nearest neighboring scales in a way that the center of mass would be the optimized scale value, making sure {x+} Ex (9 (9((x))) (x))))
C IMPLEMENTATION DETAILS
MIST auto-encoder network. The input layer of the autoencoder is 32 x 32 x C where C' is the number of color channels. We use 5 up/down-samplin non-bottleneck ResNet v1 blocks (He et al. 2015) and each ResNet block uses number of channels that doubles after each downsampling step. ResNet blocks uses 3 x 3 convolutions of stride 1 with ReL.U activation. For downsampling we use 2D max pooling with 2 2 stride and kernel. For upsampling we use 2D transposed convolutions with 2 x stride and kernel. The output layer uses a sigmoid function, and we use layer normalization before each convolution layer.
levels. Each level is made of standard
MIST classification network. We re-use the same architecture as encoder for first the task and append dense layer map the latent space to the score vector of our 10 digit classes.
Baseline unsupervised reconstruction methods. To implement the Eslami et al. (Eslami et al. 2015 baseline, we use a publicly available reimplementation | We note that Eslami et al. (Eslami

---

Under review as a conference paper at ICLR 2020
1.60
0.0450
0.0350
1.40
0.0250
1.20
0.0150
5.000e-3
1.00
0.000
30.00k
80:00K
90.00%
(a) Link
(b) [[(((xx)) (T)
(c) H.y (I)
Figure 7: Evolution of the loss and heatmap during training. (a) the task (classification) loss Ltask }) - H,, (Z) 12 for each iteration. In (c), we show how the heatmap evolves for example image from the SVHN dataset during training (from top bottom)-
and (b) the heatmap related loss [|S( (x((xx})
brighter the intensity, higher the scores; see Section
et al. 2015) originally applied their model to a dataset consisting of images of 0, 1, or 2 digits with equal probability. We found that the model failed to converge unless it was trained with examples where various number total digits exist, so for fair comparison, we populate the training set with images consisting of all numbers of digits between 0 and 9. For the Zhang et al. (hhan et al. 2018c baseline, we use the authors' implementation and hyperparameters.
D CONVERGENCE DURING TRAINING
As is typical for neural network training, our objective is non-convex and there is no guarantee that a local minimum found by gradient descent training is global minimum. Empirically, however, the optimization process is stable, as shown in Figure [ We reporpt both the training losses Ltask and [[9((x))) H. [1, and visualize how the heatmap evolves for an example image from the SVHN datset. We visualize the heatmap by showing brighter intensity values for higher scores. Each image normalized separately with their own maximum and minimum values for better visualization. In Figure 7 (c) top, early in training, one can see that keypoints are detected at random locations as the heatmaps are generated by networks with randomly initialized weights. However, as training continues, keypoints that, by chance, land on locations nearby the correct object (e.g. numbers) for certain samples random batch, and become reinforced. Ultimately, as shown (c) middle and bottom, MIST learns to detect these locations, thus learns to produce a peaky heatmap, and perform the task interest. Note that this is unsurprising, as our formulation a lifted version of this loss to allow gradient- based training.
E
NON-UNIFORM DISTRIBUTIONS
4
S
8
3
5
Figure 8: Examples with uneven distributions of digits.
Although the images show in Figure our method does not require the keypoints to be evenly spread. As shown in Figure g our method is able to successfully learn even when the digits are placed unevenly. Note that, our detector is fully
2 involve small displacements from uniformly spaced grid,
2

---

Under review as a conference paper at ICLR 2020
convolutional and local. it cannot learn the absolute location of keypoints. In fact, we weakened the randommess of the locations for fairness against (Zhang et al. 2018c), which is not designed to deal with severe displacements.


Reviews:
[{"review_id": "SkxcvDuXcH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for multi-instance object classification and reconstruction that does not require any location-based supervision. The main contribution is the introduction of a differentiable top-K region proposal that allows to train the whole model with only a supervision of the total number of instances (and their class) in the image. They test the performance of their method in simple visual tasks like cluttered MNIST, street digit recognition and finding the basis of procedural texture generation.\n\nThe paper is well written and motivated. The proposed method is clear and well formalized. Their reported results seem substantially better than the baselines they compare against. The additional experiments in their Appendix Fig. 7, analyzing the evolution of the heatmap loss, is interesting, although I think the heatmap visualization could be improved to better understand what the model is learning across iterations (improve legend to include at what iteration the heatmap is taken, give on what image this is evaluated).\n\nUnfortunately, their tasks seem quite easy, and it is hard to assess the impact of their method when working with more real-world data-sets, where the number of instances of every class is more loosely defined (we could always describe more objects in a real image from the COCO dataset for example). It seems of great importance to evaluate the limitations of their method in this direction, as the source of supervision might be too weak in the cases where the generated dataset might not have all the combinations of number of instances per image (as the cluttered MNIST has given that it’s procedurally generated). The results on SVHN are a little bit confusing, and it’s unclear what the “Supervised” method is, specially knowing that there are available methods that do obtain much better performance on this task using all the supervision available. It should also be better explained why the performance drops so drastically when the IoU threshold is increased. Finally, the texture generation experiments are very hard to interpret and are even further from real-world tasks.\n\nGiven my concerns on how this unsupervised approach can scale to real-life datasets, I suggest a weak reject, but I think the  proposed method has some interest for the community and I strongly encourage the authors to provide further evidence of performance of their method on more complex vision tasks.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJge34RuOB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The article introduces the novel MIST architecture which tries to solve the problem of multiple-instance classification and image generation from multiple objects. It employs two submodels, where the first generates a heatmap of intereting region and the second model is a task-specific model that works on image-patches, for example a classifier or an autoencoder. Both models are connected by a patch-extraction routine. The main contribution of this paper is to provide a way to propagate errors through this non-differentiable patch-extraction scheme. This is done by introducing slack-variables.\n------------------\n\nThe paper is overall relatively easy to follow and the results are very good. However, it suffers from the fact that it does not differentiate between model-architecture and the overall approach. While the main contribution is described in Section 4, the paper spends a lot of space beforehand to introduce the task-dependent models as well as the heatmap architecture - things that i can imagine will vary a lot in different applications. The real important part is how to train the model and this is unfortunately only half described. A good deal of abstraction from the network architecture would have made the paper a lot better.  Further, I think that the loss-function for the classification task does not work in the general case.\n\n\nOn my first read-through, i completely misunderstood Section 4. Here is an unsorted list of issues i had with this:\n- since E_K is not truly invertible, writing the approximate inverse as E_K^{-1} is misleading. \n- It might help to stress that you treat {x_k} as continuous and the sampling as differentiable.\n- In (7) it would be better to explicitly write E_K^{-1}(x_k) instead of introducing \\bar{h}. The line below is not clear.\n- It is also misleading, because the choice of \\bar{h}=E_K^{-1}(x_k) is not the minimizer of (5) given that all other variables are fixed. You can see this by observing that assuming that when {x_k}=E_K(H(I)) holds, we can choose all other pixels\nto be exactly the value returned by H(I).\n- I am not sure where the alternating part comes from because this usually involves taking your solution from (7) and feeding it into (6). \n- I am pretty sure that in (6)+(7), as well as lines 3+6 of the algorithm, you actually don't want to optimize for tau or eta from scratch but only perform a single SGD step. I think that is what you are doing, but right now it is written as \"find a complete new model for each batch\".\n- Since this is performed batch-wise: is x_k a variable kept between iterations or do you use H(I) for an initial estimate of x_k for the batch?\n\nRegarding the classification objective:\n-since (3) uses the MSE of the mean class-label and the mean-prediction, a dataset where all objects always appear with the exact same amount will not work since than for each image the mean label is identical.\n- Therefore, the MNIST-easy dataset should be unsolvable for the proposed architecture since every digit occurs exactly once.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "B1gBC5jEcr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a way to solve a localization task for in images without providing any localization supervision. This allows tackling image reconstruction and classification problems that involve multiple object instances. At the core of the approach is a suggestion to resolve the non-differentiable top-k selection process: It is done by introducing axuilliary variables which allows the derivation of an alternating optimization procedure for their framework.\n\nI like the paper and I think the problem it's tackeling is at the core for all many important image understanding tasks. When thinking about reasons that speak against the paper, I'm mostly concerned about the fact that the idea is not yet fully worked out:\n* The biggest unresolved aspect is the number 'k' of instances needs to be provided. Assuming real-world relevant instance detection tasks, images in the test set never have 'k' provided.\n* All patches need to be squared -- again, for real-world tasks this is not the case.\n* In the case of image reconstruction, what is the 'reconstruction error' exactly? Assuming a scene with background clutter, how would this work here? It seems that the demonstrated examples have a black background so working additively is not a problem.\n\nThe conclusion mentions 'a first step', so I understand that the authors may be aware of the shortcomings. As such, I'd like to have limitations (and potential resolutions) to be pointed out in the paper.\n\nWith respect to related work, I'm missing \"Attention-based Deep Multiple Instance Learning\", by Ilse et al. I'm not certain if it could be applied to the reconstruction task, but it seems that it should be a baseline for the classification task. When talking about weak supervision, the paper \"Weakly supervised object recognition with convolu-tional neural networks\" by Oquab should also be mentioned in the related work section.\n\nOn a more detailed level, Figure 1 has no symbols for the newtork H_n and T_tau.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two reviewers are negative on this paper while the other one is slightly positive. Overall, this paper does not make the bar of ICLR. A reject is recommended.
Decision: Reject