Weakly Supervised Virus Capsid Detection with Image-Level Annotations in Electron Microscopy Images
Authors: Hannah Kniesel
Leon Sick
Tristan Payer
Tim Bergner
Kavitha Shaga Devan
Clarissa Read
Paul Walther
Timo Ropinski
Pedro Hermosilla
Venue: ICLR 2024 poster (2024)
Keywords: Weakly Supervised Object Detection
Limited Annotation Time
Bounding Box Regression
Electron Microscopy
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: We propose an optimization strategy with shrinking receptive field to extract virus capsids directly by bounding box regression from image level annotations.

Abstract: Current state-of-the-art methods for object detection rely on annotated bounding boxes of large data sets for training. However, obtaining such annotations is expensive and can require up to hundreds of hours of manual labor. This poses a challenge, especially since such annotations can only be provided by experts, as they require knowledge about the scientific domain. To tackle this challenge, we propose a domain-specific weakly supervised object detection algorithm that only relies on image-level annotations, which are significantly easier to acquire. Our method  distills the knowledge of a pre-trained model, on the task of predicting the presence or absence of a virus in an image, to obtain a set of pseudo-labels that can be used to later train a state-of-the-art object detection model. To do so, we use an optimization approach with a shrinking receptive field to extract virus particles directly without specific network architectures. Through a set of extensive studies, we show how the proposed pseudo-labels are easier to obtain, and, more importantly, are able to outperform other existing weak labeling methods, and even ground truth labels, in cases where the time to obtain the annotation is limited.

Published as a conference paper at ICLR 2024
WEAKLY SUPERVISED VIRUS CAPSID DETECTION WITH AMAGE-LEVEL ANNOTATIONS IN ELECTRON MICROSCOPY IMAGES
Hannah Kniesel, Leon Sick, Tristan Payer, Tim Bergner, Kavitha Shaga Devan, Clarissa Read, Paul Walther, Timo Ropinski Ulm University
Pedro Hermosilla
TU Vienna
ABSTRACT
Current state-of-the-art methods for object detection rely annotated bounding boxes of large data sets for training. However, obtaining such annotations is expensive and require up to hundreds of hours of manual labor. This poses challenge, especially since such annotations can only be provided by experts, as they require knowledge about the scientific domain. To tackle this challenge, we propose a domain-specift weakly supervised object detection algorithm that only relies image-level annotations, which are significantly easier to acquire. Our method distills the knowledge of a pre-trained model, on the task of predicting the presence or absence of a virus in an image, obtain a set of pseudo-label that can be used to later train state-of-the-an object detection model. To do so, we use an optimization approach with a shrinking receptive field extract virus particles directly without specific network architectures. Through a set of extensive studies, we show how the proposed pseudo-labels are easier to obtain, and, more importantly, are able to outperform other existing weak labeling methods, and even ground truth labels, in cases where the time to obtain the annotation is limited.
INTRODUCTION
Deep learning algorithms rely on large data sets for training a model to perform a complex task. However, annotating such large data sets usually requires person to analyze each data point and label accordingly, resulting in a time-consuming process. In particular, detecting particles in Electron Microscopy (EM) images is extremely costly, since this annotation process has to be performed by an expert, which usually results in small data sets that are not well suited to train deep models. Since particle detection is key step in several scientific studies such as the analysis of the formation of infectious virions (Shaga Devan et al. 202I). catalytic investigation (Nartova et al. 2022). analysis of multi tistue histology images (Graham al. 2019), preclinical trials single particle reconstruction (Sigworth) 2015; [Shaikh et al. 2008), these studies could benefit significantly from automated object detection methods tailored towards particle detection. However, particle detection comes with additional challenges. First, it requires the need for experts to annotate the data. Second, since these areas are active research fields, they require quick adaption of the detection model to new virus mutants, particles, or imaging modalities.
To address this issue, weakly supervised algorithms (Oquab et al.) 2015 Bency et al. 2016 Zeng al. 2019 Gao et al. [202I) rely secondary task, usually classification, for which annotations easy to obtain. Then, to solve the main task of object detection, weakly supervised algorithms usually set of bounding box candidates or Region of Interests (ROIs), obtained with selective search strategy (Uijlings et 2013).. which are later filtered based on the classification score a pre-trained classification model. However, the accuracy of such object detection models highly depends on the quality and quantity of such ROI candidates (Girshick 2015), since a direct regression of bounding
Conresponning author email: hannah. knicsel @uni-ulm.de

---

Published as a conference paper at ICLR 2024
boxes based on the pre-trained classifier is not possible. This is why current weakly supervised methods in the field of particle detection in EM usually rely on more fine- seee-graind (and expensive) object-level annotations rather than image-level annotations (Devan et al. 2019 Matuszewski &
Sintorn 2019).
In our work, however, reduce annotation time by exploiting image-level annotations for virus capsid detection in EM images, by proposing a distillation method, that is able to regress the bounding box position directly from classifier pre-trained on image-level annotations. To this end, we combine a Gaussian masking strategy and domain-specific knowledge about the virus size and its shape, in order localize virus capsids on the images using an optimization algorithm informed by the pre- trained classifier. To propagate the gradients over the full input image, we initialize the Gaussian mask with a large standard deviation and progressively reduce during the optimization procedure, similar to the training mechanism used in score-based generative models (Song & Ermon] (019). By exploiting this novel approach, we are able to perform accurate particle detection, which is robust with respect to the variance of the initial ROIs. Since our approach is only relying on image level labels, the collection of a new data set for a newly discovered virus mutant or a new imaging modality can be done efficiently. To evaluate our methods, we first conducted user study comparing different types of labels which results show that our labels are easier to obtain and less prone to errors. Then, compare our approach to other weakly supervised and fully supervised approaches on five different virus types. Our results show that our approach, solely relying on image labels, does not only outperform other weakly supervised approaches but even fully supervised ones when allocating the same annotation time. Thus, within this paper, we make the following contributions:
We propose domain-specific gradient- optimization algorithm, which exploits pre-trained classifier and a Gaussian masking strategy, in order to detect virus capsids.
We introduce a class activation map guided initialization strategy to significantly reduce the computational overhead of the underlying optimization process.
We conducted a user study comparing different label types and show that image-level annotations are easier and faster to obtain, and more robust to annotation errors.
We show that our approach outperforms other weakly as well as fully supervised methods given the same annotation time.
2 RELATED WORK
Weakly supervised object detection. The requirement for fast annotation times is long-standing problem in several fields, which has made Weakly Supervised Object Localization (WSOL) and Weakly Supervised Object Detection (WSOD) an active area of research in the last few (quab et al. (015)) introduced a CNN architecture that can be moved over the input image during inference time in sliding window fashion to perform WSOD. Bazzani et al. (2016) used selective search strategy (Uijlings et al. 2013) to draw a set of bounding box candidates on the image for which the score of each box was obtained from a pre-trained classification model. [Bency et al. (2016) used hierarchical search to reduce the number of bounding box candidates and the feature map of a deep network to find the location of the object of interest. Bilen& & Vedaldi (2016) introduced Multiple Instance Learning (MIL) in an end-to-end trainable fashion. In MIL, training instances are organized in bags such that a positive bag contains at least one object of interest and negative bag does not contain any object of interest. There are many works to follow and improve upon the MIL approach
(Kantorov et al. 2016 Diba al. 2017 Tang et al.) 2017)] Cheng al. 2020:] Huang et al., 2020 Ren et al. 2020 Zeng et al. 2019 Seo et al. (002). Among these methods, the approach by Zeng jet al. (2019) stands out, as it includes refinement of the ROI candidates in the loss to obtain more accurate bounding box predictions. A similar idea for bounding box refinement was also explored by Dong et al. (2021). However, they rely on additional data sets to learn bounding box modifiers that can be applied to the data set with weak labels. Contrary to these approaches, we propose to directly regress the bounding box of the objects from the pre-trained classifier without the need for supervised tre-training different data sets, while further being robust initial ROI proposals computed by selective search (Ujlings et al. 2013) or similar methods. This makes it possible to use a smaller amount of initial ROI candidates reduce computational cost.
2

---

Published as a conference paper at ICLR 2024
In another line of work, researchers have investigated how to obtain the object's bounding box from Class Activation Maps (CAMs) of pre-trained deep neural networks (Zhou et al. (015). However, such methods have difficulties identifying specific discriminative parts of objects. To address this problem. [Singh & Lee (017)) tried to improve the quality of CAMs by randomly masking patches of the image during the training phase, to not only rely on specific features of the object during predictions. This concept was later extended by Zhang et al.] (2018) and Choe & Shim (2019), whereby both methods facilitate attention maps to mask certain regions of the image during training. Later, (uee et al. (2019) proposed a regularization loss and hierarchical classification loss to enforce discrepancy the feature maps, which allows the classifier to attend to the full extent of objects. More recently, Gao et al. (2021) investigate the attention mechanism of vision transformers (Dosovitskiy jet al. (020) to guide WSOD. Alternatively, Meng et al. (202T) aim to better capture the full object through object-aware and part-aware attention. With similar goal. WWii et al. (2022) propose mechanism that enforces inter-class feature similarity and intra-class appearance consistency, while Xu et al. 2022) use class-specific foreground and background context embeddings which act as class centroids during inference to form a more complete activation map. However, those methods still suffer from correctly identifying the full extent of objects and/or rely specific architectures which might not be suited for small data sets, as they usually occur in EM scenarios.
The most similar work to the one proposed in this paper is the work from Lu et al. (2020). They propose a secondary network to predict the geometric parameters of a mask, center and radius of an ellipsoid, which then input to another network that predicts the final mask. They show in their experiments that using the predicted geometry directly leads to poor performance. In this work instead, we show that no neural network is necessary to predict or transform such mask and by using similar ideas to the ones used in score-based generative models (Song & Ermon 2019) we can optimize directly the location of the object. Additionally, we introduce a method that is able to detect multiple instances of the same object in an image, which is not possible with the approach Lu et al. (2020).
Virus particle detection in EM. Despite the progress in WSOD in standard computer vision, its application in EM images is limited, even though the need for fast annotations in EM is of special interest. This is likely the case since low Signal to Noise Ratio (SNR) in EM images can limit the capacity of methods well performing on other imaging modalities. Further, EM data usually contains a high number of instances of the same object in one image, which are hard detect weakly supervised setup with image-level labels only. To solve similar problem in medical imaging, Dubost let al.)) introduced regression models for WSOD in 3D MRI data. In the EM domain Huang et al.. (2022) introduced a weakly supervised learning schema for finding the location of proteins in cryo- EM. However, in their weakly supervised setup, they still require a small amount of labeled training data. For detecting virus particles in EM in weakly supervised fashion Devan et al. (2019) trained a classification model on a small set of annotated bounding box crops of the HCMV nuclcocapsids. They then use a weakly supervised approach similar to Oquab et al. (2015) to detect virus particles based on their classifier. However, this approach requires images of a single virus, instead of random crops with and without virus particles. The same authors later explore the improvement of supervised virus detection by augmenting training data by a generative adversarial network (Shaga Devan et al.. 2021). One of the most promising works in weakly supervised detection and segmentation probably originates from Matuszewski & Sintorn (2018). They introduced a minimal annotation strategy for the segmentation of microscopy images: annotations of the center or center line of target object are used to generate segmentation masks. The labels for the object of interest were generated by dilating each particle annotation with a disk of 0.7x average known size of the target object. The background label, on the other hand, was created by dilating the center annotations with 1.5x the average known size of the target object. Later, the same authors (Matuszewski & Sintorn 2019) made use of the minimal labels to train an improved U-Net architecture (Ronneberger et al. 2015) for virus recognition in EM. However, all of the mentioned methods rely on more inn-ggrined annotations and/or the use of compute inefficient sliding window to obtain ROI candidates to locate the particles.
3 METHOD
Our method expects as input an EM image, I E R If the expected virus radius r, and a classifier C: WW -> R. We pre-train the classifier on image-level annotations, such that it can classify
3

---

Published as a conference paper at ICLR 2024
Input
Intieization
Opamization
Detection
Viuu Removal
Capsi
Iniielization
Virui Removel
Optiizztton
2 Capsd
Detection
Stop?
Stop?
Figure I: Left: Overview of our weakly supervised virus detection approach, working in an iterative fashion until a stopping criteria met. Right: Detailed description of our approach visualizing different steps for the detection of virus. For the Initialization of the particle position po we compute a CAM obtained through GradCAM (Selvaraju et al. (2017)) and place po at the position of the highest CAM value. During Optimization the position pt is iteratively refined, guided by the classifier output and Gaussian mask with decreasing standard deviation centered at Pt. A Detection happening once the position is converged to the exact position of the virus partiele. Finally, the input image is prepared to detect the next virus by the Virus Removal of previously detected virus particles. We check at multiple points of the virus detection pipeline, if stopping criteria is met. For more details see section 3.4
I, based on the presence and absence of virus capsids in I, in a binary manner. To locate the virus capsids, we first initialize their position Po with the location of the highest value of a CAM obtained for C()). This position is then iteratively optimized to obtain refined position over time steps t. In each step t, we mask the input image with a Gaussian mask M centered at Pt. before optimizing Pt maximize the classifier score C(I M(:))) During this optimization, we fix the weights of the classifier and only optimize the position. In order to successfully converge to the desired position, when Po is far from virus particle, the gradient computation needs to consider areas of T far from Po- Therefore, the standard deviation of the Gaussian is chosen to initially span the entire image and continuously decreases during the optimization process. Once the optimization process converges, the already detected virus particles are cut out from 1, using virus radius r, such that the optimization towards new viral particle is not misguided by already located particles. This iterative process stops when (T)) predicts the absence of viruses on the masked image. See See Figure for an overview our method.
3.1 INITIALIZATION
We compute the CAM of the input image / using the GradCAM (Selvaraju al., 2017) algorithm based on our pre-trained classifier (implementation from Gildenblat & contributors (2021) )).. Then, the center of mass of the top 1% of activations used as the initial position, Po- Thus, if there are multiple instances of the virus, the CAM can spread over multiple regions of the input image. Therefore, to within the top 1% of the activations. this is not the case, we define Po as random posítion among those top 1%.
initialize po inside of a relevant region, check if the center of mass lies
3.2 OPTIMIZATION
Given an initial position Po. we want to further optimize it using gradient descent to match the exact location of virus. To achieve this, we define fully differentiable mask M E TR W as a Gaussian function centered at Pt, where it the current optimization iteration. This mask M is defined as:
1
Iii Pz 1l
My (p.)
cxp
2T
20
where miy is the coordinate the position i, in the image, and o, is the mask's 's standard deviation Note, that the mask is normalized have integral equal to since we found this to work
4

---

Published as a conference paper at ICLR 2024
max
min
Figure 2: Visualization of the magnitude and direction of the gradients for multiple positions in the input image over the optimization process. For large values of the standard deviation, maxx large portions of the image receive gradients pointing to virus particles. However, for small values of the standard deviation, mmin, only the regions close to virus particles contain strong gradients pointing towards particles. By reducing the value of during the optimization process are able accurately find a particle in the image even if the initial starting position is far away from any virus.
better in practice. Also, before feed the masked input to the classifier, normalize it based the statistics of the pre-training data set. Then, the optimization objective is defined as:
max Pt
C(I ((:))
Mask standard deviation. In order to propagate gradients to optimize the position over the full EM image, the standard deviation of the Gaussian mask needs to be adapted for each optimization step. While a Gaussian mask with large standard deviation max pulls positions that are far from virus closer to the optimal position, Gaussian mask with small standard deviation min will generate smooth gradients for positions close virus (see gegure 2}. Therefore, we take inspiration from approaches commonly used for score generative models (Song & Ermon 2019): We start with a large standard deviation max and then reduce over the optimization process to min- Since the different EM images can have different levels of magnification, depending on the real-world virus size in nm. We choose maax such that the entire image will be visible if the mask is placed in the center of the image of the smallest magnification level. In practice, exponential decay performed the best when interpolating between max and min- |Figure 2]shows an illustration of gradient magnitude and direction at different points in image for multiple dr.
define the standard deviation
3.3
VIRUS REMOVAL.
Then, we iteratively repeat Initialization and Optimization. However, to prevent the virus detection to converge to the same position, we remove the already detected virus by masking it with a circular shape using the known virus size.
3.4 STOPPING CRITERIA
To stop the iterative detection, we consider three criteria: 1) During the Initialization step we compute the CAM and stop the virus detection when the computed CAM does not show any focus, meaning the minimum value equals the maximum value of the CAM. 2) After applying the Virus Removal step, we forward the image through the classifier. If the output score smaller than threshold stop searching for viruses in the image, as the classifier predicts no remaining viruses in the image. The value of is chosen based on the smallest threshold used for computing the Mean Average Precision (mAP) metric. 3) During the Detection step, we test if the detected region actually contains virus. We test this by masking everything in the image but the last virus detected and process this image with the pre-trained classifier.
3.5 POSTPROCESSING
Once all particles have been detected, we apply non maximum--uppprssiinn similar to the Faster- RCNN (Ren et al. 2015), to discard low-scoring virus particles that overlap with higher-scoring ones and exploit the fact that virus particles do not overlap in the image plane. Lastly, bounding box is created for each virus detected in the image, using the known size of the virus. Moreover,
5

---

Published as a conference paper at ICLR 2024
compute score for each bounding box by masking all other detected viruses in the image with circular disks and forwarding it to the pre-trained classifier.
4 EXPERIMENTS
In this section, describe the experimental setup used to validate method. To analyze results on a variety of viruses, we have focused our experiments on the following five virus types: Herpes virus, Adeno virus, Noro virus, Papilloma virus, and Rota virus. Below, will briefly discuss these with respect to imaging-relevant virus properties and data availability, before providing details on the conducted user study and discussing the obtained results.
4.1 DATA
Herpes virus. The Herpes virus causes lifelong infections in humans. It is composed of an icosahedral capsid with double-stranded DNA, a tegument (middle layer), and an outer lipid bilayer envelope. We use the data from Shaga Devan et al. (202T) which contains 359 EM images with 2860 annotated bounding boxes of the virus particles in total. We 287 images for training, 36 for validation, and 36 for testing. To approximate the size of the virus, we use values reported by Weil al. (2000) and Yu al. (201T) adjusted to account for the different image modality (Read et al. 2019) of room temperature Transmission Electron Microscopy (TEM). This results in virus size of 165nm, which also the average size in the data set.
Adeno virus. The Adeno virus is non-enveloped icosahedral capsid with dsDNA. can infect lining of the eyes, airways and lungs, intestines, urinary tract, and nervous system leading to cold-like symptoms. We use the data from Matuszewski & Sintorn (202T) containing 67 negative stain TEM images of the Adeno virus with location annotations. We approximate the virus size with 80nm as reported in literature by Goldsmith & Miller (2009).
Noro virus. The Noro virus is a small-sized, non-enveloped capsid with icosahedral geometries and single-stranded RNA. It can cause acute gastroenteritis. For this virus, we use 54 negative stain TEM images from Matuszewski & Sintorn (202T) with location annotations. We approximate the virus size with 30mm as reported in literature by Ludwig-Begal al. (202I).
Papilloma virus. The Papilloma virus common virus in humans. While it can cause small benign tumors, it can also progress to cervical cancer in high-risk forms. It is non-enveloped with icosahedral DNA. Here, we use the data from Matuszewski & Sintorn (202T) containing 31 negative stain TEM images of the Papilloma virus with location annotations. We approximate the virus size
with 50nm as reported in literature by Doorbar et al. (2015).
Rota virus. The Rota virus has a distinctive wheel-like shape: round with a double-layered capsid, non-enveloped, with double-stranded RNA (segmented RNA genome). We use the data from Matuszewski & Sintorn (202T) containing 36 negative stain TEM images the Rota virus with location annotations. We approximate the virus size with 75mm reported in literature by Yates (2014).
It be noted that the data sets of the Adeno, Noro, Papilloma and Rota virus (maximum of 67 images) are significantly smaller than the data set the Herpes virus (359 images). For all viruses, work image patches with resolution of 224 224 pixels following the standard image input size for state-of-the- image classifiers. To generate the patches we use a sliding window with no overlap.
4.2
USER STUDY
We conducted a user study to compare the cost of obtaining different types of annotations, such that we later analyze detection accuracy in relation spent annotation time. The three types of annotation we collect during this study are 1) binary labels indicating virus presence, 2) bounding boxes that precisely describe the virus location and extent, and 3) locations of the virus centers, which

---

Published as a conference paper at ICLR 2024
Timings for Annotations
Score
Location Bounding Box
0.9
Time (8]
10'
0.7
10"
6
0.5
Loc
Vieible Capsids
Figure 3: Annotation time vs. the number of particles in the image. Note the log scale.
Figure 4: User Study results for different types of annotations.
is another popular choice for collecting weak detection labels (Li et al. 2019 Matuszewski & Sintorn 2018 2019).
Study design. During the study, six experts asked annotate 85 patches of the TEM images of the Herpes virus with the three types of annotations. We provided an in-house application with a user interface designed maximize the annotation speed of the three types of labels. We permuted the order of the conditions presented to the experts by a balanced Latin square. We presented the same data all our participants during all conditions. To counterbalance learning effect, we randomized the order of the presented patches. The 85 TEM patches show a range of 1-8 visible virus capsids, which is the full range of visible Herpes capsids in the data. For more information and results see the appendix.
Task performance. We compare the performance of the participants all three different tasks. We consider the Fi score the performance metric. For the evaluation of the location and bounding box task, we use an IoU threshold of 0.5 to define a True Positive (TP). We found significant differences in the performance of the participants between all tasks (see Figure 4). This reveals the increasing complexity of annotating binary labels, location labels, and bounding box labels: While binary annotations only require the decision of whether virus present in image or not, localization and bounding box annotations require this decision for every visible particle in the image, making the annotations more prone to errors. Additionally, bounding box annotations require the definition of the size of a virus, thus increasing their complexity even more. These results support the motivation for using binary labels, as the annotations are less prone to errors.
Annotation times. Morcover, we investigate the annotation times per visible virus for all tasks. Figure 3/shows the average annotation times per patch of visible virus capsids. The time was measured between the moment showing the image and the user interaction to trigger the visualization of the next image. The average annotation times are slightly decreasing for the binary annotations when the number of visible virus capsids increases. We assume this be the case based on a simpler detection of virus capsids when their occurrence is higher. However, for both other conditions, the annotation times increase with the number of visible virus capsids. This is accountable to the need for independent annotation for every single virus.
4.3 EXPERIMENTAL SETUP
Our experiments include wide range of object detection models with different levels of supervision. First of all, we include a fully supervised object detection model (BB). Second, we follow Li et al. (2019) and Matuszewski & Sintom (2018) 2019) use minimal labels for training an object detection model (Loc). We derive the bounding boxes for training from location labels and set their sizes by the known virus size. Finally, we compare the bounding boxes resulting from our optimization process (Ours(Opi) )) and an object detection model trained with such boxes (Ours(OD). We use a ResNet-101 (He et al. 2016) as our classification model and Faster-RCNN with a ResNet-101 backbone as our object detection model. Additionally, models, SAM and CutLER, to work a weakly supervised setup: We pick images of the training set that contain virus and forward these through the pre-trate pre-trained models generate bounding boxes. We then find a suitable range of bounding box sizes on the validation set. The resulting range is used in the train set to obtain bounding boxes that later used to train an object detection model.
adapt two recent zero-shot segmentation
7

---

Published as a conference paper at ICLR 2024
Next, we compare against different weakly supervised methods: (radCAM(Sellvaaaaa al. (2017). LayerCAM Diang et al. (202T), TS- IS--AM[Gaa [ao et al.] (202I) and Reattention Su et al. (2022) 2). As current state-of-the-art methods found that ViT-based architectures can be benefcial for WSOL, we compare the use of a ViT-B/16 backbone and a ResNet-101 backbone for GradCAM as well as LayerCAM. For methods that rely on saliency maps, we compute multiple bounding boxes by thresholding the maps and deriving bounding boxes from connected regions using Bolelli et al. (2019). We choose the threshold based on the best result on the validation set and apply it to the test set. For more fair comparison, we also include the knowledge about the virus size in the compared approaches. We report the best results over several runs with different hyperparameters. An extensive evaluation can be found in the appendix A.5
In all experiments, we obtained three runs with different seeds and reported the mean and standard deviation. For all methods, we perform a parameter search to find the best measure the performance of the object detection models, we use mean average precision with an overlap of 50 % (mAPso).
To
4.4 RESULTS
To compare the three different types of annotations, we fix an annotation time budget and pick random image patches until the time is exhausted. We define the budget as the total time required to annotate the entire data set using binary labels. To compute the time cost of each image, we average the annotation times of the experts participating in the user study. Table presents the results of this experiment. It can be observed that our method is able to outperform location and bounding box labels for all viruses. In particular, Ours(OD) is outperforming all other approaches. Moreover, we can see that location labels are not able perform well for some of the virus particles due to the small number of training images.
In our comparison to different zero-shot learning approaches, found that, in the case of negative stain TEM images where the background is stained making the virus the most prominent structure, both SAM and CuLLER performed comparably well. However, when dealing with small viruses such as Noro and Papilloma, CutLER's performance was subpar, while SAM struggled particularly with the smallest virus, Noro. In general, the performance of these methods heavily depends on the sample, noise levels, and preparation method. Our approaches, on the other hand, are more stable over all data sets, leading to the best results on all viruses except for the Adeno.
In conclusion, our investigation revealed that existing weakly supervised methods faced challenges in effectively detecting viruses in EM. Despite incorporating supplementary information about virus size into all comparison approaches, their performance remained suboptimal. This limitation is likely attributed to the fact that contemporary state-of-the-art methods thrive on large data set sizes, which not available for the detection of virus capsids in EM. Furthermore, these methods are typically crafted to excel in scenarios with more object-centric data sets, whereas E. images present distinctive challenge by containing numerous object instances within single frame. Moreover, the conventional methods are not inherently equipped to handle the low signal-to-noise ratio (SNR) characteristic of EMM The prevalence of low SNR introduces inherent ambiguity in object boundaries, a challenge that can be partially mitigated by incorporating the known virus size into the methods, thereby circumventing this ambiguity. Additionally, the presence of noise and low-contrast regions in EM images poses obstacles to extracting discriminative features crucial for precise object localization. This became evident in certain classifiers trained on negative stain TEM data, leading to bias towards virus borders (see Figure 9.. Our introduced optimization, involving fixed size, contributes to more robust localization, addressing these challenges.
These findings underscore the pressing need for methodologies purposefully tailored to excel in the intricate task of virus detection in EM. The unique characteristics of EM data necessitate specialized approaches that can navigate the challenges posed by low SNR, small data set sizes and the abundance of object instances within a single image.
Reduced Annotation Time. We further investigate the impact of reducing the annotation times. For this experiment, we choose the herpes virus as it has the largest amount of annotated images as well as bounding box annotations. According to our study, this data set requires an annotation time of 19 hours to annotate bounding boxes, 17 hours to annotate location labels and 11 hours to annotate binary labels. We use the time required to annotate all available images using our binary labels as the upper bound of 100 % and reduce annotation times to 75 %, 50 %, 25 %, 10 %, and 5 %.
8

---

Published as a conference paper at ICLR 2024
Table 1: Comparison of the different methods for the different viruses reporting mAP 50-
Herpes
Adeno
Noro
Papilloma
Rota
BB
89.18 =0.95
Loc
88.13 =0.38
26.24
19.93
00.82
#0.34
27.20
116.4
06.51
#4.06
Durs(Opt)
86.98 =1.92
47.85
#11.82
54.65
14.94
70.02
22.85
71.73
#3.51
Ours
91.20 =0.24
58.28
#5.91
74.32
+1.18
78.33
+2.40
78.34
#2.15
SAM
41.34 64
44.62
43.90
08.80
#3.92
73.23
#7.02
66.71 1 #4.33
CutLER
64.95 1,98
68.49
15.44
10.72 .15.63
23.02
1..73
75.5
1:444
GradCAM ResNet
78.79
22.04
19.17
10.78
05.54
12.99
11.57
14.17
31,78 121158
LaycrCAM ResNet
78.44
12.73
16,48
19.34
05.04
11.91
10.87
15.33
31.22 1:20.07
GradCAM VIT
61.87
11188
08.00
12.12
19.31 113.64
04.03
1.452
13.12 1.737
LayerCAM VIT
68.33
=6.09
09.18
#5.04
10.82 11178
17.41
11133
09.74 #2.42
TS CAM
32.06
-1.02
39.25 13.13
14.64
+4.66
07.1 1 #3.85
43,53 43.9
Reattention
68.85 =0.62
58.49 4:222
55.09
#8.92
35.60 #13.01
59.05 #11.40
Limitad Annotation Time
Infinite Amount of Annotation Time
0.8
mAPs
0.6
mAP33
0.8
0.4-
BB
0.7
B6
Loc
Loc
0.2
Our
OumrOpr)
0.ra(00)
100%
75%
50%
25%
10%
5%
100%
75%
50%
25%
10%
5%
Amount Annotation Time
Amount of Training Data
Figure 5: Comparison of detector model using
Figure 6: Comparison of a detector model using different data set sizes.
different amnotation times.
Figure 5
presents the results of this experiment. We found that 1) Ours(OD) is able outperform other types of labels for all the time budgets. 2) Ours(Opt) can provide better performance than all other methods, including Ours(OD), when the data set is small (annotation time less than 25 %).
Infinite Annotation Time. Moreover, evaluated the performance of all four methods when infinite time for annotation is possible, but the amount of data is limited. The results are presented in Figure 6 We here elaborate again on the results of the Herpes virus, has the largest amount of annotated images as well as bounding box annotations. However, we also include results on the additional virus data sets in the appendix Table 6 can be observed that Ours( obtained Tageen
similar or slightly better performance than Loc and BB when the data set size is large. Moreover, we can see that Ours(Opt), although not able reach the performance of the other methods, is able to achieve competitive performance. However, for small data set sizes, we see that the supervised approaches start to outperform the weakly supervised approach. We believe that this has two reasons: First, the smaller data set sizes do not allow train classifier, with good localization abilitics. Additionally, training the Faster-RCNN on a data set that is small and noisy leads to worse performance. However, please note that the benefit of annotating image-level labels vanishes the absolute time for annotation already small.
5 CONCLUSION
In this paper, we proposed novel approach for virus particle detection in EM data based on weak supervision. Our approach optimizes bounding box positions of virus particles by leveraging pre-trained classifier. Gaussian masking and domain
knowledge. Furthermore, to improve the optimization, we initialize the Gaussian masks based on GradCAM hotspots. We compared the results obtained with our method to other weakly supervised approaches, as well as fully supervised ones, where we show that our method able to outperform those for the same amount of annotation time. Moreover, conducted user study that shows that binary labels are easier to obtain and more robust against errors than other annotation methods. Thus, our approach shows promise for efficient and accurate particle detection in EM images, opening avenues for practical applications in this field. In the future, we would like to analyze the applicability of our method to the localization of objects that vary in size.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
We would like thank Jens von Einem (Institute Virology, Ulm University Medical Center) for providing herpesvirus infected cells.
This work was financed by the Baden-Wirrtemberg Stiftung (BWS) for the ABEM project under grant METIDI2 -BEM.....B......................
REPRODUCIBILITY STATEMENT
The source code associated with the experiments conducted in this paper is publicly available on GitHub at the following link: experiments presented in the paper will be provided in the repository. This includes information on command-line arguments, hyperparameters, and any additional configurations necessary to reproduce the reported results.
Instructions for replicating
REFERENCES
Paul Ayres. Using subjective measures to detect variations of intrinsic cognitive load within problems. Learning and instruction, 16(5):389 400, 2006.
Loris Bazzani, Alessandra Bergamo, Dragomir Anguelov, and Lorenzo Torresani. Self-taught object localization with deep networks. In 2016 IEEE winter conference on applications of computer vision (WACV), pp. 1 1-9. IEEE, 2016.
Archith John Bency, Heesung Kwon, Hyungtae Lee, S Karthikeyan, and BS Manjunath. Weakly supervised localization using deep feature maps. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pp. 714-731. Springer, 2016.
Hakan Bilen and Andrea Vedaldi. Weakly supervised deep detection networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2846-2854, 2016.
Federico Bolelli, Stefano Allegretti, Lorenzo Baraldi, and Costantino Grana. Spaghetti labeling: Directed acyclic graphs for block-based connected components labeling. IEEE Transactions on Image Processing, 29:1999: 1999-2012,, 2019.
Gong Cheng, Junyu Yang, Decheng Gao, Lei Guo, and Junwei Han. High-quality proposals for weakly supervised object detection. IEEE Transactions on Image Processing, 29:5794-5804, 2020.
Junsuk Choe and Hyunjung Shim. Attention-based dropout layer for weakly supervised object localization. IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Shaga Devan, Paul Walther, Jens von Einem, Timo Ropinski, Hans A Kestler, and Clarissa Read. Detection of herpesvirus capsids in transmission electron microscopy images using transfer learning. Histochemistry and cell biology, 151:101-114, 2019.
Ali Diba, Vivek Sharma, Ali Pazandeh, Hamed Pirsiavash, and Luc Van Gool. Weakly supervised cascaded convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 914-922, 2017.
Bowen Dong, Zitong Huang, Yuelin Guo, Qilong Wang, Zhenxing Niu, and Wangmeng Zuo. Boosting weakly supervised object detection via learning bounding box adjusters. IEEE/CYF International Conference Computer Vision (ICCV), 2021.
John Doorbar, Nagayasu Egawa, Heather Griffin, Christian Kranjec, and Isao Murakami. Human papillomavirus molecular biology and disease association. Reviews in medical virology, 25:2-23, 2015.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 1929, 2020.
10

---

Published as a conference paper at ICLR 2024
Florian Dubost, Hieab Adams, Pinar Yilmaz, Gerda Bortsova, Gijs van Tulder, M Arfan Ikram, Wiro Niessen, Meike W Vernooij, and Marleen de Bruijne. Weakly supervised object detection with 2d and 3d regression neural networks. Medical Image Analysis, 65:101767, 2020.
Nikita Dvornik, Julien Mairal, and Cordelia Schmid. Modeling visual context key to augmenting object detection datasets. In Proceedings of the European Conference Computer Vision (ECCV), pp. 364-380, 2018.
Debidatta Dwibedi, Ishan Misra, and Martial Hebert. Cut, paste and learn: Surprisingly easy synthesis for instance detection. In Proceedings of the IEEE international conference on computer vision, pp. 1301-1310, 2017.
Wei Gao, Fang Wan, Xingjia Pan, Zhiliang Peng, Qi Tian, Zhenjun Han, Bolei Zhou, and Qixiang Ye. Ts-cam: Token semantic coupled attention map for weakly supervised object localization. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2886-2895, 2021.
Jacob Gildenblat and contributors. Pytorch library for cam methods.
https://github.cmm com/
Tacobgi1/pytorce -aai--raddaaaa 2021.
Ross Girshick. Fast r-cnn. International Conference on Computer Vision (ICCV), 2015.
Cynthia S Goldsmith and Sara E Miller. Modern uses of electron microscopy for detection of viruses.
Clinical microbiology reviews, 22(4):552-563, 2009.
Simon Graham, Quoe Dang Vu, Shan E. Ahmed Raza, Ayesha Azam, Yee Wah Tsang, Jin Tae Kwak, and Nasir Rajpoot. Hover-net: Simultancous segmentation and classification of nuclei in multi-tissue histology images. Medical Image Analysis, 58: 18:161 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Qinwen Huang, Ye Zhou, Hsuan-FF Liu, and Alberto Bartesaghi. Weakly supervised learning for joint image denoising and protein localization in cryo-electron microscopy. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3246-3256 2022.
Zeyi Huang, Yang Zou, BVK Kumar, and Dong Huang. Comprehensive attention self-distillation for weakly-supervised object detection. Advances in neural information processing systems, 33: 16797-16807, 2020.
Andrii Iudin, Paul K Korir, José Salavert-Tome Gerard J Kleywegt, and Ardan Patwardhan. Empiar: a public archive for raw electron microscopy image data. Nature methods, 13(5):387-388, 2016.
Peng-Tuc Tao Jiang, Chang-Bou Bin Zhang, Qibin Hou, Ming-Ming Cheng, and Yunchao Wei. Layercam: Ex- ploring hierarchical class activation maps for localization. IEEE Transactions on Image Processing. 30:5875-5888 2021.
Vadim Kantorov, Maxime Oquab, Minsu Cho. and Ivan Laptev. Contextlocnet: Context-aware deep network models for weakly supervised localization. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp. 350-365. Springer, 2016.
Chao Li, Xinggang Wang, Wenyu Liu, Longin Jan Latecki, Bo Wang, and Junzhou Huang. Weakly supervised mitosis detection breast histopathology images using concentric loss. Medical image analysis, 53: 13:-677, 2019.
David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60:91-110, 2004.
Weizeng Lu, Xi Jia, Weicheng Xie, Linlin Shen. Yicong Zhou, and Jinming Duan. Geometry constrained weakly supervised object localization. In European Conference on Computer Vision (ECCV), 2020.
11

---

Published as a conference paper at ICLR 2024
Louisa F Ludwig-Begall, Axel Mauroy, and Etienne Thiry. Noroviruses the state of the art, nearly fifty years after their initial discovery. Viruses, 13(8):1541, 2021.
Damian J Matuszewski and Ida-Maria Sintorn. Minimal annotation training for segmentation of microscopy images. In 2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018), pp. 387-390. IEEE, 2018.
Damian Matuszewski and Ida-Maria Sintorn. Reducing the u-net size practical scenarios: Virus recognition in electron microscopy images. Computer methods and programs in biomedicine, 178: 31-39, 2019.
Damian J Matuszewski and Ida-Maria Sintorn. Tem virus images: Benchmark dataset and deep learning classification. Computer Methods and Programs Biomedicine, 209: 009:10631,, 2021.
Meng Meng, Tianzhu Zhang, Qi Tian, Yongdong Zhang, and Feng Wu. Foreground activation maps for weakly supervised object localization. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021.
Anna V Nartova, Mikhail Yu Mashukov, Ruslan R Astakhov, Vitalii Yu Kudinov, Andrey V Matveev, and Alexey G Okunev. Particle recognition on transmission electron microscopy images using computer vision and deep learning for catalytic applications. Catalysts, 12(2):135, 2022.
Maxime Oquab, Léon Bottou, Ivan Laptev, and Josef Sivic.
Is object localization for free? weakly- supervised learning with convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 685-694, 2015.
Fred Paas, Juhani E Tuovinen, Huib Tabbers, and Pascal WM Van Gerven. Cognitive load mea- surement as a means to advance cognitive load theory. Educational psychologist, 38(1):63-7,, 2003.
Clarissa Read, Martin Schauflinger, Dimitri Nikolaenko, Paul Walther, and Jens von Einem. Regula- tion of human cytomegalovirus secondary envelopment by c-terminal tetralysine motif in pul71. Journal of Virology, 93(1 (3(3))x002444-88 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Zhongzheng Ren, Zhiding Yu, Xiaodong Yang, Ming-Yu Yu Liu, Yong Jae Lee, Alexander G Schwing, and Jan Kautz. Instance-aware, context-focused, and memory memory-efficient efficient weakly supervised object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10598-10607, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Assisted Intervention-MICCAA 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234 241. Springer, 2015.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Jinhwan Seo, Wonho Bae, Danica Sutherland, Junhyug Noh, and Daijin Kim. Object discovery via contrastive learning for weakly supervised object detection. In Computer Eision-ECCy 2022: 17th European Conference, Tel Aviv. Israel, October 23-27, 2022, Proceedings, Part XXXI, pp. 312-329. Springer, 2022.
Kavitha Shaga Devan, Paul Walther, Jens von Einem, Timo Ropinski, Hans A Kestler, and Clarissa Read. Improved automatic detection of herpesvirus secondary envelopment stages in electron microscopy by augmenting training data with synthetic labelled images generated by a generative adversarial network. Cellular Microbiology, 23(2):e 13280, 2021.
12

---

Published as a conference paper at ICLR 2024
Tanvir R Shaikh, Haixiao Gao, William T Baxter, Francisco J Asturias, Nicolas Boisset, Ardean Leith, and Joachim Frank. Spider image processing for single-particle reconstruction of biological macromolecules from electron micrographs. Nature protocols, 3(12):1941-1974, 2008.
Fred J Sigworth. Principles of cryo-em uigle-particle image processing. Journal of Electron Microscopy, 65(1):57-677 2015.
Krishna Kumar Singh and Yong Jac Lec. Hidc-and-seek: Forcing a network to be meticulous for weakly-supervised object and action localization. In IEEE International Conference Computer Vision (ICCV), 2017.
Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribu- fion. 2019.
Hui Su, Yue Ye, Zhiwei Chen, Mingli Song, and Lechao Cheng. Re-attention transformer for weakly supervised object localization. arXiv preprint arXiv:2208.01836 2022.
Peng Tang, Xinggang Wang, Xiang Bai, and Wenyu Liu. Multiple instance detection network with online instance classifier refinement. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017.
J. R. R. Uijlings, K. E. A. van de Sande, T. Gevers, and A. W. M. Smeulders. Selective search for object recognition. International Journal of Computer Vision, 2013.
Jun Wei, Sheng Wang, S Kevin Zhou, Shuguang Cui, and Zhen Li. Weakly supervised object localization through inter-class feature similarity and intra-class appearance consistency. In Computer Vision-ECCCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXX, pp. 195-210. Springer, 2022.
Tatjana Weil, Ruediger GroB, Annika Roecker, Kenny Bravo-Rodriguez, Christian Heid, Andrea Sowislok, My-HUe Le, Nelli Erwin, Mridula Dwivedi, Stephen M Bart, et al. Supramolecular mechanism of viral envelope disruption by molecular tweezers. Journal of the American Chemical Society, 140444):170 2020.
Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Rui-Wei Zhao, Tao Zhang, Xuequan Lu, and Shang Gao. Cream: Weakly supervised object localization via class re-activation mapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9437-9446, 2022.
Haolan Xue, Chang Liu, Fang Wan, Jianbin Jiao, Xiangyang Ji, and Qixiang Ye. Danet: Divergent activation for weakly supervised object localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019.
Marylynn V. Yates. Chapter twenty-eight- rotavirus. In Steven L. Percival, Marylynn V. Yates, David W. Williams, Rachel M. Chalmers, and Nicholas F. Gray (eds.), Microbi- ology of Waterborne Diseases (Second Edition), pp. 523-527. Academic Press, London, second edition edition, 2014. ISBN 978-0-12-415846-7
doi: http:///oi.org/10.0001
B978-0- 120---22-11588-77000022222 Atricile/p1//9970021188881000000
URL https: www. www.:ssieeeeeirecccoo.c ctt.s/iencomm/ciieceeeeeoo
Xuekui Yu, Sanket Shah, Manfred Lee, Wei Dai, Pierrette Lo, William Britt, Hua Zhu, Fenyong Liu, and Z Hong Zhou. Biochemical and structural characterization of the capsid-bound tegument proteins of human cytomegalovirus. Journal of structural biology, 174(3):451 460, 2011.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference computer vision, pp. 6023-6032, 2019.
Zhaoyang Zeng, Bei Liu, Jianlong Fu, Hongyang Chao, and Lei Zhang. Wsod2: Learning bottom-up and top-down objectness distillation for weakly-superviee object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8292-8300 2019.
Xiaolin Zhang, Yunchao Wei, Jiashi Feng, Yi Yang, and Thomas Huang. Adversarial complementary learning for weakly supervised object localization. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018
13

---

Published as a conference paper at ICLR 2024
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep fea- tures for discriminative localization. IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2015.
14

---

Published as a conference paper at ICLR 2024
A
APPENDIX
A.1 ADDITIONAL EXPERIMENTS
this section we present additional ablation studies. All our experiments were done using the herpes virus and results are reported on the validation set for Ours(Opt).
A.1.1 OPTIMIZATION
While investigate the impact of the initialization for the optimization in main paper (Section 4.5), Table 2] shows results obtained when using different number iteration steps for the opti- mization. As be seen, initialization not only reduces the number of needed forward passes, but also outperforms all other initialization schemes for all numbers of iterations. Based on these findings, have chosen maximum number of iterations during optimization.
Method
Iterations
mAP50
Forward Passes
2.93 1: 0.73
5 0
25
6.15 = 0.92
117 #
Random
50
11.63 1.36
179 #7
100
36.9 + 1.33
289 + 19
200
60.24 + 2.29
518
57.86 + 4.94
145 +
25
63.39 5.75
169 + 76
Selective Search
50
64.76 1: 6.68
186 1
100
65.44 7.49
232 1
200
66.77 + 7.28
344 # 142
68.66 3.14
1 +
25
77.58 + 1.23
32
GradCAM
50
80.35 0.8
64 2
100
81.4 - 0.91
128
200
82.4 + 0.25
255 9
Table 2: Comparison of different position initialization methods. Iterations denotes the maximum number of steps used for optimizing a single position. As expected, we found, that a random initialization will require more optimization steps to converge to good position. An initialization with selective search already reduces the amount of optimization steps needed, while our proposed initialization, based GradCAM, reduces the needed optimization steps the most, while the same time requires fewest forward passes.
To conclude, our introduced optimization can converge to the correct position of the virus, even though the initialization of the position is not close to virus particle. However, combining the optimization with a good initialization scheme, like the proposed GradCAM initialization, the number of iterations can be reduced, saving computation time.
A.1.2 INITIALIZATION
We evaluate the impact GradCAM initialization compared to selective search (Uijlings et 2013) random initialization, and greedy version of our optimization approach: the sliding window. For the sliding window, we use all possible positions the image separated by distance of 0.125 r of each other as bounding box candidates. For the selective search, we pick the highest scoring region proposal that has a size bigger than 0.8x virus size and smaller than 1.2x virus size. Lastly, for GradCAM we use the initialization schema introduced in /section 3 Additionally, to compare the computational cost of the approaches, we report the average number of forward passes through the pre-trained network when virus has been detected. Results can be found in Table 3
We found that random initialization and selective search are not performing well when compared to GradCAM. The sliding window underperforms when compared to our method since its performance
15

---

Published as a conference paper at ICLR 2024
Table Comparison of different approaches initial- ize the bounding box position to optimize.
Table 4: Evaluation of the standard devia- tion of the Gaussian mask.
mAP 50
Forward Passes
MAP50
Sliding Window
78.91 #1.73
3834 #70
2 1 1x
68.68 #2 02
Random
60.24 #2,29
518 #14
75.66 #3.24
Selective Search
66.77 1:728
344 1142
0.5 X
75.75 1:260
GradCAM
82.40 10.25
255 1:9
0.25 x r
75.36 1:2.83
Table 5: Comparison of different backbones for our approach reporting mAP
Herpes
Adeno
Noro
Papilloma
Rota
Ours Ours(Opt) ResNet
86.98 +1.92
47.85 +11.8
54.65 +4.91
70.02 2.88
71.73 #3.51
Ours Opt) VIT
48.66 +07.40
07.46 +05.21
10.44
#09.06
05.67 =08 05 04.50 #3.51
highly depends on the distance between consecutive positions. When comparing the number of forward passes, we can see that the sliding window approach requires the most forward passes of the classifier. Moreover, we can see that selective search requires fewer forward passes than random initialization, indicating that it converges faster. GradCAM requires the least forward passes while obtaining the best performance.
A.1.3 GAUSSIAN STANDARD DEVIATION
We investigate the influence of min the Gaussian mask. Given the virus radius we found that min 0.5 x r gives the best results (see Table 4).
A.2 BACKBONE ARCHITECTURES
We found that most state of the approaches ViT (Dosovitskiy et al., 2020) backbones to achieve superior performance. We hence compare the use of a ViT-B/16 backbone compared to a ResNet-101 backbone. We found, that given the small virus data sets VIT backbones were not able to perform well (see Table 5).
A.3 INFINITE ANNOTATION TIME
In line with the paragraph Infintee Annotation Time (Section 4.4) the main paper, we evaluate our approach when all available data is needed and hence the annotation times are not balanced. For results see Table 6
In line with our findings in the main paper in Section 4.4, fully supervision is outperforming weakly supervised approach when the data set sizes become small, as it is the case for the Adeno virus. the Noro virus, the Papilloma virus and the Rota virus. For large data set sizes (the Herpes virus) the annotation of more complex labels like location annotations or bounding boxes do not improve the result.
Discussion. Considering limited amount of available data, the experiment with the herpes data set suggests, that the annotation of image level labels is beneficial in two cases: First, when the data set is too small to sufficiently train detector model, our pseudo labels, Ours(Opt), that do not rely on the training of state of the art detector models, but only require a well performing classifier, outperform object level annotations. Second, when the data set is large enough pseudo labels are performing well enough to train state of the art object detection model, Ours(OD), that en par with similar model that has been trained on object level annotations, suggesting that annotation time can be reduced when only annotating image level labels instead of object level labels.
However, for the 10% and 25% split of the Herpes virus, well the Adeno virus, Noro virus, Papilloma virus and Rota virus, bounding boxes or location labels seem to be beneficial. We believe this to be the case, since the size of the data set is sufficient train detector model. However, small
16

---

Published as a conference paper at ICLR 2024
Table 6: Comparison of the different methods for the different viruses reporting mAP We here evaluate the approaches for limited data, instead of limited annotation time.
Herpes
Adeno
Noro
Papilloma
Rota
BB
90.77 #0.99
Loc
90.11 1 #0.48
80.25
+3.54
89.43 +1.12
91.74
=0.17
89.51
=1.76
Ours Opt)
86.98 11.92
47.85
111.2
54.65 14.94
70.02
2.85
71.73
13.51
Ours (D)
91.20 10.24
58.28
15.91
74.32 11118
78.33
22.4
78.34
22.1
errors in the pseudo labels can cause the detector to perform worse, based on the rather small training data sets, This means, that the performance of our method heavily depends on the performance of the classifier. Qualitative results (Figure 17) on the virus data sets suggest that our pscudo labels (Ours(Opts)) are prone to detect False Positive (FP). While their score low, the RaNter-RCNN (Ours( OD)) is able to differentiate between TP and FP. However, when the data set size is sufficiently big to train a Faster-RCNN, bounding box labels do not share this disadvantage and result in better performing Faster-RCNN, With increased data set sizes, the detector will perform better and better. Additionally, as mentioned above, the error of the pseudo labels gets smaller, since the location ability of the classifier increases, and the large amount of data is more forgiving for small errors. This finally results in Ours( OD), performing en par with methods that rely on object level annotations.
However, we argue that the most limiting factor is not the availability of data but the annotation time required to generate labels, which is why we emphasize on the results of the limited annotation times. Raw EM data is available in large amounts, for example in data bases such EMPIAR (ludin et al.) 2016). This data base has grown immensely (ludin et al.] 2016) based on the current research topics bio-medicine. Despite this, the data does not come with annotation labels. Results in our main paper suggest, that when the limiting factor is the annotation time, one should annotate image level labels and use our suggested approach to optimize for the position of the virus particles. Additionally, we would like to point out, that when the data set sizes become small, the benefit of annotating image level labels vanishes, as the absolute time for annotation is already quite small.
A.4 INFLUENCE OF APPROXIMATED OBJECT SIZE
As our method additionally requires the knowledge of the objects size, we conduct an experiment on the influence of the error in the objects size. We therefore corrupt the known object size by 10%, 20% and 30%. As expected, we found a decrease in performance with increased error of the objects size (see Table 7p. However,
would like to highlight, that in the case of virus detection, the sizes can most often be retrieved from literature. Additionally, an accurate approximation the virus size can be retrieved from measuring only few instances the standard deviation of the virus size can be neglected.
Table 7: Influence of error in size approximation of the object reporting mAPso-
Error
0%
10%
20%
30%
Ours( Opt)
86.98 11.2
82.38 +02.31
79.14 +03.10
67.31 +02.03
A.5 WEAKLY SUPERVISED COMPETITORS USING VIRUS SIZF
To be able to make a more fair comparison to other state of the art approaches, we include the known virus size into existing methods. We test two different ways of doing this and also check their combination. First, we try to remove noise by filtering detected boxes based on the known virus size. Therefore, define& size range based the virus size v, such that the size of the detected boxes falls in the range of E [[1 r) x v, (1 + r) x v). Next, we additionally replace the detected boxes with boxes of the known virus size. Results can be seen in Table 8
17

---

Published as a conference paper at ICLR 2024
Table 8: Comparison of different ways incorporate the known virus size into existing weakly supervised approaches reporting mAPso- We incorporate filtering the detected bounding boxes by the virus size (F) and replacing the detected boxes with boxes of the actual virus size (R).
F
R
Herpes
Adeno
Noro
Papilloma
Rota
X
X
57.28 #3.15
12.21
=8.05
01.96
=1.02
04.70 #2.35
19.20 + 19.64
GradCAM
X
V
78.79
12.04
15.18
28.51
05.54
2.99
11.57 14.17
31.28
1: 21.53
ResNet
/
61.34
+2.85
14.39
=1.09
02.67
=2 11
06.90 +3.37
28.56
+18.82
/
76.60
12.57
19.17
10.78
03.99
22.58
09.20 13.27
31.78
1. 21.58
X
x
56.38
13.82
09.69
16.58
01.88 1..94
05. 10 1:226
26.
J. 19.62
LayerCAM
X
V
78.44
22.73
16.48
=9.31
05.04 =1.9
10.87 #5
31.22 + 20.07
ResNet
61.74
11.33
12.92
19.18
02.34
22.15
05.47 12.57
29.20
1.9.98
/ V
76.46
14.69
16.26
11.47
03.74 22.45
08.47
1.405
29.27 1. 20.48
X X
44.00
115.31
00.66
10.53
08.98
L9.38
01.24 1..4
03.52 1. 3.98
GradCAM
X
58.52
+10.39
08.00
22.12
19.31
13.64
04.03 +4.52
13.12 +
VI
/
X
57.02
113.09
04.91
13.67
13.45 10.05
03. 14 1.889
09.02 1.707
V
V
61.87
+1187
06.4 =4.58
16.24
13.38
02.96 + 09
09.02 77.07
X X
49.22
#7.96
00.50
=0.27
03.51
=3
04.65 +3
01.02 + 0.17
LayerCAM VIT
65.47
17.43
09.18
1..64
10,82
11.78
17.41 111.33
06.43 1. 1.76
V
X
61.27
#7.32
02.88
#3.67
06.33
=6.65
06.33 #6.66
09.74 4:242
/
68.33 V
03.67
08.62
08.62 11..2
09.74
X X
07.57
05.34
04.20
01.07
05.50 2.12
TS CAM
V
18,40
+1.99
11.54
#2.92
11.10 23.18
07.11
+3 86
16.67
+ 6,38
/
32.06 X
11.21 24.69
25.03
19.56
10.94 4.774
04.03
1.201
32.48
1:3.99
/
1.002
39,25
#4.13
14.64
4.66
05.77
1:3.4
43.53
4:3.93
X 37.93
1.4.3
30.81
16.12
25.41
1.99
14.61
1.9
24.52 4: 12.64
Reattention
X
V
42.72
#4.3
58.49
22:22
48.87
=6.77
31.39
117.1
41.05 + 16.5
V
X
65.05
42.37
44.29
28.13
1:2.9
45.96 15.85
68.85
40.62
57.58
=1.32
55.09
=8 92
35.6 113.01
59.05
#11.4
Ours Opt)
86.98
+1.92
47.85
=1182
54.65
=4.94
70.02 42.86
71.73
#3.51
Ours (OD)
91.20 1:0.24
58.28
#5.91
74.32
1.18
78.33 1:2.40
78.34
4:2.5
For an easier comparison, we also include the results of our method in the table. Here, the virus size used like explained in the main paper.
We found that incorporating the size into other weakly supervised methods leads to strong perfor- mance increase. general, Reattention was performing the best out of other weakly supervised approaches. Still, our methods were mostly able to outperfom other approaches by large margin. For more the depth discussion we refer to the main paper subsection 4.4]
18

---

Published as a conference paper at ICLR 2024
Herpes
Adeno
Noro
Papilloma
Rota
DoG + response
18. 16
15.09
15.53
21.78
21.33
DoG
14.27
09.54
08.44
18.62
02.89
Table 9: Standard computer vision does not require the annotation of large data sets making it especially feasible for EM. Additionally, we can inform the process with the known virus size, further pushing its performance. However, we found that due to the high levels of noise DoG features fail to reliably detect keypoints.
Table 10: Evaluation of the loss computation.
Table 11:
Evaluation of the Table 12: Evaluation of the score computation.
masking strategy.
Loss
mAP50
Masking
mAP500
Masking
mAP50
Score
75.75 +2.60
Zeros
72.01 #2232
Mask Bkg.
76.72 1.66
Logits
76.72 11.6
Inpainting
75.70 1.77
Mask Other
80.35 #0.80
Mean
76.72 1..62
Table 13: Evaluation of Gaussian masking.
Loss
50
PDF 76.22 + 3.23
no PDF 75.87 + 2.29
A.6 VIRUS DETECTION USING DoG
As annotated EM data is limited, another way for virus detection is the use of standard computer vision like Differences of Gaussians (DoG) for object detection. This method only requires small validation set to find suitable parameters. However, in our experiments we found that DoG is not able reliably detect virus particles in noisy EM data. For keypoint detection we follow Lowe (2004). Additionally, we use the virus size to define range of valid keypoint sizes to filter keypoints. We tune the contrast threshold as well as the size range on the validation data set and report the numbers on the test data set. We tried two variants. One uses the maximum score for all detected keypoints and the second uses the keypoint response as the score of the bounding box. We got the best results by normalizing the keypoint response with the highest response in the data set. Results can be found in Table We report mAP30 similar to the main paper.
We found that DoG did not perform well detect the virus particles. We believe that this has multiple reasons: Depending on the state of the virus, the contrast of the capsid varies. Finding a suitable contrast threshold is therefore not trivial. Additionally, there virus capsids that are hardly visible (based on lower contrast and noise). Lastly, we believe that DoG features by themselves are not well suited work EM images, these images usually quite noisy. Using machine learning solve the problem is more robust choice, first, to also detect virus particles that are not dominant, and second, transfer the method between different EM-modalities (which can come with different levels of noise), as the features are learned, rather than hand-crafted.
A.7 ABLATIONS
In this section, we present additional ablation studies.
Gaussian Masking Ablation In main paper, conduct all experiments by using the Gaussian PDF, since we found this to work better in practice. Table 13 shows the obtained numbers for a direct comparison of PDF and PDE, based on which we made this decision.
19

---

Published as a conference paper at ICLR 2024
Loss Computation We compare the use of the classifier's logits to the classifier's score input to the loss function for the bounding box optimization. We found that when using the logits, gradients can be propagated more favorably (sce Table IO).
Masking Strategy We evaluate different masking strategies: masking with zeros, masking with the mean the data set, and an inpainting technique that pastes the content of background patch (Dvorik et al., 22188) Dwibedi et al., 2017)] Yun et al. 2019). We found that masking with the mean provides the best results (see Table 1T).
Score Computation We evaluate different ways compute the score of the resulting bounding box. For one, we mask everything but the detected virus and forward this through the pre-trained classifier retrieve the score. Second, we mask all other detected virus particles and forward the masked image through the classifier. We found that the second approach works better (see Table 12)
B ADDITIONAL INFORMATION ON DATASET DISTRIBUTION
We provide an in depth overview of the data set distribution that have been used in the main paper.
Annotation time
100% Loc
75%
50%
Amount
Bin
BB
Bin
Locc
BB
Bin
Loc
BB
Images
22987
14306 1391
12936
17240
10725
9656
11497
7172
6473
Virus
2235
1248
1693
1050
957
1122
669
621
Annotation time
25%
10%
5%
Amount
Bin
Loc
BB
Bin
Loc
BB
Bin
Loc
BB
Images Virus
5748
3607
3294
2303
1444
1323
1149
718
649
571
309
287
202
121
111
102
66
61
Table 14: Absolute numbers of the herpes data set as they have been used in Table and Figure 5 in the main paper.
C USER STUDY
C. MENTAL EFFORT
We additionally investigate subjective measure of cognitive load by asking for the difficulty of the task (Ayres) 2006) as well the mental load (Paas al. 2003) the participant experienced during the task. Right after the completion of each task (binary, location, and bounding box annotation), the participants were asked to rate their mental load as well as the difficulty the task based on nine-point Likert scale. Since the participants of our study were well- well-trained biologists, they found the recognition virus particles not challenging. Therefore, the differences difficulty between the tasks were small, being only significantly different between the binary and bounding box labels. However, even if the tasks were not difficult for the participants, their mental load after the annotation process was different for different types of annotation, since their complexities requires more effort to locate all viruses on the image than simply identify if there is a virus, and even effort to draw the enclosing bounding box of all viruses. For mental effort, were able to see significant difference in mental load between all types of annotations, binary, location, and bounding boxes.
different. It
C.2 STUDY DESIGN
Here provide additional details regarding user study setup. During user study, six experts were asked to annotate 85 patches of TEM images of the Herpes virus with the three types of annotations: 1) Binary annotations (Bin), indicating the presence of virus particles. 2) Location
20

---

Published as a conference paper at ICLR 2024
annotations (Loc), indicating the center of every virus particle. 3) Bounding boxes (BB), showing the size and position of every virus particle. The participants got paid 10 per hour.
Every condition was structured into a training phase and main phase: During the training phase, the participants were asked to read short instructions on how to use the tool.
For the Binary condition the following instruction was presented:
When you see virus capsid, use the up key to mark that there is virus in the image. When you don't see a virus, use the down key to mark that there no virus. Once you press Up or Down, the next image will be shown.
For the Location condition the following instruction was presented:
When you see virus capsid, use the mouse to mark the center of the capsid. Note: not all images will contain a capsid. If there are multiple capsids visible, make sure you mark the location of all capsids. If you click right on a location, the location will be removed. If you click, hold and move with the left mouse, the location will be moved. To get to the next image, use the right arrow key. If there is no virus visible the patch, you can skip to the next image.
For the Bounding Box condition the following instruction was presented:
When you see a virus capsid, use the mouse to draw a bounding box around the virus capsid by pressing the left mouse and holding. Note: not all images will contain virus. If there are multiple virus capsids visible, make sure you draw a bounding box for each of the visible virus capsids. To move an existing box left click (inside the bounding box that supposed to be moved), hold and move with the mouse. To remove bounding box right click inside the according bounding box. To get to the next image, use the right arrow key. there is no virus visible the patch, you can skip to the next image.
Based on these instructions, the participants were asked to test every functionality of the tool by annotating schematic images (see Figure 7). The participants then labelled 25 TEM images, again extracted from the Herpes virus data, for training purposes. We made sure, that none of the training images were used during the actual annotation task. As a last step of the training phase the participants asked if they felt comfortable using the tool. All answered with "yes". During the main phase, the participants annotated 85 TEM images. Finally, they were asked to report their mental load as well as the task difficulty using a nine point Likert scale.
As stated in the main paper (Section 4.2), we permute the order of the three conditions by a balanced Latin square. To minimize bias due to the presented data, reuse the TEM images for all three conditions. To then counterbalance a learning effect on the presented data, we randomize the order of the TEM images during every condition.
C.3 ANNOTATION TOOL
The annotation tool was designed similar fashion for labeling the bounding boxes, location labels or binary labels (see Figure 8);
Once the participants were ready, they could start the labeling with a click on button. Then, the first image to annotate was shown. For the bounding box annotations, the participants were able to draw bounding boxes using the mouse. These bounding boxes could be edited by changing the size and the position. Once the participants were finished, the right key could be used to get to the next image to label.
For the location condition, the participants were again asked to use the mouse mark location single click. Using drag and drop, this location could be moved. Again, when the participants were finished with one image label, they could get the next image using the right key.
For the binary case was possible to implement the interaction with the labeling tool with the keyboard only. Pressing the up-key was labeling the presented image patch as "presence of virus".
21

---

Published as a conference paper at ICLR 2024
Hart this image Tonnaine sIMA?
Park DVE "contark
then move the location to the grean
Dige & Voatan mase every roo onde.
Locaton
Drow bound box arsu
arcis ore the beuncin
Drea toarding bex amond every red
Dow boundiee prouid tox iac flls arcend gree
Bounding
Figure We prepare schematic images to annotate during a small scale training phase of the annotation tool. For each condition, made sure that the participant tested all functionalities by following the given instructions. Additionally, 25 TEM images are annotated to get familiar with the tool.
while pressing the down-key marked it as "absence of virus". Since this interaction did not require confirmation of the labelling, the next image label was presented soon as either the up or down key was pressed.
For simplicity, the user was able go back to the previous image. To make sure that the participants were not able to just skip through the images, only allowed a "next" click, 0.2 seconds after the image was shown. With 0.2 seconds being the threshold to perceive single image by human (similar to frame rates in videos).
C.4 EVALUATION
As stated the main paper, for finding significant differences we first conduct a Shapiro Wilk test for normality and the Levene's test for equal variances. Depending the outcome, make use of paired t-test or the Wilcoxon signed-rank test. We report significant results, when the p-value smaller than 0.05. For completeness we list statistical findings here and include further evaluations.
C.4.1 TASK PERFORMANCE
For the evaluation the task performance, we consider precision, recall and F1 score. Mean and standard deviation computed over all participants over all images to annotate. We consider bounding boxes with loU 0.5 as TP. For location annotations, we use the known virus radius of 165nm, as described in the main paper in Section 4.1, derive bounding boxes.
22

---

Published as a conference paper at ICLR 2024
Binary
Location
Bounding Box
Figure 8: Annotation tool for the three different tasks of annotating binary labels, location labels and bounding boxes.
23

---

Published as a conference paper at ICLR 2024
Classification In first evaluation, converted all annotation labels to classification labels. We do so, by classifying all annotations, that contain at least one location or bounding box as "presence of virus", while all other annotations are converted to "absence of virus". We did not find significant difference between the tasks, when considering this evaluation on classification labels. This allows for the conclusion, that all three tools equally well designed to classify the images.
Results of the Levene's s test can be found in Table 15a We report the metrics and the outcome of the Shapiro Wilk test for normality in Table 15b
The final statistical results can be seen Table 15c
Table 15:
We converted all annotations to classification labels and compared the results. We report the p-value of the conducted statistical tests for all metrics. The p-value is the result of paired t-test or the Wilcoxon signed-rank test, depending on the outcome of the Shapiro Wilk test for normality and the Levene's s test for equal variances.
(a) P-value the Levene's ' test for equal variances.
(h) report different performance metrics and run a Shapiro Wilk test for normality for every metric.
Bin Loc BB
Bin
Loc
BB
Precision
0.58
Precision
98.83 + 1.26
95.72 4: 5.39
96.45 +: 6.13
Recall
0.91
Shapiro Wilk
0.0411
0.0412
0.0009
Fl
0.84
Recall
95.24 1: 5.13
95.18 3.53
94.75 4.96
Shapiro Wilk
0.1613
0.0661
0.0056
Fl
96.9 2.39 0.2381
95.3 = 2.75
95.37 = 3.59
Shapiro Wilk
0.4402
0.0380
(c) We did not find significant differences, indicating that there is no expected bias due to the annotation tools.
Bin Loc
Bin BB
Loc BB
Precision
0.2733
0.4652
0.4652
Recall
0.9859
0.5625
0.8927
F1
0.2487
0.5001
1.0
Location Second, convert the bounding box annotations to location annotations and test for significant difference in the performance of the participants. To do so, we compute the center of the bounding box as location and place a bounding box with the known virus size of 165nm. We include mean and standard deviation for precision, recall and Fl score (see Table 16b). Additionally, results of the Levene's test for equal variances can be found in Table 16a We did not find a significant difference in the performance of the participants (see Table 16c). This allows for the conclusion, that the annotation tools for the location and bounding boxes are equally well designed.
Comparison While the first two evaluations indicate that there is no expected bias due to the annotation tools, this evaluation considers an actual comparison between all three tasks.
We test for equal variances using the Levene's test (see Table 17ap. We report precision, recall and F1 score for the performance of the participants annotating binary labels, location labels and bounding boxes, as well as the statisties of the Shapiro Wilk test for normality (see Table 17b). As stated in Section 4.2 in the main paper, we found significant difference between all three tasks (see Table 17ch. For completeness we list statistical findings here.
Mental Load As described in the appendix, we were also asking the participants to rate their subjective measure mental load and the difficulty of the task. For completeness, we here show a detailed statistical evaluation, which follows the same scheme as the previous one: First, we test for equal variances with the help of the Levene's 's test. Next, we test for normality by using the Shapiro Wilk test and report the median of the Likert scales. Finally, check for significant differences using a paired t-test or the Wilcoxon signed-rank test, depending on the outcomes of the previous tests.
24

---

Published as a conference paper at ICLR 2024
Table 16: We converted all bounding box annotations localization annotations and compared the results. We report the p-value of the conducted statistical tests for all metrics. The p-value is the result of paired I-test or the Wilcoxon signed-rank test, depending on the outcome of the Shapiro Wilk test for normality and the Levene's 's test for equal variances.
(a) P-value Levene's (b) report different performance metrics and run a Shapiro Wilk test for normality for every metric.
(c) not find signifi- cant differences, indicating that there is no expected bias due to the annotation tools.
test for equal variances.
Loc BB
Loc
BB
Loc BB
Precision
0.94
Precision
88.51 +7.22
89.22 #6.81
Precision Recall F1
0.6858
Recall
0,91
Shapiro Wilk
0.0251
1068
0.0864
F1
0.93
0.3127
Recall
88.48 #4477
86.44
#4.91
Shapiro Wilk
0.7427
0.1568
$8.19 #3.31
87.51 #3.32
Shapiro Wilk
0.1739
0.1572
Table 17: We directly compare all annotations by reporting precision, recall and Fl metrics. We report the p-value of the conducted statistical tests for all metries. Significant differences are marked in bold. The p-value is the result of paired t-test or the Wilcoxon signed-rank test, depending on the outcome of the Shapiro Wilk test for normality and the Levene's for equal variances.
(a) P-value of the Levene's test for equal variances.
(b) We report different performance metries and run Shapiro Wilk test for normality for every metric.
Bin Loc BB
Bin
Loc
BB
Precision
0.0472
Precision
98.83 +1.2
88.51 #7.22
71.12 12..17
Recall
0.2442
Shapiro Wilk
0.0411
0.0251
0.2794
F1
0.0308
Recall
95.24 +: 5.13
88.48 +447
69.04 +11.2
Shapiro Wilk
0.1613
0.7427
0.4118
F1
96.9 +: 2.39
88.19 18.19 43.31
69.85 111.3
Shapiro Wilk
0.2381
0.1572
0.5273
(c) We found significant differences between all three tasks indicating correlation between annotation erro and task complexity.
Bin Loc
Bin BB
Loc BB
Precision
0.0312
0.0312
0.0312
Recall
0.1253
0.0025
0.0064
Fl
0.0040
0.0312
0.0312
25

---

Published as a conference paper at ICLR 2024
Table 18:
We compare the mental load and task difficulty over all tasks with Likert scale from one (low) to nine (high). We report the p-value of the conducted statistical tests for all metries. Significant differences are marked in bold. The p-value is the result of paired t-test or the Wilcoxon signed-rank test, depending on the outcome of the Shapiro Wilk test for normality and the Levene's test for equal variances.
(a) P-value the Levene's 's test for equal variances.
(b) We report the median of the nine point likert scale and run Shapiro Wilk test for normality for every metric.
Bin Loc BB
Bin
Loc
BB
Mental Load Difficulty
0.5772
Median Shapiro Wilk
2.0
4.5
6.0
Mental Load
0.0645
0.1670
0.2517
0.1492
Median
1.0
3.0
3.5
Difficulty
Shapiro Wilk
0.0063
0.3515
0.9605
(c) We found significant differences between all three tasks regarding the reported mental load. Additionally, we found significant differences in difficulty annotating bounding boxes or binary labels.
Bin Loc
Bin BB
Loc BB
Mental Load Difficulty
0.0117
0.0047
0.0335
0.0655
0.0421
0.2031
26

---

Published as a conference paper at ICLR 2024
D QUALITATIVE RESULTS
D.1
ROBUSTNESS OF OPTIMIZATION APPROACH
16%
Ours (Opt) ResNet
98%
31%
99%
87%
8%
99%
96% %
98%
98%
99%
99%
GradCAM ResNet
99%
98%
99%
99%
99 %
99%
Figure 9: Even though there is a clear bias visible towards the boarder of the virus (which is most likely due the imaging modality of negative stain TEM), approach is able converge suitable positions of the virus based on the introduced optimization strategy. The GradCAM approach, which was applied to the same classifier, on the other hand, is able to produce well fitting bounding boxes.
27

---

Published as a conference paper at ICLR 2024
D.2
COMPARISON TO WEAKLY SUPERVISED APPROACHES
Herpes
Adeno
Noro
Papilloma
Rota
95%44%
89%
Ours(Opp)
98%
28%
29%
92%
92%
56%
96%
97%
98%
99%
13%
9.7%%
39%
73%
Reattention
96%
99%
99%699%
99%
99%
99%
99%
98%
99%
99%
99%
99%
99%
99%
99%
99%
999
99%
TS-CAM
99%
99%
99%9%%
99%
99%
99%
99%
99%
99%
99%
99%
99%
65%
LayerCAM VIT
65%
39% 99%
99%
0.2%
99%
99%
99%
99%
95%
65%
99%
99%
65%
99%
99% 99%
GradCAM NIT
95%
99%
6.5%
9-%
99%
99%
99%
99%
99%
99%
95%
99%
6.5%
99%
LayerCAM Reaker
99%
99%
30%
98%
GradC AM nodie
D9SS
99%
99%
95%
99%
99%
99%
Figure 10: We compare pseudolabel generation (Ours(Opt)) to other weakly supervised ap- proaches and show qualitative results for all viruses.
28

---

Published as a conference paper at ICLR 2024
D.3 VIRUS SIZE FOR SAM AND CUTLER
SAM
CutLer
No Filter
Filter
Figure 11: For a fair comparison to our approach we also inform the bounding boxes generated by Segment Anything Model (SAM) and Cut and Learn (CutLER) about the known virus size. As shown above, we are hence able to reduce the amount of FPs
29

---

Published as a conference paper at ICLR 2024
D.4
COMPARISON TO SAM AND CUTLER
Ground Truth
Ours(Opt)
SAM
CutLer
Figure 12: qualitative comparison of the pseudolabels generated by SAM, CutLER and Ours(Opt). We found that CutLER and SAM are able to detect enveloped virus capsids comparably comparably well (row 4). We believe that this is the case due to the enclosing membrane around the capsid, that clearly outlines the object. However, failure cases of SAM and Cutl ER include the detection of virus-unrelated vesicles (row 2) as well as the inability to detect naked virus capsids (row 2+4).
30

---

Published as a conference paper at ICLR 2024
Ground Truth
Ours(Opt)
SAM
CutLer
100%
100%
100%
9936
50%
ZC0%
Figure 13: A qualitative comparison of the pseudolabels generated by SAM, CutLER and Ours(Opt) for the Adeno virus. While all approaches are able to perform well on high contrast virus particles (top row) SAM and CutLER fail to detect low contrast viruses. Additionally, unrelated, similar shaped, high contrast objects are also detected, due to the zero shot learning of the base model.
31

---

Published as a conference paper at ICLR 2024
Ground Truth
Ours(Opt)
SAM
CutLer
1100%
61%
107%
100%
100%
100%s
100%
200%S
S00%
20%
51%
Figure 14: A qualitative comparison of the pseudolabels generated by SAM, CutLER and Ours(Opt) for the Noro virus. As the data of the Noro virus is rather noisy, we can here the main benefits of our approach being able to detect almost not visible virus particles (top row). Still, on higher contrast virus particles, again SAM and CutLER perform comparably well.
32

---

Published as a conference paper at ICLR 2024
Ground Truth
Ours(Opt)
SAM
CutLer
109%
100%
ICONA
100%
100%
100%
95%
200
Figure 15: A qualitative comparison of the pseudolabels generated by SAM, CutLER and Ours for the Papilloma virus. For the this virus SAM well as CutLER struggle detect multiple objects in an image. While SAM is still able to perform well on some images, CutLER usually only detect single instance.
33

---

Published as a conference paper at ICLR 2024
Ground Truth
Ours(Opt)
SAM
CutLer
100%
200%
100%
94%
zorrs
100%
Figure 16: A qualitative comparison of the pseudolabels generated by SAM, CutLER and Ours for the Rota virus. We can observe similar behaviours of the pseudolabels as for the other virus particles.
34

---

Published as a conference paper at ICLR 2024
Ground Truth
Ours(Opt)
Ours(OD)
Location
Bounding Box
Herpes
Adeno
Noro
Papilloma
Rota
Figure 17: Qualitative results of the predictions of the Faster-RCNN, trained on bounding box labels, location labels and image level labels. Additionally, we show predictions of our weakly supervised approach Ours Opt) directly the test set. We found that Ours(Opt) tends to predict FP. However, these FPs are detected with low score, allowing the Faster-RCNN to learn a differentiation between FPs and TPs. This explains the superior performance of Ours(OD) in comparison with Ours (Ors))
35


Reviews:
[{"review_id": "pcnJGVhMZM", "reviewer": "Reviewer_xKLD", "summary": "This paper presents a domain-specific weakly supervised object detection method that relies on image-level annotations instead of bounding boxes. They use a pre-trained model to generate pseudo-labels for training, showing that these labels outperform other weak labeling methods and even ground truth labels in time-constrained scenarios.", "questions": "The current setup with Gaussian as a prior assumes that the object to detect is in a round shape. How easily it can be extended to different objects and how accurately it will work?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- the paper is well-written and easily to follow. \n- it proposed an relevantly simple but effective method for an impactful task. In their experimenst, aurthors sucessfully demonstrated the supriority over the consider baselines, includig supervised method as well as zero-shot learning with large scale pretrained models. \n- the authors utilized the spatial information and explored an novel way to refine the localization neural networks provide.", "weaknesses": "The proposed method has potential to work for not only electron microscope images but other medical images. It will be interesting and also brings broader impact if authors can provide discussions around this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jI2Mu3gUrf", "reviewer": "Reviewer_6zGs", "summary": "This work presents a method for weakly-supervised object detection (WSOL) of virus capsids in EM images, which can be used for rapid curation of bounding boxes. The overview of this method (presented in Figure 1) uses an iterative process in which: 1) Grad-CAM from a pretrained encoder is use to output saliency maps of the highest-scoring virus location, 2)  gradient descent is used to optimize the location of the virus, 3) virus is masked out using known information about the virus size. This process repeats until all viruses are removed (referenced as Ours (Opt)), with the bounding boxes created using this process usable for developing weakly-supervised object detectors (Ours (OD)). Comparisons against human annotators (weakly-supervised binary annotation, location, bounding box) and self-supervised detectors were performed, with comparison against human annotators (with and without time constraints) also performed.", "questions": "See above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "Overall, this work presents a very unique methodology and study design for curating bounding boxes in EM images. A contribution not emphasized in this work is the simplicity of the method, using a very intuitive heuristic that outperforms current unsupervised, deep learning-based detectors such as SAM and CUTLER. Though specific to EM, I believe the uniqueness and simplicity of this work would still be of interest to the computer vision community. The related work section is all comprehensive, and the authors of this work reference related works in weakly-supervised and self-supervised object detection very well.", "weaknesses": "- Though the related work section provides a comprehensive overview of current progress in WSOL methods, was there a reason why this work does not compare against other WSOL methods such as Xu et al. [1] (CREAM), Wei et al. [2] (ISIC), and other more recent works such as LOCATE [3] and GenPromp [4]? Though specific to EM, many other works in the WSOL domain can also be readily adapted.\n- In addition to lack of comparisons, one of the main limitations of this work that may prevent broader interest in the ICLR community is that the proposed method is too specific to EM and is not evaluated on diverse tasks. Though EM is unique compared to natural images which are generally more object-centric, other modalities such as histopathology and multiplexed imaging share similar characteristics (as noted in [5]), with the image scale is objective with units per pixel being fixed. Specifically, the contributions of this work would be strengthened if shown that a simpler heuristic can also be created for other imaging domains.\n- Following other works which have found pretrained Vision Transformers (ViTs) to be strong in WSOL [4,5,6], was there as a reason why a ResNet-101 was used for classification instead of a ViT? Moreover, was the DINO-ViT used in CUTLER trained using EM images, or was it using a pretrained checkpoint from ImageNet? As ViTs have been also found to have natural fit for microscopy images [5], it would be interesting to explore how the a DINO-ViT for EM images would: 1) improve the WSOL results of this work, and 2) improve the CUTLER baseline reported in this work.\n- Though this work is well-written, it was difficult to understand the training dataset and the downstream dataset for evaluation and annotator labeling. Though described in text, including a table with the distribution of labels for train and test may be simpler to communicate.\n\nOverall, I found this work to have a unique contribution computer vision in proposing simpler techniques that outperform more complicated solutions that are more complex to train and underperform in domain-specific areas. At the same time, I feel that the scope of the work is too narrow for ICLR, and lacks comparisons to other WSOL works. I still feel that this work has many strengths, and believe that it would make a timely contribution in conferences specific to computer vision such as CVPR and ECCV where approaches for solving domain-specific challenges would be more broadly appreciated.\n\n1. Xu, J., Hou, J., Zhang, Y., Feng, R., Zhao, R.W., Zhang, T., Lu, X. and Gao, S., 2022. Cream: Weakly supervised object localization via class re-activation mapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 9437-9446).\n2. Wei, J., Wang, S., Zhou, S.K., Cui, S. and Li, Z., 2022, October. Weakly supervised object localization through inter-class feature similarity and intra-class appearance consistency. In European Conference on Computer Vision (pp. 195-210). Cham: Springer Nature Switzerland.\n3. Li, G., Jampani, V., Sun, D. and Sevilla-Lara, L., 2023. LOCATE: Localize and Transfer Object Parts for Weakly Supervised Affordance Grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10922-10931).\n4. Zhao, Y., Ye, Q., Wu, W., Shen, C. and Wan, F., 2023. Generative prompt model for weakly supervised object localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 6351-6361).\n5. Chen, R.J., Chen, C., Li, Y., Chen, T.Y., Trister, A.D., Krishnan, R.G. and Mahmood, F., 2022. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 16144-16155).\n6. Murtaza, S., Belharbi, S., Pedersoli, M., Sarraf, A. and Granger, E., 2023. Discriminative sampling of proposals in self-supervised transformers for weakly supervised object localization. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 155-165).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hZspqTp3hb", "reviewer": "Reviewer_zGyY", "summary": "The manuscript presents a class activation map (CAM)-based weakly supervised learning method for virus particle detection in electron microscopy images. Specifically, it first uses a pre-trained classifier to obtain an initial position of a virus using GradCAM (Selvaraju et al. 2017), and then iteratively refines the position with a Gaussian mask with a dynamic standard deviation. It repeats this process for each virus until all the viruses are detected in the input image. The proposed method is evaluated on 5 electron microscopy image datasets, and the experimental results are promising.", "questions": "1. The proposed method is based on the GradCAM method. What if the GradCAM does not provide good initializations or even wrong saliency maps? What are the effects of inaccurate saliency map creation on the quality of the pseudo-labels generated by the proposed method? \n\n2. During the optimization process of the proposed method, i.e., solving Equation (2), is the classifier C fixed and not updated? If so, is the optimization of Equation (2) simply to find the position that has the highest value in the prediction map C(I * M(p_t)) at each time step? But if not, what algorithm is used to optimize Equation (2)? \n\n3. During the postprocessing, the method uses non-maximum suppression to eliminate virus particles that have low detection scores. Are the detection scores of virus particles obtained from the initial CAM map or the prediction map from the Gaussian-filtered input, C(I * M(p_t))?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper introduces a simple yet promising method for weakly supervised object detection. Meanwhile, it conducts extensive ablation studies to show that the proposed weakly supervised method can outperform other more fine-grained annotation-based approaches (e.g., bounding box and point annotations), given a certain time budget.\n\n2. The paper designs a specific user study to demonstrate the effectiveness and efficiency of the proposed method.", "weaknesses": "1. In the experiments, the proposed method is not compared with other state-of-the-art weakly supervised learning methods, such as Zeng et al. 2019, Wei et al. 2022, and Lu et al. 2020. In addition, it is not compared with other CAM-based weakly supervised object detection methods in the experiments, such as Xu et al. 2022. Without a comparison with recent state of the art, it is difficult to determine the superiority of the proposed methods over other approaches.\n\n2. The method requires the object size to be known in advance. This needs additional effort to estimate the size of target objects before applying the method. It would be helpful to provide an in-depth discussion on this design (probably also including the effects of using different estimated object sizes).  \n\n3. It seems that the proposed method needs to repeat the optimization process (i.e., solving Equation (2)) for each virus. The time cost may be high if there is a large number of viruses in the input image.\n\n4. The method is evaluated on only virus detection in electron microscopy images, where viruses do not overlap. Thus, the method may not generalize to object detection (e.g., cell or nuclei detection) in other microscopy imaging modalities, such as hematoxylin and eosin (H&E) or immunohistochemistry (IHC) stained brightfield microscopy images, and fluorescence mages, which often have touching or overlapping cells or nuclei. In addition, the repeated optimization for each object would be expensive for H&E or IHC images that typically have thousands of or even more cells/nuclei.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0dKIxthvr0", "reviewer": "Reviewer_KB1H", "summary": "This paper addresses the challenge of expensive and time-consuming annotation requirements for training state-of-the-art object detection models. The authors propose a domain-specific weakly supervised object detection algorithm that leverages image-level annotations instead of annotated bounding boxes. By distilling the knowledge of a pre-trained model focused on virus presence/absence prediction, the proposed approach generates a set of pseudo-labels that can be used to train an object detection model effectively. The method utilizes an optimization approach with a shrinking receptive field, enabling the extraction of virus particles directly without relying on specific network architectures.", "questions": "In general, I find this work to be commendable. However, there are certain limitations that should be addressed for further improvement. Specifically, it is crucial to include testing with DETR to evaluate the effectiveness of transformer-based architectures. Additionally, at least providing a thorough discussion on the low SNR problem would significantly enhance the quality of the paper. If these questions are well solved, I would like happy to raise my score.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Addressing Expensive Annotation Requirement: The paper tackles the challenge of acquiring costly bounding box annotations by proposing a weakly supervised approach that relies on image-level annotations. This significantly reduces the manual labor and time required for annotation, making the training process more efficient.\n- Extensive Comparative Studies: The authors conduct comprehensive studies to evaluate the effectiveness of the proposed pseudo-labels. The results demonstrate that the generated pseudo-labels outperform other weak labeling methods and even ground truth annotations in scenarios where annotation time is limited. This indicates the superiority and practical value of the proposed approach.", "weaknesses": "- Limited Model Exploration. The paper primarily focuses on using Faster-RCNN with a ResNet-101 backbone as the detection model. It would be beneficial for the authors to consider exploring other models, such as DETR, to evaluate their effectiveness in the proposed approach.\n- Lack of Discussion on Low Signal to Noise Ratio (SNR) in EM Images: While the authors mention that low SNR in EM images can impact the performance of methods designed for other imaging modalities, there is a lack of in-depth discussion, algorithm design, and experiments addressing how to mitigate the low SNR problem and how it specifically affects the capacity of weakly supervised object detection (WSOD) methods in the EM scenario. Further exploration and discussion of this property would be valuable to enhance the understanding and applicability of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)