dataesr/scientific-paragraphs-categorization huggingface dataset:
S5 , S6, Supplementary Note 3). Close to the focus, best-in-class software was near the CRLB (within 25 %), but significant deviations from the CRLB occurred > 200 nm (Fig. S6 ). This could be due to difficulty in distinguishing signal from false positives away from focus. Astigmatic software performance dropped for the challenging high spot density datasets (Fig. 2A, 3 ). For the high SNR high spot density dataset (best software, SMolPhot), localization error increased and Jaccard index decreased significantly compared to the low density condition (lateral RMSE best HD 51 nm vs best LD 27 nm, axial RMSE best HD 66 nm vs best LD 29 nm, Jaccard index best HD 66 % vs best LD 85 %). Inspection of the super-resolved images (Fig. S7 ) nevertheless shows qualitatively acceptable results for the HD dataset, particularly in the lateral dimension.
In some circumstances, the performance reduction at 10x higher spot density could be acceptable for 10x faster, potentially live-cellcompatible, imaging speed. We also observed a large spread of software performance for the high density datasets, probably because a significant fraction of the software packages were primarily designed for low density conditions. We observed poor performance for the most challenging low SNR high spot density astigmatism dataset (Fig. 2A,   In general software localization performance was close to the CRLB (Fig. S6 ). We observed that performance of the software away from the focal plane is relatively uniform (Fig. 4A , S4A), and best-in-class Z-range at high SNR was large at 1180 nm (Fig. S4A , Supplementary Table  2 ). Double helix imaging may show less software-to-software variation and larger Z-range at low spot density than astigmatic imaging because the PSF shape and intensity are fairly constant as a function of Z; unlike astigmatic imaging, where spot size, shape and intensity vary greatly as a function of Z (Fig.
Performance of 2D software We next assessed the performance of 2D SMLM software. For the pseudo-ER 2D dataset at low density, best-in-class software (ADCG  23  ) performed substantially better than the class average (Fig. S10 , S11, lateral RMSE 31 nm vs 36 nm average, Jaccard index 90 % best vs 72 %). Low density results for the brighter fluorophore microtubules dataset were similar to the dimmer pseudo-ER dataset (Fig. S10 , S12 best software SMolPhot). For the very high density 2D dataset, which had 25x higher spot density than the LD dataset, best-in-class software (ADCG) showed excellent performance (Fig. S10 , lateral RMSE, 45. 5 nm, Jaccard index 75%). Best-in-class performance (ADCG) on the dimmer fluorophore data at high spot density was also strong (Fig. S10 , best lateral RMSE 51 nm, best Jaccard index 70 %).
2. Single emitter fitting software is usually built on a multi-step strategy of detection, spot localization, and optional spot rejection. The detection step finds bright spots in noisy images on the pixel grid. The selection of candidates is usually performed by local maximum search after a denoising filter. Others rely on more complex algorithms like the wavelet transform (WaveTracer 26 ). We did not observe software ranking to depend noticeably on the choice of optimization scheme: least-square, weighted least-square or maximum-likelihood estimator.
3. Multi-emitter fitting software groups clusters of overlapping spots and simultaneously fits multiple model PSFs to the data. Typically, fitted spots are added to the cluster until a stopping condition is met  4, 5. This leads to improved localization performance at high spot density, at the cost of reduced speed. This class of software (e. g. , 3D-DAOSTORM  11  , CSpline, PeakFit, ThunderSTORM  27  ) was amongst the top performers in each 2D and 3D competition category. As expected, single-and multiple-emitter fitting methods both performed well on low density data. For the 2D challenge, multi-emitter fitting showed a clear advantage over single emitter fitting at high density. Surprisingly however, welltuned single-emitter fitting algorithms slightly outperformed multi-emitter algorithms for 3D high density conditions (e.
4. Compressed sensing algorithms. One subset of these algorithms utilize deconvolution with sparsity constraints to reconstruct super-resolved images  [28] [29] [30]. Although deconvolution approaches can give good results, they are limited by the necessary use of a sub-pixel grid; increased localization precision requires smaller grid resolution, which must be balanced against increased computational time. Recent approaches address this issue by localizing the point sources in a gridless manner under some sparsity constraint (ADCG, SMfit, SOLAR_STORM, TVSTORM  31  ). This software class consistently gave the overall best performance for 2D high-density (ADCG 1 st , FALCON  30  2 nd , SMfit 3 rd ).
5. Other approaches. Of the alternative algorithmic approaches used, the annihilating filter-based method LEAP  32  gave good performance for biplane imaging. Recently, we received the first challenge submission from a deep learning SMLM software (DECODE); these promising preliminary results are available on the competition website. Post-hoc temporal grouping-Because molecule on-time is stochastically distributed across multiple frames, a common post-processing approach to improve localization precision is to group molecules detected multiple times in adjacent frames, and average their position  33  (Supplementary Note 4). Temporal grouping was used by the top performers (including SMolPhot, MIATool  34  and SMAP-2018), and is visibly apparent as a more punctate super-resolved image (Fig. 4A ).
Choice of PSF model-Most software used a variant of Gaussian PSF model. A few participants designed more accurate PSF models. Either diffraction theory was used (MIATool, LEAP) or spline fitting of an analytical function to the experimental PSF was adopted (CSpline, SMAP-2018). Although simple Gaussian model PSFs were sufficient to obtain best-in-class performance for the 2D and astigmatic modalities (ADCG, PeakFit, SMolPhot), top results for the more optically complex biplane and double helix modalities were exclusively software using non-Gaussian PSF models (SMAP-2018, CSpline, MIATool, LEAP). Multi-algorithm packages-Several software packages take a Swiss army knife approach of integrating multiple optional localization algorithms into one program, to be flexible enough to suit various experimental conditions  19, 27.
SMAP-2018 and ThunderSTORM achieved strong across-the-board performance supporting this rationale. Software run time-Software run time is important both for ease of use and real time analysis. We did not observe correlation between software localization performance (Efficiency) and software run time (Fig. S13A ). We thus created an alternative ranking metric, Efficiency-Runtime, which gave 25 % weighting to run time (Supplementary Note 2. 7, Fig  S13B ). Many good performers in the efficiency-only ranking were relatively fast and thus retained good ranking (SMAP-2018, SMolPhot, 3D-DAOSTORM). Interestingly, two software packages highly optimized for speed gained top ranking in this analysis: pSMLM-3D  35  and QC-STORM. Diagnostic tools for software and algorithm performance-During our analysis, we frequently noticed common types of deviation between software results and ground truth which were easily diagnosed by visual inspection (Fig.
Assessment on real STORM data We investigated the performance of a representative subset of astigmatism software on real STORM datasets of well-characterized test structures, microtubules and nuclear pore complex, NPC (Fig. 4B, S17 ). This qualitative assessment was consistent with findings for simulated data. No performance difference between single and multi-emitter fitters was observed, which is not surprising since spot density in these datasets was low. Relatively poor software performance was immediately obvious from visual inspection (QuickPALM). Temporal grouping noticeably improved resolution (3D-DAOSTORM, CSpline, MIAtool, SMAP-2018). Interestingly, although Gaussian/ Bessel PSF modelling software (3D-DAOSTORM, MIATool, ThunderSTORM) gave high resolution images, software which explicitly modelled the non-ideal experimental PSF via spline fitting (CSpline, SMAP-2018) gave noticeably improved resolution of fine structural features such as the top and bottom of the NPC (Fig.
The second major conclusion is that localization software that explicitly includes the experimental PSF in the fitting model gives a significant performance increase for 3D SMLM. For the more optically complex biplane and double helix modalities in particular, the best results were from software that incorporated non-Gaussian PSF models (SMAP-2018, CSpline, MIATool). This result also highlights the importance of accurate PSF modelling in 3D SMLM simulations. The performance advantage of experimental PSF fitting software would not have been observable had simulations been generated with a simple Gaussian PSF. We can also make an overall comparison between 3D modalities, taking into account software performance. We stress that these comparisons apply to microscope PSFs similar to those tested here; for example, additional PSF engineering could improve results of any modality.
We also found that several software packages for astigmatic or biplane imaging gave adequate performance for the challenging case of high molecule densities, as long as the image SNR was high. Current software packages gave poor performance when molecule density was high and image SNR was low. These results indicate that with current algorithms high density 3D SMLM performance is mediocre at high SNR and poor at low SNR. Surprisingly, multi-emitter fitting did not show significant improvement over well-tuned single emitter fitting for the 3D high-density datasets; this may indicate that potential for improvement remains in this category. Many software packages did not apply temporal grouping  33  , resulting in reduced software performance. Since temporal grouping is a simple step for maximum precision, we urge all software developers to integrate this approach into their software as an optional final step in the localization process.
The second 2D localization microscopy challenge provided the opportunity to reassess the state of the field. The performance of best-in-class 2D software over a range of conditions, at both high and low spot density, was very strong. Interestingly, the top three performers in the 2D high density condition were all compressed sensing algorithms (ADCG, FALCON, SMfit). In low density 2D conditions, the best single-emitter, multi-emitter and compressed sensing algorithms all gave comparable, excellent, performance. We speculate that performance in the low spot density 2D category might now be near optimal levels. We look forward to new competition submissions using approaches not yet represented in the software challenge. In addition to the elegant HAWK preprocessing technique  36  , deeplearning-based SMLM algorithms show great promise  [37] [38] [39] [40]  , especially for modelling complex point spread functions  38  or analyzing high emitter density data  40.
CHALLENGE ORGANIZATION We first ran the 3D SMLM software challenge as a time limited competition, with a results session hosted as a special session of the 6 th Annual Single Molecule Localization Microscopy Symposium in August 2016. The competition has now been converted to a permanent software challenge accepting new submissions. Special thanks is due to the software SMAP and 3D-WTM  25  that participated in all eight categories (density x modality). The current list of participants is at: http://bigwww. epfl. ch/smlm/challenge2016/ index. html?p=participants All datasets, methods, participations, and results of the challenge 2016 made available at http://bigwww. epfl. ch/smlm/challenge2016/. Software for simulation and analysis is hosted on the competition GitHub repository: https://github.
Structure, noise levels and spot densities Structure. : The synthetic datasets were designed to be similar to images derived from real cellular structures. We defined mathematical models for cellular structures that imitate cytoskeletal filaments such as microtubules and larger tubular structures such as the endoplasmic reticulum or mitochondria (Fig. S18A ). These structures have a tubular shape in the 3D space. For the 3D competition, we simulated synthetic 25 nm diameter microtubules (Fig. 1 ). Psuedo-microtubules are defined with their central axis elongating in a 3D space having an average outer diameter of 25 nm with an inner, hollow tube of 15 nm diameter. For the 2D competition, in addition to synthetic microtubules (MT), we simulated larger diameter 150 nm cylinders, called pseudo-endoplasmic reticulum (pseudo-ER), designed to approximate larger cellular structures such as mitochondria and the endoplasmic reticulum (ER) (Fig.
Spot density. : As performance at different density of active emitters is a key challenge for SMLM software, we generated 3D competition datasets at both sparse emitter density (0. 25 mol. [molecule] μm -2 ), 3D LD and high emitter density (2. 5 mol. μm -2 ), 3D HD. For the 2D competition, we generated a sparse (0. 5 mol. μm -2 ), 2D LD, and very high density dataset (5 mol. μm -2 ), 2D HD. Together, these simulated conditions closely resemble experimental 3D and 2D data under a range of challenging conditions of SNR, spot density, axial thickness and structure summarized in Supplementary Table  3. In addition, we provide simulated z-stacks of bright beads for software calibration. The competition datasets (Supplementary Table  4 ) are available online on the competition website.
Experimental Point Spread Function-Model PSFs, stored as high resolution look up tables, were derived from experimentally measured PSFs. Although the algorithmic approach is distinct, the concept of accurately modelling the experimental PSF based on calibration data bears relation to the PSF phase retrieval approach previously employed by Flanser and coworkers  47. Images of fluorescent beads were recorded for each modality (Supplementary Table  5 ). Signal to noise ratio of recorded PSFs was maximized in all cases by maximizing exposure time and averaging over several frames to increase dynamic range. To acquire experimental PSFs, we took 100 nm Tetraspek beads (Invitrogen) adsorbed to #1. 5 (170 μm thick) coverglass, imaged in water. The excitation wavelength was between 640 nm and 647 nm, and a Cy5 emission filter was used.
Data acquisition parameters for each modality are listed in Supplementary Table  5. The experimental PSFs used to generate the simulated data are available on the competition website. As the goal of this study was to compare software obtained on typical SMLM microscopes, we deliberately chose PSFs representative of common implementations of each 3D modality. However, additional PSF engineering should improve results of any specific modality, for example adaptive-optics corrected astigmatism  48  , or reduced Z-range, higher SNR DH-PSF designs  21. The experimental point spread functions used here were measured for fluorescent beads adsorbed to the microscope cover slip, and should be appropriate simulations of SMLM data acquired within a few microns of the cover slip. Performing SMLM imaging at greater depths, e.
SOFTWARE ASSESSMENT 3. 1 Protocol-Each localization file submitted by the participants was manually checked for erroneous systematic errors in the definition of the dataset coordinate system, such as offsets, XY axis flips or clear scaling errors. Datasets were then programmatically standardized into a consistent output format. All modifications are publicly available. If required, the modifications consisted of columns reordering, reversing axes, XY axis swap, and shifting the lateral positions by a half camera pixel. The assessment pipeline includes three main parts: localization processing, the pairing between true and estimated localization and the metrics calculations. The first one depends on the assessment settings. There are two switchable properties: photon thresholding and wobble correction.
Their combinations yield four different assessment settings. Up to 64 assessment runs per software were possible (i. e. , 4 modalities, 4 datasets per modality). For any setting, we excluded the fluorophores within a lateral distance of 450 nm from the border. This value corresponds to the radius of the largest PSF, i. e. , Double Helix. The activations too close from the border are more difficult to localize and could bias the results. The pairing between true and estimated localizations was performed frame by frame. For every frame, we identified the localizations that are close enough to a ground-truth position as true-positives (TP), the spurious localizations as false-positives (FP) and the undetected molecules as false-negatives (FN). The procedure matches two sets of localizations. We deployed the presorted nearest-neighbor search for its efficiency, with a linking threshold of 250 nm.
The results are effectively similar to the computationally intensive Hungarian algorithm  7. Photon thresholding: A photon threshold was required primarily due to the use of a realistic fluorophore blinking model. Since a fluorophore could activate/ bleach at any point in a simulated frame, this led to many frames containing very dim, undetectable localizations, e. g. , where a molecule had been active for one or more frames previously, and then bleached during the first 5 % of a frame. These fractional localizations should also be present but practically undetectable in an experimental dataset.
2. Jaccard index (JAC, %). In addition to localization precision, SMLM image resolution depends critically on number of localized molecules  53  , so it is crucial for SMLM software to accurately detect a large fraction of molecules in a dataset, and minimize false localizations. For every frame, we identified the localizations that are close enough to a ground-truth position as true-positives (TP), the spurious localizations as false-positives (FP) and the undetected molecules as false-negatives (FN). We then computed the Jaccard index %), which measures the fraction of correctly detected molecules in a dataset, JAC = 100 TP TP + FP + FN.
Baseline Localization Software-We developed a minimalist Java tool software that performs localizations of bright emitters on the 4 modalities of the challenge 2016: 2D, Astigmatism, Double-Helix, and Biplane. This SMLM_BaselineLocalization software is only designed to establish the performance baseline for the SMLM challenge. It has intentionally limited lines of code and relies only on few threshold parameters to localize particles. It has basic calibration tool that has to run on a z-stack of beads to find the linear f(x) relation between the axial position Z and the shape of the bead.
• Biplane: Z = f (W left -W right ), where W left and W right are respectively an estimation of the size of the spots in left and the right plane. The Java code is available: https://github. com/SMLM-Challenge/Challenge2016 2 ) and best-in-class (colored marker with gold star) software performance for all competition modalities. AS, astigmatism; DH, double helix; BP, biplane.
ACKNOWLEDGEMENTS Authors acknowledge the following funding sources: a  Newcastle University   Research Fellowship  and a  Wellcome Trust & Royal Society   Sir Henry Dale Fellowship  grant number  206670/Z/17/Z  to SH; an  European Research Council (ERC)  under the European Union's  Horizon 2020 research and innovation programme , Grant Agreement no. 692726  to DS, TAP, MU; UK  BBSRC  grants  BB/M022374/1 ,  BB/P027431/1 ,  BB/R000697/1  grant and  MRC  grants  MC-UU-12018/2 ,  MR/K015826/1  to RH;  European Research Council (ERC)  grant  CoG-724489 , CellStructure to JR; FranceBioImaging infrastructure  ANR-10-INBS-04  to J. -B. S;  National Institutes of Health  grant  1R15GM128166-01  to GMH; and  NSF   SBIR grants   1353638 ,  1534745  to  Double Helix LLC. We thank  R. Piestun at  University of Colorado  for providing DH-PSF phase mask designs to  Double Helix LLC.
2 ), and 24 hours of antibiotic exposure was selected in order to sufficiently increase fluorescence intensity of sensitive colonies relative to insensitive ones. Upon completion of colony staining, membranes were imaged for green fluorescence, which revealed fluorescence of colonies consisting predominantly of dead cells, and masking of colonies containing cells that had tolerated ciprofloxacin treatment (Fig. 2b, c , Supplementary Fig. 3a-c ). Fluorescence intensities of colonies were quantified using a custom image analysis pipeline in MATLAB (see Methods), and strains sensitive to killing by ciprofloxacin were identified (Supplementary Table  1 ). Strains with fluorescence values 3σ or greater than the mean of the entire Keio collection were classified as hits (Fig. 2d , Supplementary Fig.
Cells were then 10-fold serially diluted in sterile PBS, plated onto solid LB agar, and incubated overnight at 37°C. For instances where colony area was calculated, colony edges were detected using a custom MATLAB script in ImageJ, and the number of pixels within each was determined. The number of pixels within each 13 mm disk was also determined, and the areas of colonies in mm 2 were calculated using the known area of the disk as a pixel-scaling factor. KCN was purchased from Sigma-Aldrich (60178). Growth and killing of bacterial colonies by heat on nitrocellulose membranes. For heat-treatment experiments, overnight cultures of E. coli BW25113 were grown as described above, seeded onto nitrocellulose disks, and grown for 24 hours on solid LB agar at 37°C. Disks were subsequently removed from solid LB media, placed into empty sterile petri dishes (351029; Corning), and incubated at 80°C for the noted durations in a VWR Gravity convention oven (VWR).
After ciprofloxacin treatment, membranes were transferred to LB agar supplemented with 50 µg/ml 2,3,5-triphenyl-2H-tetrazolium chloride and incubated at 37°C for one hour. After colony staining, membranes were fluorescently imaged at excitation and emission wavelengths for Cy3 using an Azure c400 imaging system to differentiate live and dead colonies based on intrinsic fluorescence properties of E. coli. Membrane images were then analyzed using a custom image analysis pipeline in MATLAB. Here, imported. tif files were first converted to greyscale and then binarized using an Otsu global image threshold to locate the placement of the nitrocellulose membrane. Since the location and angle of the membrane varied slightly between each image, rotational and spatial transformations were normalized to a rectangular grid aligned with the membrane.
g. , nitrocellulose membrane without the colonies) was used to adjust the background of each image. Finally, background-adjusted colony intensities were normalized by row, column, and plate average to account for edge-specific and image-specific fluorescence effects. Colonies were then sorted by the corresponding Keio collection gene list, and average intensities were compared between experiments. For all image analysis, image processing was used only to generate image masks where designated. Raw images within the masked regions were used for all data quantification. Once fluorescence intensities had been calculated for each strain in duplicate, ciprofloxacin-treated sample fluorescence values were divided by the corresponding values for their untreated controls to obtain a fold-change in strain sensitivity to ciprofloxacin.