dataesr/scientific-paragraphs-categorization huggingface dataset:
5  and 6 ). These metrics allow the user to improve the performance of a given model by tuning its hyperparameters or exploring the range of applicability to different data from which it was trained (generalisation). The QC section typically has two parts. In the first part, the performance metrics shown to users are loss curves for model training and validation (Fig. 8a ). These allow users to determine if the tested model overfits during training, identifiable by an increasing divergence between the validation and training loss. This divergence appears if the model learns features too specific to the training dataset instead of general features applicable to all similar datasets, therefore preventing it from generalising to unseen data, a common problem for DL networks  62. However, even a model that performs well on the validation data can produce unwanted results when used on unseen data, ultimately making the model unreliable.
The YOLOv2 notebook uses the mean average precision score (mAP)  65, 66  and F1 score, reflecting the validity of the bounding box positions and the corresponding classification (Fig. 8d ). These metrics and our implementations are described in detail in Supplementary Information (Supplementary Note 2). Data augmentation and transfer learning. In ZeroCostDL4Mic, we also enabled several important functionalities that facilitate and improve the applicability of each DL approach. First, we implemented the possibility to enable data augmentation (Augmentor  32  , imgaug) which can artificially expand the image diversity of a dataset. It commonly consists of applying a set of image transformations to both source and target data in the training dataset, such as rotation or vertical/horizontal flipping but more complex transformations can also be used, such as shearing.
For instance, merely flipping (horizontal and vertical) and rotating all the images in a dataset by 90°will increase the size of a dataset by 8. Data augmentation may improve the generalisation of a model by amplifying diversity in the dataset. This may be especially useful if the available dataset is small, which can occur if it is expensive or time-consuming to generate. For instance, when training YOLOv2, with our test dataset, we found that the model performance improved when performing data augmentation on the training images and bounding boxes (Fig. 9a-c ). In addition, we included the option to perform transfer learning  33  in all the ZeroCostDL4Mic notebooks. Transfer learning allows us to take advantage of pretrained models, from model "zoos" for instance, by re-using previously learned features within these models and speeding up and improving the training process (Supplementary Note 4).
e. , image segmentation with CDeep3M 10 , DeepCell Kiosk 15 or DeepMIB  16  ) or rely on local servers or paid-for services (as the authors of U-Net have implemented  14  , or with ImJoy 18 ). Some methods can already be trained on the widely used Fiji platform  21  (DenoiSeg 11 , Noise2-Void 2D 8 ). However, for most users, the latter methods are only feasible for use if their machines have GPU acceleration without which training times can take 10's to 100's of hours. We believe that ZeroCostDL4Mic fills these gaps and enables affordable and versatile DL-deployment capabilities. It also differs from existing solutions by providing a wide variety of DL tasks within a standardised and user-friendly platform to carry out end-to-end DL workflows: (1) installing relevant computational components, (2)  loading training datasets, (3) training models with tailored data, (4) quantitatively validating the performance of models and (  5 ) deploying validated models on new data.
This can lead to the creation of higher performance and/or more general models. Access to these platforms will only require a small financial investment to train DL models. For instance, FloydHub access costs $9 per month plus $1. 2 per GPU hour (as of December 2020). However, as it can take multiple training sessions to find a suitable set of parameters for an individual model and dataset, we recommend first-time users familiarise themselves with training models and explore network suitability using Google Colab initially before moving to paid-for platforms. One remaining challenge lies in handling, curating, and annotating datasets, especially when performing segmentation tasks: the appropriate preparation of training datasets is always associated with human hours cost, disregarding the platform used for training DL networks.
However, we would like to highlight that creating segmentation training datasets can be significantly accelerated by initially training models first with a small number of images  67, 68. These models can then generate masks that can be refined by users (i. e. , in Fiji 21 ) before being used for training to obtain a high-performance model  5, 7, 8, 11, 23, 69. This bootstrapping approach can be carried out iteratively, thereby increasing the network performance progressively while increasing the amount of training data available  56. Another challenge associated with DL is enabling model versioning to ensure reproducibility. Indeed, a DL model's performance is affected by the training dataset, the network and training parameters but also all the underlying dependencies. To mitigate this issue, in ZeroCostDL4Mic, we provide each trained model with a thorough report that contains all parameters used for training (the type of the data, network parameters, essential package versions) and the model performance (assessed via the quality control section).
MDA-MB-231 cells were provided by ATCC. DCIS. COM cells were provided by J. F. Marshall (Barts Cancer Institute, Queen Mary University of London, London, England, UK). HeLa cells were provided by ECACC. U-251 glioma cells were provided by J. Ivaska (University of Turku, Turku, Finland). A2780 cells were a kind gift of P. Caswell (University of Manchester, Manchester, UK). Python programming. We developed our platform on Jupyter Notebooks as a Python interactive environment. The notebooks were developed with Google Colab, which includes several pre-installed packages. Important packages used in ZeroCostDL4Mic include Numpy  76  , Keras (https://github. com/keras-team/keras), TensorFlow  77  , Augmentor  32  , tifffile (https://github. com/cgohlke/tifffile), elasticdeform  28  and imgaug. U-Net training dataset.
The training datasets used for segmentation in the U-net ZeroCostDL4Mic notebooks are publicly available EM datasets. For 2D segmentation, this was a neuronal membrane segmentation dataset from the ISBI challenge 2012  78, 79  and for 3D segmentation from the mitochondrial segmentation dataset available from EPFL (https://www. epfl. ch/labs/cvlab/data/data-em/). Datasets for segmentation tasks can also be created manually. This requires target images that have been segmented by an expert using drawing tools, e. g. in ImageJ/ Fiji  21, 57  , to draw outlines around the structures of interest. For training in the notebook, the source (raw EM image) and target (8-bit mask obtained from expert drawing) images were placed in separate folders, with each source image having a corresponding target image with the same name.
StarDist training dataset. DCIS. COM lifeact-RFP cells were labelled using 0. 5 μM SiR-DNA (SiR-Hoechst, Tetu-bio, Cat number: SC007) for 2 h. The cells were then imaged live for 14 h on a spinning-disk confocal microscope (one picture every 10 min). The spinning-disk confocal microscope used was a Marianas spinning-disk imaging system with a Yokogawa CSU-W1 scanning unit on an inverted Zeiss Axio Observer Z1 microscope (Intelligent Imaging Innovations, Inc. ) equipped with a ×20 (NA 0. 8) air, Plan-Apochromat objective (Zeiss). To generate the StarDist training dataset, mask images were generated manually in Fiji. Briefly, the outlines of each nucleus were drawn using the freehands selection tool and added to the ROI manager. Once all outlines were stored in the ROI manager, the LOCI plugin (https://imagej.
net/LOCI) was used to create an ROI map. These ROI map images were then used as the mask images to train StarDist. To automatically track cells, we sequentially used StarDist and the Fiji plugin TrackMate  56. Briefly, using our StarDist notebook, nuclei from live-cell imaging data were detected, and tracking files containing the coordinate of their centre (marked by a dot) were generated (StarDist notebook section 6). These tracking files were then used as input for TrackMate  38. We also provide in the ZeroCostDL4Mic GitHub page a Fiji macro to batch analyse a folder containing multiple tracking files. Cell tracks were then analysed using the online tool Motility lab (http://www. motilitylab. net/)  80. Noise2Void training datasets. The 2D dataset provided with our notebooks was generated by plating U-251 glioma cells expressing endogenously tagged paxillin-GFP on fibronectin-coated polyacrylamide gels (stiffness 9.
6 kPa)  72. Cells were then recorded live using a spinning-disk confocal microscope equipped with a long working distance of ×63 (NA 1. 15 water, LD C-Apochromat) objective (Zeiss). The 3D dataset provided with our notebooks was generated by recording A2780 ovarian carcinoma cell, transiently expressing lifeact-RFP (to visualise the actin cytoskeleton), migration on fibroblast-generated cell-derived matrices  81. The cell-derived matrices were labelled using Alexa Fluor 488-recombinant fibronectin and the images acquired using a spinning-disk confocal microscope equipped with a 63x oil (NA 1. 4 oil, Plan-Apochromat, M27 with DIC III Prism) objective (Zeiss). For both datasets, the spinning-disk confocal microscope used was a Marianas spinning-disk imaging system with a Yokogawa CSU-W1 scanning unit on an inverted Zeiss Axio Observer Z1 microscope controlled by SlideBook 6 (Intelligent Imaging Innovations, Inc.
). Images were acquired using a Photometrics Evolve, a back-illuminated EMCCD camera (512 × 512 pixels). CARE training datasets. Briefly, DCIS. COM lifeact-RFP cells were plated on hightolerance glass-bottom dishes (MatTek Corporation, coverslip 1. 5) and were allowed to reach confluence. Cells were then fixed and permeabilized simultaneously using a solution of 4% (wt/vol) paraformaldehyde and 0. 25% (vol/vol) Triton X-100 for 10 min. Cells were then washed with PBS, quenched using a solution of 1 M glycine for 30 min, and incubated with phalloidin-488 (1/200 in PBS; Cat number: A12379; Thermo Fisher Scientific) at 4 °C until imaging (overnight). Just before imaging using SIM, samples were washed three times in PBS and mounted in Vectashield (Vectorlabs). The SIM system used was Delta-Vision OMX v4 (GE Healthcare Life Sciences) fitted with a ×60 Plan-Apochromat objective lens, 1.
42 NA (immersion oil RI of 1. 516) used in SIM illumination mode (five phases and three rotations). Emitted light was collected on a front-illuminated pco. edge sCMOS (pixel size 6. 5 μm, read-out speed 95 MHz; PCO AG) controlled by SoftWorx. In the provided dataset, the high signal-to-noise ratio images were acquired from the phalloidin-488 staining using acquisition parameters optimal to obtain high-quality SIM images (in this case, 50 ms of exposure time, 10% laser power). In contrast, the low signal-to-noise ratio images were acquired from the LifeAct-RFP channel using acquisition parameters more suitable for live-cell imaging (in this case, 100 ms of exposure time, 1% laser power). The dataset provided with the 2D CARE notebooks are maximum intensity projections of the collected data.
Label-free prediction (fnet) training dataset. The dataset provided for training label-free prediction notebook was designed to predict a mitochondrial marker from bright-field images. Before the acquisition, fixed HeLa ATCC cells were permeabilised and blocked using 0. 25% Triton X-100 (Sigma-Aldrich, Germany) and 3% IgG-free bovine serum albumin (BSA, Carl Roth, Germany) in PBS for 1. 5 h. Cells were labelled for TOM20 using 5 μg/ml rabbit anti-TOM20 primary antibody (sc-11415, Santa Cruz, USA) and 10 μg/ml donkey-anti-rabbit-secondary antibody (Alexa Fluor 594 conjugated, A32754, Thermo Fisher, USA) in PBS containing 0. 1% Triton X-100 and 1% BSA for 1. 5 h each. Samples were rinsed twice and washed thrice with PBS (5 min) after each incubation step. Image stacks were acquired on a Leica SP8 confocal microscope (Leica Microsystems, Germany) bearing a ×63, 1.
40 NA oil objective (Leica HC PL APO). The pixel size was set to 90 nm in XY-dimensions, and 150 nm in Z (32 slices) and fluorescence image stacks were recorded using 561-nm laser excitation and collected by a (photomultiplier tube) PMT. The corresponding transmitted light image stack was recorded in parallel using a transmitted light PMT. We acquired 25 3D stacks with dimensions of 1024 × 1024 × 32. To create the training set each image stack was split into four stacks with dimensions of 512 × 512 × 32, giving a dataset of 100 images of which 92 were used for training and 8 unseen for testing and quality control (see Table ). The raw data were converted into. tif file format and split into stacks of the respective channels (fluorescence and transmitted light). To prepare a training set, stacks were split into individual folders by channel.
To create matched training pairs, the signal files (transmitted light) and their respective targets (fluorescence) must be in the same order in the irrespective folders. It is therefore advisable to number source-target pairs or to give the files the same names. Deep-STORM training and example dataset. For Deep-STORM, training data and test data can be generated via SMLM data simulations that mimic the experimental data type that needs to be subsequently analysed. This is directly possible within the notebook that we provide. The example experimental data that we provide was obtained from a glial cell in a culture of rat hippocampal neurons, fixed and stained with phalloidin-Alexa Fluor 647  82  , then imaged on a Nikon N-STORM microscope using ×100, NA 1. 49 TIRF objective (256 × 256 pixels with 160-nm pixel size, acquiring 59,900 frames at 15 ms/frame) controlled by NIS-Elements (version 4.
4). In order to simulate a higher density of emitters, we binned the data in groups of four frames using Fiji (Grouped z-project/Sum slices plugin) leading to 14,975 frames (BIN4), or groups of ten frames leading to 5990 frames (BIN10). The binning was performed on a 32-bits dynamic range. In order to train the model for the reconstruction of the BIN4 and the BIN10 dataset, 20 frames of a 64 × 64 pixels field-of-view were simulated with parameters matching the experimental datasets closely. The parameter lists are shown in Table  1. The localisation files obtained from our Deep-STORM notebook were subsequently imported in ThunderSTORM 51 for drift correction using crosscorrelation analysis  83. For DNA-PAINT imaging of the tubulin cytoskeleton, U2OS cells were fixed and immuno-labelled with mouse-anti-β-tubulin primary antibody (32-2600, Thermo Fisher) and DNA-conjugated (P1-docking strand) secondary antibody.
DNA-PAINT imaging was performed using a Nikon N-STORM system bearing a Nikon Apo TIRF ×100 oil immersion objective (1. 49 NA). In all, 2 nM P1-ATTO655 (8 nt duplex, Eurofins) in PBS pH 8. 2 + 500 mM NaCl was added to the sample, resulting in a labelling density significantly higher than used for conventional DNA-PAINT imaging. Fluorophores were excited in highly inclined and laminated optical sheet mode using ~1 kW/cm² 647-nm laser illumination. In total, 2000 frames were recorded at 10-Hz frame rate and an EM gain of 200. Parameters for simulating the training data (TUB) were estimated from the raw movie or ThunderSTORM ME-MLE reconstruction of a few frames and are listed in Table  1. The training was performed for 30 epochs and 1750 steps per epoch using a batch size of 4. Both localisations obtained from Deep-STORM and ThunderSTORM analysis were corrected for drift using the cross-correlation function in ThunderSTORM.
Emission events that are split onto subsequent frames were merged, applying a 40 nm distance threshold (no dark frame allowed). CycleGAN training dataset. U2OS cells were plated on fibronectin-coated glassbottom dishes (MatTek Corporation) for 2 h before methanol fixation at -20 °C for 5 min. Fixed samples were washed three times using PBS and stained with an anti-tubulin antibody (clone 12G10, Developmental Studies Hybridoma Bank) for 35 min at room temperature. Samples were then washed thrice with PBS and incubated with an anti-mouse secondary antibody conjugated to Alexa488 (Thermo Fisher Scientific. Cat number: R37114). Stained samples were washed thrice with PBS and kept at 4 °C, in PBS, until imaging. The SDC dataset was acquired using a Marianas spinning disk equipped with a Yokogawa CSU-W1 scanning unit on an inverted Zeiss Axio Observer Z1 microscope and a ×100 (NA 1.
4 oil, Plan-Apochromat, M27) objective and controlled by SlideBook 6 (Intelligent Imaging Innovations, Inc. ). Images were acquired using a Photometrics Evolve, a back-illuminated EMCCD camera (512 × 512 pixels). For each field of view, 200 images were acquired. The SDC images used to train CycleGAN were generated by performing average projections of the collected images. To maintain a uniform pixel size across the SDC, SIM, and Fluctuation-based super-resolution (FBSR) images, the SDC images were magnified by four using a bilinear interpolation. The fluctuation-based super-resolution dataset was acquired by processing the SDC images using the latest implementation of NanoJ-SRRF  55  within the ImageJ software  21, 57. This new version of SRRF (super-resolution radial fluctuations) is available upon request and will be openly available for download soon.
The SRRF settings were chosen so that the least amount of errors were present in the reconstructed images (estimated using SQUIRREL  52  ) and were as followed: "vibration correction", on; radius, 2; sensitivity, 2; magnification, 4; temporal analysis, average; intensity weighting, on; macro-pixel patterning correction, on. The SIM dataset was acquired using a DeltaVision OMX v4 (GE Healthcare Life Sciences) fitted with a ×60 Plan-Apochromat objective lens, 1. 42 NA (immersion oil RI of 1. 512) used in SIM illumination mode (five phases and three rotations). Emitted light was collected on a front-illuminated pco. edge sCMOS (pixel size 6. 5 μm, read-out speed 95 MHz; PCO AG) controlled by SoftWorx. Each dataset was augmented by five by randomly cropping (original size 2048 × 2048 px, crop size 1280 × 1280), flipping and rotating the original images.
This augmentation pipeline was generated using Augmentor  32  , for which we also provide a ZeroCostDL4Mic notebook. pix2pix training dataset. DCIS. COM lifeact-RFP cells were incubated for 2 h with 0. 5 μM SiR-DNA (SiR-Hoechst, Tetu-bio, Cat Number: SC007) before being imaged live for 14 h (1 picture every 10 min) using a spinning-disk confocal microscope. The spinning-disk confocal microscope used was a Marianas spinning-disk imaging system with a Yokogawa CSU-W1 scanning unit on an inverted Zeiss Axio Observer Z1 microscope (Intelligent Imaging Innovations, Inc. ) equipped with a ×20 (NA 0. 8) air, Plan-Apochromat objective (Zeiss). Statistics and reproducibility. Unless otherwise specified, all experiments were performed once. Here, we are assessing the performance of computer algorithms and not the variability of biological systems.
Acknowledgements First and foremost, we would like to thank  Dr. Martin Weigert  from the  Swiss Federal Institute of Technology (EPFL)  and  Dr. Uwe Schmidt  from the  Max Planck Institute of Molecular Cell Biology and Genetics (MPI-CBG ), who pioneered a considerable portion of the technology this work is based on and whose ethos in making deep learning more accessible for microscopy helped inspire this work. Dr Schmidt  has kindly given key feedback during the preparation of the paper. All the network architectures and tasks presented here originate from already published work, having been edited and prepared for  Google Colab  to simplify their uptake by novice users. When using the Zer-oCostDL4Mic platform, please cite the original publications associated with each network. We would like to also thank  Estibaliz Gómez de Mariscal  and  Ignacio Arganda-Carreras  for providing code and support to export and use  U-Net  ( 2D ) and Deep-STORM models trained in ZeroCostDL4Mic in DeepImageJ.
This work was funded by grants from the  UK Medical Research Council  ( MR/K015826/1 ) (R. H. ), the  Wellcome Trust  ( 203276/Z/16/Z ) (R. H. ), the  Gulbenkian Foundation (R. H. ) , This  project has also received funding from the  European Research Council (ERC)  under the European Union's  Horizon 2020 research and innovation programme  (grant agreement No. [101001332]) (R. H. ) and an  European Molecular Biology Organization (EMBO)   Installation Grant  ( EMBO-2020-IG-4734 ) (R. H. ). R. F. L. would like to acknowledge the support of the  MRC Skills development fellowship  ( MR/T027924/1 ). This work was also supported by grants awarded by the  Academy of Finland  (to G. J. and P. K. M. ), the  Sigrid Juselius Foundation  (to G. J. and P. K. M. ), the  University of Turku foundation and Turku Doctoral Program in Molecular Medicine (TuDMM) (to S.
H. P. ),  Åbo Akademi University Research Foundation  ( CoE CellMech ; G. J. ) and by  Drug Discovery and Diagnostics strategic funding to Åbo Akademi University  (G. J. ). We thank  Dr Aki Stubb  for providing us with the raw data used to showcase Noise2Void  2D. The Cell Imaging and  Cytometry Core facility  ( Turku Bioscience, University of Turku, Åbo Akademi University, and Biocenter Finland ) and  Turku Bioimaging  are acknowledged for their services and instrumentation and expertise. M. L is supported by  Victoriastiftelsen (FI). A. K. , T. -O. B. and F. J. funded by the  German Research Foundation (DFG)  under the code  JU3110/1-1  and  German Federal Ministry of Research and Education  under the code  01IS18026C , ScaDS2. M. H. and C. S. acknowledge funding by the  German Science Foundation  (grant nr.
Data availability All our example datasets are available for download in Zenodo (Links also available on our GitHub page): CARE 3D  85  , CARE 2D 86 , Noise2Void 3D 87 , Noise2Void 2D 88 , Deep-STORM  89  , CycleGAN 90 , pix2pix 91 , YOLOv2 92 , StarDist 2D  93  , label-free prediction fnet  94. The datasets used to train 2D U-net were originally published as part of the ISBI 2012 segmentation challenge  32, 79  and were retrieved for this work from https://github. com/zhixuhao/unet. The dataset used for 3D segmentation in U-Net 3D is publicly available from the page of the École polytechnique fédérale de Lausanne (EPFL): https:// www. epfl. ch/labs/cvlab/data/data-em/. Source data are provided with this paper.
Global scale strategy: Scanning a surface of interest (the shell-scan) We estimate the biological surface of interest by performing a pre-scan of the sample, collecting only ~0. 1% of the voxels. Such a fractional pre-scan is a negligible contribution to the total light dose received by the biological sample. The full method for surface estimation is detailed in the methods section, associated Supplementary Fig. S2 , and supplementary text. Briefly, the fluorescence data collected in the prescan is first normalized to remove spatial inhomogeneities of the intensity distribution (equation (1), Fig. S2a, b ). We detect bright points from this normalized intensity distribution and remove bright points that are not part of the most densely populated surface through filtering approaches, including local RANSAC polynomial fits on overlapping windows (Fig.