dataesr/scientific-paragraphs-categorization huggingface dataset:
In Vivo Pharmacokinetics For pharmacokinetics, whole blood was collected 5, 10, 15 min, 1, 2, 4, 8, 16, 24, 48, 72 h, 7 and 14 days after injection. Among the 63 measurements of PFOB in blood, 21 records (33%) were below the Lower Limit of Quantification (LLOQ) and were excluded from the pharmacokinetic analysis (one mouse at the 24 h sampling point and all mice at the 48 h to 14 days sampling points). The retained data are summarized in Table  S3. The median PFOB concentration-time profile, overlaid with observed data points, is displayed in Figure  4a  (same time points under the LLOQ were excluded). The pharmacokinetic curve indicates that droplets had a circulation time consistent with an efficient BBB crossing after FUSmediated BBB disruption. The peak plasma concentration of PFOB (C max = 25.
After BBB opening, mice were sacrificed, and undelivered droplets were washed from the harvesting. Brain tissue penetration of DiD-labeled droplets was first observed in four mice 1 h after BBB opening, after brain sectioning and using fluorescence microscopy. Delivery of droplets across the BBB was successful in all mice (n = 4). As can be seen from coronal brain slices (Figure  5b ), DiD fluorescence signal from STE was clearly observed only in the hemisphere subjected to BBB disruption. By contrast, no fluorescence signal was observed in the hemisphere with non-disrupted BBB. Some assays were performed to assess the potential extravasation of STE after i. v. injection and FUS exposure without SonoVue ® injection, i. e. , without BBB disruption. Interestingly, no DiD fluorescent signal from STE was observed on the brain (data not shown), suggesting that there is no DiD extravasation under these US conditions.
Task Description The Memory Game includes questions testing both paired association and recognition memory, depending on the distractor photos present (Figure  3 ). In a recognition question, the 2 distractor photos are not part of any pairs seen in training and have not been previously viewed. In a paired association question, 1 of the distractors will be the matching photo for a prompt not currently being displayed and so will have been previously viewed. The images used have been selected based on their shape and appearance from the Bank of Standardized stimuli (BOSS) database created by Dr Mathieu Brodeur.
Task Development We initially developed the Memory Game for testing using E-Prime (Psychology Software Tools) and administered it to children with NDD in our developmental neurology clinic, using a touch screen laptop under the supervision of trained undergraduate neuroscience students. This allowed us to optimize the instruction video and the choice of images, and refine the task. For instance, we found that having a voiceover during the instruction video made it easier for the participants and caregivers to follow. The choice of the pictures used for the paired association and recognition testing, which were standardized in the BOSS database, was reviewed for general understandability and quality assurance with caregivers and for formatting (orientation, color, and shapes) with 2 psychologists from our team (SAW and JP).
Similarly, we changed the position of the "correct response" from left, middle, and right to prevent a "false success" if, for instance, an individual always chose a given position. We also sought feedback from caregivers present in the room during the testing about the flow used and challenges in understandability. In addition to general comments, we ensured that the participants understood the instruction video and the training and testing phases of the Memory Game. Once established in E-Prime software, a widely used program for stimulus presentation  [77] , the final version of the task was discussed with the programmers (Jeffrey Van Alstine, Arthur Schuiltz, Department of Education from University of Alberta) and developed for web-based testing on tablets. The Memory Game underwent several revisions to optimize responsiveness.
Coding Approach The Memory Game consists of 3 parts: the front-end game interface, the researcher interface, and the application programming interface (API) that both communicate with. The front-end game interface is static HTML running a custom web application built with jQuery 1. 11 (OpenJS Foundation) and reliant on the YouTube iFrame API. The researcher interface is a small custom PHP 7 application with no dependencies. The API itself is constructed on the PHP Slim framework v3. 4, uses PHPMailer v5. 3 to send confirmation or reminder emails through a local university SMTP server, and in turn saves the data in a MySQL (Oracle Corporation) database with any identifying information (email addresses) encrypted such that it can only be decoded and read by researchers with the correct key. The Memory Game is web based and can be accessed in  [78].
Data Analysis From the game component, the percentage of correct responses was recorded, as well as the time required to answer each question as reaction time (RT). The performance of participants with NDD was compared against the corresponding age group of the TD participants. The groups were broken down into 2 age groups: 5-9 years and 10-14 years. The percent correct score is given as a percentage of questions answered correctly. The performance on day 1 and day 2 was plotted. A line joining those 2 was drawn and the slope was used to estimate memory decay. RT is the average time participants took to answer the questions and is given in ms. Unpaired t tests with assumed Gaussian distribution (using parametric test), no assumption about consistent SDs (the Welch t test), and no correction for multiple comparisons were performed (P value * <.
Moreover, despite being potentially easier by being image-based rather than text-based paired association tasks, we recognize that many individuals with NDD may still not be able to perform the testing. In order to be of extensive use, the tutorial video will need to be translated, as well as some of the buttons, but the pictures used will be readily reusable. This study also measured performance across different age groups but does not provide longitudinal data about memory performance in function of previous performance. Therefore, longitudinal follow-up will be important. We also recognize the importance of including individuals with diverse NDD etiology or a larger sample size of individuals, with a given specific NDD diagnosis to further assess diversity in the future and correlate the Memory Game findings with other cognitive tests such as for attention, working memory, intelligence quotient, or simpler form of plasticity (sensitization and habituation).
The Human Protein Atlas (HPA) Subcellular Section has generated the first subcellular proteome map of human cells, consisting of a publicly available dataset of 83,762 confocal microscopy images detailing the subcellular localization of 13,041 proteins (HPAv21)  6  across a multitude of cell lines. Classifying the subcellular protein locations from these images is currently hampered by various technical challenges. First, around 55% of human proteins localize to multiple subcellular compartments, indicating that they might be involved in several biological processes  [6] [7] [8]  , making it a multi-label classification problem. Second, the distribution of proteins in these compartments is highly disproportionate by up to five orders of magnitude (HPAv21)  6  , causing an extreme class imbalance.
The key to success for deep learning models has been their ability to learn complex visual features from large quantities of labeled data (that is, supervised learning). Given that labeled data are expensive, there has been an increasing trend among deep learning researchers to develop models that can take advantage of unlabeled or weakly labeled data (that is, weakly supervised learning). For example, class activation map (CAM)-guided approaches  18  and weakly supervised adaptations of mask R-CNNs (region-based CNNs)  19, 20  were designed to localize signal regions and assign precise object labels given coarse image-level labels. In 2018 the Human Protein Atlas Image Classification competition focused on crowdsourcing machine learning solutions for multi-label image classification for a class-imbalanced dataset  15.
AnAlysis NATure MeTHodS of a sub-pattern in the nucleolus  21  or construction of a multiscale hierarchical cell model by using the image features derived from the classifier  22. It is a great demonstration of how competitions can be used as a tool to perform large-scale investigation of state-of-the-art methods by defining a clear research problem in the setting of a competition. As a continuation, deep learning models for generating single-cell protein labels will provide higher-quality feature maps, and enable studies of localization heterogeneity and dynamics in single cells and potentially facilitate the discovery of new subtle, rare or transient patterns that cannot reliably be observed across entire cell populations. This motivated us to host another competition to crowdsource solutions for using our HPA image dataset annotated with image-level labels in a weakly supervised setting to generate single-cell-level protein localization labels.
In this paper we present the design and analysis of the results from the competition Human Protein Atlas -Single-Cell Classification hosted on the Kaggle platform (https://www. kaggle. com/c/hpa-single-cell-image-classification). Given images, image-level labels and pretrained segmentation models, the participants were required to submit a solution able to generate a segmentation mask and single or multiple labels for each single cell in every image in a hidden dataset. Over 3 months, 757 teams registered a total of 19,058 submissions. The top-ranking teams were awarded with cash prizes: US$12,000, $8,000 and $5,000 for first, second and third place, respectively. The winning models were able to capture relevant biological features, and greatly improved the state of the art for classification of subcellular protein patterns.
Results Competition design and evaluation metrics. The aim of this competition was to develop computational models to classify subcellular protein localization patterns for single cells in microscopy images, given a training set with only image-level labels (Fig. 1a ). Successful submissions consisted of an individual segmentation mask and the protein localization labels (one or more of 19 classes) for every cell in each image of the test dataset. A training and test dataset was assembled to facilitate effective model training and to alleviate somewhat the extreme class imbalance in the HPA image dataset (Methods). The training dataset consisted of 21,806 confocal microscope images of 17 different human cell lines and proteins encoded by 7,807 different genes. Furthermore, participants had the possibility to use any publicly available external dataset or pretrained model, including the full HPA Subcellular Section dataset (HPAv20: 82,495 images)  6  (Supplementary Tables  1  and 2 ), as well as the pretrained in-house HPACellSegmentator (https://github.
com/CellProfiling/HPA-Cell-Segmentation) or any other segmentation tool (for example, Cellpose  23  ). Each image corresponds to a field of view consisting of multiple cells (average, 20 cells per image) where the protein of interest has been visualized together with three reference markers for the nucleus, microtubules and endoplasmic reticulum (ER). The images in the training dataset were annotated with the standard HPA annotation pipeline  6, 24  , in which one or multiple labels were assigned to each image by assessing the localization patterns of all of the cells. The labels for each cell in the image are therefore not guaranteed to be precise because of single-cell heterogeneity. That is, in the same image of a genetically identical population, individual cells can have different protein localization patterns (Fig.
2 ). This makes the data weakly labeled and the classification task weakly supervised. To evaluate the performance of the participating models, a test dataset of 41,597 cells in 1,776 images with pronounced single-cell heterogeneity was manually annotated for single-cell labels (Methods) and used as the test dataset (Supplementary Tables  1  and 2 ). The test set was divided into a public test set with 31% of the images (n = 559) and a private test set with 69% of the images (n = 1,217), which has never been published before (Supplementary Table  1 ). Importantly, the participants can only see the images in the public test set and do not have access to any single-cell labels. In contrast to our previous Kaggle competition  15  , we chose to make this a code competition, in which participants were required to predict on test images by executing their source code via the Kaggle Notebook environment.
68 ± 0. 10), Nuclear Membrane (AP 0. 68 ± 0. 10), Microtubules (AP 0. 63 ± 0. 09) and Nucleoli (AP 0. 63 ± 0. 09). Classes with fewer training samples, highly varied location distribution and merged classes such as Negative (AP 0. 34 ± 0. 11), Centrosome (AP 0. 33 ± 0. 07), as well as Vesicles and cytosolic punctate pattern (AP 0. 31 ± 0. 06) achieved lower AP scores. Mitotic Spindle had the highest spread of performance in the top 50 teams (AP 0. 38 ± 0. 23), which probably stems from the approach of manual labeling of rare classes that some teams adopted (Supplementary Table  5 ). Interestingly, Fig. 1  | Challenge overview. a, For training, participants were given access to a well-balanced training set and the public HPA dataset, which consists of fluorescent images with four channels and corresponding image-level labels.
To evaluate the solutions' performance, a private (hidden) dataset of 1,776 images containing 41,000 single cells with corresponding single-cell labels was used. b, Overview of the main approaches: image-level models take in images and predict image-level labels, which can be combined with CAMs of different classes and segmented cell regions to give the final cell-level labels. Image-level models can also be trained on whole images, but predict on bag-of-cell tiles at inference to produce the cell labels. Cell-level models take in single-cell crops and predict single-cell labels. ER, which was provided as a reference channel, achieved a medium performance in the public test set (AP 0. 42 ± 0. 10) and the worst performance in the private test set (AP 0. 25 ± 0. 07). This difference could be attributed to the use of 3-or 4-image channel inputs by the teams.
Overview of approaches. Most solutions used one of two main approaches, hereby referred to as 'cell-level models' and 'image-level models' (Fig. 1b  and Supplementary Table  6 ). Image-level models take in whole images and predict image-level locations. To obtain single-cell labels, teams either combined activation maps of individual classes with segmented cell regions or used a bag-of-cells approach, which concatenates augmentations of one single cell into an image input. Cell-level models require segmented cell inputs to produce single-cell labels. The solution complexity varied greatly, with, for example, the ensembling of four single models by team 4, and 18 single models by team 2. Most teams used traditional networks as backbones and did specific modifications to achieve higher performance on this specific dataset.
4, 5  and Supplementary Note 1). The results of image-level prediction and cell-level predictions were combined either by addition or multiplication. To handle the extreme class imbalance, several teams used weighed loss or oversampling. The most popular loss in this competition was focal loss  33  , which addresses the large class imbalance in the training and validation datasets by decaying the scaling factor when the confidence of that class increases. The addition of external data and hand-labeled data also improved the performance for rare classes such as Mitotic Spindle. Given that most teams made use of the HPACellSegmentator (Supplementary Table  5 ), with several community-developed improvements such as faster runtime, no significant correlation can be seen between segmentation score and average precision score (Methods, Supplementary Fig.
Discussion Organizing a citizen science competition with the aim to produce useful models requires deep knowledge of the data, the current state of the art of relevant DNNs, and a clear view of the desired aim of the competition (that is, the area where innovation is needed). With this in mind, the challenge description, training and test datasets and evaluation metrics must be assembled to best assist participants to reach the competition goal under the given restrictions. There is a risk that the winning solutions are ensembles of weak learners, much too complex to be deployed, or that the models are unable to capture relevant biological features, making them irrelevant for future use. To mitigate these pitfalls we need to guarantee fair metrics and fair data. For metrics, we chose mAP, which focuses on overall model performance, rather than the threshold-specific performance for each class in the case of the more commonly used F1 score.
For data, we ensure that the single-cell labels are accurate (90% agreement between human annotators, Methods) and that the dataset reflects the normal physiological proportion of subcellular patterns. Given that the competition is time constrained, we made the conscious choice to reduce the magnitude of class imbalance from five to two orders of magnitude (Supplementary Table  1 ). And given that no single-cell labels were provided for training, participants relied on their public leaderboard score (that is, the public test set) for validation. The careful division between the public and private test sets ensures the same proportional class imbalance. This led to a very minor shake-up in the final ranking, supporting our fair competition design of datasets and metrics. Throughout the competition we observed great community spirit and an open science approach.
Methods Segmentation model. For segmentation of single cells, a pretrained DPN (dual path network) U-Net, which was the winning architecture in the 2018 Data Science Bowl  43  , was further trained on HPA data. For each image, Nuclei, Microtubules and Endoplasmic Reticulum channels were used as 3-channel input and manually annotated cell masks were used as ground truth. Each channel had a frame size of 2,048 × 2,048 pixels and was acquired with a ×63 objective with a pixel resolution of 80 nm. The DPN U-Net neural network was trained and validated on a dataset of 266 HPA Cell Atlas images from 28 cell lines, which were split into training (257 images) and validation (9 images) datasets. During inference, a pretrained DPN U-Net for nuclei segmentation  43  was used to generate nuclei mask images, followed by customized post-processing steps with the watershed algorithm to obtain the individual nuclei.
Then, using the DPN U-Net trained on our cell images, we obtained the cell masks. The nuclei obtained in the first step were combined with the cell mask in a watershed-based post-processing step to produce the segmented cells. The activation function was softmax, and the loss function was a combination of BCE (binary cross entropy) and Dice Loss. The model was trained using one Nvidia GeForce RTX 2070 graphics card until convergence, which took approximately 1. 5 h (with pretrained weights). The source code for the DPN U-Net segmentation model is available at https://github. com/cellProfiling/ hpa-cell-segmentation. Data and pretrained weights are available at https://zenodo. org/record/4665863. Single-cell annotation pipeline. The annotation of ground truth segmentation masks and single-cell labels followed a two-step procedure.
In the first step, a pretrained DPN U-Net  43  -based model (HPACellSegmentator; https://github. com/CellProfiling/HPA-Cell-Segmentation), with weights trained on HPA public images, was used to generate baseline segmentation masks for all images in the test set. Every image was subsequently manually inspected and adjusted for inaccurate cell masks. In the second step, individual cells were annotated by experts from the HPA team. The cells were presented to the annotator one by one, and no metadata were shown except the image-level labels. Each cell was shown in the bounding box region of its segmentation mask, padded by 50 pixels on all sides. The order of cells to annotate was randomized so that cells in the same image were not presented in sequence. A total of 10% of all cells were annotated by multiple people to control for annotation consistency.
In this overlapped single-cell set, labels are 90% consistent between multiple annotators. (This agreement is much higher compared with chance overlap, for which the probability of two annotators choosing the same organelle label out of 30 labels is 0. 1% and that for choosing two of the same organelle labels out of 30 is 0. 0001%, while some cells have up to four annotators and some cells have more than two labels, or 67% agreement between four pathologists for surgical pathology reports  44. ) Label processing and final competition dataset. In the annotation process, each single cell was classified into one or more of the 30 organelle labels or Negative/ Unspecific. These labels were: Nucleoplasm, Nuclear membrane, Nucleoli, Nucleolar fibrillar center, Nuclear speckles, Nuclear bodies, Endoplasmic reticulum, Golgi apparatus, Peroxisomes, Endosomes, Lysosomes, Intermediate filaments, Actin filaments, Focal adhesion sites, Microtubules, Microtubule ends, Mitotic spindle, Centrosome, Lipid droplets, Plasma membrane, Cell Junctions, Mitochondria, Aggresome, Cytosol, Cytoplasmic bodies, Rods and Rings, Cleavage furrow, Centriolar satellite, Vesicles, Mitotic chromosome, and Negative-Unspecific.
Statistical analysis. Plotting and statistical analysis were performed with Python 3, NumPy, Pandas, scikit-learn, seaborn, Matplotlib and plotnine. Overall participation and performance (members per team, submissions, public and private scores for each submission and so on) were extracted from Meta Kaggle (https://www. kaggle. com/datasets/kaggle/meta-kaggle). To test the correlation between mAP scores and segmentation IOU, Pearson correlation was performed. First, the IOU of matched cells (at a threshold of 0. 6) was calculated for the best submission of the top 50 teams. (Most top teams chose two submissions for consideration for final scoring. For this analysis, the highest scored submissions were manually selected for each team that did not choose specific submissions. ) For the same submissions, average precision per class and average precision per private and/or public test set were calculated using the TensorFlow Object Detection application programming interface (https://github.
Acknowledgements The authors thank all of the participants of the  Human Protein Atlas (HPA) Image Classification competition and acknowledge the staff at Kaggle , especially  P. Culliton  and  M. Demkin , for providing the competition platform that enabled this study. The staff of the  HPA program  provided valuable contributions, such as data storage and management, and  J. Fall  helped with project administrative tasks. The competition prizes were funded by  Kaggle. Research funding was provided by the  Knut and Alice Wallenberg Foundation  (grant no. 2021. 0346 , 2021. 0189 , 2018. 0172 ), the  Erling Persson Foundation  and the  Swedish Research Council  (grant no. 2017-05327 ), and Stanford startup funding to E. L.
Data availability The dataset used for the HPA competition is available at https://www. kaggle. com/c/hpa-single-cell-image-classification. The external dataset HPAv20 is publicly available at https://v20. proteinatlas. org/. A script for downloading the dataset is available at https://www. kaggle. com/lnhtrang/ hpa-public-data-download-and-hpacellseg.