dataesr/scientific-paragraphs-categorization huggingface dataset:
Dataset The breast ultrasound dataset is from Al-Dhabyani et al. [27] , which is composed of 780 images from 600 women divided into three categories: 133 normal images, 437 benign images, and 210 malignant images. In this work, the normal and benign images are combined into one class, and the malignant images are another class, corresponding to the binary classification of the primary task. Both the benign and malignant images in the dataset are labeled with segmentation masks for the tumor areas that can be used to generate the patch-token labels for the secondary task. Figure  4  shows some sample images and their masks in the original dataset. The morphological characteristics of the breast ultrasound images are shown in Table  1. In general, the malignant lesions are more irregular in shape than the benign lesions, and they are more likely to be confused with normal human tissue.
The image sizes in the original dataset are not standardized. In order to be able to input the ViT models, we unified the image sizes to 384 × 384 by reshaping it. We randomly selected 118 (about 15% of the total samples) images as the test set, of which 33 images are malignant, 65 images are benign, and 20 images are normal. The remaining 664 images are used as training samples.
Experimental Setting Our experimental platform is Torch 1. 10 and Python 3. 7. 7 @ an RTX 3090Ti GPU. All models haved trained with the Adam  [28]  optimizer, with an initial learning rate of 1 × 10 -4 and beta = (0. 9, 0. 999). The transformer encoder in the standard ViT contains 12 encoder blocks, and the dimension of all tokens is set to D = 768. For comparison, the ViT-Patch did not make any changes to the other modules of the standard ViT except for adding the MLP header to the patch tokens. All MLP heads used in the ViT and ViT-Patch have the same structure, containing a hidden layer of size 4D with the GeLU activation. All ViT-based models in our experiments are trained for 50 epochs. To verify the effects of different patch divisions on the performances of ViT-based models, we divided the 384 × 384 ultrasound images into patches of 3 × 3, 6 × 6, 12 × 12, and 24 × 24, forming patches of sizes S = 128, 64, 32, and 16, respectively.
Experimental Results We validate the performance of the ViT-Patch for whole-image-based malignant identification and patch-based tumor localization, based on the breast ultrasound dataset. The experimental results for malignant identification are compared with the standard ViT, and some traditional models which have been widely used for comparison. The learning curves of the standard ViT and ViT-Patch models are compared, to validate the improvement of the proposed architecture for model training. In addition, the differences between the deep token features learned by the standard ViT and the ViT-Patch models are analyzed to show the optimization of the proposed architecture for self-attention-based feature learning. Malignant identification: The primary task (malignant identification) is evaluated using Accuracy (ACC) and Sensitivity (SEN), ACC = TP + TN TP + TN + FN + FP (11) SEN = TP TP + FN ( 12 ) where TP indicates the number of True Positive samples, FP indicates False Positive, FN indicates False Negative, and TN indicates True Negative.
g. , optimizing the RPN or ROI modules of the Faster-RCNN model  [33]. The DICE coefficient is used to quantitatively evaluate the secondary task, DICE = 2TP FP + 2TP + FN (13) Table  3  shows the DICE results for the ViT-Patch with different patch sizes. In contrast to the performance on the primary task, the ViT-Patch with the largest patch size (S = 128) has the best DICE. The overall trend is that models with small patch sizes have a large deviation between the predicted mosaic maps and the ground truth. However, it must be noted that small patches imply higher spatial resolution. The difficulty of the second task itself increases dramatically as the patches size decreases.
Conclusions This paper has proposed an improved ViT architecture, ViT-Patch, which introduces a secondary task on the patch tokens, in addition to the primary task on the class token. The standard ViT used for breast ultrasound image classification only constructs one MLP head on the class token for supervised training, while a larger number of patch tokens are in suspense. This unbalanced architecture tends to lead to an overfitting of the ViT models on small training datasets. The ViT-Patch constructs an MLP head for each patch token (with shared weights across patches), and introduces supervised information on whether each patch overlaps with the tumor areas as a secondary task. The improved architecture not only gives the model the ability to localize tumors based on patches, but also improves on the performance of the primary task (i.
e. , malignant identification) on the class token, because more task-related supervised information is provided. The ViT-Patch is validated on the public breast ultrasound image dataset. The experimental results show that in the malignant identification task, the ViT-Patch performs significantly better than the standard ViT at all patch sizes, and the ViT-Patch models of small patch sizes also outperform the CNN-based baseline models. Besides malignant identification, the ViT-Patch can provide patch-based mosaic maps for tumor localization. In the experimental part, we also compared and analyzed the learning curves and the learned token maps of the ViT and ViT-Patch, which demonstrate that the ViT-Patch architecture can improve the generalization of the ViT model and optimize the feature learning at deep levels.
e. , Natural Language Processing, Artificial Intelligence, Machine Learning, Deep Neural Networks and entity recognition in clinical narratives)  [15] [16] [17] [18] [19] [20]. For the purpose of this study, we collected phenomic and genomic data from unsolved RD cases in a standardized and machine-readable format, and ran a series of one-to-all comparisons, including cases and known RD in order to raise diagnostic hypotheses based on re-analysis of candidate causative variants. These were then resubmitted to clinicians for further investigation and validation. Ultimately, this led to the identification of a formerly undescribed disease, or to the identification of unreported manifestations of known RD. The final goal was to return a clinical diagnosis to the patients. This article aims to describe the methodology used in our phenotype similarity-based pipeline by providing illustrative examples of its application.
MATERIALS AND METHODS Data management The overall Solve-RD data management is documented  [7, 8]. Briefly, after submission to the RD-Connect GPAP, data are shared within the Solve-RD consortium via the European Genome-Phenome Archive (EGA)  [25]. We retrieved data for our analysis through a cloud-based central database RD3 (rare disease data about data) using the MOLGENIS open-source data platform  [26]. RD-Connect GPAP ensures data standardisation, pseudonymisation and harmonisation according to GA4GH-approved standards (Global Alliance for Genomics and Health)  [27] , in a computer-readable format (Phenopacket)  [28]  which enables exchange of phenotypic and family information.
Ontologies used To ensure data standardization, the Human Phenotype Ontology (HPO)  [22] , the Orphanet Rare Disease Ontology (ORDO)  [23]  and HOOM (HPO-ORDO Ontological Module)  [24]  were used. HPO provides a standardized vocabulary of phenotypic abnormalities encountered in human disease. For this study, we used the HPO 2020-10-12 release (including 15,656 unique HPO terms). ORDO is a structured vocabulary for rare diseases derived from the Orphanet knowledge base including relationships between diseases and between diseases and genes. This study uses ORDO v3. 1, Dec 2020 including 9338 active clinical entities. Clinical entities in ORDO are designated in this paper by their unique identifier in Orphanet, the ORPHAcode. HOOM is an ORDO module that qualifies the association between a clinical entity and its HPO-based phenotypic abnormalities according to their frequency.
Data and models employed Study population. Solve-RD collects phenomics and genomics data from patient cohorts  [7]  and releases them as data freezes. The present study uses data selected from the 2020 Data Freeze 1 which contains data (phenotype, pedigree, genotype) of 8370 cases affected by a RD and submitted by the four ERNs  [7]  to the RD-Connect Genome-Phenome Analysis Platform (GPAP)  [21]. This initial dataset contained 1101 "solved cases" (patients annotated as "solved" in the GPAP platform by the data submitter) and 7269 "unsolved cases" (patients annotated as "unsolved" by the data submitter, therefore with an inconclusive whole exome/ genome sequencing WES/WGS result). Our study population was built by filtering the cases as depicted in Fig. 1A:  (i) 644 cases with no causative gene were removed from the solved cases population and added to the unsolved cases population, as cases without a causative gene are considered unsolved in this analysis; (ii) cases without phenotype annotations that are not suitable for a phenotypic comparison were removed from both populations (72 solved and 3289 unsolved cases).
Thus, the resulting final population included 385 solved cases and 4624 unsolved cases (4. 6% and 55. 2% of the data freeze 1 respectively). Reference data. Reference data is obtained after applying the following filtering to ORDO (Fig. 1B ) where "groups of disorders" and "inactive clinical entities" were removed (see Orphanet's definitions in Additional informations file). Of the resulting active entities (6171 disorders and 993 subtypes of disorder) only entities associated with (a) at least one gene (3789 entities) and (b) at least one HPO term (3955 entities) are kept. Thus, the resulting 2259 entities (ORPHAcodes) which satisfy (a) and (b) served as reference data (31. 5% of Orphanet active clinical entities).
Workflow design As schematized in Fig. 2 , after having obtained similarity calculations we applied a workflow which is divided into three complementary approaches A, B and C, each using a triggering case (solved case) as a starting point for the purpose of validating the methodology (Fig. 3 ). Through in-house Python 3. 8 scripts, each approach generates a cluster around each point of interest thus maximizing the one-to-all phenotypic similarity explorations. Each point, which is a case or an ORPHAcode related to the triggering case, can hold genetic information if it exists and a list of genes is submitted to RD-Connect GPAP for identification of candidate variants  [8]. A-ORPHAcodes around the triggering case. This approach aims to check that the solved case phenotype (i) clinically fits a known RD phenotype (ORPHAcode) retrieved with the 50-first similar ORPHAcodes and (ii) is caused by mutations in the same gene (A1).
226 G > C, p. Ala76Pro, (Supplementary Table  A  and B )) seemed like a promising non-sense variant, even if not available on ClinVar. It is located in the part of the gene coding for the protein motor region and it was found in a case at rank 16 of similarity, with a phenotypic description of pure spastic paraplegia. Despite the fact that the parent's DNA was not available and no functional analysis could be carried out, this finding will likely solve the case. The SPAST missense variant identified (c. 134 C > A;p. Pro45Gln, (Supplementary Table  A  and B )) in an unsolved case within the SPAST cluster bears conflicting evidence of interpretation in reference databases and it does not segregate in the other symptomatic siblings; therefore, this case could not be solved by this approach. The variant found in the TBL1XR1 gene (c.
Five KIF5A candidate variants were found in four unsolved cases in a SPG10 (ORPHA:100991)-centred cluster. Despite their classification as likely pathogenic in ClinVar, they were discarded because they are not in the motor region of KIF5A gene, suggesting that they might have been erroneously classed as likely pathogenic in the reference databases. In addition, for one unsolved case clinically consistent with SPG10 phenotype, a variant (c. 1373 C > T, (p. Ser458Phe) was identified. This case has been published, and although the variant in KIF5A was discussed by the authors, the case was still labelled as unsolved  [40]. Indeed, clinicians agreed on the diagnostic hypothesis but suggested that the study of further cases is needed before certifying the variant's pathogenicity. No variants were found in the CASQ1-related ORPHAcodescentric clusters.
Furthermore, not every ORPHAcode in the Orphanet knowledge database already has phenotypic annotations, as manually curated annotations are a long, ongoing process, and hence some could not be used by similarity calculations. Nevertheless, it is clear that the process can become more effective as the study population grows and phenotypic/genotypic annotations are added and improved. Secondly, most of RD are multi-systemic disorders, however we have observed that phenotypic annotations are often influenced by the annotating physician's medical specialty, thus a bias in calculations could be introduced because of missing discriminant HPO terms. Similarly, we have noticed the trend of using a set of "coarse grain" HPO terms to the detriment of lower and more specific HPO terms, compromising the specificity of the case annotation.
This methodology enables the user to rapidly filter genomic data from thousands of datasets thanks to a specific gene list generated for each case and each approach as an output of the similarity calculations. A list of candidate variants based on any pre-defined filtering step is thus produced: in this case rare variants with a high or moderate impact at the protein level. The whole process is scalable and can be automated. Current limitations lie in data interpretation as the list of candidate variants can be fairly long (>10 variants per case) in approaches comparing up to 50 individuals / disease entities. The elevated number of cases assessed made it impossible for clinicians to evaluate all variants, therefore we had to restrict the submission of results to variants classified as likely pathogenic or pathogenic according to ACMG criteria, missing variants of uncertain significance that could be re-classified as pathogenic after expert evaluation.
ACKNOWLEDGMENTS The authors would like to thank the data submitters of our triggering cases,  Vincenzo Nigro ,  Rebecca Schule ,  Laurence Faivre , and all the clinicians who attended the four workshops for their participation and data interpretation. This study was made possible thanks to the European Reference Network(s) ERN-ITHACA, ERN-RND, ERN-Euro NMD and  ERN-GENTURIS  (https://ec. europa. eu/health/ern/networks_en). The authors are grateful to  Charlotte Rodwell  who contributed to the organisation of the work presented in this article as well as proof-reading.
This is interpreted in terms of an active ocean modifying the nearsurface atmospheric state, driving convection, mixing and entrainment of air from the free troposphere into the MABL. The estimated values of the oceanatmosphere coupling at the ocean small-scale are then used to develop a linear and SST-based downscaling method aiming to include and further investigate the impact of these fine-scale SST features into an available low-resolution latent heat flux (LHF) data set. The results show that they induce a significant increase of LHF (30% to 40% per °C of SST). We identify two mechanisms causing such a large increase of LHF: (1) the thermodynamic contribution that only includes the increase in LHF with larger SSTs associated with the Clausius-Clapeyron dependence of saturating water vapor pressure on SST and (2) the dynamical contribution related to the change in vertical stratification of the MABL as a consequence of SST anomalies.
For instance,  Cronin et al. (2019)  stress that the current ocean observing system does not meet the necessary requirements to accurately measure THFs, mainly due to a lack of global coverage of surface humidity and air temperature. Other studies highlight the challenges in representing THFs as bulk quantities using the MOST  (Fairall et al. , 2003; Edson et al. , 2013)  as the uncertainties in these parametrizations are large. Moreover, when such bulk formulations are integrated over the ocean basins they cause a large imbalance in the energy budgets  (Yu, 2019). As a consequence, there is a far-reaching ongoing effort to properly define and tune these THFs in numerical weather predictions as well as in Earth System models (ESMs). Differences of the order of 15% have been observed between model THF data sets and have been mainly attributed to the choice of parametrization scheme of the bulk formulae.
(2020)  highlight that the cold wakes of tropical cyclones have a similar imprint on the surface fluxes and the lower atmosphere. In fact, the cold wakes reduce the THFs, slowing down the surface wind and reducing cloud fraction and rainfall, by means of a cross-track secondary circulation  (Pasquero et al. , 2021). In-situ observations, despite being sparse and intermittent, provide unique and essential data to understand the physical processes at play. The EUREC 4 A (ElUcidating the RolE of Clouds-Circulation Coupling in ClimAte, www. eurec4a. eu) initiative  (Bony et al. , 2017)  aims to advance the understanding of the interplay between clouds, convection and circulation (and their role in climate) with an unprecedented extensive observational effort. The core activity of EUREC 4 A has been a 6-week field campaign between the 12th of January and the 23rd of February 2020 in the north-western tropical Atlantic.
28 times the length of the day  (Clayson et al. , 2014). Finally, to compute surface turbulent fluxes from near-surface variables, SeaFlux makes use of a neural network version of COARE3. 0 algorithm  (Fairall et al. , 2003)  which has been optimized for speed of use of processing. No additional parametrizations such as those taking into account the effects of waves or sea spray are considered. This data set has a spatial resolution of 25 km × 25 km and a time resolution of 1 hour. The variables we use in the present study are 2 m specific humidity (q 2m ), 10 m wind speed (U 10m ), 2 m air temperature (T 2m ), SST and LHF. Only daily averages of all the variables are considered in this study.
ERA5 The ERA5 reanalysis  (Hersbach et al. , 2020)  embodies a detailed record of the global atmosphere, land surface and ocean waves from 1950 onward. It does not include the coupling with an ocean model, but it is forced at the lower boundary by a prescribed SST and seaice concentration. In particular, it exploits various flavors of the Hadley Centre Sea Ice and Sea Surface Temperature (HadISST) data set as well as the Climate Change Initiative (ESA-CCI) SST v1. 1  (Merchant et al. , 2014)  up to 2007. The Operational Sea Surface Temperature and Ice Analysis (OSTIA) is considered for the modern period  (Donlon et al. , 2012). ERA5 has a spatial resolution of 0. 25°× 0. 25°and a time resolution of 1 hour. The variables used are 2 m dew point temperature, SST, T 2m , 10 m wind components, sea-level pressure (SLP) and LHF.
For simplicity, all these parameters are set to the constant reference values presented in Table  1. No rain rate or wave-related corrections are considered and fluxes are computed from skin temperatures, with the exception of WRF data where bulk SST is considered. Therefore, the cool skin correction to the SST is not applied to SeaFlux and ERA5 and is considered in WRF. As stated in the Introduction, the relation between atmospheric and near-surface oceanic variables is quite different depending on the spatial scale considered, with the atmosphere generally forcing the ocean at the large scale and the ocean forcing the atmosphere at the mesoscale and below. In order to separate the large from the small-scale patterns, U 10m , q 2m , T 2m and SST are filtered with an isotropic Gaussian filter as described in the Appendix.
Variable Value SLP (hPa) 1015 SW (W m -2 ) 150 LW (W m -2 ) 370 z u (m) 10 z t (m) 2 z q (m) 2 Mean latitude (°N) 12. 5 as a U , a q and a T respectively and their statistical significance is assessed using a two-sided t test. These coupling coefficients are computed taking into account the residual fields at all (daily) time steps in a subset of the grid points in the EUREC 4 A-OA region. A sub-sampling has been introduced in order to consider spatially uncorrelated data, as detailed in the Appendix. A generic coupling coefficient is written as a y = ∂ y 0 ∂ SST 0 : (3) The coupling coefficients are the core of the downscaling linear algorithm proposed in this study since they enclose all air-sea interaction processes. Given a certain SeaFlux/ERA5 variable (y) we build a new data set of the same variable including fine-scale SST features (y HR ) as y HR = y + a y DSST, (4) where DSST stands for the SST correction and represents the deviation of the high-resolution SST field with respect to the coarse SeaFlux/ERA5 SSTs.
In order to ensure that the domain-averaged SST correction is zero and that the area-weighted means of the variables in the EUREC 4 A-OA region are conserved, DSST takes the following expression DSST = (SST HR -〈 SST HR 〉 ) -(SST -〈 SST 〉 ) : (5) Angle brackets denote the spatial average over the EUREC 4 A-OA region (W), namely 〈 SST 〉 = 1 W j j ðð W dxdy SST(x, y), with W j j = ðð W dxdy : (6) Hence, to refine an existing LHF data set we first remap the original SeaFlux/ERA5 U 10m , q 2m , T 2m and SST to the MUR-JPL grid using the closest neighbor technique with Climate Data Operator (CDO) as described in  Schulzweida et al. (2019). This is to avoid interpolated values which could potentially affect the results. Then, from these remapped data we compute LHF by means of COARE3. 5 (LHF LR ).
Finally, we apply eq. 4 to the remapped SeaFlux/ERA5 variables and introduce them as an input in the COARE3. 5 function. This operation provides the refined LHF data set (LHF HR ). To validate the downscaling procedure explained above and infer confidence in our results, numerical simulations using the WRF model have been used. Each simulation of a ten member ensemble is run from the 1st of December 2019 to the end of February 2020, forced at the lateral boundaries by ERA5 hourly data. At the lower boundary, the MUR-JPL SST product interpolated on the WRF numerical grid is used (in the following referred to as SST WRF ). The month of December 2019 is used as a spin-up and is not considered in the following analyses. The validation consists in creating lower resolution data starting from WRF outputs on the original model grid (at 3 km of resolution), that are then refined through the downscaling procedure described above.
The obtained LHF HR field can be then compared with the original LHF WRF computed using the COARE3. 5 formulation directly from WRF variables at 3 km grid spacing. To do so, we have produced lower resolution data, named SeaFlux-like data, by filtering WRF fields U 10m , q 2m , T 2m and SST with a Gaussian filter considering s = 24km. Such filtered fields are found to have spectral properties that are similar to the original SeaFlux data (not shown). These SeaFlux-like variables correspond to y variables in eq. 2, that we thus handle as if they were the original SeaFlux variables: we first compute the associated coupling coefficient and then we apply eq. 4. In the latter step, DSST is defined considering SST HR ≡ SST WRF.