dataesr/scientific-paragraphs-categorization huggingface dataset:
Instead of exploring the effect of hard negative sampling  [16, 31] , in this paper we propose to solve these problems with Cross-Level Instance-Group Discrimination (CLD)  [37] , which aims to deal with highly correlated datasets. Furthermore, we hypothesize that top view UAV imagery should be invariant to geometric transformations, e. g. , rotation. From this perspective, we propose to apply extra geometric transformation to contrastive model, which captures invariant information of UAV images introduced by different augmentations. We propose an SSL model to pretrain wildlife recognition models based on contrastive learning. Our work build on the work of MoCo  [13, 5]  and Cross-Level Instance-Group Discrimination (CLD) [37]: • We propose a methodology for image-level wildlife recognition with a reduced number of annotations by self-supervised pretraining.
Related Work Self-Supervised Learning in UAV imagery. Unlike in the field of classic computer vision, self-supervised learning of aerial images has not yet been fully studied. Stojnic et al. [32]  apply Contrastive Multiview Coding  [35]  to learn aerial image representations on both RGB and multispectral remote sensing images. [17]  proposes a method based on contrastive learning with different image augmentations. [33]  analyze different pretext tasks, e. g. , inpainting  [27] , context prediction [9], and contrastive learning with different image augmentations on remote sensing dataset. Besides,  [1]  use geo-location classification as the pretext task. The encoder is trained by predicting the global geo-location of input image. Tong et al. [36]  generate pseudo-labels of unlabeled UAV imagry data to improve the downstream classification accuracy.
As illustrated in Figure  2 , the MoCo model consists of five parts: A stochastic data augmentation module T [4] transforms one given input image I into different augmented views with randomly applied augmentations, denoted I i and I + for query q i and positive key k +  [13]. We sequentially apply five augmentations (Base and Color Aug. in Table  1 ) similar as MoCo v2  [5]. A base encoder f q for query q maps the augmented views into feature space: f q : I i , → z i , where I i ∈ R C×W ×H , z i ∈ R s , where z denotes the s-D encoded representation. The parameters are updated by backpropagation. This takes the form of a CNN in our case. A momentum-updated encoder f k for keys k shares the same structure with base encoder f q and is initialized with the same parameters. However, they are not learned through backpropagation during training; instead, the parameters of f k (•) are updated with a momentum mechanism  [13].
Study Area and Data In this work, we use the data from [19], consisting of RGB aerial images acquired with a SenseFly eBee * UAV over the Kuzikus Wildlife Reserve in Namibia † by the SAVMAP consortium ‡. The UAV's flight height varied between 120 and 160m, resulting in a resolution of 4 to 8 cm with the given camera (Canon PowerShot S110). The images were annotated with bounding boxes for animals in a crowdsourcing campaign led by MicroMappers § ; these annotations were then refined in several iterations by the authors. This resulted in a total of 1183 animals. We derived the Kuzikus Wildlife Dataset Pre-training (KWD-Pre) and Kuzikus Wildlife Dataset Long-Tail distributed (KWD-LT) for pre-traning and fine-tuning/downstream task. Technical Challenges. Most contrastive models are trained on curated dataset with unique characteristics, e.
g. , ImageNet  [7]. In these datasets, images contain only a single object which is located in the center of the image (object-centric). And objects have discriminative visual features. The datasets also have uniformly distributed classes. In contrast, domain-specific datasets (e. g. , our KWD) contain less discriminative visual feature, making it hard to distinguish between similar and small objects e. g. , small trees, wildlife from the top view. KWD-Pre. We apply the same patches creating procedure as described in  [19]. We randomly crop 15 patches for every original 4000 × 3000 image. The size of each patch is 256 × 256 pixels to save memory and have a larger batch size. We randomly crop 15 extra patches if one image contains animals. Cropping this way increases the chances of extracting patches containing animals for training, but we * https://www.
According to our prior knowledge, the images in the KWD dataset are primarily composed of artefacts as follows: animal, tree, grass, tire mark (road), animal beneath tree, and dead tree trunk, as illustrated in Figure  3. Ideally, the number of all possible clusters is therefore C 0 6 + C 1 6 + C 2 6 + C 3 6 + C 4 6 + C 5 6 + C 6 6 = 2 6 = 64. Among all elements, animals beneath tree are hard to recognize; also dead tree trunks are easy confused with animals. Meanwhile, with a batch size of 64, it might not yield all 64 combinations, but we still use 64 clusters to give the model enough freedom. We train the MCC1 model with different hyper-parameters for 200 epochs and select the best hyper-parameter combination based on accuracy on the validation set of the downstream recognition task. Downstream Recognition Task.
We verify different models by applying linear classification on encoded image representations. We follow the same common linear classification protocol as  [13]. We first perform self-supervised pretraining on KWD-Pre dataset. Then we perform two kinds of experiments: (1) frozen features: we freeze the output features of the global average pooling layer of a ResNet and train a linear classifier (a fully-connected layer followed by softmax)  [13]  in a supervised way on our KWD-LT downstream task dataset; (2) end-to-end: we fine-tune the base encoder and linear classifier by softmax loss instead of contrastive loss. And we report the linear classification top-1 accuracy on the KWD-LT validation set, as well as recall and precision for foreground class.
However, the accuracy of MCC2 in epoch 200 is lower than that in epoch 150. That might imply that adding geometric transformation might cause the problem of overfitting. And feature visualization in Figure  5  show that CLD with geometric augmentation (MCC2) converges faster and better towards a more distinctive feature representation than MCC0 and MCC1. Results of Recognition Task. Self-supervised pretraining on our dataset can utilize annotations far more efficiently than supervised pretraining on ImageNet. As shown in Table  6 , fine-tuning the encoder end-to-end will completely destroy the capacity of ImageNet pretrained model. Even though we freeze the feature representations, the model perform poorly with small fraction of labeled instances. However, fine-tuning only linear classifier on frozen features with 1% and 10% annotations outperforms fine-tuning the encoder end-to-end.
When we only train the linear classifier with 1% of labels, MCC2 outperforms MCC0 by 9. 4% and MCC1 by 11. 5%. For MCC1, fine-tuning encoder with 10% annotations outperforms the recognition accuracy with full annotations. For MCC2, we need 20% annotations to get a better result. Meanwhile, for MCC1, fine-tuning the encoder with 20% of labeled instances will have the same performance with fine-tuning ImageNet pretrained model (Sup2) with full labeled instances. Whereas for MCC2, only 10% labeled data is required to outperform the Sup2 supervised model. The results show that SSL model can learn more information through geometric invariant mapping and capturing geometric invariant information can benefit UAV top view imagery task. Influence of Label Fraction. The results of Sup1, Sup2, MCC0, and MCC1 in Table  5  and of these frozen features in Table  6  show that increasing the fraction of labels used can improve the performance of linear classification.
We provide a detailed analysis in the large scale UK BioBank resource and a specifically designed software package for deriving CN estimates from NGS data that are robust enough to be used for CN-GWAS. We use these methods to perform genome wide CN-GWAS analysis across 78 human traits discovering 862 genetic associations that are likely to contribute strongly to trait distributions based solely on their CN or by acting in concert with other genetic variation. Finally, we undertake an analysis comparing CNV and SNP association signals across the same traits and samples, defining specific CNV association classes based on whether they could be detected using standard SNP-GWAS in the UK Biobank.
Although it is widely accepted that CNV can contribute significantly to differences in human traits, to date, methods for large scale CNV to phenotype association studies, ie the equivalent of GWAS for CNVs, rather than SNPs have been hampered by a number of factors, including methodological difficulties  (27) , the availability of sufficiently large datasets and the ability to interpret complex rearrangements from sequencing data  (28, 29). Although CNVs have been a major component of routine clinical medical genetics screening for over a decade the interpretation of individual CNV events remains challenging  (30, 31) , with most clinical testing laboratories routinely finding potentially pathogenic CNVs in patients with intellectual disability, autism spectrum disorders, and/or multiple congenital anomalies  (32) (33) (34).
Finally, the impact of CNVs in rare diseases is likely to be large, whereas one should expect weaker effects of all variation in more common diseases, consistent with their polygenic behaviour. A large number of CNV specific genotype-phenotype correlations have been observed in relatively small-scale studies of specific patient groups  (43)  or by collaborative efforts to share genetic data for rare disease  (44) , however CNVs have also been associated with a number of complex diseases  (45, 46). More recently large scale CNV association testing from datasets such as the UK Biobank have found a number of highly significant loci in relation to certain human traits  (47)  and previous studies focused on cognitive traits such as schizophrenia  (48)  and autism  (49)  have demonstrated the utility of leveraging single-nucleotide polymorphism (SNP) genotyping arrays to search for novel CNV associations.
Copy Number Variation in 200,629 individuals from the UK Biobank To identify exon resolution CNV regions across a large population of individuals from NGS data in the UK Biobank we developed a suite of flexible, highly scalable CNV analysis tools known as -CNest (Methods). Within this package we include a robust CNV caller as well as a set of tools and novel approaches to CNV association testing genome wide in discovery mode. A central component of these methods which is of critical importance for any accurate CNV detection, is the selection of appropriate reference datasets and normalisation procedures by modelling specific noise characteristics of WES/WGS, for example the presence and scale of genomic waves, to generate optimised relative copy number measurements across very large numbers of samples (Methods).
Triple X is a condition caused by random error during reproductive cell division and is found in approximately 1 in 1000 women. Although triple X has been associated with a number of trait differences, it can often go undiagnosed and depending on other social factors may never give rise to any noticeable problems  (62). We also detected 51 datasets which show an unusual level of chromosome X coverage and can not be reliably assigned to either double (female) or single X (male). These we assume to be due to both inconsistent capture of chromosomal baits and potential mosaic sex chromosome events (ie. mosaic XXY). Sex chromosome aneuploidy is not a focus of this study and we simply exclude all samples that could not be reliably assigned to either double or single X based on their coverage profiles (Figure  1A ); these copy number sex chromosome calls will be returned to the UK BioBank for further investigation by other investigators.
The examples vary from well established CNV loci (eg, LPA with heart disease loci) through partially understood CNV loci (eg the UGT1A gene in porphyrin and bilirubin metabolism) to very credible associations to paralogues with the same phenotype (the RCHE gene in pregnancy complications) or credible novel alleles in a gene with robust association to a phenotype (HERC gene with hair colour). There are numerous other examples in supplementary information, and the full information of the discovery processed here is available both via UK BioBank return of results and also via GWAS Catalogue. Even so, we have chosen only a subset of phenotypes present in UK BioBank, itself only one cohort; to enable broader discovery by others we have released CNest as an open source package and provided portable workflows consistent with GA4GH standards for other groups to use.
Sample Cohort and Phenotypes For this study, we used 200,624 Whole Exome Sequencing datasets from the UK Biobank 200k release generated using the IDT xGen Exome Research Panel v1. 0 including supplemental probes and sequenced with dual-indexed 75x75 bp paired-end reads on the Illumina NovaSeq 6000 platform using S2 and S4 flow cells  (168). We used the aligned CRAM files from the OQFE pipeline which aligned and duplicate-marked all raw sequencing data (FASTQs) against the full GRCh38 reference in an alt-aware manner as described in the original FE manuscript  (169). These aligned sequence datasets were used as the primary input in the CNest pipeline (Details below) for exome-wide copy number estimation and CNV calling. Phenotypes were extracted and linked to the copy number data under UK Biobank application number 49978, resulting in a total of 78 different traits (24 quantitative and 54 binary) that we tested for CNV association (Supplementary Table  1 ).
This is because all datasets are different and will contain a variety of sex classification types (Figure  1A ), as standard across large numbers of samples we observe 3 types of sex chromosome dosage exclusion types and classify samples into either male or female, or ambiguous low, ambiguous mid, ambiguous high. All samples that are not classified as either male or female are removed from the subsequent steps. The next crucial step is to derive sample specific dynamic reference datasets, briefly, CNest uses an optimisation process to select groups of appropriate datasets to make up individual references (or baseline) estimates across the genome for each sample individually. During these process CNest estimates the overall correlation of coverage information between all samples, applies a wavelet model for estimating the scale of genomic waves, and implements a dose response optimisation using sex mismatched samples and the expected single copy dosage change.
This process has been designed to be extremely efficient across very large numbers of samples and results in a ranked list of which samples are most correlated in terms of certain coverage patterns and noise characteristics which are assumed to be the ideal set of samples to generate the baseline estimate. The only parameter needed to be decided on at this stage is the total number of datasets to use to generate the dynamic references, for the UK Biobank 200K release we elected to use 2,000 samples within each of the ~200K dynamic reference datasets. Although it may, in some cases, be preferable to allow the dynamic reference sets to be made up of different numbers of individual samples (which is possible using CNest) we decided to fix this number across all datasets as we wanted to minimise any potential biases within the resulting copy number estimates that could be introduced due to using differentially sized reference datasets.
Following the dynamic reference selection process, the median coverage for all genomic locations across all relevant reference datasets is calculated for each sample individually for matched, mismatched, and mixed sex classifications. These data values are stored in the custom CNest binary format to allow fast random access across the genome and across different sample sets. Finally, the coverage information and reference estimates are transformed into the log2 ratio space and median normalised using the median log2 ratio excluding sex chromosomes for sex matched, mismatched and mixed estimates. These estimates are again saved back into the CNest binary file format to allow efficient extraction during the next steps of CNV detection and CNV GWAS testing. For CNV calling CNest implements a single custom designed 3 state Hidden Markov Model (HMM) to call losses and gains, the basic implementation of which can be found (https://github.
com/tf2/ViteRbi). In our hands this HMM model has been highly reliable across a number of different detection applications, and it is extremely efficient in terms of speed and memory usage. We apply a few important steps during the HMM training (using the EM algorithm) to further improve the reliability of the model. Primarily these include training the model independently for each sample by using all the log2 ratio estimates of all samples that made up the dynamic reference for that sample. By doing this we aim to increase the accuracy of the transition probabilities by giving the EM algorithm sight of large numbers of highly similar datasets. Importantly the log2 ratio estimates that we use during this training phase are those generated using the sex mismatched dynamic reference, ensuring that there is always at least one large single copy number loss/gain event present in all training datasets.
Finally, we calculate loss, gain and overall CNV frequency and standard errors for each CNVE resulting in a set of CNV regions across the genome with frequency estimates that can be used in some of the subsequent analyses (PCA and Association testing). On top of the frequency estimate for all merged CNVEs we also assign frequency measures to individual bait regions where we can calculate the frequency that each bait is included within any copy number event. This gives us two different sets of CNV type and frequency datasets that can be used to perform principal component analysis (PCA) across the sample space. PCA is often used in SNP GWAS to control for population structure and other technical (or sample level) variation that is less well understood but that is important to control for during genome wide association scans.
Similarly, it is important to correct for larger scale differences between copy number estimates across large datasets for CNV GWAS analysis. Often during SNP based PCA the sites get filtered or subsampled to allow efficient PCA to be performed, for CNV analysis it seems important that we can run PCA analysis using both commonly variable CNV regions and rare regions separately. This is due to an observation that when including all CNV sites in PCA, the commonly variable positions tend to suck out a lot of variation and get overrepresented in the first PCs. We used iterative PCA to perform several different types of PCA using both bait level and CNV call level copy number information stratified by frequency estimates. Overall, we find that PCA based on commonly variable positions are better able to capture sample level information such as population structure, whereas PCA based on rare regions can account for cryptic sample differences which are likely due to certain noise properties of the data that we were unable to accurately model during the previous steps.
used CNest (full source code available: https://github. com/tf2/CNest. git) to carry out large scale copy number estimation in the UK Biobank 200k WES release. This program was designed to provide accurate copy number estimation from very large NGS (WES and WGS) datasets. The first required step is to extract read coverage information for all genomic locations of interest, to do this CNest makes use of the samtools and htslib libraries (170) implementing a custom coverage extraction method that importantly filters reads based on several samtools alignment flags. The main flags of interest are BAM_FPROPER_PAIR, BAM_FDUP and BAM_FSECONDARY where we ensure that aligned reads have a MAPQ greater than 1, are primary alignements with proper pairs and are not PCR duplicates (CNest docs).
Code availability The main CNest code base and docker setup can be found here: https://github. com/tf2/CNest This repository contains all the source code and a docker setup as well as a link to a NextFlow workflow. There is a WDL workflow in development as well as a detailed tutorial for getting CNest up and running across a diverse set of computational infrastructure including cloud based systems. This tutorial and the additional workflow implementations will be linked to from the CNest main repository as soon as available.
JASPAR (http://jaspar. genereg. net/) is an openaccess database containing manually curated, nonredundant transcription factor (TF) binding profiles for TFs across six taxonomic groups. In this 9th release, we expanded the CORE collection with 341 new profiles (148 for plants, 101 for vertebrates, 85 for urochordates, and 7 for insects), which corresponds to a 19% expansion over the previous release. We added 298 new profiles to the Unvalidated collection when no orthogonal evidence was found in the literature. All the profiles were clustered to provide familial binding profiles for each taxonomic group. Moreover, we revised the structural classification of DNA binding domains to consider plant-specific TFs. This re-lease introduces word clouds to represent the scientific knowledge associated with each TF.
We updated the genome tracks of TFBSs predicted with JASPAR profiles in eight organisms; the human TFBS predictions can be visualized as native tracks in the UCSC Genome Browser. Finally, we provide a new tool to perform JASPAR TFBS enrichment analysis in userprovided genomic regions. All the data is accessible through the JASPAR website, its associated RESTful API, the R/Bioconductor data package, and a new Python package, pyJASPAR, that facilitates serverless access to the data.
PFMs, which summarize occurrences of each nucleotide at each position in a set of observed TF-DNA interactions, are largely and most commonly used to capture TF binding specificities. Unlike the simple consensusbased models, PFMs can be transformed to probabilistic or energy-based models to obtain position weight matrices (PWMs) (or position-specific scoring matrices (PSSMs)) that can be used to scan any DNA sequence and predict TF-BSs with sum weights above a defined threshold (reviewed in  (17) (18) (19) ). Hence, TF binding preferences can be repre-sented as PFMs, which can be interpreted as TF binding profiles or motifs. In this manuscript, we will use the term PFM, motif, and TF binding profile interchangeably. JASPAR is a popular and regularly maintained openaccess and manually curated database storing TF binding preferences as PFMs.
The JASPAR CORE collection provides non-redundant binding preferences for TFs (one versioned profile per TF per taxon, except when a TF has multiple DNA-binding preferences) across 6 taxa: urochordata, vertebrates, plants, insects, and fungi. Inclusion of new profiles requires orthogonal evidence for the binding preferences of the TFs, which is rigorously evaluated by our expert curators. To complement the CORE collection, we previously introduced the Unvalidated collection to store high-quality TF-binding profiles that are lacking orthogonal supporting evidence in the literature  (17). Beyond the high-quality TF binding profiles and metadata stored in JASPAR, the popularity of the database originates from its simplicity, the tools embedded in its web-interface, and the multitude of popular resources and tools directly integrating JASPAR profiles.