dataesr/scientific-paragraphs-categorization huggingface dataset:
Since 1) and 3) are costly, the current strategy is to develop functional link methods and to use data from 1) and 3) as reference to evaluate them. However, there is still no consensus on the best functional link method to date, and method comparison remain seldom. Here, we compared the relative performances of three recent methods for the identification of enhancer-gene links, TargetFinder, Average-Rank, and the ABC model, using the three latest benchmarks from the field: a reference that combines 3D and eQTL data, called BENGI, and two genetic screening references, called CRiFF and CRiSPRi. Overall, none of the three methods performed best on the three references. CRiFF and CRISPRi reference sets are likely more reliable, but CRiFF is not genome-wide and CRiFF and CRISPRi are mostly available on the K562 cancer cell line.
Broadly speaking, non-supervised methods use correlations between functional genomic signals present at enhancers and promoters across many cell types. Distance between promoters and enhancers as well as correlation thresholds are determined heuristically and the evaluation of the accuracy of the method is done a posteriori using external reference data (most often 3D or genetic)  (Ernst et al. (2011)  For illustration purposes, an example of such unsupervised/heuristic methods, the open chromatin correlation method, is provided in Figure  2  (see Material and Methods for details). The second category of methods uses machine learning approaches such as random forests or neural networks. They consist in training a model to discriminate true vs false E/G based on distinctive features from the 1D data they use, from a reference dataset of known E/G (ground positives, most often a combination of 1D data for enhancer and promoter identification and 3D or genetic data for the relationship identification), and a dataset of unsupported E/G (ground negatives), as a negative control.
Here potential enhancers are all distal enhancer elements (distal enhancer like signature elements or dELS) of the ENCODE registry of candidate cis-regulatory elements (cCREs)  (Moore et al. (2020b) ). TargetFinder defines true (ground positive) E/G based on 3D data (HiC) and learns features associated to those using gradient boosting. The learnt features are as diverse as open chromatin, methylation, histone marks or transcription factors, and can both be taken from enhancer and promoter regions and from the window between them  (Whalen et al. (2016) ). Indeed its authors showed that features located in enhancer-promoter windows (EPW) are also predictive of true E/G relationships and should be incorporated in the model. The two first link methods mentioned above also propose their own reference/evaluation datasets.
(2020a) 's reference set. It is made of 651 ground positive and 24,576 ground negative E/G relationships, and we will call it CRISPRi (Table  1 ). Moore et al. 's reference set is entitled BENGI (Benchmark of candidate Enhancer-Gene Interactions) and is made of sets of E/G active and inactive in different cell lines according to different types of data (3D, genetic). We focus our evaluation on the GM12878 cell line, which has the largest amount of annotation data, with 6 sets of active and inactive E/G available. The active E/G sets result from the processing of four types of 3D data, Hi-C  (Rao et al. (2014) ) and promoter capture Hi-C  (Mifsud et al. (2015) ) data and ChiA-PET of polymerase II and CTCF  (Tang et al. (2015) ) data, and of eQTL data from two different studies, GEUVADIS  (Lappalainen et al.
(2013) ) and GTEx  (Consortium et al. (2015) ). The sets of ground negatives are built by taking, for each enhancer of a positive set, all the genes not connected to it in the positive set and lying within the 95 percentile of the positive set distances from it. The number of ground positive and negative E/G obtained are indicated in Table  1. Since 3D and eQTL data are not specifically generated to identify E/G relationships, the BENGI reference sets are expected to be overall less reliable than the CRiFF and the CRISPRi reference sets. However, the fact that BENGI provides genome-wide information is an advantage over CRiFF. Given all these data we proceeded to the evaluation of the ABC model, the Average-Rank method, the distance method, and TargetFinder, on all three reference datasets: BENGI, CRiFF and CRISPRi.
Altogether, these positive controls confirmed the validity of the pipeline we implemented. Figure  3  further shows low AUPR values for the first three methods on the six BENGI datasets, and that TargetFinder performs best overall, followed by Average-Rank, distance and ABC model. Note that TargetFinder is much more precise than sensitive, and performs much better on HiC and CTCF sets. This last result can be explained by the fact that TargetFinder learns true E/G based on HiC data. Nevertheless, Figure  4A  shows larger AUPRs for the three last methods on the CRiFF set, and that the ABC model performs best (AUPR = 0. 63), before the distance method and finally the Average-Rank method. Contrary to its result on BENGI, TargetFinder does not perform well on CRiFF. When comparing the performances of the methods on CRISPRi with respect to CRiFF, if the methods that perform best (ABC model) and worst (TargetFinder) are the same, it has to be noted that Average-Rank performs better than distance and that the AUPRs are globally lower (Figure  4B ).
Discussion The poor performance (small precision values even for small recall values) of the ABC model on the BENGI sets could be due to the fact that BENGI's underlying data (HiC, promoter capture HiC, ChiA-PET and RNA-seq) were not specifically designed to identify E/G relationships. For instance, some E/G relationships may not need spatial proximity or the presence of CTCF to operate (Ray-Jones and Spivakov (  2021 )). Likewise the presence of an eQTL in a predicted enhancer does not necessarily imply the presence of an E/G relationship. The poor performances of the Average-Rank and the TargetFinder methods (small precision values even for small recall values) on the CRiFF and CRISPRi data are more difficult to explain as these techniques should be quite exhaustive in identifying the enhancers of a given gene.
Therefore if we really had to select an E/G relationship identification method, then we would choose the one that performs best on the CRiFF and CRISPRi data, namely, the ABC model. Another, more practical, reason to select the ABC model over the Average-Rank and TargetFinder methods is that a dedicated and more importantly well documented software has been made   2  for more details). TABLE 2 TargetFinder's performances on BENGI, CRiFF and CRISPRi. #Predicted refers to the number of positive (TP + FP) and negative (TN + FN) predictions that were also in the reference set, and is used to compute Precision. Ground positives pes. refers to the total number of ground positive elements in the reference set, while Ground Positives opt. refers to the subset of those that were also in the set of E/G relationships TargetFinder started from.
Pairwise chromatin accessibility correlation across cell types In order to illustrate unsupervised/heuristic enhancer/gene identification methods, we chose the simplest one, the pairwise chromatin accessibility correlation across cell types, and represented it on Figure  2. The actions that led to this figure are the following: we first downloaded the ENCODE uniformly processed read alignments (bam files) of DNAse-seq data (single end) from 10 cell types: stomach, HepG2, K562, thymus, adrenal gland, small intestine, GM12878, IMR-90, heart and H1-hESC, with accession numbers provided in Table  3. We then called the chromatin accessibility peaks from the mapped reads in each cell type using macs2 7 (Figure  2 ; Table  3 ). We obtained from about 60,000 (GM12878) to about 200,000 (IMR-90) peaks per cell type.
By concatenating, sorting and merging on the genome the peaks called in each cell type using bedtools merge, we then obtained 473,766 consensus peaks across all cell types. We then quantified the chromatin accessibility of the 473,766 consensus peaks in each cell type by simply counting the number of mapped reads of each cell type overlapping each consensus peak using bedtools intersect, and normalized the number of reads of each peak in each cell type by the total number of mapped reads in peaks for this cell type. Finally we computed the consensus peak pairwise Pearson correlation between the log10 of the normalized chromatin accessibility across the 10 cell types of these peaks for all pairs of peaks less distant than 500 kb using a script that we wrote: compute_correlations. py 8. We then only considered as E/G relationships, the pairs of peaks with a correlation above 0.
Method evaluation In addition to evaluating the ABC model on BENGI and the Average-Rank and the TargetFinder methods on CRiFF, and since Moore et al. provided the code of the Average-Rank method, and Fulco et al. the code to run the ABC model, we decided to try and reproduce the evaluation of the Average-Rank method on BENGI and of the ABC model on CRiFF. We also used the code of the distance method provided by Moore et al. (with some modifications), to evaluate the baseline distance method on BENGI, CRiFF and CRISPRi (Tables 1, 2; Figures  3, 4 ). In total we evaluated four methods, the ABC model, the Average-Rank, the distance and the TargetFinder methods on three references sets, BENGI, CRiFF and CRISPRi (Figures  3, 4 ). It has to be noted that contrary to the other methods, TargetFinder's predictions were downloaded directly from its authors's website 10 , therefore only allowing us to compute a single pair of (precision, recall) values, and not AUPR curves.
In fact, we used two different ways to compute TargetFinder's recall, an optimistic and a pessimistic one, which led to two different dots for this tool in the evaluation plots (see sections below about TargetFinder). In addition, since the code to generate the Precision-Recall curves and the AUPRs was not provided in the papers, we generated our own R code to make these plots using existing R packages. The code used to perform all these analyses was stored in Jupyter notebooks that we provide below, together with additional details about these analyses.
Method evaluation on BENGI The Moore et al. 's code, reference data and annotation were first downloaded from the BENGI github repository 11. More precisely the Scripts directory included, on the one hand the scripts to make the BENGI sets, and on the other hand the scripts to run the evaluation of the methods on a given BENGI set (note that other cell types than GM12878 were provided). It is important to bear in mind that the script corresponding to a method was not a generic script allowing to retrieve all the E/G relationships called by this method in a particular cell type, but rather only produces evaluation data of this method on a given BENGI set, i. e. , attaches to each true and false E/G of a BENGI set, the score of the method's associated prediction (to be used to draw the Precision-Recall curves and compute the AUPRs).
Our modification consisted in adding two additional columns to this tabulated file, one for the enhancer id and one for the gene id, this for an easier downstream fusion with the evaluation result of the Sheffield method. For this purpose we also had to modify the Run-Distance-Method. sh script so that it sorts the 4 column tabulated file provided by the python script according to the enhancer id and the gene id. After running the evaluation script we plotted the Precision-Recall curves using existing R packages. The following Jupyter notebook provides all the necessary information for evaluating the distance method on the BENGI sets 12.
This script takes as input the BENGI set and its version, and outputs a 7 column tabulated file including for each E/G of the BENGI set, 1 or 0 according to whether this E/G is true or false in BENGI, the average rank score, the distance score, the correlation score, the distance rank, the correlation rank and the average rank between the distance and the correlation. Here we also had to modify the bash script to make it run but more importantly to correct a bug. The exact process and modifications are provided in 15. do so, we first had to convert the enhancer and promoter coordinates of the TargetFinder's prediction file into cCRE-ELS (candidate cis-regulatory elements with enhancerlike signature) ids and gene ids respectively. For that we used bedtools intersect on the K562 cCRE-ELS file and the Gencode v19 TSS file from Moore et al.
Acknowledgments We would like to thank the authors of Fulco et al. (2019); Moore et al. (2020a) for the detailed Material and methods' section of their papers, that allowed us to reproduce the curves and AUPRs of Figure 3A of Fulco et al. (2019), and the part of Figure S2 corresponding to the GM12878 cell line (all pairs, natural ratio) for Moore et al. (2020a). We would also like to thank the authors of Fulco et al. (2019) for their reactivity answering the questions we had about their study,  Nathalie Vialaneix  from  INRAE  for her advice on statistical analyses and  Cédric Cabau  from  INRAE  for his technical help. Finally we would like to thank the Toulouse genotoul bioinformatics plateform, and in particular  Marie-Stéphane Trotard  and  Didier Laborie , for their very efficient computing infrastructure and their help in installing and running software.
10 https://github. com/shwhalen/targetfinder 11 https://github. com/weng-lab/BENGI 12 https://hoellin. github. io/eg/notes_BENGI/distance_method/distance_ evaluation_with_code. html 13 http://big. databio. org/papers/RED/supplement/ 14 https://hoellin. github. io/eg/notes_BENGI/dnase_expression_ correlation/correlation_method_with_code. html 15 https://hoellin. github. io/eg/notes_BENGI/avg_rank_method/avg_rank_ method_with_code. html\#partial-reimplementation-of-run-averageranksh Frontiers in Bioinformatics frontiersin. org.
The thermal cycling and data collection were performed on Quantstudio 5 Real-Time PCR instrument (Thermo Fisher), using the thermal cycling protocol 2 min at 50 °C, 5 min at 94 °C, (15 s at 94 °C, 15 s at 56 °C, 30 s at 68 °C) × 45, 5 min at 68 °C, followed by melting curve from 65 to 95 °C. The concentration of nucleic acids was calibrated using 10-fold serial dilutions of corresponding standard DNA templates ranging from 1 fM to 1 nM. Data were analyzed using the Quantstudio Design and Analysis software v1. 4. 3 Software (Thermo Fisher). Flow Cytometry. The liposome suspension (1-3 μL) was diluted in 300 μL (final volume) of PB buffer consisting of 20 mM HEPES-KOH, pH 7. 6, 180 mM potassium glutamate, and 14 mM magnesium acetate. To remove any remaining beads or large debris, the diluted liposome suspension was gently filtered through the 35 μm nylon mesh of the cell-strainer cap from the 5 mL round-bottom polystyrene test tubes (Falcon).
Data were analyzed using Cytobank (https://community. cytobank. org/). Raw data were pre-processed as described in Figure  S11  to filter out possible aggregates and debris. Confocal Microscopy and Image Analysis. A custommade glass imaging chamber was functionalized with BSAbiotin:BSA and Neutravidin as previously described. 17 The liposome suspension (3-7 μL) was supplemented with PB buffer to a maximum volume of 7 μL and transferred into the functionalized chamber. The LactC2-GFP probe was used at a final concentration of ∼260 nM. After 30 to 60 min incubation at room temperature to let the liposomes sediment, the sample was imaged with a Nikon A1R Laser scanning confocal microscope equipped with a ×100 objective and operated via the NIS Elements software (Nikon). The laser lines 488 nm (for MinC-eGFP), 514 nm (for YFP), and 561 nm (for DHPE-Texas Red and LactC2-mCherry) were used in combination with appropriate emission filters.
The purified DNA was quantified by the Nanodrop 2000c spectrophotometer (Isogen Life Science). The enrichment efficiency of ori-yfp over ori-minD was calculated as where N A is the Avogadro constant. A CADGE reaction may occur in a liposome if one or more copies of linear DNA template are encapsulated, whose corresponding probability is given by = P k e ( 1) 1 (6) With expCADGE, the concentration of p2-p3-plasmid largely exceeds that of ori-GOI such that only the concentration of oritemplate limits the percentage of liposomes exhibiting CADGE: P(k p2-p3-plasmid ≥ 1) × P(k ori-GOI ≥ 1) ≈ P(k ori-GOI ≥ 1). Statistics. Box and whiskers plots in Figures  2, 4 , and 5 have the following characteristics: middle line is the median, the whiskers of the plot are drawn from the 10 th percentile up to the 90 th percentile, any data point outside the whiskers is drawn as an individual point.
We demonstrate that linked-reads longrange information can substantially improve local assembly results on several linked-read datasets of large eukaryote genomes. We also show that it can be used for various local assembly use cases, such as gap-filling between scaffolds or alternative allele reconstruction of large insertion variants. MTG-Link performs local assemblies of specific loci, using linked-read data. For all use cases (specific locus assembly, intra-scaffold and inter-scaffold gap-fillings, as well as alternative allele reconstruction of large insertion variants), the locus of interest is defined by two coordinates on a reference genome, indicating its left and right flanking sequences, and the sequence in-between to be assembled is referred to as the target sequence. Here, the reference genome corresponds to our genomic coordinate reference, regardless of its assembly quality or proximity to the re-sequenced individual.
The input of MTG-Link is a set of linked-reads, the target flanking sequences and coordinates in GFA format (genome graph format, with the flanking sequences identified as "segment" elements (S lines) and the targets identified as "gap" elements (G lines)) and an indexed BAM file obtained after mapping the linked-reads onto the reference genome. It outputs the set of assembled target sequences in Fasta format, as well as an assembly graph file in GFA format, complementing the input GFA file with the obtained assembled sequences. In MTG-Link, each target sequence is processed independently in a three-steps process: (i) read subsampling using the barcode information of the linked-read dataset, (ii) local assembly by de Bruijn graph traversal and (iii) qualitative evaluation of the obtained assembled sequence.
Read subsampling The purpose of the read subsampling step is to extract the linked-reads whose barcode is observed in flanking regions surrounding the target sequence (Figure  1a ). First, a barcode list is computed from the reads mapped on both flanking sequences. This selection is performed based on two user-defined parameters, the flanking region size and the minimum number of occurrences in these regions for a barcode to be retained (-flank and -occ options). Then, the whole read file is searched for reads having the selected barcodes. In order to perform these two tasks efficiently given the large file sizes, we rely on the LRez toolkit  [32]  for barcode-based indexing and query of linked-read files.
Implementation and availability We provide an implementation of this method named MTG-Link, freely available at https:// github. com/anne-gcd/MTG-Link under the GNU Affero GPL licence. MTG-Link is also available as a Bioconda package. MTG-Link is written in Python 3. The main steps are implemented in a modular way, allowing the user to start or re-run the program from previous intermediate results. As an example, the first step (read subsampling step) is not to be repeated if we want to perform the local assembly on the same input files but with different assembly parameters values. In order to speed up the process, the data handling and analysis are set up in a multi-threaded manner, thus allowing multiple target sequences to be processed simultaneously. Additional Python scripts for converting input and output files to the desirable formats are provided.
Materials 3. 1 Linked-read datasets 3. 1. 1 stLFR Homo sapiens dataset MTG-Link was applied on a human dataset obtained from the individual HG002 with the stLFR technology, from the Genome In A Bottle resources [19] (FTP links are given in Supplementary Material; Additional file 1: Section S1. 1). After the removal of the PCR duplicates, this dataset is composed of approx. 1. 5 billion 100 bp Illumina reads, with an effective read depth of 47X. We considered the assembly GRCh37 (hg19 version) as the human reference genome.
10x Genomics Heliconius numata datasets MTG-Link was also applied on linked-read datasets obtained from 12 individual genomes of the butterfly Heliconius numata, with the 10x Genomics Chromium technology (BioProject PRJNA676017)  [5]. The number of reads in each dataset is approx. 110 million, with an effective coverage ranging from 20X to 47X. For each individual, we used as reference genomes their draft genome assemblies obtained with the Supernova assembler  [22]  and available under the same BioProject ID. Experiments on randomly selected loci were performed on individual 37 (read depth of 40X).
Evaluation on randomly selected loci To evaluate the accuracy of MTG-Link, we performed experiments where the target sequence to be assembled was a sequence already present in the reference genome at a locus randomly sampled. In such cases, the true sequence to be assembled is known and can be compared to the assembler's output, in order to compute quality metrics. Four target sizes were tested (1, 5, 10 and 20 Kb) on 63 and 57 randomly selected loci respectively for the human stLFR dataset and the butterfly 10x Genomics dataset (individual 37). MTG-Link was applied on all of these targets, testing different flanking region sizes (5, 10 and 15 Kb). Assembled sequences were aligned to their reference sequence using Blastn  [34]. The assembled sequences having more than 90% identity and coverage with the reference sequence were labelled as "successful", otherwise they were considered as "erroneous".
Real use case: Reconstruction of large insertion variant sequences MTG-Link was used to reconstruct the alternative sequences of large novel insertion variants using the stLFR Homo sapiens dataset. From the gold standard SV callset of Genome In A Bottle obtained on the HG002 individual  [35] , we selected all insertion calls that were larger than 250 bp and annotated as "novel insertion" in the  Delage et al. study [36]. This resulted in a set of 151 insertion calls, ranging in insertion size from 250 bp to 27,920 bp, and consisting of 104 insertions with a homozygous genotype and 47 insertions with a heterozygous genotype in HG002. The repeated nature of the context of the insertion site was retrieved from the Delage et al. study  [36]. As the reported position of the insertion site may be imprecise, we attempted to reconstruct the insertion sequence by including 50 bp on either side of the insertion site.
Assessing MTG-Link results on evaluation datasets 4. 1. 1 Assembly of randomly selected loci MTG-Link was first applied on several evaluation datasets, composed of real linked-read data (an stLFR H. sapiens dataset and a 10x Genomics H. numata dataset) and randomly selected loci for which the sequence to assemble is known, to precisely estimate its success rate and assembly accuracy. The results obtained with MTG-Link for several target size (1, 5, 10 and 20 Kb) are presented in Figure  2. For small targets (1 Kb), MTG-Link returns an assembly for most targets with a good accuracy. Notably, for the human dataset, 87. 3% of the 1 Kb targets are successfully assembled (success rate) and MTG-Link did not return any incorrect assemblies (e. g. with more than 10% of sequence or size divergence with the reference sequence), giving an assembly accuracy of 100%.