dataesr/scientific-paragraphs-categorization huggingface dataset:
System Descriptions HITS The HITS team participated in Tasks 1-2 and Tasks 3, with two separate systems. Their approach for Tasks 1 and 2 was languagespecific, by fine-tuning monolingual or multilingual transformer-based models per corpus-for corpora with a training set. Their classifier architecture was based on pretrained models (various BERT or RoBERTa based for the monolingual models, XLM-RoBERTa-base for the multilingual), fine-tuned with a bidirectional LSTM network with a CRF layer (BiLSTM-CRF,  Huang et al. 2015). They implemented an adversarial training strategy, which introduced small perturbations to the original inputs in order to help the trained model generalize better. For corpora without a training set (the surprise and OOD ones), they used their previously fine-tuned models of the same language and framework.
For Task 3, the team submitted a system composed of two fine-tuned transformer-based models (as in Tasks 1-2, BERT or RoBERTa based for the monolingual models, XLM-RoBERTa-base/large for the multilingual). For large corpora, a corpusspecific fine-tuned classifier was used, based on monolingual or multilingual models. However, they aggregated smaller corpora in a joint training approach based on their frameworks, and then fine-tuned a multilingual model for classificationand also used those for corpora without a training set. They also implemented the adversarial training strategy for this task, for specific datasets.
MELODI: DisCut and DiscReT The MELODI team submitted two systems to handle Tasks 1-2 and Task 3 respectively: DisCut and DiscReT. The former system is a revised version of the team's 2021 submission  (Kamaladdini Ezzabady et al. , 2021). The main modifications to DisCut included a shift to a single multilingual language model to accommodate all languages (XLM-RoBERTa-large was chosen,  Conneau et al. 2020) , and the use of a simple linear layer for classification, replacing the character-level CNN and token-level LSTM used in the 2021 version. Additionally, the team experimented with layer freezing, finding an overall optimum for the large language model when layers 0-5 of 24 were frozen. Both Tasks 1 and 2 were handled as BIO-encoded sequence labeling, and no additional features beyond sentence splits were used (for the plain text scenario, Trankit was used to preprocess the data, see  Nguyen et al.
2021). For Task 3, MELODI submitted DiscReT, which was unique in not only using a multilingual language model for all languages (this time choosing mBERT-base-cased) but also training jointly on all datasets after performing label lower-casing and selective merging to reduce the total of possible labels from 163 to 135 across datasets. Their models are fine-tuned and fitted with a fine-tuned Adapter. Adapters  (Houlsby et al. , 2019)  offer a lightweight alternative built on transformers that expose only a subset of parameters to fine-tuning, reaching comparable results to fully fine-tuned transformers. The system did not use additional features, except for encoding the relation direction information by permuting the order of input sequences to always begin with the source argument of the relation (meaning sequences were transposed from their natural order for relations of the form 1<2).
DiscoFLAN DiscoFlan is based on the Flan-T5 generative language model, itself a fine-tuning of the T5 model on a large set of additional tasks  (Chung et al. , 2022). The basic principle of this family of models is to encode an instruction in natural language input to resolve a given NLP task, and to learn to decode it as the answer. In the case of discourse relation classification within DIS-RPT, this is implemented in DiscoFlan by finetuning Flan-T5 and encoding the instruction "what discourse relation holds between sent1 and sent2: sent1 <text> sent2 <text>" in various languages, and learning to decode the discourse relation label. A post-processing step tried to match an output token to an existing label, or select the majority class if the output cannot be mapped. The majority class is computed on the training set, or the dev set for the OOD corpora that do not have training sets.
Results Task 1: EDU Segmentation Table  3  shows the EDU Segmentation scores of the two submitted systems. The comparison between the two systems for the Treebanked track indicates very similar results, with the winner being DisCut (a mean F1 score of 91. 87) from the MELODI team. Both systems used rather similar architectures, and the main difference was the language model used as backbone: always XLM-RoBERTa large for MELODI, and for HITS a language model was specifically chosen for the target language. As illustrated here, it seems that the hyper-parameter tuning including freezing layers and/or the use of a large version of RoBERTa allows performance to be on par with the specific base models. Major improvements were observed for nld. rst. nldt (MELODI +4 points), spa. rst. rststb (+2), spa.
rst. sctb (+3), and zho. rst. sctb (+2). However, these variations should be taken with precaution as we noticed an important variance of the scores when reproducing the results, especially for small-sized corpora. In general, scores are high, and the performance of DisCut is better than the ones obtained by the winning system DisCoDisCo in 2021  (Gessler et al. , 2021) , with a mean score of 91. 77 when only considering the corpora used in 2021 against 91. 48 for DisCoDisCo (for the Treebanked track). See the paper describing the MELODI results for a full comparison. Additionally, this year's mean scores are not far from the 2021 ones, despite the addition of the new corpora and one OOD dataset (eng. dep. covdtb). This demonstrates some ro-bustness of the approaches as well as the consistencies of the new annotations.
Conclusion The DISRPT 2023 shared task was very challenging, with the addition of datasets from a new framework, in new languages, and 4 OOD surprise datasets without training partitions. The submitted systems still demonstrated rather high performance for EDU segmentation, with room for improvement for some corpora / languages / domains. However, further research and error analysis are needed to better understand not only what could be missing in the current models, but also what could be improved in some annotation projects, especially for example when EDU boundaries do not match sentence segmentation. Connective detection has been shown to be far from a solved task, with specific challenges for speech or dialogue data and generalizability to new domains. Finally, challenges are still significant for discourse relation classification.
Competitors proposed original and attractive strategies to combine corpora due to data scarcity, but the label set explosion is a major obstacle as well as for analyzing the results. We hope that this work will bring new research and discussion in increasing convergence and cohesion of frameworks and annotation projects. We encourage researchers in the field to use the DISRPT data as a benchmark to evaluate their systems in the future in order to provide a realistic view of the robustness and generalization ability of their approaches.
Acknowledgements This work is partially supported by the  AnDiaMO  project ( ANR-21-CE23-0020 ) and the  ANR  ( ANR-19-PI3A-0004 ) through the  AI Interdisciplinary Institute, ANITI , as part of  France's "Investing for the Future -PIA3" program. This work is also partially supported by the  SLANT  project ( ANR-19-CE23-0022 ) and the  ANR  grant  SUMM-RE  ( ANR-20-CE23-0017 ). Chloé Braud  and  Philippe Muller  are part of the programme DesCartes and are also supported by the  National Research Foundation, Prime Minister's Office, Singapore  under its  Campus for Research Excellence and Technological Enterprise (CREATE)   programme.
Marker catalog and genome-wide density To obtain a comprehensive catalog of SNPs (Data S1), we generated high-coverage HiFi data (ds1, $203 coverage, n = 5) for five H. r. rustica individuals (Table  S11A ) and aligned them using bHirRus1 as reference. We complemented this information with all the publicly available genomic data for the species (Figure  3A ; Table  S12 ), including two Illumina whole-genome sequencing (WGS) datasets  2, 39  (ds2 and ds3. 1, $6. 83, n = 159) and four ddRAD datasets  2, 14, 40, 41  (ds3. 2 through ds6, $0. 073; n = 1,162). Despite the fewer individuals in HiFi WGS, the average SNP density and distribution (Figures  3B  and S5 , light blue track; 142. 37 SNPs/10 kbp; Table  S13 ) was comparable to the one computed for Illumina WGS (Figures  3B  and S5 , dark blue track; 160.
8 SNPs/10 kbp; HiFi WGS: 53 2. 34 SNPs/10 kbp; Illumina WGS: 53. 3 SNPs/10 kbp). As expected, ddRAD exhibited very localized peaks of SNPs (0. 8 SNPs/10 kbp; Figures  3B  and S5 , red track). Particularly, ddRAD identified an extremely low number of SNPs on chromosome Z (0. 27 SNPs/10 kbp) and no SNPs on microchromosome 33 (Figure  S5 ). As observed in other bird species,  43, 44  we detected a positive correlation between chromosome GC content and SNP density in all datasets (Data S1).
QUANTIFICATION AND STATISTICAL ANALYSIS Mitogenome assembly A de novo assembly of the barn swallow mitogenome was generated from 10X reads, which were firstly trimmed with the proc-ess_10xReads. py script from proc10xG (https://github. com/ucdavis-bioinformatics/proc10xG) with -a and -b 16 parameters. Trimmed reads were aligned to the Chelidonia assembly  16  with bowtie2  68  and unmapped reads were extracted. NOVOplasty  69  was run with default parameters (read length = 151, insert size = 300) to assemble the mitogenome de novo from the unmapped reads. The mitogenome annotation was performed with MITOS2. 70 As sanity check, we aligned and mapped our complete mitochondrial sequence to the Hirundo r. rustica mitochondrial Reference Sequence (HrrRS, GenBank accession number MZ905359), which is included in a companion study on barn swallow mitogenome relationships.
Reference genome assembly Prior to the assembly, Genomescope2. 0 21 was used to estimate genome size, heterozygosity and repeat content through statistical analyses of k-mer profiles in unassembled sequencing data. Genomescope2. 0 21 was run online (http://qb. cshl. edu/genomescope/ genomescope2. 0/) starting from the k-mer (31 bp) histogram generated with Meryl 25 using the 10X linked reads with barcodes (i. e. the first 23 bp of the forward read) trimmed off. Newly generated sequencing data were combined with PacBio CLR long reads and Bionano optical maps already available for the same individual. 16 The assembly was performed on the DNAnexus cloud-based informatic platform for genomic data analyses (https://www. dnanexus. com/) using the VGP standard genome assembly pipeline 1. 6  12  (https://github.
To join proximal contigs, 10x linked reads were aligned to the p1 intermediate in two rounds and an adjacency matrix was produced from the barcodes using scaff10X v2. 0-2. 1 (https://github. com/wtsi-hpag/Scaff10X). Two scaffolding rounds were performed with options -matrix 2000 -reads 12 -link 10 and then -matrix 2000 -reads 8 -link 10. Contigs were then joined with 100 bp gaps ('N's). The resulting s1 intermediate was then scaffolded with Bionano DLS optical maps  16  using Bionano Solve v3. 2. 1 in non-haplotype assembly mode with a DLE-1 one enzyme non-nicking approach, obtaining s2. Finally, Hi-C reads from Arima were aligned to the s2 intermediate with the Arima Genomics mapping pipeline (https://github. com/ArimaGenomics/mapping_pipeline). Forward and reverse reads were aligned independently with BWA-MEM  75  with the -B8 parameter and filtered with a minimum mapping quality of 10.
BUSCO was run with the vertebrata_odb10 database and 'chicken' as training species for gene prediction on bHirRus1 and Chelidonia to assess differences in functional completeness, but also on the alternate assembly and the assembly pipeline intermediates c1, p1 and p2, to assess whether purge_dups  74  removed unintended sequences from the primary assembly. The BUSCO results were manually evaluated to detect missing genes in bHirRus1 that were found in the other assemblies, and could, therefore, be recovered. Nucleotidenucleotide BLAST 2. 10. 1+  81 was used to search in bHirRus1 the sequence of the missing genes retrieved from the corresponding assembly. These genes were erroneously not detected by BUSCO in bHirRus1. To confirm the presence of the genes found with BLAST and rescue the remaining bHirRus1 missing genes from the other assemblies, the scaffold or contig sequences containing the predicted BUSCO genes were aligned to bHirRus1 with MUMMer NUCmer.
Chromosome size estimations from karyotype images Chromosomes sizes were estimated from four karyotype images using the chromosome_size software (https://git. mpi-cbg. de/ dibrov/chromosome_size#example). The average size value was calculated for each chromosome. Sizes were correlated with the assembly chromosome sizes using Spearman nonparametric rank test. 124 eutral model estimation PHAST v1. 5  33  was used in combination with the HAL toolkit  89  for the selection analyses. An alignment in the MAF format was extracted for each bHirRus1 chromosome from the Cactus HAL output using hal2maf  89  with the -noAncestors and -onlyOrthologs options. The MAFs were post-processed with maf_stream merge_dups consensus (https://github. com/joelarmstrong/maf_stream), as previously described. 15 The non-conserved neutral model was trained from fourfold degenerate (4d) sites in the coding regions of the barn swallow annotation.
35, 129 Briefly, CDS that fall within bHirRus1 chromosomes were extracted from the NCBI gff3 annotation file. msa_view  33  was used to extract 4d codons and 4d sites from each MAF separately, using the correspondent CDS coordinates. The combined 4d sites were used with phyloFit  33  (-subst-mod REV -EM) to generate the neutral model. The command used to estimate the neutral model can be found on GitHub (https://github. com/SwallowGenomics/BarnSwallow/blob/main/Analyses/Selection% 20analysis/neutral_model_estimation. txt).
22 -expected-length 8). Following Craig et al. ,  37  windows that overlapped for more than 20% with an assembly gap were removed, and all bases that fell into gaps were filtered out. Correlations between phyloP conserved elements and phastCons CEs as the number of elements per 10kb windows were computed with the Spearman correlation rank test. 124 The commands used for this analysis can be found on GitHub (https://github. com/SwallowGenomics/BarnSwallow/blob/main/Analyses/Selection%20analysis/phastCons_analysis. txt).
Candidate gene detection To calculate the percentage of conserved and accelerated bases in bHiRus1 we considered how many chromosomal bases (1,082,536,200 bp) were detected as conserved and accelerated by both phyloP and PhastCons (Table  S6A ). To detect candidate genes, we intersected the conserved and accelerated bases detected with each annotated class extracted with GenomicFeatures. Bases overlapping with more than one feature were hierarchically assigned based on their first appearance  37, 131  in this order: CDS, 5 0 UTR, 3 0 UTR, intronic, intergenic. Genes without identified orthologs (''LOC'' genes) were discarded. The commands used for this analysis can be found on GitHub (https://github. com/SwallowGenomics/BarnSwallow/tree/main/Analyses/GenomicFeatures) Gene ontology enrichment analysis The gene ontology (GO) analysis was performed on the top 500 genes with the most overlaps with phyloP accelerated and conserved sites using the Generally Applicable Gene-set Enrichment (GAGE) method  90  (gage R package).
GAGE detects enrichment for genes' functions (GO terms) in the tested datasets with respect to a broader dataset. A GO term is considered enriched in the tested dataset when the associated p value after FDR correction (q-value) is <0. 05. Previous to gage analysis, bioMart  91  R package was used to retrieve correspondence between the zebra finch and human Ensembl IDs and associate the latter with GO terms. The zebra finch annotation was used as the broader complete dataset since the barn swallow could not be found on Ensembl yet at the time of the analysis. Human genes were used since annotation with GO terms should be more accurate. The script used can be found on the project GitHub page (https://github. com/SwallowGenomics/BarnSwallow/tree/main/Analyses/Gene_ontology). camk2n2 tree construction To look at differences in camk2n2 transcript between species with different levels of association with humans, the transcript sequences of 38 species were downloaded from NCBI (Table  S31 ) and aligned with Muscle on MEGA.
Datasets used To generate the catalog of genetic variants, five Italian barn swallow individuals were sampled. HMW DNA was extracted from the blood samples and sequenced with PacBio HiFi technology (see ''HiFi reads processing for SNP catalogue, titration and phasing experiment'' section for a detailed description of the generation and processing of HiFi data). Then, all publicly available datasets (Table  S12 ) were used to complement our newly generated HiFi reads set and generate a comprehensive genetic marker catalog for the barn swallow. Raw reads from public datasets were downloaded using fasterq-dump v2. 9. 1 from SRA Toolkit (https:// github. com/ncbi/sra-tools). The data were single-end, except WGS data in ds2 and ds3. 1 and ddRAD data in ds5. Quality control was performed on all raw reads using Fastqc v0.
11. 9 (https://www. bioinformatics. babraham. ac. uk/projects/fastqc/) and Multiqc v1. 9  93  (https://github. com/ewels/MultiQC). Low quality bases were trimmed using Cutadapt v2. 10 94 (Figure  S8 ). BBDuk, from BBMap v38. 18  95  was used to remove Illumina adapters (k = 23, max mismatches = 1). Fastq files were aligned to bHirRus1 reference genome using bowtie2 v2. 4. 1. 68 The unmasked genome was used as reference. For WGS data, duplicated reads were removed using the Picard MarkDuplicates tool v2. 23. 4 (http://broadinstitute. github. io/picard/). Samtools v1. 9  79  (https://github. com/samtools/ samtools) was used to sort and index alignments. Alignment files generated from paired-end genomic data were further processed with Bam clipOverlap software v1. 0. 14 (https://genome. sph. umich.
edu/wiki/BamUtil:_clipOverlap) to trim overlaps between paired reads. The complete pipeline used to download and align reads is available on the project github page (https://github. com/ SwallowGenomics/BarnSwallow/blob/main/Analyses/popgen_data_download_alignment/popgen_pipeline. bash). Variant calling and filtering Freebayes v1. 3. 1  77  (https://github. com/freebayes/freebayes) was used to call variants. To reduce computational time, a script adapted from the VGP assembly pipeline (https://github. com/VGP/vgp-assembly/blob/master/pipeline/freebayes-polish/freebayes_v1. 3. sh) was used to parallelize the process by subsetting the reference genome by scaffolds. Variants were called with the options -min-mapping-quality 10 -min-base-quality 20 -populations (all other parameters were left to default).
Due to the lower sequencing coverage, -min-alternate-count 0 was used for ds6. The coordinates of the repetitive regions were extracted from the masked reference genome with a python script (https://gist. github. com/danielecook/cfaa5c359d99bcad3200) and the unmasked regions identified with bedtools v2. 29. 2  87  using the complement command. All vcf files were first filtered to remove variants falling within repetitive regions, multiallelic SNPs and indels. Variants were then split by population, and further filtering steps and thresholds are detailed in Table  S21. We removed sites showing more than twice the mean read depth across samples (INFO/DP field). In the vcf generated by Freebayes, genotype quality is expressed as QR (quality reference) and QA (quality alternate). We marked as missing all genotypes in which both values were below the threshold reported in Table  S21.
For FMT/DP filtering, we used as maximum value twice the average DP value and we approximated the 5% quantile of the distribution to set the minimum value. Individuals presenting a high amount of missing data (>70%) were discarded (Table  S21 ). Variants were also filtered for minor allele frequency (maf) with the usual 5% threshold and average fraction of missing sites among individuals (Table  S21 ). All filters were applied using bcftools v1. 1  79  (https://github. com/samtools/bcftools) with the view and filter commands, except the removal of variants falling within repetitive regions, performed with bedtools v2. 29. 2  87  using the intersect command and the coordinates of the unmasked regions previously identified. Standard statistics from the vcf files (in particular average site depth and average individual depth) were calculated using VCFtools v.
0. 1. 16  96 (https://github. com/vcftools/vcftools). An example of the complete set of commands used to filter variants (from ds2. 2) can be found here (https://github. com/SwallowGenomics/BarnSwallow/ blob/main/Analyses/variants%20filtering/filtering_commands. txt). To compare variant identification achieved with a linear genome (bHirRus1) and with the pangenome, we used the raw vcf file generated by Freebayes with the options -min-mapping-quality 10 -min-base-quality 20, extracting the 16 Illumina WGS samples relative to ds3. 1. Only biallelic SNPs were kept for the comparison. Bcftools v1. 1  79  (https://github. com/samtools/bcftools) was used to manipulate the vcf file and extract the genomic region corresponding to the camk2n2 gene. To validate variants from reads aligned to bHirRus1, IGV  97  was used for visual inspection.
SNP statistics and correlations with genomic features For all the analyses described in this subsection and the following one (''SNP density plotting''), all datasets generated with the same sequencing technology were combined (HiFi WGS; Illumina WGS; Illumina ddRAD). SNP density for each chromosome (excluding unlocalized/unplaced scaffolds) was computed on 10 kbp windows and SNPs were counted using bedtools v2. 29. 2  87  with the coverage -counts option. The average SNP density values across all chromosomes for each sequencing technology was calculated in R using the weighted mean function. Mean value was weighted for the window size to take into account truncated windows potentially present at chromosome ends. For the HiFi dataset (ds1) also a 5x downsampled HiFi dataset was generated (see ''HiFi reads processing for SNP catalogue, titration and phasing experiment'' section, ''titration of HiFi reads'' subsection, first titration experiment) considering the 20x read coverage of each sample (except for the A2 sample, starting from 15x) as the truth set (variants from the 5x reads set were intersected with variants from the 20x reads set using bedtools v2.
Resource For each chromosome and dataset, SNPs falling in intervals corresponding to genic, intergenic, exonic and intronic regions as determined from NCBI annotation were counted using bedtools v2. 29. 2  87  with the coverage -counts option (Data S1). To analyze correlations between SNP density and GC content in our catalog, the GC content was calculated using bedtools v2. 29. 2  87  with the -nuc option on 10 kbp windows and SNPs were counted every 10 kbp window. Correlation was tested in R computing the Spearman nonparametric rank test  124  with the R function cor. test. Unlocalized/unplaced scaffolds were excluded from the analysis. bedtools v2. 29. 2  87  was used to divide the genome in 10 kbp windows, using the makewindows command with the -w 10,000 flag.