dataesr/scientific-paragraphs-categorization huggingface dataset:
Maintaining Consistency in Treebank Annotations We start with an extended clean-up of the NAra-biziV1 formatting, which involves reinstating missing part-of-speech tags and rectifying Conllu formatting discrepancies. Then, we embark on general error mining in the lexical and syntactical annotation and correction phase. We implement this stage using semi-automated methods. We do not change the UD tagsets used in the original treebank. Error Mining We use the UD validator Vr2. 11 3  , a tool designed to assess the annotation of treebanks in UD and ensure compliance with the UD specifications. The validator is specifically employed to detect common errors, such as invalid dependency relations, incorrect part-of-speech tags, and inconsistent usage of features like tense and aspect. By leveraging the UD validator, we guarantee that our dataset is syntactically consistent and conforms to the standards established by the UD project.
These changes encompass correcting cycle and projectivity issues and removing duplicates. We also use Errator  (Wisniewski, 2018) , a data mining tool, to pinpoint inconsistencies in our dataset. It implements the annotation principle presented by  Boyd et al. (2008) , which suggests that if two identical word sequences have different annotations, one is likely erroneous. We remove the duplicated sentences when the text field is an exact match and fix duplicated sentence identification for different sentences. We also fixed some problems with the original text, such as Arabic characters encoding and sentence boundaries. Tokenization We address tokenization concerns to uphold consistency in the NArabizi Treebank annotations. Furthermore, we introduce targeted adjustments to resolve issues related to segmenting specific word classes, including conjunctions, interjections (e.
Annotation Methodology for NER and Offensive Language Detection Named Entity Recognition Our NER annotation guidelines are based on the revised tokenization of the NArabizi treebank, which ensures consistency between token-level annotations, an essential aspect of multi-task learning. We use the Inception tool  (Klie et al. , 2018)  for our manual annotation by two native speakers, adhering to the IOB2 Scheme  (Tjong Kim Sang and Veenstra, 1999). Each word is labeled with a tag indicating whether it is at the beginning, inside, or outside of a named entity. In case of disagreement between annotators, the multiple annotations were subsequently discussed until agreement was reached, and one annotation was selected to be retained. We extend the FTB NER  (Sagot et al. , 2012)  French treebank annotations.
Our annotation contains the following NE types: PER for real or fictional persons, ORG for organizations, LOC for locations, COMP for companies, and OTH for brands, events, and products. In cases of ambiguity between products and companies, we adhere to the decision made in the FTB dataset. For person names, we exclude grammatical or contextual words from the mention. We annotate football teams as organizations, and we annotate mentions of "Allah" or "Rabi" as PERderivA. The PERderiv annotation is applied to groups of individuals who originate from or share the same location. Country names are consistently labeled as locations, irrespective of the context. TV channels and ambiguous brand names are annotated as companies, while religious groups are not designated entities. The names of football stadiums are classified under OTH, whereas journal names are identified as organizations.
see Table  10  in the appendix A). Our schemes also differ in named entities' scope, as we split contracted forms, ours only cover the nominal phrase parts. Regarding nouns, such as "bled", which means country, some are annotated as entity GPE in NERDz, which is not the case in our dataset. Also, the names of stadiums are annotated as LOC in NERDz while they are considered OTH in our dataset. Similarly, for "equipe nationale", which means national team is annotated ORG in NERDz, while we do not consider it as an entity, following the FTB NER's guidelines. Added to annotator divergences, this may explain the differences in the count of the entities. Offensive Language Classification The annotation process for offensive language classification was conducted manually by three annotators with diverse backgrounds.
The annotators consisted of two females and one male, each bringing unique expertise to the task. One female annotator is a Ph. D. student in NLP, the other is a Ph. D. student in political sciences, and the male annotator is an engineer with in-depth knowledge of North African football, a prominent topic in the dataset. The annotators were asked to annotate every sentence as offensive (OFF) or non-offensive (NOT-OFF). Offensive posts included any form of unacceptable language, targeted offense (veiled or direct), insults, threats, profane language, and swear words. To maintain objectivity and minimize potential bias, the annotators were not granted access to the other annotators' work and were not allowed to discuss their annotations with one another. This approach ensured the independence of their judgments, allowing for a more reliable evaluation of the offensive language classification process.
Dataset Evaluation We evaluate the NarabiziV2 dataset on UD parsing tasks and NER using standard transfer learning architectures on which we vary the pre-trained language model and the tokenization scenario. New NArabizi CharacterBert Model Following  Riabi et al. (2021) , we train a CharacterBERT (El  Boukkouri et al. , 2020)  model, a characterbased BERT variant, on a NArabizi new filtered corpus. The authors demonstrate that Character-BERT achieves significant results when dealing with noisy data while being extremely data efficient. We improve the initial pre-training dataset used by  Riabi et al. (2021)  by more stringently filtering non-NArabizi examples from the 99k instances provided by  Seddah et al. (2020) , as well as incorporating new samples from the CTAB corpus  (Amara et al.
, 2021)  and 12k comments extracted from various Facebook and forum posts, mostly in the Tunisian dialect taken from different datasets listed by  Younes et al. (2020). This results in a 111k sentence corpus. To exclude non-NArabizi content, we first use a language detection tool  (Nakatani, 2010)  with a 0. 9 confidence threshold to eliminate text in French, English, Hindi, Indonesian, and Russian, which are commonly found in mixed Arabizi data. Following the filtering process, a bootstrap sampling method is adopted to randomly select a subset of the remaining text for manual annotation. This annotated text is then used to train an SVM classifier for NArabizi detection. The final dataset, containing 91k annotated text instances after deduplication, focuses on North African Arabizi text. We make this corpus publicly available.
Sub-word Models We also evaluate the performance of subword-based language models, monolingual and multilingual. For the multilingual subword-based language model, we use mBERT, the multilingual version of BERT  (Devlin et al. , 2018). It is trained on data from Wikipedia in 104 different languages, including French and Arabic. Muller et al. (2020)  demonstrated that such a model could be transferred to NArabizi to some degree. Finally, our monolingual model is DziriBERT  (Abdaoui et al. , 2021) , a monolingual BERT model trained on 1. 2M tweets from major and highlypopulated Algerian cities scrapped using a set of popular keywords in the Algerian spoken dialect in both Arabic and Latin scripts.
Training on NArabiziV2 enhances the average scores for UPOS, UAS, and LAS by 3. 5 points, illustrating the favorable outcomes of the refinements introduced in the NAra-biziV2 dataset. This observation is further substan- tiated by examining the performance of Character-BERT and mBERT, reinforcing the validity of the noted improvements. A comparative analysis of the results for models trained and tested on NArabiziV1, denoted by the blue box, and those for models trained and tested on NArabiziV2, denoted by the red box, reveals that NArabiziV2 generally yields superior evaluation scores. This observation underlines the impact of the treebank's consistency on the overall performance of the models. When we test on NAra-biziV1, the model trained on NArabiziV1 gets better results than the model trained on NArabiziV2.
Results for NER and Offensive Language Detection NER Table  5  presents the results for NER 6. The CharacterBERT model achieves the highest F1 scores for LOC and OTH categories, as well as the best performance for PERderiv and PERderivA. On the other hand, the DziriBERT model outperforms the other models in the ORG and PER categories. It is important to note that the performance varies significantly across the different categories, reflecting the diverse challenges posed by each entity type. For instance, some categories contain named entities with variations of the same word, such as "Allah"/"Alah"/"Elah", which translates into God for PERderivA. Since CharacterBERT uses character-level information, it is more robust to noise, which explains the high performances for those entities. Offensive Language Detection The imbalance between non-offensive and offensive instances is challenging during the models' training and eval-uation.
For example, we fail to train mBERT as it only predicts non-offensive labels corresponding to the majority class. This can also be explained by how hard the distinction between offensive and nonoffensive content is without context and external knowledge, as explained before. This also raises the question of how relevant is the backgrounds of the annotators for the offensive detection dataset  (Basile et al. , 2020; Uma et al. , 2021; Almanea and Poesio, 2022). 6 Discussion.
Impact of the Pre-training Corpus In Appendix A, we present the results of all our experiments using the CharacterBERT model trained by  Riabi et al. (2021). We observe a heterogeneous improvement in performance, with predominantly better outcomes for our CharacterBERT. We hypothesize that the impact of filtering the training data may not be overly beneficial, possibly due to some smoothing during the training process. Both models' final training data sizes are comparable: 99k for CharacterBERT  (Riabi et al. , 2021)  and 91k for our CharacterBERT. Nevertheless, we believe this new corpus can be a valuable resource for this language.
Conclusion In this paper, we present a comprehensive study on the development and refinement of the NArabizi Treebank  (Seddah et al. , 2020)  by improving its annotations, consistency, and tokenization, as well as providing new annotations for NER and offensive language. Our work contributes to the enhancement of the NArabizi Treebank, making it a valuable resource for research on low-resource languages and user-generated content with high variability. We explore the impact of tokenization on the refined NArabizi treebank, employing the UDPipe tokenizer for our evaluation. The results demonstrate the tokenizer's effectiveness in handling various tokens and multiword expressions. Our experiments show that training and testing on the NArabiziv2 improve the UD tasks performances. Furthermore, we show the impact of the tokenization for NER and UD tasks, and we report results using predicted tokenization for evaluation to estimate the models' performance on raw data.
Future research could emphasize expanding the NArabizi Treebank towards other dialects and ex-amining the treebank's potential applications in various NLP tasks. Our dataset is made freely available as part of the new version of the Narabizi Treebank 9. The next release will additionally contain a set of other sentence translations prepared by a Tunisian speaker. These translations will be interesting for cross-dialect studies, given that the Narabizi corpus is predominantly made of Algerian dialect.
2022b). Chaining consists of finding maximal ordered subsets of seeds that all agree on a certain genomic location  (Jain et al. 2022a ); seeds often have spurious matches due to their short lengths. However, even the most recent long-read alignment tools are bottlenecks in analysis pipelines  (Berger and Yu 2022). For instance, the popular minimap2 software requires 12 CPU hours to map a typical PacBio HiFi dataset to the human genome, and Winnowmap2 requires 15 CPU days, preventing both real-time analysis of sequencing data  (Loose et al. 2016 ) and efficient re-analysis of previouslysequenced data collections  (Edgar et al. 2022). A significant part of the minimap2 and Winnowmap2 running times are in their seeding and chaining steps  (Kalikar et al. 2022). These state-of-the-art long-read alignment tools are sensitive and accurate, but their underlying seed constructs (k-mers) are tailored to noisy reads.
Our approach borrows from natural language processing where the tokens of the k-mers are the minimizers instead of base-pair letters. We asked whether long-read mapping can be sped up by newly substituting k-mer seeds with k-min-mers that occur uniquely in the genome. In this work, we evaluate the extent to which k-min-mers can act as suitable seeds for accurate long-read alignment, and the performance of mapquik in making use of them for HiFi-read alignment. Our work represents the promise of unique, inexact seeds, such as k-min-mers, for ultra-fast long-read mapping and beyond.
In recent years, many research groups focusing on low-level and/or hardware-specific acceleration have proposed ways to accelerate minimap2. mm2-fast  (Kalikar et al. 2022) , a CPU acceleration of minimap2 developed by Intel, achieved a 1. 5× acceleration over minimap2, end-to-end on HiFi reads. The bulk of the speed-up was obtained in the alignment phase, not in the seeding-chaining phase (Figure  1  in  Kalikar et al. 2022). The domain-specific language Seq was developed to speed up genomic sequence analysis, and achieved two orders of magnitude improvement in the homology table reconstruction of the CORA read mapper  (Yörükoğlu et al. 2016; Shajii et al. 2021). There also exist efficient implementations that employ specialized hardware. mm2-ax  (Sadasivan et al. 2023 ), a recent GPU acceleration of minimap2 developed by NVIDIA, achieved 2.
Datasets and mapping evaluation We used the complete human reference genome CHM13v2. 0 for our evaluations. We constructed a simulated dataset of long reads with 99% base-level accuracy and 24 kbp mean length using In order to map a query sequence using the reference index (top right, nucleotide C in blue denotes a sequencing error), mappers compute the minimizers on the query sequence (c), and find matches between the minimizers of the query and those in the reference index. Once minimizer matches are found, minimap2 and Winnowmap2 perform a colinear chaining step to output a high-scoring set of matches, using dynamic programming (d). In contrast, mapquik (bottom, green shaded) indexes reference sequences by generating k-min-mers, k consecutive, randomlyselected minimizers of length (k = 3, = 2) (e), and storing only the k-min-mers that appear exactly once in the reference (f).
1) at 30× coverage using the same protocol as the human simulated reads. mapquik was run with default parameters (k = 5, = 31, δ = 0. 01, β = 4, µ = 11, ε = 2000); we provide an extensive evaluation of the parameters k, , and δ in Supplemental Note S8 and Supplemental Figure  S3 , and of β and µ in Figure  3. All other tools were run with default parameters in HiFi mapping mode. Command lines and versions are given in Supplemental Note S2. For simulated reads, we assessed mapping accuracy using the mapeval command of the paftools software distributed in the minimap2 package. A read is considered to be correctly mapped if the intersection between the true and mapped reference intervals is at least 10% of their union. For real reads, we evaluated the concordance between the alignments of minimap2 and mapquik using a custom script (experiments/intersect_pafs.
py in the GitHub repository and Supplemental Code), similar to mapeval. In our evaluation of both the simulated and real datasets, we focus on reads of the highest mapping quality (Q60). Mappers report a mapping quality metric for each read, indicating their confidence that the read is mapped at the right location, as an integer between 0 and 60 where 60 corresponds to the highest confidence. Reads with low mapping quality are less frequent and are often removed in downstream applications (e. g. , the popular variant calling pipeline GATK  (McKenna et al. 2010 ) filters out reads with mapQ ≤ 20 by default). With minimap2 results, mapeval reports only two mapping quality groups (Q0 and Q60), with no value in-between. With mapquik, we report a mapQ of 60 if the output pseudo-chain has score ≥ µ or length ≥ β, and 0 otherwise.
mapquik achieves faster and accurate mapping of HiFi reads to the human and maize genomes Table  1  demonstrates the enhanced overall performance of mapquik and other evaluated methods (minimap2, mm2-fast, Winnowmap2, and concurrently-developed BLEND) on mapping simulated and real PacBio HiFi reads to the human and maize genomes. We demonstrate mapquik's 37× and 975× relative speedup over state-of-the-art mapping methods minimap2 and Winnowmap2, respectively, for both simulated and real HiFi reads from the human genome. We further apply mapquik to mapping simulated HiFi reads from the highly-repetitive maize genome, and demonstrate a 410× speed-up over minimap2, with remarkably no loss of sensitivity nor accuracy. On the simulated human dataset, mapquik is 54× faster than minimap2. mapquik maps 95.
8% of reads at a mapQ score of 60 (Q60), indicating a high-confidence match, with 2 errors. In contrast, other mappers map 96. 1% to 98. 6% of reads at Q60 with also no/almost no error. On the real dataset from HG002, a similar trend is observed. mapquik outperforms all mappers except Winnowmap2 in terms of percentage of reads mapped (96. 1%), as opposed to 92. 2 to 97. 9% for the other tools. The concordance of minimap2 and mapquik mappings is 99. 8% on the Q60 reads mapped by minimap2. In Supplemental Table  S1 , we provide the fraction of the reference genome covered by each tool for all experiments. All mappers required less than 14 gigabytes of memory on the human genome (MashMap2 was not further evaluated as it took over 13 wall-clock hours on the simulated human dataset, and does not output mapping quality scores.
) A highlight of mapquik is a 410× mapping speed-up compared to minimap2 on the maize genome. Remarkably, this speed-up comes with near-perfect precision for mapquik and no loss of sensitivity, as mapquik reports the second highest number of mapped reads at mapping quality 60 across all tools after Winnowmap2. Of note, Winnowmap2 has faster performance on maize than the human genome and than that of minimap2 on the maize genome. We further investigated why some reads were mapped at lower qualities than Q60 or were not mapped at all. Out of 58,004 reads from the simulated human dataset that were not mapped at Q60 by mapquik, 94. 4% of these reads intersected with centromeric/satellite regions of chromosomes, as reported by BEDTools (Quinlan and Hall 2010) using the chm13v2. 0_censat_v2. 0. bed annotation from the Telomere-to-Telomere (T2T) consortium.
2004 ), different in nature from k-min-mers. mapquik indexing is 9× faster than minimap2 and 6× faster than BLEND. The mapquik seeding strategy provides ultra-fast construction of an index that records unique k-min-mer positions across the reference genome. This index is of independent interest, for example, for indexing larger databases such as RefSeq (W  Li et al. 2021). We anticipate that this index will have other uses beyond long-read mapping, such as large-scale sequence search. We have already demonstrated the usefulness of performing k-min-mer search in anti-microbial resistance tracking  (Ekim et al. 2021) , albeit in earlier work that did not benefit from the speed-up of such an index presented here. Table  2. Indexing a reference human genome using mapquik and other evaluated methods (minimap2, mm2-fast, Winnowmap2, BLEND).
Robustness of mapping shorter reads with mapquik We next asked how mapquik would perform on smaller reads than those offered by HiFi, which are currently available for lengths ranging from 10 kbp to 25 kbp. Future technologies may also offer long high-accuracy reads, although HiFi is currently the only high-throughput solution available. To determine whether mapquik is also suitable for mapping HiFi reads shorter than 24 kbp, we tested mapquik on other datasets with overall shorter reads using both real and simulated data. To assess mapquik's robustness in mapping reads of varying length, we ran mapquik on real HG002 HiFi 16 kbp length reads from DeepConsensus  (Baid et al. 2022) , using identical parameters as in the run with 24 kbp read length. mapquik's performance on 16 kbp reads is strikingly similar to that on 24 kbp reads: the running time increased by only 4%, memory usage remains nearly identical, and the concordance with minimap2 remains 99.
Efficient mapping of long reads in minimizer space This research is partially based upon work supported by the  National Science Foundation  under Grants  DBI-2138585  and  OAC-1931531  (to P. M. ). Research reported in this publication was also supported by the  National Institutes of Health  under Grants  NIH   R01HG010959  (for B. E. to B. B. ),  NIH   1R35GM141861  (to B. B. ), and  NIH   R01GM146462  (to P. M. ). This project has received funding from the  European Union 's  Horizon 2020 research and innovation programme  under the  Marie Skłodowska-Curie  grants agreements No. 872539  and  956229  (R. C. ). R. C. was supported by  ANR Transipedia, SeqDigger, Inception  and  PRAIRIE grants  ( ANR-18-CE45-0020 ,  ANR-19-CE45-0008 ,  PIA/ANR16-CONV-0005 ,  ANR-19-P3IA-0001 ). The content is solely the responsibility of the authors and does not necessarily represent the official views of the  National Institutes of Health.
In this paper, we introduce a novel end-to-end multimodal video captioning framework based on cross-modal fusion of visual and textual data. The proposed approach integrates a modality-attention module, which captures the visual-textual inter-model relationships using cross-correlation. Further, we integrate temporal attention into the features obtained from a 3D CNN to learn the contextual information in the video using task-oriented training. In addition, we incorporate an auxiliary task that employs a contrastive loss function to enhance the model's generalization capability and foster a deeper understanding of the intermodal relationships and underlying semantics. The task involves comparing the multimodal representation of the video-transcript with the caption representation, facilitating improved performance and knowledge transfer within the model.
Finally, a transformer architecture is used to effectively capture and encode the interdependencies between the text and video information using attention mechanisms. During the decoding phase, the transformer allows the model to attend to relevant elements in the encoded features, effectively capturing long-range dependencies and ultimately generating semantically meaningful captions. The experimental evaluation, carried out on the MSRVTT benchmark, validates the proposed methodology, which achieves BLEU4, ROUGE, and METEOR scores of 0. 4408, 0. 6291 and 0. 3082, respectively. When compared to the state-of-the-art methods, the proposed approach shows superior performance, with gains in performance ranging from 1. 21% to 1. 52% across the three metrics considered.