dataesr/scientific-paragraphs-categorization huggingface dataset:
Text Summarization is a popular task and an active area of research for the Natural Language Processing community. It requires accounting for long input texts, a characteristic which poses computational challenges for neural models. Moreover, real-world documents come in a variety of complex, visually-rich, layouts. This information is of great relevance, whether to highlight salient content or to encode long-range interactions between textual passages. Yet, all publicly available summarization datasets only provide plain text content. To facilitate research on how to exploit visual/layout information to better capture longrange dependencies in summarization models, we present LoRaLay, a collection of datasets for long-range summarization with accompanying visual/layout information. We extend existing and popular English datasets (arXiv and PubMed) with visual/layout information and propose four novel datasets -consistently built from scholar resources -covering French, Spanish, Portuguese, and Korean languages.
, 2019; Graliński et al. , 2020; Mathew et al. , 2021). Nonetheless, a remaining limitation is that these (transformer-based) approaches are not suitable for processing long documents, the quadratic complexity of self-attention constraining their use to short sequences. Such models are hence unable to encode global context (e. g. long-range dependencies among text blocks). Focusing on compressing the most relevant information from long texts to short summaries, the Text Summarization task naturally lends itself to benefit from such global context. Notice that, in practice, the limitations linked to sequence length are also amplified by the lack of visual/layout information in the existing datasets. Therefore, in this work, we aim at spurring further research on how to incorporate multimodal information to better capture long-range dependencies.
Our contributions can be summarized as follows: • We extend two popular datasets for long-range summarization, arXiv and PubMed  (Cohan et al. , 2018) , by including visual and layout information -thus allowing direct comparison with previous works; • We release 4 additional layout-aware summarization datasets (128K documents), covering French, Spanish, Portuguese, and Korean languages; • We provide baselines including adapted architectures for multi-modal long-range summarization, and report results showing that  (1)  performance is far from being optimal; and (2) layout provides valuable information. All the datasets are available on HuggingFace. 1 2 Related Work 2. 1 Layout/Visually-rich Datasets Document Understanding covers problems that involve reading and interpreting visually-rich documents (in contrast to plain texts), requiring comprehending the conveyed multimodal information.
Hence, several tasks with a central layout aspect have been proposed by the document understanding community. Key Information Extraction tasks consist in extracting the values of a given set of keys, e. g. , the total amount in a receipt or the date in a form. In such tasks, documents have a layout structure that is crucial for their interpretation. Notable datasets include FUNSD  (Jaume et al. , 2019)  for form understanding in scanned documents, and SROIE  (Huang et al. , 2019) , as well as CORD  (Park et al. , 2019) , for information extraction from receipts. Graliński et al. (2020)  elicit progress on deeper and more complex Key Information Extraction by introducing the Kleister datasets, a collection of business documents with varying lengths, released as PDF files. However, the documents in Kleister often contain single-column layouts, which are simpler than the various multi-column layouts considered in LoRaLay.
Document VQA is another popular document understanding task that requires processing multimodal information (e. g. , text, layout, font style, images) conveyed by a document to be able to answer questions about a visually rich document (e. g. , What is the date given at the top left of the form?, Whose picture is given in this figure ?). The DocVQA dataset  (Mathew et al. , 2021)  and InfographicsVQA  (Mathew et al. , 2022)   The vast majority of summarization datasets only deal with plain text documents. As opposed to other Document Understanding tasks (e. g. , form understanding, visual QA) in which the placement of text on the page and/or visual components are the main source of information needed to find the desired data  (Borchmann et al. , 2021) , text plays a predominant role in document summarization.
However, guidelines for summarizing texts -espe-cially long ones -often recommend roughly previewing them to break them down into their major sections  (Toprak and Almacioglu, 2009; Luo et al. , 2019). This suggests that NLP systems might leverage multimodal information in documents. Miculicich and Han (2022)  propose a two-stage method which detects text segments and incorporates this information in an extractive summarization model. Cao and Wang (2022)  collect a new dataset for long and structure-aware document summarization, consisting of 21k documents written in English and extracted from WikiProject Biography. Although not all documents are explicitly organized into clearly defined sections, the great majority contains layout and visual clues (e. g. , a physical organization into paragraphs, bigger headings/subheadings) which help structure their textual contents and facilitate reading.
Datasets Construction Inspired by the way the arXiv and PubMed datasets were built  (Cohan et al. , 2018) , we construct our corpus from research papers, with abstracts as ground-truth summaries. As the PDF format allows simultaneous access to textual, visual and layout information, we collect PDF files to construct our datasets, and provide their URLs. 2 For each language, we select a repository that contains a high number of academic articles (in the order of hundreds of thousands) and provides easy access to abstracts. More precisely, we chose the following repositories: Further, we provide enhanced versions of the arXiv and PubMed datasets, respectively denoted as arXiv-Lay and PubMed-Lay, for which layout information is provided.
Collecting the Data Extended Datasets The arXiv and PubMed datasets  (Cohan et al. , 2018)  contain long scientific research papers extracted from the arXiv and PubMed repositories. We augment them by providing their PDFs, allowing access to layout and visual information. As the abstracts contained in the original datasets are all lowercased, we do not reuse them, but rather extract the raw abstracts using the corresponding APIs. Note that we were unable to retrieve all the original documents. For the most part, we failed to retrieve the corresponding abstracts, as they did not necessarily match the ones contained in the PDF files (due to e. g. PDF-parsing errors). We also found that some PDF files were unavailable, while others were corrupted or scanned documents. 6  In total, about 39% (35%) of the original documents in arXiv (PubMed) were lost.
arXiv-Lay The original arXiv dataset  (Cohan et al. , 2018)  was constructed by converting the L A T E X files to plain text. To be consistent with the other datasets -for which L A T E X files are not available -we instead use the PDF files to extract both text and layout elements. For each document contained in the original dataset, we fetch (when possible) the corresponding PDF file using Google Cloud Storage buckets. As opposed to the original procedure, we do not remove tables nor discard sections that follow the conclusion. We retrieve the corresponding abstracts from a metadata file provided by Kaggle. 7  PubMed-Lay For PubMed, we use the PMC OAI Service 8  to retrieve abstracts and PDF files. HAL We use the HAL API 9  to download research papers written in French. To avoid excessively long (e.
g. theses) or short (e. g. posters) documents, extraction is restricted to journal and conference papers. SciELO Using Scrapy, 10  we crawl the following SciELO collections: Ecuador, Colombia, Paraguay, Uruguay, Bolivia, Peru, Portugal, Spain and Brazil. We download documents written either in Spanish or Portuguese, according to the metadata, obtaining two distinct datasets: SciELO-ES (Spanish) and SciELO-PT (Portuguese). KoreaScience Similarly, we scrape the Korea-Science website to extract research papers. We limit search results to documents whose publishers' names contain the word Korean. This rule was designed after sampling documents in the repository, and is the simplest way to get a good proportion of papers written in Korean. 11  Further, search is restricted to papers published between 2012 and 2021, as recent publications are more likely to have digital-born, searchable PDFs.
Data Pre-processing For each corpus, we use the 95th percentile of the page distribution as an upper bound to filter out documents with too many pages, while the 5th (1st for HAL and SciELO) percentile of the summary length distribution is used as a minimum threshold to remove documents whose abstracts are too short. As our baselines do not consider visual information, we only extract text and layout from the PDF files. Layout is incorporated by providing the spatial position of each word in a document page image, represented by its bounding box (x 0 , y 0 , x 1 , y 1 ), where (x 0 , y 0 ) and (x 1 , y 1 ) respectively denote the coordinates of the top-left and bottom-right corners. Using the PDF rendering library Poppler 12  , text and word bounding boxes are extracted from each PDF, and the sequence order is recovered based on heuristics around the document layout (e.
g. , tables, columns). Abstracts are then removed by searching for exact matches; when no exact match is found, we use fuzzysearch 13  and regex 14  to find near matches. 15  For the non-English datasets, documents might contain several abstracts, written in different languages. To avoid information leakage, we retrieve the abstract of each document in every language available -according to the API for HAL or the websites for SciELO and KoreaScience -and remove them using the same strategy as for the main language. In the case an abstract cannot be found, we discard the document to prevent any unforeseen leakage. The dataset construction process is illustrated in Section A in the Appendix.
Datasets Statistics The statistics of our proposed datasets, along with those computed on existing summarization datasets of long documents  (Cohan et al. , 2018; Sharma et al. , 2019)  are reported in Table  1. We see that document lengths are comparable or greater than for the arXiv, PubMed and BigPatent datasets. For arXiv-Lay and PubMed-Lay, we retain the original train/validation/splits and try to reconstruct them as faithfully to the originals as possible. For the new datasets, we order documents based on their publication dates and provide splits following a chronological ordering. For HAL and Korea-Science, we retain 3% of the articles as validation data, 3% as test, and the remaining as training data. To match the number of validation/test documents in HAL and KoreaScience, we split the data into 90% for training, 5% for validation and 5% for test, for both SciELO datasets.
Pegasus is an encoder-decoder model pre-trained using gapsentences generation, making it a state-of-the-art model for abstractive summarization. For the non-English datasets, we rely on a finetuned MBART as our baseline. MBART  (Liu et al. , 2020 ) is a multilingual sequence-to-sequence model pretrained on large-scale monolingual corpora in many languages using the BART objective  (Lewis et al. , 2019). We use its extension, MBART-50  (Tang et al. , 2020) , 17  which is created from the original MBART by extending its embeddings layers and pre-training it on a total of 50 languages. Both Pegasus and MBART are limited to a maximum sequence length of 1,024 tokens, which is well below the median length of each dataset.
Long-range, text-only models To process longer sequences, we leverage BigBird  (Zaheer et al. , 2020) , a sparse-attention based Transformer which reduces the quadratic dependency to a linear one. For arXiv-Lay and PubMed-Lay, we initialize Big-Bird from Pegasus  (Zaheer et al. , 2020)  and for the non-English datasets, we use the weights of MBART. The resulting models are referred to as BigBird-Pegasus and BigBird-MBART. For both models, BigBird sparse attention is used only in the encoder. Both models can handle up to 4,096 inputs tokens, which is greater than the median length in PubMed-Lay, HAL and KoreaScience. Long-range, layout-aware models We also include layout information in long-range text-only models. Similarly to layout-aware models with standard input size, we integrate layout information into our long-range models by encoding each token's spatial position in the page.
Implementation Details We initialize our Pegasus-based and MBART-based models with, respectively, the google/pegasus-large and facebook/mbart-large-50 checkpoints shared through the Hugging Face Model Hub. As for T5 and LED, we use the weights from t5-base and allenai/led-base-16384, respectively. 18 Following  Zhang et al. (2020)  and  Zaheer et al. (2020) , we fine-tune our models up to 74k (100k) steps on arXiv-Lay (PubMed-Lay). On HAL, the total number of steps is set to 100k, while it is de- creased to 50k for the other non-English datasets. 19  For each model, we select the checkpoint with the best validation loss. For Pegasus and MBART models, inputs are truncated at 1,024 tokens. For BigBird-Pegasus models, we follow Zaheer et al. (  2020 ) and set the maximum input length at 3,072 tokens.
As the median input length is much greater in almost every non-English dataset, we increase the maximum input length to 4,096 tokens for BigBird-MBART models. Output length is restricted to 256 tokens for all models, which is enough to fully capture at least 50% of the summaries in each dataset. For evaluation, we use beam search and report a single run for each model and dataset. Following  Zhang et al. (2020) ;  Zaheer et al. (2020) , we set the number of beams to 8 for Pegasus-based models, and 5 for BigBird-Pegasus-based models. For the non-English datasets, we set it to 5 for all models, for fair comparison. For all experiments, we use a length penalty of 0. 8. For more implementation details, see Section B. 1 in the Appendix.
General Results In Table  3 , we report the ROUGE-L scores obtained on arXiv and PubMed datasets (reported by  Zaheer et al. (2020) ), as well as on the corresponding layout-augmented counterparts we release. 20  On arXiv-Lay and PubMed-Lay, we observe that, while the addition of layout to Pegasus does not improve the ROUGE-L scores, there are gains in integrating layout information into BigBird-Pegasus. To assess whether these gains are significant, we perform significance analysis at the 0. 05 level using bootstrap, and estimate a ROUGE-L thresh- 19 We tested different values for the number of steps (10k, 25k, 50k, 100k) and chose the one that gave the best validation scores for MBART. 20 For detailed results, please refer to Section C. 1 in the Appendix. old that predicts when improvements are significant.
r. t. the scores obtained on the original datasets. This can be explained by two factors. First, our extended  datasets contain less training data due to the inability to process all original documents. Secondly, the settings are different: while the original arXiv and PubMed datasets contain clear discourse information (e. g. , each section is delimited by markers) obtained from L A T E X files, documents in our extended versions are built by parsing raw PDF files. Therefore, the task is more challenging for text-only baselines, as they have no access to the discourse structure of documents, which further underlines the importance of taking the structural information, brought by visual cues, into account. Table  4  presents the ROUGE-L scores reported on the non-English datasets. On HAL, we note that BigBird-MBART does not benefit from layout.
After investigation, we hypothesize that this is due to the larger presence of single-column and simple layouts, which makes layout integration less needed. On both SciELO datasets, we notice that combining layout with long-range modeling brings substantial improvements over MBART. Fur-ther, we find that the plain-text BigBird models do not improve over the layout-aware Pegasus and MBART on arXiv-Lay and SciELO-ES, demonstrating that simply capturing more context does not always suffice. Regarding performance on Ko-reaScience, we can see a significant drop in performance for every model w. r. t the other non-English datasets. At first glance, we notice a high amount of English segments (e. g. , tables, figure captions, scientific concepts) in documents in KoreaScience. To investigate this, we use the cld2 library 21 to detect the language in each non-English document.
We consider the percent confidence of the top-1 matching language as an indicator of the presence of the main language (i. e. , French, Spanish, Portuguese or Korean) in a document, and average the results to obtain a score for the whole dataset. Table  5  reports the average percent confidence obtained on each split, for each dataset. We find that the percentage of text written in the main language in KoreaScience (i. e. , Korean) is smaller than in other datasets. As the MBART-based models expect only one language in a document (the information is encoded using a special token), we claim the strong presence of non-Korean segments in KoreaScience causes them to suffer from interference problems. Therefore, we highlight that KoreaScience is a more challenging dataset, and we hope our work will boost research on better long-range, multimodal and multilingual models.
22  Based on the aforementioned factors, the scores obtained by each model are then grouped by quartile range, and averaged over each range, see Figure  1. On arXiv-Lay, we find that layout brings most improvement when dealing with the 25% longest documents and summaries, while, for both datasets, layout is least beneficial for the shortest documents and summaries. These results corroborate our claim that layout can bring important information about long-range context. Concerning the third factor, we see, on PubMed-Lay, that layout is most helpful for documents that have the widest ranges of font sizes, showcasing the advantage of using layout to capture salient information.
Conclusion We have presented LoRaLay, a set of large-scale datasets for long-range and layout-aware text summarization. LoRaLay provides the research community with 4 novel multimodal corpora covering French, Spanish, Portuguese, and Korean languages, built from scientific articles. Furthermore, it includes additional layout and visual information for existing long-range summarization datasets (arXiv and PubMed). We provide adapted architectures merging layout-aware and long-range models, and show the importance of layout information in capturing long-range dependencies.
Limitations The proposed corpus is limited to a single domain, that of scientific literature. Such limitation arguably extends to the layout diversity of documents. In terms of risks, we acknowledge the presence of Personally Identifiable Information such as author names and affiliations; nonetheless, such informa-  A. 1 Extended Datasets -Lost Documents Figure  3  provides details on the amount of original documents lost in the process of augmenting arXiv and PubMed with layout/visual information. We observe four types of failures, and provide numbers for each type: • The link to the document's PDF file is not provided (Unavailable PDF); • The PDF file is corrupted (i. e. , cannot be opened) (Corrupted PDF); • The document is not digital-born, making it impossible to parse it with PDF parsing tools ( Scanned PDF); • The document's abstract cannot be found in the PDF (Irretrievable Abstract).
, 2018; Houssaye, Waskow, et al. , 2016; Plasse et al. , 2019; Wagstaffe et al. , 2022). In line with these latter works, this study aims to improve our understanding of the relationships between posture and bone microstructure in reptiles. Using mid-diaphyseal cross-sections of extant and extinct reptile femora, we test for geometric and microanatomical differences between taxa that reflect locomotion and posture. Typically, regardless of allometric and environmental factors (which will be considered in detail), we would expect bipeds to have greater bone compactness (proportion of bone tissue in a skeletal element) than quadrupeds, as body mass is more evenly distributed in the latter (two weight-bearing limbs vs four). We use the data we obtain to construct phylogenetically informed models linking locomotion, posture and microanatomy, built on an extant and extinct sample, which then allow us to infer the posture of Palaeozoic and Mesozoic amniotes that document various phases of posture diversification in that clade.
| Data acquisition We measured different geometric and microanatomical parameters on femoral shaft cross-sections. To do so, we relied mainly on CT scan data that we retrieved from the literature and from MorphoSource. We scanned some of the specimens on the tomography platforms of the Muséum national d'histoire naturelle and the Université de Montpellier. We also used unpublished images of histological sections from the study by  Quemeneur et al. (2013). In the literature, the reference plane for the sections is traditionally located at the TA B L E 2 List of the extinct taxa for which we inferred the posture. midpoint of the bone or at the level of the ossification centre  (Amson & Kolb, 2016; Houssaye et al. , 2018; Quemeneur et al. , 2013). For CT scan data, we decided to locate the reference plane where the perimeter of the shaft is the smallest, because this is an area where mechanical stresses are important  (Beck et al.
nodes for each tree. We then removed Tinamus major from the trees as it was not part of our sample of reptile femoral cross-sections initially. For the statistical analyses, we also needed reference trees that included the taxa for which we wanted to infer posture. We set the diver- Concerning the relationship between ornithopods, we used the consensus tree from  Strickson et al. (2016). We considered the hypsilophodont in our sample as a member of clade Elasmaria based on Herne et al. (  2019 ) and inserted it between Jeholosaurus and Hypsilophodon. The Wonthaggi Formation encompassing the Flat Rocks locality where the hypsilophodont was recovered  (Woodward et al. , 2018)   (Sames, 2008). We then used the cal-3TimePaleoPhy function of the R package paleotree to calibrate this topology over time based on the FADs and LADs mentioned above.
| Phylogenetic flexible discriminant analyses We imported our reptile dataset into R and log transformed the variables, with the exception of ratios (Ecc and SR). To avoid future complications related to the existence of singular variance-covariance matrices, we generated a dissimilarity matrix for each dataset from the correlation coefficients of the microanatomical variables before performing a hierarchical cluster analysis to identify and eliminate highly correlated variables (correlation coefficient above 0. 95). 9 variables out of 13 were retained for our dataset (Pe min , Ecc, SR, C obs , P, S, RS, RPSD, RSSD). Linear discriminant analysis (LDA), classically used to separate groups from explanatory variables and to make predictions, cannot remove phylogenetic bias  (Motani & Schmitz, 2011).