dataesr/scientific-paragraphs-categorization huggingface dataset:
This process begins by selecting the top consecutive words from the temporary database. For each pair of words, the second word is combined with other consecutive words that follow it and possess the highest probability. This technique generates sentences that effectively represent the content of the given papers. The process above is repeated for each consecutive word, resulting in a list of sentences. Finally, the probability of each sentence is calculated using a formula derived from the Markov chain theorem. Using the Markov Chain technique, the query modeling phase generates well-formed sentences for search queries. This approach enhances the effectiveness of the MASSPR system by ensuring the relevance and coherence of the queries, ultimately leading to more accurate recommendations. The most significant values will be chosen to accurately represent the given papers.
Searching articles module The Searching Articles Module is tasked with searching for scientific papers on the web using the generated queries. It comprises three main components. First, the web searching agents, functioning as spiders, utilize the queries to search the web for relevant scientific papers. The inner architecture of the searching agent is depicted in Figure  4. The process begins by importing queries from the query buffer. The agent then performs a search task, crawling web pages and extracting URLs that are deemed relevant. Each discovered article is subsequently forwarded to the collecting agents in the next module. To facilitate web navigation, we leverage the Jsoup Java library, which provides a user-friendly API for fetching URLs, extracting data, and manipulating HTML using HTML5 DOM methods and CSS selectors.
Second, the article collecting agents are responsible for aggregating the downloaded articles obtained from the previous module. These agents scrape articles and extract relevant data stored in a well-structured and unified format within the articles buffer. Finally, the articles buffer acts as a temporary database that houses the gathered scientific papers from the web. The agents within our search module are autonomous, meaning they can make judgments regarding the collected material. These judgments are based on the Bayes probabilities computed by the preceding model. Figure  5  presents the sub-architecture of the searching articles module, providing an overview of its internal structure and component interactions. Through the collaborative efforts of the web searching agents, article collecting agents, and the articles buffer, the Searching Articles Module effectively retrieves scientific papers from the web and stores them in a structured manner.
The agents' autonomy allows for intelligent decision-making, while utilizing the Jsoup library streamlines the web navigation process. The Searching Articles Module consists of a team of agents that function as web spiders, utilizing the queries to search the web for relevant scientific papers. These agents begin their process by importing queries from a query buffer, and then initiate search tasks across the World Wide Web. They navigate through various websites, automatically retrieving targeted web pages. The agents extract relevant URLs from each webpage during their web crawling process. These URLs serve as potential sources for finding scientific papers. The agents explore these URLs, downloading the associated web pages for further analysis. Once the web pages are downloaded, the agents transfer them to the collecting agents in the subsequent module for further processing and aggregation.
Data preparation They must be prepared before feeding the articles from the buffer database into the learning model. The data for the learning model is divided into two parts: the inputs and the outputs for the classification model. Both of these parts require processing to meet the requirements of the multi-label classification model. Reference Extraction and preparation (output model): The references from all citing documents are extracted and represented by IDs corresponding to the ranking of the papers in the buffer. Subsequently, a binarization transformation is applied to prepare the output for the multi-label, multi-output classification model. During the learning stage, this binarization transformation involves training a regressor or binary classifier for each class. It requires converting the multi-class labels into binary labels, indicating whether an instance belongs or does not belong to a particular class.
GUI front-end module and GUI back-end engine The GUI Front-End module serves as the user interface for interacting with the system, while the GUI Back-End Engine module plays a crucial role in providing the necessary functionalities to support the GUI Front-End module. Acting as a bridge between different modules in MASSPR, such as the Query Module and Filtering Module, the GUI Back-End Engine module performs the following roles: • Execution of visualization functions: The GUI Back-End Engine module handles the execution of all visualization functions required for the smooth operation of the GUI Front-End module. • Integration with the Query Module: This module facilitates the transfer of scientific articles uploaded by the user to the Query Module, enabling the generation of search queries based on the user's input.
DEVELOPMENT Our objective is to enhance a system that assists students and researchers in automatically collecting and filtering published scientific papers from the Internet, while recommending the most pertinent ones within their research domains. To achieve this, we outline the following goals of MASSPR, which aim to empower users to: (1) Perform document-based search using input documents: Users can input their desired documents into the system, and the system will utilize these documents to conduct searches for related scientific papers. (2) Choose the scientific platform for the search: Users can choose the scientific platform or database where the search will be conducted, allowing them to tailor the search to their specific needs. (3) Provide the most pertinent document as the output: The system employs advanced algorithms and machine learning techniques to analyze and rank the retrieved scientific papers, ensuring that the most relevant and significant documents are presented to the users.
(4) Present the details of the retrieved related documents: Along with the recommendation of pertinent scientific papers, the system provides comprehensive details about each retrieved document. This includes information such as the title, abstract, author, keywords, and other relevant details that aid users in assessing the relevance and significance of the papers. By achieving these goals, MASSPR empowers users to efficiently search for and access valuable scientific papers, saving them time and effort in their research endeavors. To create the MASSPR tool, we utilized the Python programming language and the Jupyter IDE. We also employed the Java programming language and the NetBeans IDE as the development environment.
Transfer learning Transformer-based pre-trained language models have demonstrated remarkable achievements across various natural language processing (NLP) tasks. The development of these models commenced with GPT and BERT, both of which rely on transformers, self-supervised learning, and transfer learning. By leveraging the power of transformers, pre-trained language models acquire universal language representations through self-supervised learning, and subsequently transfer this knowledge to downstream tasks. This approach offers the advantage of furnishing downstream models with valuable foundational knowledge, obviating the need to train them from scratch  [10]. (1) doc2query/all-t5-base-v1 The T5-base model underwent training using the MS MARCO Passage Dataset, comprising approximately 500,000 actual search queries sourced from Bing and their corresponding relevant passages.
This library includes various state-of-the-art models capable of transforming text into vector representations. Sentence-BERT utilizes the BERT encoder within a Siamese architecture, training it for similarity comparison tasks. While Sentence-BERT can be trained on data from different domains to cater to various tasks, there isn't a specific encoder within the library that is specifically trained for encoding Java code. However, the generic pre-trained all-MiniLM-L6-v2 encoder is available. This encoder is trained on a diverse, extensive dataset of training pairs to support multiple domains  [13]. Table  1  contains details about the all-MiniLM-L6-v2 model.
GUI front-end and GUI back-end module To commence the development of the MASSPR system, we prioritize the construction of its core, which encompasses all essential modules and their respective functions. The Graphical User Interface (GUI) Front-End module serves as the platform for user interaction, and we leverage the Java programming language along with JavaFX libraries to create a user-friendly GUI that caters to the specific needs of our users. Our tool is designed for desktop software usage, providing a seamless experience. The MASSPR interface offers a range of notable features, a few of which are highlighted below: • Time-saving capabilities: Users can streamline their search tasks by utilizing customizable web search agents tailored to their specific requirements. This feature ensures efficient and targeted searches, saving valuable time.
• Query Module: The Query Module facilitates uploading a collection papers and generates precise search queries based on the uploaded content. This allows users to obtain relevant results quickly and accurately. • Easy access to recommended papers: Users can conveniently browse through recommended scientific papers and export them to their local machine repository. This feature enables easy access to important research materials. The GUI Back-End Engine module in MASSPR is the central hub for various internal modules, such as the query and recommendation modules. It plays multiple roles, including executing visualization functions essential for the proper functioning the GUI Front-End module. Additionally, it sends the scientific articles uploaded by the user to the Query Module for query modeling.
Searching module The Searching module in our implementation is developed using Java programs and utilizes the Jsoup Java library. Jsoup provides a convenient API for our agents to browse the Internet, offering features for fetching URLs, extracting data, and manipulating HTML using HTML5 DOM methods and CSS selectors. Furthermore, we have an article collection agent dedicated to aggregating the downloaded articles. Table  2  presents information about 10 academic search engines that we crawled, namely ACMDL1, Base2, CiteSeerX3, MDPI4, Nature5, SciELO6, TaylorFrancais7, ResearchGate8, PubMed Central9, and Eric10. Each agent is assigned to crawl a specific site within the academic search engine and extract article information based on the queries received from the query buffer. Table  3  displays the data obtained through web crawling, including title, type, authors, journal, publication date, access link, citation information, abstract, keywords, and references.
Input model preparation For the input of our classification model, we employed the NLTK (Natural Language Toolkit) Python library to preprocess the article context. Initially, we converted the articles' context to lowercase to ensure consistency in the text representation. Then, we removed numbers, punctuation marks, and whitespace from the article context. This helped eliminate non-alphabetic characters that may not contribute to the classification task. Following that, we utilized the default set of stop words provided by the NLTK library to remove commonly occurring words in the English language that typically do not carry significant meaning for our classification task. Stop words such as "the," "is," and "and" were eliminated to reduce noise in the text data. Lastly, we performed stemming, which transforms words with similar semantics into a standard form.
Once the preprocessing step is complete, we encode the context of the preprocessed articles. We utilize the all-MiniLM-L6-v2 transformer model from the sentencetransformers Python library to accomplish this. This allows us to obtain embedding vectors that are prepared and suitable for use as input in the learning classification model. The provided dataset in Table  4  is a sample dataset used in the learning model. This dataset was acquired from the search module and included the article title and abstract as the context. It is important to note that this dataset is in its raw form, before undergoing preparation steps such as preprocessing and embedding. On the other hand, the subsequent table, labeled Table  5 , showcases the same dataset after preparation, including preprocessing and embedding.
Output model preparation To apply the OneVsRestClassifier for multilabel classification tasks, it is essential to format the output labels (in this case, citations) as a 2D binary (0/1) matrix. This process, known as binarization, can be accomplished using techniques like one-hot encoding and the make_column_transformer transformers in Python. These methods enable the conversion of the label matrix into a binary representation suitable for subsequent machine learning operations. Table  6  presents the citations' IDs before binarization, while Table  7  showcases the citations' IDs after the binarization process. Please note that the provided information is a revised version of the original text. It is important to proofread and ensure the accuracy of the content before finalizing it. One-Hot Encoding (OHE) is a widely used in data mining tasks to convert categorical features into numerical representations.
Analysis of the results Multi-label classification performance metrics: In the case of multi-label classification problems, predictions for each instance consist of multiple labels. The effectiveness of classifiers in such scenarios can be evaluated by calculating the average score of an evaluation metric or by directly comparing the scores for each class. Our research employed two commonly used performance metrics: hamming loss and Jaccard similarity. These metrics provide a robust means to assess the performance of classifiers. Table  10  presents the results of applying three classification models (SGD Classifier, Logistic Regression, and SVM) to the dataset. Specifically, it illustrates the Jaccard scores achieved by these models. The classifiers were trained and assessed using a five-fold cross-validation approach to ensure reliable evaluation.
CONCLUSIONS This project aims to develop a tool called MASSPR, designed to assist researchers in handling time-consuming daily tasks associated with finding relevant papers for their studies. Instead of manually browsing the web, our tool automates the process by utilizing search engines to retrieve scientific articles. It can search the web, download papers, and apply intelligent mechanisms to filter and recommend the most relevant documents. One key feature of the MASSPR tool is its ability to address the Cold-start issue for new users. By leveraging an innovative approach, our system generates search queries based on a user's given paper or a simple search sentence, assisting new researchers in finding the most relevant documents for their specific needs. The MASSPR architecture consists of six main modules: the Graphic User Interface (GUI) Front-end modules, GUI Back-end Engine, Query Modelling Module, Searching Module, Filtering Module, and Database Module.
The implementation utilizes the Java programming language along with various APIs and libraries. The tool is cross-platform and compatible with Windows, Linux, and macOS. The initial version of our tool has received positive feedback from many users, and incorporating recommendation algorithms has significantly improved the quality of the final recommendations. These findings have substantial implications for researchers seeking to save time by automatically collecting and storing scientific papers on their personal computers. Additionally, the MASSPR tool proves valuable in real-time data set collection of articles. However, it's important to note that the current version has limitations, such as not fully utilizing the advantages of collaborative-based filtering and disregarding contextual information that influences user preferences as their needs change.
Description All-Round Model Tuned for Many Use-Cases Trained on a Large and Diverse Dataset of Over 1 Billion Training Pairs Base Model nreimers/MiniLM-1. 6-H384-uncased Max Sequence Length 256 Dimensions 384 Normalized Embeddings true Suitable Score Functions Dot-product (util dot_score), Cosine-similarity (util. cos sim), Euclidean distance Size NO MB Peoling Mean Pooling Training Data 18+ training pairs for details see model card Model Card https://huggingface. co/sentence-transformers/all-MiniLM-L6-v2.
Driven by the development and upscaling of fast genome sequencing and assembly pipelines, the number of protein-coding sequences deposited in public protein sequence databases is increasing exponentially. Recently, the dramatic success of deep learning-based approaches applied to protein structure prediction has done the same for protein structures. We are now entering a new era in protein sequence and structure annotation, with hundreds of millions of predicted protein structures made available through the AlphaFold database. These models cover most of the catalogued natural proteins, including those difficult to annotate for function or putative biological role based on standard, homologybased approaches. In this work, we quantified how much of such "dark matter" of the natural protein universe was structurally illuminated by AlphaFold2 and modelled this diversity as an interactive sequence similarity network that can be navigated at https://uniprot3d.
Manual curation of such functional annotation is carried out by those assembling the genomes and by biocurators  (Boutet et al. , 2016)  most of which is incorporated into automated annotation pipelines such as InterPro  (Paysan-Lafosse et al. , 2023). However, despite the great success of such approaches, only 83% of UniProt sequences are covered by InterPro, and many of these correspond to Domain of Unknown Function (DUF) domains, indicating that there is a large number of protein sequences that remain functionally unannotated and unclassified. In 2009, it was predicted that circa 20% of the proteins deposited in sequence databases represented such "dark matter" of the protein universe  (Levitt, 2009). Some of these proteins may just correspond to highly divergent forms of known protein families and thus lie beyond the detection horizon of the automated, homology-based methods employed; others could belong to so far undescribed protein families with yet-to-be determined molecular or biological functions.
The 3D structure of a protein is an important source of information about its molecular mechanism or biological role. Obtaining such data experimentally is, however, an expensive and timeconsuming process. As of March 2023, only about 200'000 experimental 3D structures of proteins and protein complexes are known and made available through the Protein Data Bank (PDB)  (Bittrich et al. , 2023) , which contrasts with the more than 320 million unique protein sequences in UniRef100  (Suzek et al. , 2015). Computational protein structure prediction is a way of bridging this gap, but with the current wealth of experimentally determined protein structures, homology-based approaches can expand structural coverage only up to a certain point  (Bienert et al. , 2017). On one hand, there are not enough experimentally determined templates that cover the full wealth of sequenced proteins and, from the other, even when a remote homolog of known structure is available, the lower the sequence similarity the higher the uncertainty in the predicted model.
Deep-learning-based approaches have recently circumvented this limitation by achieving extremely high levels of accuracy for proteins without any known homologs of known structure. The main player here was AlphaFold2, a deep neural network that performs end-to-end protein structure prediction based on evolutionary, physical and geometric information  (Jumper et al. , 2021). Its success drove its developers at DeepMind, in collaboration with the EMBL-EBI, to produce predicted structural models for all natural proteins catalogued in the UniProt Knowledgebase (UniProtKB) and set up the AlphaFold database (AFDB)  (Varadi et al. , 2022). AFDB version 4 (AFDBv4) includes structural models for about 215 million protein sequences and excludes those from viruses and those with more than 1'300 residues (2'700 for selected proteomes).
This means that it provides models not only for wellstudied proteins whose 3D structure is unknown, but also for all of those that make the "dark matter" of protein sequence databases. In this work, we combine sequence, structure and genomic context similarities with deep learning-based function prediction tools to shed light into these proteins and the reasons why some of them remain unannotated based solely on sequence similarity and homology-based approaches. For that, we first revised the proportion of such sequences in UniProtKB and constructed, for the first time, a sequence similarity network of all catalogued proteins with high quality predicted structural models. We then combined our results with protein structure information as predicted by AlphaFold2 and evaluated how much structural novelty is encompassed in these proteins, using an evaluation of substructure compositions based on natural language processing.
Functional darkness in the UniProt and AlphaFold databases As of August 2022, there were more than 520 million unique protein sequences in UniProt (i. e. , UniRef100 clusters). This includes more than 500'000 manually curated proteins in Swiss-Prot, more than 220 million sequences in TrEMBL and more than 300 million sequences in UniParc from other sources not yet included in UniProtKB. The total number of sequences drops drastically to circa 50 million unique sequences if one filters the UniRef100 dataset to a maximum sequence identity of 50% (UniRef50). We define the "functional brightness" of a given protein as the full-length coverage with annotations of its close homologs, with 0% meaning "dark" and 100% meaning "bright" and a UniRef50 cluster being as "bright" as the "brightest" sequence it encompasses (Fig.
Large-scale sequence similarity network of AFDB90 For the remainder of the work, we focused only on those dark UniRef50 clusters with high average pLDDT representatives, as these represent protein clusters that can be more confidently studied at a structural level. While UniRef50 clustering provides us with groups of sequences that are overall identical at the sequence level, such clustering does not reach the family and superfamily levels and does not account for local similarities. Thus, to search for those UniRef50 clusters that belong to the same families and superfamilies and identify those that are locally identical, we constructed a largescale sequence similarity network of all Uniref50 clusters whose structural representative has an average pLDDT > 90 (the AFDB90 dataset). This corresponds to 6'136'321 UniRef50 clusters and, thus, to only 11% of the initial set of UniRef50 clusters.