dataesr/scientific-paragraphs-categorization huggingface dataset:
With such conditions, we can observe a gradual shifting in feature importance from the ones informative to classify cell lines to the ones informative to classify patients and cell lines (Fig. 3d  and Additional file 1: Figs. S3d-4d ). The algorithm stops when it can no longer incorporate the patients into one or the other group given the cutoff of P = 0. 6. ML analyse was done with Python 3. 7. 3 based on scikit-learn version 0. 21. 2. To select the more efficient features that were able to separate B-like from A-like patients, we used Boruta package (0. 3) implemented in python. We ran it 10 times with different random states, on the 217 features related to splicing and kept the ones that were present at least 7 times on 10. We ended with 25 AS features. Considering only these 25 AS features, we applied TSNE function from manifold package (with perplexity = 20) to 3 other datasets of basal cell lines (n = 56) to check the features were sufficient to distinguish spatially these cell lines according to their labels.
Breast cancer annotation Basal B and A cells were labelled according to literature: Neve et al. [28] , Kao et al. [33] , Marcotte et al. [107] , Dai et al. [108]. PAM50 intrinsic subtype was retrieved from https://www. cell. com/cancer-cell/fulltext/S1535-6108(1 8)30119-3  [109]. Claudin Low status was defined with script downloaded from https://github. com/clfougner/ClaudinLow/ blob/master/Code/TCGA. r [110]  using dataset from http://download. cbioportal. org/brca_tcga_pan_can_ atlas_2018. tar. gz [111, 112].
Survival analysis Log-rank tests were performed using the functions surv and survfit from R package (survival v3. 1. 8). A different survival was considered significative if log rank test p value was < 0. 05. Coxph function was also used for univariate Cox regression analysis in order to compute Hazard Ratio and 95% Interval of confidence. Kaplan-Meier curve was plotted using function ggsurvplot from R package survminer (0. 4. 6). Plots were truncated at 5 years, but the analyses were conducted using all of the data. All endpoints used for survival analysis in this study were retrieved from this study  [113].
Availability of data and materials All datasets are available in the Gene Expression Omnibus (GEO): GSE75489, GSE58381, GSE75491, GSE61220, PRJEB25042, GSE74881, GSE75492, PRJNA523380, PRJNA297219, PRJNA210428, PRJNA251383, PRJEB30617 (detailed in Additional file 2: Table  S1 ) and The Cancer Genome Atlas (TCGA) repositories upon request (https://www. ncbi. nlm. nih. gov/projects/gap/cgibin/study. cgi?study_id=phs000178. v11. p8).
Introduction Most of the cellular machinery and biological processes are fulfilled by biomolecular associations, particularly proteinprotein interactions (PPI). The versatile, adaptable, and specific nature of proteins gives them the ability to create networks that govern virtually all intra-and inter-cellular activities. Analyzing these networks enlightens the relative importance of proteins in different organisms and communities. It improves our understanding of physiopathological mechanisms and helps us to decipher gene-disease-drug associations and find therapeutic treatments  (25, 33, 30, 13, 37). The ever increasing growth of protein sequential, structural, and functional information  (39, 38, 3, 10)  and of experimental evidence for PPI  (27, 26, 29, 28, 14)  has stimulated the development of databases and web-interfaces for curating, inferring, and browsing PPI networks.
A PPI is not a simple edge between two nodes but can be viewed as a multi-edge, where the multiplicity comes from the different sources of evidence, associated with a set of weights reflecting either a property from the source or the reliability of the evidence. Graphs formed upon diverse sources may explain the existence of multiple interaction sites on the surface of a protein  (18, 8). Additionally, it is desirable that such graphs encode information about non-interacting pairs. While most of the databases describe interactions without accounting for any underlying physical information  (5, 16, 9, 12, 20) , a handful of them build upon protein 3D structural evidence, e. g. Interactome3D  (23) , PPI3D  (6) , and 3did  (24). Structural determination techniques provide experimental biologists with atomic-level resolution about the formation of protein complexes.
Types of (non-)interactions and inference algorithm Relations between protein chains are represented by six types of edges (Figure  1B ): • type 1: observed interaction, i. e. the two chains show a physical contact in a known complex • type 2: interaction inferred by homology where one of the chains is in physical contact with a homolog of the other chain • type 3: interaction inferred by homology where some homologs of the two chains are in physical contact • type 4: HIPPIE annotated interaction • type 5: NEGATOME annotated non-interaction • type 6: User-defined interaction The absence of an edge indicates missing supporting evidence for an interaction or a non-interaction. LEVELNET infers the three first types of edges directly from the PDB ("PDB" source in Figure  1B ). To achieve computational and memory efficiency, we pre-computed two databases, one of physical interfaces, and one of physical and homology-transferred interactions: Database of physical interfaces.
We first downloaded all PDB entries from the June 2020 release, and identified the residues at the interface between every pair of protein chains in the same biological assembly (from X-ray crystallography and cryogenic electron microscopy) or NMR model using INTBuilder  (7). PDB entries with more than 100 chains or with a resolution lower than 5Å were discarded. Protein chains smaller than 20 residues or with more than 20% of unknown residues were also discarded. Two residues were considered as in contact if the distance between any of their atoms was less than 5Å. Database of physical and homology-transferred interactions. We generated an optimized database of interactions containing all necessary information for inferring type 1, type 2, and type 3 edges. We determined the set of observed physical contacts (type 1: chain-chain interactions) by looping over all possible pairs of chains in PDB complexes and verifying the existence of a contact in our previously computed database of physical interfaces.
Processing an input query Starting from the input query, LEVELNET will interrogate the pre-computed database of physical and homologytransferred interactions to create type 1, 2 and 3 edges. For each chain pair in the query, if there is a chain-chain interaction, LEVELNET creates a type 1 edge. If not, then if there is a chain-cluster interaction it creates a type 2 edge, otherwise if there is a cluster-cluster interaction, it creates a type 3 edge. LEVELNET will also interrogate HIPPIE and Negatome to infer type 4 and 5 edges. Finally, if the user specified some relationships between the input chains, LEVELNET will integrate them in the output multi-layered network as type 6 edges.
LEVELNET implementation details To generate the PDB interaction database, we developed a pipeline in Python that processes PDB protein complexes, their interfaces, and chain clusters by sequence identity. We created an interactive environment based on recent advances in web development including HTML5 and related technologies such as D3, JavaScript, Vue, and SVG. HTML5, CSS3 and Vue are used for the front-end and provide a stylish and user-friendly interactive interface. Data visualization is performed using D3 (4) and SVG. Codes developed in JavaScript and Python process databases and user queries. The whole pipeline is optimized to respond to queries rapidly.
Accounting for protein co-localization Next, we focused on the canonical Wnt signaling pathway, which regulates gene transcription by passing signals from the cell surface receptors to the nucleus. A large number of X-ray, NMR and cryoEM structures characterizing this pathway are available and were recently reviewed in  (1). The authors provide a list of 85 PDB codes organized according to their cellular localization and role: secreted positive or negative regulators, transmembrane proteins, cytoplasmic proteins and intranuclear proteins. Starting from the list and the co-localization annotations, LEVELNET built a network comprising 235 nodes linked by 2734 edges inferred from the PDB and by homology transfer above 30% sequence identity. By spatially arranging the nodes according to their co-localization, LEVELNET gives an overview of the full pathway, where proteins sharing the same location and role are grouped together and where interactions between groups are clearly distinguishable (Figure  3 ).
Upon merging the nodes sharing more than 30% sequence identity, the (E) For the RuBiSCo complex, we also show its 3D structure (PDB code: 1GK8), with the small and large subunits in red and blue respectively. (F) Subnetwork comparison of two layers, corresponding to observed interaction in the PDB (blue) and non-interactions from Negatome (in black) for the cytochrome b6f enzyme (1Q90). (G) Two views of the structure 1Q90. size of the network reduces dramatically down to less than 50 nodes. This non-redundant version allows reasoning at the protein level, toward identifying the parts of the pathway that are described by structural information and the parts where such information is missing (Figure  3 , compare the inset with Figure  1  from (1)). Several complexes formed in the membrane are clearly identifiable, for instance between Dickkopfs (DKKs), Kremen (KREM1) and LRP6, between R-spondins (RSPOs) and LGR family receptors, and between Wnt proteins (WNTs) and frizzled receptors (FZDs) (Figure  3 , grey panel in inset, from left to right).
Conclusion Fast and accurate, LEVELNET is a valuable asset for the community to explore protein interactions. It is useful for the biologists interested in the physical contacts of a particular protein or a set of proteins as well as for those who develop and assess computational predictive approaches for interface, partner and complex predictions. It provides a convenient mean to account for different types of relationships between proteins, e. g. functional annotations, cell co-localization, spatiotemporal proteomics, or co-occurrences in publications, and investigate how the latter are correlated to physical interactions. When using this environment, the user must take into account the functional class of the partners when transferring interactions. Future developments will concern the integration of other experimental databases, node selection and merging based on annotations from Uniprot, SCOP or CATH, support of multi-chain proteins and protein-nucleic acid interactions.
SUPPLEMENTARY DATA Discovery of self-and cross-interactions. The variation of the topology of the networks with respect to the interactions between each layer and corresponding benchmark network is shown in Supplementary Figure  S2. Self-and crossinteractions are considered separately and the difference is calculated as the percentage of the maximum possible connections: n and n 2 (fully connected graph), respectively. The only difference between physical contact and benchmark networks is the self-interactions. These self-interactions are caused by the identical copies of the same chain found in a biological assembly that are in contact. For both benchmarks, more than half of the nodes can interact with themselves at sequence identity above 95%. In case of cross-interactions, ZDockv5 with AA and AS included shows higher differences for both unbound and bound annotations compared to other benchmarks.
Modern random access schemes Classical four-way handshake based scheduling approaches are neither scalable nor efficient for mMTC services where a massive number of transmitters send data intermittently and possibly with unpredictable traffic pattern. GF solutions based on the preemptive assignment of resources as proposed in 5G, also may become inefficient or entail unwanted latency. Uncoordinated random access solutions, where devices transmit data directly without first acquiring a transmission grant, appear as natural alternatives. In recent years, modern random access schemes  [58]  have been proposed focusing on simple repetition of the transmitted messages and the use of IC receivers  [59, 60]. More recently advanced multi-user code constructions and MUD algorithms have been investigated  [61] [62] [63]  under the common umbrella of unsourced random access.
Further investigations on some of the practical implications is ongoing to render these very schemes as fundamental enablers for 6G MTC scenarios. Open problems in the research community include, detecting user activity, maintaining user time-synchronization, and keeping receiver complexity under control. Finally, the tight connection between PHY and the medium access policy can only be harnessed by a dedicated error correcting code design. 5G NR employs low-density parity-check (LDPC) codes and polar codes as error correcting codes for eMBB services and the transmission of control information, respectively. However, none of these schemes are directly suitable for mMTC services, and require to be substantially adapted. The main challenges in this case are: (1) constructing efficient codes for short packet sizes ( ∼ 100 -1000 bits); (2) achieving significant error detection capability at low overheads  [64] ; and (3) designing decoding algorithms that can operate with limited/outdated or no CSI  [65].
Hence, moving towards 6G, there is a need to introduce codes specifically tailored for mMTC services that addresses the emerging challenges. An instance of the achievable gains of a dedicated error correcting code is reported in Fig. 4 , focusing on the slot-asynchronous  4  GF access  [67]  case. The plot exemplifies how the standardized 5G codes, as well as an enhanced version thereof relying on an optimal interleaving to protect packet portions more prone to interference (the 5G perm. curve), can be significantly outperformed by a tailored design. In this case, improvements are brought by a LDPC code (the LDPC Design curve) specifically devised to cope with the interference structure induced by the medium access control strategy (more details can be found in  [67, 68] ).
4  Slot-asynchronous random access schemes performance. Slot-asynchronous random access schemes adopting time diversity, i. e. copies of the same physical layer packet are transmitted with random delays, and interference cancellation. It is shown that a dedicated low-density parity check (LDPC) code design outperforms standardized 5G error correcting codes. Details on the modern random access scheme and the code design can be found in  [67, 68]  respectively be envisioned for mMTC towards 6G. Greater scalability can be achieved by exploiting the differences in the traffic and QoS characteristics across the different slices to jointly optimize the allocation of resources and meet the diverse requirements  [70]. Such cross layer optimization also needs to consider the coexistence among multiple numerologies as well as orthogonal/non-orthogonal waveforms, and can be efficiently addressed using AI-inspired solutions.
We present PyTorch Geometric Temporal a deep learning framework combining state-of-the-art machine learning algorithms for neural spatiotemporal signal processing. The main goal of the library is to make temporal geometric deep learning available for researchers and machine learning practitioners in a unified easyto-use framework. PyTorch Geometric Temporal was created with foundations on existing libraries in the PyTorch eco-system, streamlined neural network layer definitions, temporal snapshot generators for batching, and integrated benchmark datasets. These features are illustrated with a tutorial-like case study. Experiments demonstrate the predictive performance of the models implemented in the library on real world problems such as epidemiological forecasting, ride-hail demand prediction and web-traffic management.
Besides limiting assumptions about the input data, these off-the-shelf libraries are not designed to operate on spatiotemporal data. Present work. We propose PyTorch Geometric Temporal, an open-source Python library for spatiotemporal machine learning. We designed PyTorch Geometric Temporal with a simple and consistent API inspired by the software architecture of existing widely used geometric deep learning libraries from the PyTorch ecosystem  [15, 40]. Our framework was built by applying simple design principles consistently. The framework reuses existing neural network layers in a modular manner, models have a limited number of public methods and hyperparameters can be inspected. Spatiotemporal signal iterators ingest data memory efficiently in widely used scientific computing formats and return those in a PyTorch compatible format.
The design principles in combination with the test coverage, documentation, practical tutorials, continuous integration, package indexing and frequent releases make the framework an end-user friendly spatiotemporal machine learning system. The experimental evaluation of the framework entails node level regression tasks on datasets released exclusively with the framework. Specifically, we compare the predictive performance of spatiotemporal graph neural networks on epidemiological forecasting, demand planning, web traffic management and social media interaction prediction tasks. Synthetic experiments show that with the right batching strategy PyTorch Geometric Temporal is highly scalable and benefits from GPU accelerated computing. Our contributions. The main contributions of our work can be summarized as: • We publicly release PyTorch Geometric Temporal the first deep learning library for parametric spatiotemporal machine learning models.
• We provide data loaders and iterators with PyTorch Geometric Temporal which can handle spatiotemporal datasets. • We release new spatiotemporal benchmark datasets from the renewable energy production, epidemiological reporting, goods delivery and web traffic forecasting domains. • We evaluate the spatiotemporal forecasting capabilities of the neural and parametric machine learning models available in PyTorch Geometric Temporal on real world datasets. The remainder of the paper has the following structure. In Section 2 we overview important preliminaries and the related work about temporal and geometric deep learning and the characteristics of related open-source machine learning software. The main design principles of PyTorch Geometric Temporal are discussed in Section 3 with a practical example.
We demonstrate the forecasting capabilities of the framework in Section 4 where we also evaluate the scalability of the library on various commodity hardware. We conclude in Section 5 where we summarize the results. The source code of PyTorch Geometric Temporal is publicly available at https://github. com/benedekrozemberczki/pytorch_geometric_ temporal; the Python package can be installed via the Python Package Index. Detailed documentation is accessible at https://pytorchgeometric-temporal. readthedocs. io/.
Graph Representation Learning Software The current graph representation learning software ecosystem which allows academic research and industrial deployment extends open-source auto-differentiation libraries such as TensorFlow  [1] , PyTorch  [41] , MxNet  [11]  and JAX  [16, 28]. Our work does the same as we build on the PyTorch Geometric ecosystem. We summarized the characteristics of these libraries in Table  2  which allows for comparing frameworks based on the backend, presence of supervised training functionalities, presence of temporal models and GPU support. Our proposed framework is the only one to date which allows  Library Backend Supervised Temporal GPU PT Geometric  [15]  PT ✔ ✘ ✔ Geometric2DR [49] PT ✘ ✘ ✔ CogDL [9] PT ✔ ✘ ✔ Spektral [21] TF ✔ ✘ ✔ TF Geometric [27] TF ✔ ✘ ✔ StellarGraph [13] TF ✔ ✘ ✔ DGL [67] TF/PT/MX ✔ ✘ ✔ DIG [34] PT ✔ ✘ ✔ Jraph [18] JAX ✔ ✘ ✔ Graph-Learn [62] Custom ✔ ✘ ✔ GEM [19] TF ✘ ✘ ✔ DynamicGEM [20] TF ✘ ✔ ✔ OpenNE [57] Custom ✘ ✘ ✘ Karate Club [46] Custom ✘ ✘ ✘ Our Work PT ✔ ✔ ✔.
Spatiotemporal Data Analytics Software The open-source ecosystem for spatiotemporal data processing consists of specialized database systems, basic analytical tools and advanced machine learning libraries. We summarized the characteristics of the most popular libraries in Table  3  with respect to the year of release, purpose of the framework, source code language and GPU support. First, it is evident that most spatiotemporal data processing tools are fairly new and there is much space for contributions in each subcategory. Second, the database systems are written in highperformance languages while the analytics and machine learning oriented tools have a pure Python/R design or a wrapper written in these languages. Finally, the use of GPU acceleration is not widespread which alludes to the fact that current spatiotemporal data processing tools might have a scalability issue.
Library Year Purpose Language GPU GeoWave  [60]  2016 Database Java ✘ StacSpec  [23]  2017 Database Javascript ✘ MobilityDB  [69]  2019 Database C ✘ PyStac  [44]  2020 Database Python ✘ StaRs  [42]  2017 Analytics R ✘ CuSpatial  [56]  2019 Analytics Python ✔ PySAL  [43]  2017 Machine Learning Python ✘ STDMTMB  [2]  2018 Machine Learning R ✘ Our work 2021 Machine Learning Python ✔.
Non-proliferation of classes. The framework reuses the existing high level neural network layer classes as building blocks from the PyTorch and PyTorch Geometric ecosystems. The goal of the library is not to replace the existing frameworks. This design strategy makes sure that the number of auxiliary classes in the framework is kept low and that the framework interfaces well with the rest of the ecosystem. 3. 1. 2 Hyperparameter inspection and type hinting. The neural network layers do not have default hyperparameter settings as some of these have to be set in a dataset dependent manner. In order to help with this, the layer hyperparameters are stored as public class attributes and they are available for inspection. Moreover, the constructors of the neural network layers use type hinting which helps the end-users to set the hyperparameters.
Data Structures The design of PyTorch Geometric Temporal required the introduction of custom data structures which can efficiently store the datasets and provide temporally ordered snapshots for batching. [58]  in memory, but returned as a PyTorch Geometric Data object instance  [15]  by the Spatiotemporal Signal Iterators when these are iterated on. This design choice hedges against the proliferation of classes and exploits the existing and widely used compact data structures from the PyTorch ecosystem  [40].
Train-Test Splitting. As part of the library we provide a temporal train-test splitting function which creates train and test snapshot iterators from a Spatiotemporal Signal Iterator given a test dataset ratio. This parameter of the splitting function decides the fraction of data that is separated from the end of the spatiotemporal graph snapshot sequence for testing. The returned iterators have the same type as the input iterator. Importantly, this splitting does not influence the applicability of widely used semi-supervised model training strategies such as node masking.
Design in Practice Case Study: Cumulative Model Training on CPU In the following we overview a simple end-to-end machine learning pipeline designed with PyTorch Geometric Temporal. These code snippets solve a practical epidemiological forecasting problempredicting the weekly number of chickenpox cases in Hungary  [47]. The pipeline consists of data preparation, model definition, training and evaluation phases. [32]  and a fully connected layer with a single neuron (lines 8-9). In the forward pass method of the neural network the model uses the vertex features, edges and the optional edge weights (line 11). The initial recurrent graph convolution based aggregation (line 12) is followed by a rectified linear unit activation function  [37]  and dropout  [53]  for regularization (lines  [13] [14].