dataesr/scientific-paragraphs-categorization huggingface dataset:
During the training, the weights of the PLM are affected along with W a. We developed these experiments with CamemBertForSequenceClassification of the transformer package. To make the probability outputs of this linear regression layer comparable with the label outputs from the GAN-BERT classifier, we fixed the threshold of 0. 5. For the predicted probability greater than 0. 5, we considered it as an observation, else as a non-observation. It is worth mentioning that we used 0. 5 as a threshold to simplify  the comparison with the PLM plus GAN-BERT classification. When applying the PLM-only classification to other datasets in other domains, we might need to find an optimal threshold depending on the real needs for precision or recall. Based on the results presented in the study of  Jiang et al.
Experimental setup For our experiments, we used GAN-BERT's latest PyTorch implementation, which is compatible with the transformer package. We fixed the max sequence length of the PLM to 128. We fixed the number of hidden layers in G and in D to 1 and the size of G's input noisy vectors to 100. We used the following learning rate combinations (D, G): (5e-5, 5e-5), (1e-5, 1e-5), and (5e-6, 1e-6). We applied the AdamW  (Loshchilov and Hutter, 2017)  optimizer with and without a cosine scheduler. To limit the number of variables, we https://github. com/crux /ganbert-pytorch conduct two groups of experiments. In the first group, we fixed the batch size per GPU to 32 and epochs 30. We trained the GAN-BERT architectures over increasing labeled data sizes  (16, 32, 64, 128, 256, and 512)  and unlabeled data sizes  (1,024, 4,096, and 8,192).
By comparing the maximum F1 scores of each configuration during the training in Figures  2 3 4 , we believe the performance of the classifiers on both the validation and test sets to be continuous and relatively stable in a period, once the training converges. In other words, overfitting will not immediately cause huge drops. It is worth pointing out that, on the unbalanced validation and test sets, the F1 score (the objective of our classification task) and the binary cross entropy loss (the objective of GAN-BERT's training) are not completely aligned and may lead to suboptimal convergence. Compared to PLM-only classification, the PLM plus GAN-BERT setting improves the scores over the validation and test sets of unseen hazards with 32, 64, 128, and 256 training data. In Figure  3 , we depict the performance with varying unlabeled data sizes.
The instability of the GAN-BERT setting with ChouBERT models Even though the fine-tuning of pre-trained transformer-based language models such as BERT has achieved state-of-the-art results on NLP tasks, fine-tuning is still an unstable process. Training the same model with multiple random seeds can result in different performances on a task as described in the study of  Mosbach et al. (2021). This training instability is the reason why we have not found the best labeled/unlabeled ratio to maximize the performance. In Figure  5 , we illustrate the training losses of the discriminator and the generator when given different sizes of labeled data with fixed unlabeled data size and learning rate. The training with ChouBERT models has more difficulties to converge than with CamemBERT. Thus, we explored the evolution of different losses and the classifiers' performance metrics on the validation and test sets in Figures  6, 7 , where the discriminators' losses with ChouBERT-16 take more epochs to decrease than with CamemBERT.
It is immediately clear that discriminators' losses had the same shape as L sup. In particular, to present the evolution of L G feat at the same scale as the other losses, we multiplied its value by 10 to draw its line. Compared with ChouBERT-16, the L sup had more difficulties decreasing than with CamemBERT. We interpret the increase of L G feat as the generator catching up with the fine-tuning of PLM and the decrease of L G feat toward its initial value as the end of the major changes of fine-tuning. According to the authors of SSGAN  (Salimans et al. , 2016) , "in practice, L unsup will only help if it is not trivial to minimize for our classifier and we thus need to train G to approximate the data distribution, " which explains that, while the L unsup of D and G converge at the same rhythm with CamemBERT and with ChouBERT-16, the troubled decrease of L sup with ChouBERT-16 renders worse F1 scores than those with CamemBERT.
Interestingly, when repeating the experiments with the same hyperparameters, the "troubled decrease" of L sup does not always happen, but statistically, most of them can happen with ChouBERT models, especially ChouBERT-16. Our strategies against the "troubled decrease" include the following: • Using a smaller learning rate with more training epochs at the cost of computational resources (see  Ta et al. , 2022 ). • Applying a smaller learning rate to G than to D (see  Heusel et al. , 2017 ). • Applying schedulers and down-sampling the majority class to balance the training data-in our case, the upsampling proposed by the original code of GAN-BERT does not help. With the optimizations mentioned above, L sup with CamemBERT decreases at a steady pace and "troubled decrease" happens less often with ChouBERT models.
Conclusion In this article, we demonstrate that combining further-pretrained language models and GAN-BERT benefits from the generalizability of the domain-specific PLM to classify unseen hazards. We also demonstrate that training such a combination may also suffer from extra instabilities compared to using GAN-BERT with CamemBERT, a general PLM. Our results validate that the GAN-BERT setting improves the task of natural hazard classification for datasets containing between 32 and 256 instances of labeled data. Based on our experimental studies, we give our suggestions to reduce the instability such as: (1) The L sup needs a certain minimum number of steps to decrease to zero. For a fixed batch size, adding unlabeled data makes more training steps to go through in each epoch, consequently allowing L sup to decrease at a similar pace as L unsup.
When the number of unlabeled data is limited, using smaller batch sizes and training for more epochs is also a good approach. (2) If the task is not too domain-specific, in other words, when the further pre-trained language model cannot significantly outperform the general language model in the PLM-only classification, using a general language model with the GAN-BERT setting is safer. On the other hand, if the task is highly domain-specific, it is better to apply schedulers, downsample the majority class to balance the training data, and use smaller learning rates to train GAN-BERT with furtherpre-trained language models. (3) We need to choose a suitable PLM. We proved that ChouBERT-32 outperforms ChouBERT-16 in an SSGAN setting. The perspectives and developments are numerous to increase the stability of domain-specific text classification using GAN-BERT; for example, how to further pre-train PLMs to adapt better SSGAN setting is yet to investigate.
Our synthetic index is based on the main indicators about incoming flows from abroad (FDIs, workers, tourists, university students). Secondly, we illustrate and discuss the results of its application to the case of the international attractiveness of the Italian regions, based on an 8-year long dataset. The findings reveal that the North-South gap, the wellknown Italian spatial pattern, is not fully satisfying to read and interpreter the geographical imbalances in terms of multidimensional territorial attractiveness. Thanks also to the calculation of the Moran's Index, we show the complexity of the Italian economic geography.
The experts were selected amongst academics and representatives of the Italian institutions dealing with research on regional economic development, thus being able to assess the contribution of the various types of flows to regional economic I x = x -min (x) max (x) -min (x) 1 3 A new territorial attractiveness index at the international… development. They were mainly, but not exclusively, economists 16 ; in any case, they had a broad multidisciplinary background, focused not just on one of the four specific topics associated with the four types of flows taken into consideration (tourism, FDI, immigration, university education). In this respect, our goal was indeed to avoid having a group of experts who were too much biased in favour of one of these topics. Since a database on such profiles is not available in Italy, experts, according to a non-probabilistic sampling principle, were identified by selecting them amongst the members of some of the major national research institutions working on the matter under consideration.
As an alternative aggregation method, as a more complex than weighted average, we also created the composite index using exploratory factor analysis by the method of principal component analysis (PCA). Despite the difficulties of temporal comparability of such an index, it fulfils the necessary conditions of applications  (Mazziotta and Pareto 2019)  and it is well-adapted to our approach, and it applies a complex aggregation instead of simple one  (Mazziotta and Pareto 2017). The four basic indicators are positively correlated, and they can be used without normalisation related to negative polarity. We tested the dataset by the usual precondition methods: Kaiser-Meyer-Olkin (KMO) and Bartlett's Test of Sphericity, depending on the period, KMO values were over 0. 65 and the p value of the Bartlett's Test < 0.
Background: The growing availability of large genomic datasets presents an opportunity to discover novel metabolic pathways and enzymatic reactions profitable for industrial or synthetic biological applications. Efforts to identify new enzyme functions in this substantial number of sequences cannot be achieved without the help of bioinformatics tools and the development of new strategies. The classical way to assign a function to a gene uses sequence similarity. However, another way is to mine databases to identify conserved gene clusters (i. e. syntenies) as, in prokaryotic genomes, genes involved in the same pathway are frequently encoded in a single locus with an operonic organisation. This Genomic Context (GC) conservation is considered as a reliable indicator of functional relationships, and thus is a promising approach to improve the gene function prediction.
Methods. Here we present NetSyn (Network Synteny), a tool, which aims to cluster protein sequences according to the similarity of their genomic context rather than their sequence similarity. Starting from a set of protein sequences of interest, NetSyn retrieves neighbouring genes from the corresponding genomes as well as their protein sequence. Homologous protein families are then computed to measure synteny conservation between each pair of input sequences using a GC score. A network is then created where nodes represent the input proteins and edges the fact that two proteins share a common GC. The weight of the edges corresponds to the synteny conservation score. The network is then partitioned into clusters of proteins sharing a high degree of synteny conservation. Results. As a proof of concept, we used NetSyn on two different datasets.
Introduction The advance of genome sequencing technology has created an ever increasing gap between protein sequences and annotations  (Galperin and Koonin, 2010). It is currently estimated that 25% of the families defined in the PFAM database are annotated as "unknown function"  (Mudgal et al. , 2015). Moreover, this issue is worsened by current automatic annotation procedures that still have a high false positive rate. This is especially true for methods that rely only on sequence similarity which lead to some protein families having a level of mis-annotation as high as 80%  (Schnoes et al. , 2009). To overcome the lack or error annotation, innovative strategies use a combination of genome analysis and metabolic information  (Chen et al. , 2011; Kharchenko et al. , 2006; Smith et al. , 2012; Yamanishi et al.
e. organisms from the same genus). Other efforts have been carried out to incorporate information from gene context to integrative approach. For instance, the STRING database  (Szklarczyk et al. , 2017)  stores all the known and predicted protein-protein interactions from different sources. These interactions can be physical (i. e. heterodimer) or indirect (e. g. gene regulation, signal relay mechanism) and can be represented by a graph where nodes correspond to the proteins and the edges to the interaction. This network highlights the interactions between different protein families. Another example is the Enzyme Function Initiative (EFI)  (Zallot et al. , 2019)  which has developed a method based on the Genomic Neighborhood Network (GNN). Starting with a list of query sequences, a Sequences Similarity Network (SSN) is created, in which groups of highly connected nodes are considered as a cluster.
Our method quantifies genomic context conservation of proteins taken from different genomes. The advantage of NetSyn over others methods, is the fact that the sequence comparison is based not on sequence similarity but only on genomic context conservation. This important originality allows the genomic comparison of both a set of homologous sequences like a whole sequence family, as well as a set of non-homologous sequences like several proteomes. NetSyn was tested on two kinds of dataset. One set contains sequences from an homologous family. NetSyn was able to create iso-functional sub-families and pinpoint putative new pathways. The second set contains sequences from three non-homologous families, involved in the same degradation pathway. NetSyn was able to retrieve this known pathway in different organisms.
Workflow and principles of NetSyn NetSyn is a workflow developed in Python 3 and is divided into four steps: i) downloading the data, ii) extracting the genomic context, iii) computing the synteny conservation, iiii) computing a synteny network. NetSyn accepts as input a list of UniProt accession numbers (hereafter called target proteins). The workflow makes requests to the UniProt rest API to retrieve the corresponding nucleic accession number and then download the genome assembly file (hereafter called the genome file) from the European Nucleotide Archive (ENA) web site. Genomes files are used to extract the GC for each target protein (Fig. S1 ). But one can use his own data that are not stored in UniProt and ENA. In that case, NetSyn takes as input a correspondence file (Fig. S2 ), which is in a tabulation-separated value format with 5 mandatory fields: -protein_AC: the protein accession number in the genome file -protein_AC_field: the name of the field (qualifier) from which the protein accession number can be found in the genome file -nucleic_AC: the accession number of the genome file -nucleic_File_Format: the format of the genome file (EMBL or Genbank format) -nucleic_File_Path: the path to the locally stored genome file.
3 , the selected window size was 11, therefore 5 genes before and 5 genes after are shown. In this view, homologous genes are colored with the same color. If the user pass his cursor above the one the gene, a pop up appears and shows information from the NetSyn computation and the genome file: the family id (MMseqs2 family identifier), strain name, protein accession number, Uniprot accession number, gene name, locus tag, product and EC_number (Fig. S4 ). The lower panel gives information about the computation of homologous protein families by MMSEQ, for the selected NetSyn cluster. For each MMseqs2 family determined, it gives the number of selected proteins belonging to this family, the number of different species, strains and genomes from which the selected proteins come.
Dividing homologous family NetSyn was tested on the β-keto acid cleavage enzymes family (BKACE). This family, initially called DUF849 (Domain of Unknown Function) in the PFAM database, was extensively characterized using high-throughput enzymatic screening  (Bastard et al. , 2014). The profiles of activity issued from the enzymatic screening of 124 representatives on 20 substrates showed a high correlation with the profiles of active sites. Indeed, the 725 proteins of the BKACE family were divided into 7 main groups based on structural classification of active sites by the Active Sites Modeling and Clustering (ASMC) method (de  Melo-Minardi et al. , 2010). As ASMC groups show a good match with in vitro enzymatic activities, our aim here is to verify NetSyn clustering can identify the ASMC groups, and thus check if NetSyn can cluster iso-functional enzymes.
All these clusters show a good fit with the ASMC groups (named here after G1 to G7) meaning that between 77 to 100% of their members belong to the same ASMC group. The only exception is the Louvain cluster 12 where only 55% of its sequences belong to the group G1. Cluster 12 is the only cluster that contains sequences from all the ASMC groups (Fig. 4 ). The network generated by NetSyn on BKACE data is available for visualization at https://doi. org/10. 5281/zenodo. 5746499 By definition, NetSyn clusters proteins according to their metabolic context while ASMC gathers proteins according to their active sites, which directly reflects the type of substrate they can catalyze (negative, positive, polar or hydrophobic substrates). Group G7 is the perfect example for illustrating this difference.
Interconnectivities of families in metabolic pathways A great advantage of NetSyn is its ability to handle genes with no homology but functionally related. Indeed, half the genes of prokaryotes and a minor fraction of genes in eukaryotes form groups of physically clustered genes on chromosomes that are related in function  (Nützmann et al. , 2018). The most studied gene organization is the operon  (Jacob and Monod, 1961)  in which genes are transcribed as single polycistronic mRNA. Operons are often conserved across species by vertical inheritance, and thus operons can be computationally predicted. More recently an operon-like structure, grouping CAZymes (for Carbohydrate Active enZymes), transcriptional regulators and transporters, has been highlighted  (Bjursell et al. , 2006). In Bacteroidetes, this system is called Polysaccharide Utilization Locus (PULs) and is composed of gene clusters that encode enzyme and protein ensembles required for the saccharification of complex carbohydrates.
, 1997)  We found 20 organisms containing homologs for the 3 key genes present in the Xygul in Cellvibrio Japonicus (Table  S4 ). The complete proteome of these organisms were downloaded from UniProt and we used DBCAN  (Zhang et al. , 2018)  to retrieve all the GHs. Then we selected all proteins annotated as GH31, GH35 or GH95 in these proteomes, which give a set of 194 UniProt entries and use this set as input for NetSyn. It took 47 second on one CPU to compute the final graph. The final network is available at https://zenodo. org/record/5705596. Only 136 input entries had a conserved genomic context and have been clusterized into 34 clusters with the Walktrap algorithm (Fig  5    It should be noted that the Cellvibrio japonicus Xygul locis lacks a xyloglucanase activity (EC 3. 2. 1. 151) for the degradation of the xyloglucan backbone  (Larsbrink et al.
Their grouping is due to their genomic context conservation which reflects their functional conservation. But functional characterisation is needed to confirm our prediction. NetSyn is a great tool to help functional assignment of protein sequences that are still of unknown function. Classical tools used sequence comparison to assign functions. However, even with recent improvement in homolog search accuracy  (Buchfink et al. , 2015) , there are still 23 % of protein families with uncharacterized function in databases  (Mistry et al. , 2021). On the other hand, many known activities have not been associated with a protein sequence. In their review,  Sorokina et al. ,  show that more than 1000 EC numbers, called orphan reactions, are not linked to a specific gene  (Sorokina et al. , 2014). To link genes with orphans reactions, methods based on protein sequence similarity are not suitable, but methods able to group proteins according to their function without taking into account sequence similarity is required.
NetSyn, on the contrary, disregards database classification, but uses MMseqs2 to create its own homologous families. To our knowledge, NetSyn is the only tool able to handle a set of proteins without any common evolutionary origin whereas the other methods start from an homologous family. NetSyn paves the way to new and innovatives tools to solve the challenging task of predicting new enzymatic activities. The main limitation of NetSyn is the fact that it may be biased by a selection of sequences from organisms classified at the same low taxonomic range (i. e organisms from the same genus). Indeed the lack of genomic diversity in the input set might lead to detected a synteny only due to closely related organisms rather than functional conservation through evolution. To circumvent this issue, we proposed to reduce the redundancy by merging nodes belonging to the same NetSyn clusters which share the same criteria like a taxonomic rank or a given metadata.
The main guiding idea of NetSyn is that genes sharing a common genomic context are more likely to share the same function. But this comparison must be independent of their sequence similarities. Here, the relationship between two proteins in the network depends only on the number of homologous genes into their corresponding genomic context. Therefore NetSyn is a suitable tool to explore either the functional diversity of enzyme families, either to highlight conserved loci containing genes of different evolutionary origins implied into the same pathway or in the same degradation/biosynthesis process. NetSyn is freely available at https://github. com/labgem/netsyn.
, 2017; Refsgaard et al. , 2022; Matthews et al. , 2022 Matthews et al. , , 2023a;; Moller et al. , 2014; Kim et al. , 2020). WaTEM/SEDEM (W/S) is the most widely applied spatially-distributed model to predict the long-term soil erosion and sediment delivery at the catchment scale in a spatially distributed manner  (Alatorre et al. , 2010; Van Oost et al. , 2000; van Oost et al. , 2002; Borrelli et al. , 2018a Borrelli et al. , , 2021)). Its fundamental conceptions embed the RUSLE equation into a catchment-scale model by coupling it with a spatially-explicit sediment transport module to simulate erosion, deposition, and the portion of sediment yield delivered to stream channels by rill and interrill erosion processes  (Van Rompaey et al. , 2001; Van Oost et al. , 2000; van Oost et al. , 2002).
Catchment data subsamples from the EUSEDcollab database Understanding the capacity of W/S to simulate the inter-annual variability of soil erosion-sediment delivery events requires catchment sediment yield measurements at a suitably small spatial scale (some suggest roughly < 10 km 2 ), ideally below the scale at which the signal of hillslope erosion is strongly confounded by other erosion (e. g. gullying, mass wasting, bank erosion) or in-channel processes  (de Vente and Poesen, 2005). Knowledge on the catchment erosion processes is likewise useful to understand if the dominant sediment sources match those simulated by W/S. We selected catchments meeting this criteria from the EUSEDcollab database  (Matthews et al. , 2023b) , compiled with the specific purpose of providing publicly available catchment data suitable for soil erosion and sediment delivery modelling.