dataesr/scientific-paragraphs-categorization huggingface dataset:
Two webservices both provide a REST API to interact with the database: one for inserting called Flask Web Service Server (FWSS, in python and using the framework Flask) and one to query the metadata called EPOS GLASS Framework (in Java). A tool in python, monitor-dgw, was developed for managing the station metadata received from the M3G and to convert them to a JSON format understood by FWSS. The index-GD and RunQC tools (in python and perl) are used for extracting and generating File and Quality Control metadata and sending them to FWSS for insertion in the database. RunQC itself uses Anubis (https://gnutsoftware. com/software/anubis/) to analyze the RINEX file quality and to generate the RINEX File Quality Information metadata. As described in section 2. 2. 2, the metadata are synchronized between the nodes of the EPOS-GNSS Data Node Network which is done with the EPOS Synchronization System.
An optional GUI is included to manage the metadata specific to each Data Nodes. Finally, in order to explore and download the data and their associated metadata, a web and a command line client, called GLASS web client and Pyglass, respectively, were developed for interacting with the EPOS GLASS Framework webservice. The GLASS Web client notably serves also as the GUI for the Data Gateway. The access to the products is done through a particular GLASS node that interacts with the user through a GUI managed by another dedicated webservice (see section 2. 3. 3). This node stores station metadata and an extended version of the database to hold information on the derived GNSS products. Problems related to the GLASS service orientated architecture built on web services are discussed further in  Manteigueiro et al.
GLASS status and future development GLASS, although fully operational, is in constant development to add new features and correct bugs internally identified or reported by the users. To facilitate testing and development, a virtual machine image is available with all software installed and configured that can be downloaded at https://gitlab. com/gpseurope/, where all GLASS software is publicly available. Currently, work is underway to seamlessly add hourly high-rate (1 Hz) RINEX data into the system. Tools for monitoring more closely the node network have been implemented and are currently under review. More data and metadata such as DOIs and Quality Control XML files provided by Anubis are also planned to be made available soon.
Rui Fernandes et al. ensured that the node does not already exist outside of EPOS and that it will only host EPOS data, else the designed system will fail. Once decisions are made, the node and configuration letters can be sent to the node infrastructure coordinator, and software can be installed, either with a virtual machine with all the necessary software or directly on a server. The node should access the data repository to be able to generate the file and quality control metadata. It is also necessary to check that the server on which the data is located can run the python and perl software necessary to generate the metadata, and that the web service dedicated to insert the metadata into the database is accessible from this server. The database itself must be accessible from and to the Data Gateway.
-checked RINEX data and station metadata of EPOS-GNSS stations; Operation of EPOS-GNSS Data Gateway and monitoring of the node infrastructure and synchronizations PRODUCT DISSEMINATION Computation and provision of positions, velocities, and time series of EPOS-GNSS stations, and Strain rates; Provision of EUREF GNSS data and products; Operation of EPOS-GNSS Product Portal SOFTWARE PROVISION Maintenance and Development of GLASS software: databases, tools & webservices.
Fast and reliable evaluation metrics are key to R&D progress. While traditional natural language generation metrics are fast, they are not very reliable. Conversely, new metrics based on large pretrained language models are much more reliable, but require significant computational resources. In this paper, we propose Fru-galScore, an approach to learn a fixed, low cost version of any expensive NLG metric, while retaining most of its original performance. Experiments with BERTScore and MoverScore on summarization and translation show that Fru-galScore is on par with the original metrics (and sometimes better), while having several orders of magnitude less parameters and running several times faster. On average over all learned metrics, tasks, and variants, FrugalScore retains 96. 8% of the performance, runs 24 times faster, and has 35 times less parameters than the original metrics.
Introduction Automatic evaluation metrics are the only way to monitor the training of, evaluate, and compare across models in a systematic, large-scale way, and are thus a critical component of the research and development ecosystem in machine learning. To get adopted in practice, evaluation metrics need to be both reliable and affordable, i. e. , fast and easy to compute. While some metrics meet these criteria, such as precision and recall in information retrieval, root mean square error in regression, etc. , finding suitable metrics is still an open problem in the field of Natural Language Generation (NLG)  (Novikova et al. , 2017). *Equal contribution 1 https://github. com/moussaKam/FrugalScore Indeed, historical n-gram matching metrics such as ROUGE  (Lin, 2004)  for summarization, BLEU  (Papineni et al.
, 2002)  and METEOR  (Banerjee and Lavie, 2005)  for translation, while affordable, are not very reliable, as they are based on surface-form matching only, i. e. , lexical similarity, and have thus no sense of semantic similarity. For instance, it makes little sense to use ROUGE for the evaluation of abstractive summarization systems (which are becoming the norm), or whenever the generated text paraphrases the original text. Following the advent of transfer learning in NLP, new NLG metrics based on large pretrained language models have recently been proposed, such as BERTScore  (Zhang et al. , 2019)  and MoverScore  (Zhao et al. , 2019). By relying on contextual embeddings, these metrics capture semantics and are therefore much more reliable. However, due to the sheer size of the underlying models, these metrics pose environmental issues  (Strubell et al.
, 2019) , take time to compute, and require access to significant computational resources, so they are not accessible by everyone in the NLP community. For example, we were not able to run some of the best variants of BERTScore 2  , based on DeBERTa-Large and DeBERTa-XLarge  (He et al. , 2020)  on a 12GB GPU. Even when enough GPU memory is available, relying on such large models is still associated with extended runtimes, which can impede the progress of experiments when used once or more per epoch for validation and monitoring purposes. To address this problem, we propose in this paper FrugalScore, an approach to learn a lightweight version of BERTScore, MoverScore, and more generally any metric based on a large pretrained language model. Our contributions can be summarized as follows: 1) Our compact models have several orders of magnitude less parameters than the original metrics and run several times faster, while retaining most of their original performance.
, 2015). BERTR  (Mathur et al. , 2019)  computes approximate recall based on the pairwise cosine similarity between the BERT embeddings  (Devlin et al. , 2018)  of the words in automatic and reference translations. Mark-Evaluate  (Mordido and Meinel, 2020 ) is a family of metrics that consider contextualized word or sentence embeddings derived from BERT as population samples, to evaluate language generation with population estimation methods used in ecology. Finally, the recently introduced BERTScore  (Zhang et al. , 2019)  and MoverScore  (Zhao et al. , 2019)  are general-purpose NLG evaluation metrics that are becoming widely used. The main difference between BERTScore and MoverScore lies in the function used to compute the similarity between the representations of the two sequences x = ⟨x 1 ,.
RUSE  (Shimanaka et al. , 2018)  evaluates machine translation with a neural regressor based on universal sentence embeddings (e. g. , InferSent  (Conneau et al. , 2017) ). NUBIA  (Kane et al. , 2020)  consists of three modules: a feature extractor based on RoBERTa  (Liu et al. , 2019)  and  GPT-2 (Radford et al. , 2019)  fine-tuned on language evaluation tasks, an aggregator trained to predict the quality of the hypothesis given the reference using the extracted features, and a calibrator mapping all predictions between 0 and 1. Differences. Like the aforementioned efforts, Fru-galScore is a learned metric. However, it does not rely on any intermediate or handcrafted features, and, most importantly, it does not require training on human annotations. Supervision in FrugalScore is conducted on a synthetic dataset, as a trick to expose and learn the internal mapping of the unsupervised metrics to be learned.
Previous studies on KD mainly focused on classification problems  (Gou et al. , 2021). A few attempts have been made on regression problems  (Chen et al. , 2017; Saputra et al. , 2019; Takamoto et al. , 2020) , in which special losses were proposed to train the student with respect to both the teacher's regression outputs and ground truth scores. Different from conventional distillation, our work is more similar to data-free KD  (Kang and Kang, 2021) , where the student is trained in the absence of the dataset used to train the teacher. To transfer knowledge, we first create a synthetic dataset by annotating sequence pairs with a large model (teacher), and then train a miniature model (student) on that dataset, in an offline and regression setting.
Differences with BLEURT A work closely related to ours is BLEURT  (Sellam et al. , 2020). However, there are a number of significant differences with our approach. First, BLEURT continues the pretraining of an already pretrained BERT-based model on a synthetic dataset in a selfsupervised way, whereas FrugalScore is directly trained to learn the scores of the metric of interest, in a supervised fashion. Also, BLEURT's synthetic dataset is made by perturbing Wikipedia sentences with mask-filling, backtranslation, and word dropping, whereas we use other data sources than Wikipedia such as summarization and translation datasets, and only NLG models to induce perturbations. When creating its synthetic dataset, BLEURT automatically annotates the (original, perturbed) sequence pairs with numerical and categorical "signals": BLEU, ROUGE, BERTscore, backtranslation likelihood, textual entailment (probability of three labels: entail, contradict, and neutral, given by BERT fine-tuned on MNLI), and backtranslation flag.
On the other hand, FrugalScore simply and directly annotates the sequence pairs with the metric to be learned. After pretraining, BLEURT is fine-tuned on human judgments, in a way similar to the supervised metrics described in subsection 2. 2. BLEURT does not learn to generate a scalar until that final finetuning phase, so it cannot be used as a metric before that. Conversely, FrugalScore is trained from the start to be a metric, and the fine-tuning phase is optional. Also, BLEURT was designed for the evaluation of translation. The authors only test whether it can be applied to a different task by experimenting on the WebNLG (data-to-text) dataset  (Gardent et al. , 2017). Conversely, we focus on learning general text similarity metrics (e. g. , BERTscore and MoverScore), so FrugalScore is task-agnostic by design.
Our approach Developing FrugalScore requires three phases, one of which is optional. Phase 1. We create a synthetic dataset (see subsection 3. 1) by sampling pairs of more or less related sequences and annotating them with the expensive metrics to be learned. This is a one-time operation that does not need to be repeated regardless of the model used in Phase 2. Phase 2. We continue the pretraining (see subsection 3. 2) of a miniature pretrained language model on the synthetic dataset built by Phase 1. Here, the miniature model learns the internal mapping of the expensive metric, including any similarity function applied to the representations. Note that a different miniature is trained for each metric to be learned (we leave learning metric combinations as future work). The miniature can then be used in inference mode to generate scores for any never-seen pair of sequences.
Synthetic dataset The objective here was to generate pairs of sequences mimicking the (reference, candidate) pairs found in NLG datasets, which are usually semantically related and in many cases paraphrasing one another. We sampled our sequences from a variety of data sources, listed next. Summarization. For each document in the wellknown CNN/DailyMail dataset  (Nallapati et al. , 2016) , our goal was to generate several summaries differing in terms of structure and quality. To this purpose, we used different pretrained seq2seq summarization models: BART-base and BART-large  (Lewis et al. , 2019) , mBART  (Liu et al. , 2020), and BARThez (Kamal Eddine et al. , 2021). BART is a seq2seq autoencoder with a Transformer architecture. The four models were fine-tuned for one epoch on 50k examples randomly sampled from the training set of CNN/DM, and were used to generate summaries for the whole training set of 287,112 documents, using greedy decoding.
The lengths of the spans were sampled from a Poisson distribution (λ = 3). 50% of the tokens within the input segment were masked and 20% of the masked text was replaced with random tokens (creating pathological examples to increase the robustness of the learned metric). The sentences in the input segment were then shuffled. We finally used a BART-Base checkpoint 4  from the Fairseq library  (Ott et al. , 2019)  to try to reconstruct the perturbed versions of the original sequences, hence creating variants of them. Annotating pairs. We sampled 4. 5M sequence pairs uniformly from each aforelisted source. These pairs were then annotated with the metrics to be learned. Note that this is a one-time operation that does not need to be repeated regardless of which models are trained downstream. In this work, we experimented with two recent expensive NLG metrics that rely on large pretrained language models, BERTScore  (Zhang et al.
Metric learning We continue the pretraining of three BERT miniatures 5  on our synthetic dataset: BERT-Tiny (L = 2, H = 128), BERT-Small (L = 4, H = 512) and BERT-Medium (L = 8, H = 512), where L is the number of layers and H is the dimension of the embedding space. These models have respectively 25 times, 3. 78 times, and 2. 64 times less parameters than BERT-base. The concept of BERT miniatures was introduced by  Turc et al. (2019)  to test whether pretraining small models from scratch was competitive to distilling very large models. The miniature models have already been pretrained on masked language model and next sentence prediction objectives. We continue pretraining using the standard method introduced by  Devlin et al. (2018). We concatenate the two sequences x = ⟨x 1 ,. , x k ⟩ and y = ⟨y 1 ,.
, y l ⟩ in a given pair, separating them with a special [SEP] token. A special [CLS] token is also added at the beginning of the resulting sequence. The sequence of contextualised embeddings ⟨z [CLS] , x 1 ,. x k , z [SEP] , y 1 ,. , y l ⟩ is then obtained. We finally add a fully connected layer on top, that linearly projects the z [CLS] vector to a scalar s. The model is trained to minimize the mean square error (MSE) loss between the learned metric s i and the metric to be learned ŝi (i. e. , the annotation of the pair): l = 1 N N i=1 ||s i -ŝi || 2 (5) When pretraining is over, the models can be further fine-tuned on smaller human-annotated datasets as shown in section 6, or directly used to generate scores for unseen examples as shown in section 4. Setup. We use a batch size of 32 and the Adam optimizer (Kingma and Ba, 2014) with a learning rate of 3 × 10 -5 , linear decay, and a warm-up for 6% of the total training steps, and we train the model for three epochs.
Experiments In this section, FrugalScore is used in inference mode to generate scores directly after pretraining, i. e. , no fine-tuning is performed (see section 6 for fine-tuning results). We evaluate on two text generation tasks: summarization and translation. We use evaluation datasets containing (reference, candidate) sequence pairs annotated with human scores assessing the quality of the candidates given the references. We measure the effectiveness of FrugalScore by measuring the Pearson correlation of its scores with the human judgments and comparing it to that of the original metrics. We also take the number of parameters and the runtime into account. Text summarization. We use 4 different multidocument summarization datasets from the Text Analysis Conference (TAC) 6  : TAC-2008, TAC-2009, TAC-2010 and TAC-2011.
These datasets respectively contain 48, 44, 46 and 44 clusters of documents and 58, 55, 43 and 51 systems are used to generate summaries. Each cluster forms a topic to be summarized and has 4 reference summaries. There are approximately 10k pairs in each dataset. Each pair is annotated with two human judgment scores: the Pyramid Score  (Harnly et al. , 2005)  and the Responsiveness  (Dang et al. , 2008). The former measures the proportion of important semantic units (SCUs) in the reference summaries captured by the system summary, while the latter reflects the content coverage and the readability of each summary. Machine translation. Our evaluation corpus is from the WMT-2019 7 shared task  (Li et al. , 2019). We consider all the to-English pairs: Chinese, Czech, German, Finnish, Russian, Lithuanian and Kazakh to English.
Results Table  1  reports the results averaged over the 4 TAC datasets and the 7 WMT to-English language pairs. Details are provided in Appendices A and B. We benchmarked the metrics in terms of Pearson correlations with human scores, runtimes, and numbers of parameters. We used two approaches to compute the Pearson correlations: summary-level (or segment-level) and system-level. In the former approach, a score is attributed to each of the output candidates, while in the latter approach, one single overall score is attributed to the system (by averaging its individual scores). Rows a to c correspond to BERTScore with BERT miniatures as the underlying model. They are simple baselines added for the sake of comparison, to see what we get when BERTScore is used with the same number of parameters as Fru-galScore.
Rows d to g correspond to the expensive metrics that are learned by FrugalScore (in the respective sections of the bottom half of the table). They are BERTScore and MoverScore metrics where the underlying model is a large pretrained language model: BERT-Base (L = 12, H = 512), RoBERTa-Large (L = 24, H = 1024)  (Liu et al. , 2019) , and DeBERTa-XLarge (L = 24, H = 1536)  (He et al. , 2020). Finally, rows i to xii correspond to Fru-galScore. Subscripts refer to row labels and indicate which metric-model combination was used to annotate pairs. I. e. , FrugalScore d learned the metric of row d, i. e. , BERTScore with BERT-Base. First, results show that all FrugalScores, regardless of which metric they learned, significantly outperform the BERTScores with miniature models. These results suggest that FrugalScore is a better approach than using an existing metric with a lightweight underlying model.
The reason for this is probably that in FrugalScore, the knowledge of the original unsupervised metric (based on a large model) is explicitly transferred to the miniature via the continuation of its pretraining on the synthetic dataset. That is, the miniature is actually learning a metric. Whereas, on the other hand, plugging a compressed version of a general-purpose language model into the original unsupervised metric just makes it lose expressiveness and capacity. Second, we can clearly see that FrugalScore retains most of the performance of the original metric, while running several times faster and reducing the number of parameters by several orders of magnitude. On average over all metrics, tasks, and miniatures, FrugalScore retains 96. 8% of the original performance, runs 24 times faster, and has 35 times less parameters.
g. , row vi → viii → ix), and sometimes, FrugalScore with the smallest miniature (BERT-Tiny) is superior (e. g. rows i and x). This finding suggests that the impact of the pretrained language model size is task-dependent. To sum up, results clearly show the effectiveness of FrugalScore in learning a cheaper, lighter, and faster version of the original metrics, while retaining most of their original performance. The system-level correlations, provided in Appendices C and D, corroborate these positive results. We also provide the correlations between the original and the learned metrics in Appendices E and F. It is interesting to note that a greater correlation with the original metric is not always associated with a better performance. E. g. , the tiny version of FrugalScore g is the best (row x), while it is the less correlated with the original metric.
Fine-tuning on human annotations We test two hypotheses in this section: (1) whether fine-tuning on a human-annotated dataset is beneficial, and (2) when fine-tuning on human annotations, whether continuing pretraining on our synthetic dataset is useful. Because we cannot use the same dataset for finetuning and evaluation, we fine-tune a BERT-Small on each year of TAC 2008-2011 for 4 epochs, using two other years as the validation set, and the remaining year as the test set. The best epoch is selected based on validation performance. We use a batch size of 32 and a learning rate of 2e-5 that linearly decreases to zero. Finally, we experi- ment with two scenarios: fine-tuning the miniature directly without continuing its pretraining on our synthetic dataset, and fine-tuning it after the pretraining continuation (with annotations generated by BERTScore-BERT-Base).
Results. Results are reported in Table  2  in terms of summary-level Pearson correlations with human evaluations (Pyramid), averaged over 3 runs with different random seeds. First, it is obvious that everywhere, continuing the pretraining on our synthetic dataset leads to a significant boost in performance. This is in accordance with  Sellam et al. (2020) , who found that pretraining was beneficial even in a supervised setting. Second, even if a direct comparison is not possible, we can remark when looking at the TAC Pyramid score of row ii) in Table  1  (FrugalScore d -BERT-Small) that fine-tuning after pretraining seems very beneficial too. Indeed, after fine-tuning, we reach on average 71, 67. 5, 68. 7, and 69. 2 (depending on the split), which represents overall a gain of 4. 4 points over the non-fine-tuned model (score of 64.
Impact of data sources To test the importance of each data source introduced in subsection 3. 1, we created a training set containing sequence pairs uniformly and equally sampled from each source. We annotated these pairs with the BERTScore-BERT-Base metric and we used them to continue the pretraining of a BERT-Small miniature. We also considered pairs drawn at random from the pairs generated with the other strategies. The motivation for random pairs was to sample "negative examples", as seeing only "positive examples" (pairs of related sequences) could bias the learned metric towards considering any two unrelated sequences as similar. We then continued the pretraining of the BERT-Small miniature four times, excluding each time the pairs coming from a specific data source. We evaluated the learned metric on TAC-2008 to 2011 and on WMT-2019.