dataesr/scientific-paragraphs-categorization huggingface dataset:
, 2015b)  or XSUM  (Narayan et al. , 2018a). Further, we provide recommended train/validation/test splits following a chronological ordering based on the articles' publication dates. In our experiments below, we train/evaluate the models on the training/test splits obtained in this manner. Specifically, we use: data from 2010 to 2018, included, for training; data for 2019 (~10% of the dataset) for validation (up to May 2019) and test (May-December 2019). While this choice is arguably more challenging, due to the possible emergence of new topics over time, we consider it as the realistic scenario a successful summarization system should be able to deal with. Incidentally, this also bring the advantage of excluding most cases of leakage across languages: it prevents a model, for instance, from seeing a training sample describing an important event in one language, and then being submitted for inference a similar article in another language, published around the same time and dealing with the same event.
Dataset Statistics We report statistics for each language in ML-SUM in Table  1 , including those computed on the CNN/Daily Mail dataset (English) for quick comparison. MLSUM provides a comparable amount of data for all languages, with the exception of Russian with ten times less training samples. Important characteristics for summarization datasets are the length of articles and summaries, the vocabulary size, and a proxy for abstractiveness, namely the percentage of novel n-grams between the article and its human summary. From Coupled with the significantly lower amount of articles available from its online source, the task can be seen as more challenging for Russian than for the other languages in MLSUM. Conversely, similar characteristics are shared among other languages, for instance French and German.
Extractive summarization models Oracle Extracts the sentences, within the input text, that maximise a given metric (in our experiments, ROUGE-L) given the reference summary. It is an indication of the maximum one could achieve with extractive summarization. In this work, we rely on the implementation of  Narayan et al. (2018b). Random In order to elaborate and compare the performances of the different models across languages, it is useful to include an unbiased model as a point of reference. To that purpose, we define a simple random extractive model that randomly extracts N words from the source document, with N fixed as the average length of the summary. Lead-3 Simply selects the three first sentences from the input text. Sharma et al. (2019) , among others, showed that this is a robust baseline for several summarization datasets such as CNN/DM, NYT and BIGPATENT.
Results and Discussion The results presented below allow us to compare the models across languages, and investigate or hypothesize where their performance variations may come from. We can distinguish the following factors to explain differences in the results: 1. Differences in the data, independently from the language, such as the structure of the article, the abstractiveness of the summaries, or the quantity of data; 2. Differences due to the language itself -either due to metric biases (e. g. due to a different morphological type) or to biases inherent to the model. While the first fold of differences have more to do with domain adaptation, the second fold motivates further the development of multilingual datasets, since they are the only mean to study such phenomenon. Turning to the observed results, we report in Table 2 the ROUGE-L and METEOR scores obtained by each model for all languages.
We note that the overall order of systems (for each language) is preserved when using either metric (modulo some swaps between Lead 3 and Pointer Generator, but with relatively close scores). Russian, the low-resource language in MLSUM For all experimental setups, the performance on Russian is comparatively low. This can be explained by at least two factors. First, the corpus is the most abstractive (see Table  1 , limiting the performance figures obtained for the extractive models (Random, LEAD-3, and Oracle). Second, one order of magnitude less training data is available for Russian than for the other MLSUM languages, a fact which can explain the impressive improvement of performance (+66% in terms of ROUGE-L, see Table  2 ) between a not pretrained model (Pointer Generator) and a pretrained model (M-BERT).
Abstractiveness of the datasets The Oracle performance can be considered as the upper limit for an extractive model since it extracts the sentences that provide the best ROUGE-L. We can observe that while being similar for English and German, and to some extent Turkish, the Oracle performance is lower for French or Spanish. However, as described in figure  1 , the percentage of novel words are similar for German (14. 96), French (15. 21) and Spanish (15. 34). This may indicate that the relevant information to extract from the article is more spread among sentences for Spanish and French than for German. This is confirmed with the results of Lead-3: German and English have a much higher. regardless Oracle. It is particularly surprising to see the low performance on German whereas, for this language, Lead-3 has a comparatively higher performance.
The case of TextRank In Table  3 , we report the performance ratio between TextRank and Pointer Generator on our corpus, as well as on CNN/DM and two other English corpora (DUC and NewsRoom). TextRank has a performance close to the Pointer Generator on English corpora (ratio between 0. 85 to 1. 21) but not in other languages (ratio between 0. 37 to 0. 65). This suggests that this model, despite its generic and unsupervised nature, might be highly biased towards English. The benefits of pretraining We hypothesize that the closer an unsupervised model performance to its maximum limit, the less improvement would come from pretraining. In Figure  3 , we plot the improvement rate from TextRank to Oracle, against that of Pointer-Generator to M-BERT. Looking at the correlation emerging from the plot, the hypothesis appears to hold true for all languages, including Russian -not plotted for scaling reasons (x = 808; y = 40), with the exception of English.
This exception is probably due to the aforementioned bias of TextRank towards the English language. Pointer Generator and M-BERT Finally, we observe in our results that M-BERT always outperforms the Pointer Generator. However, the ratio is not homogeneous across the different languages as reported in Table  3. In particular, the improvement for German is much more important than the one for French. Interestingly, this observation is in line with the results reported for Machine Translation: the Transformer  (Vaswani et al. , 2017)  outperforms significantly ConvS2S  (Gehring et al. , 2017)  for English to German but obtains comparable results for English to French -see Table  2  in  Vaswani et al. (2017). Neither model is pretrained, nor based on LSTM  (Hochreiter and Schmidhuber, 1997) , and they both use BPE tokenization  (Shibata et al.
Possible derivative usages of MLSUM Multilingual Question Answering Originally, CNN/DM was a Question Answering dataset  (Hermann et al. , 2015a). The hypothesis is that the information in the summary is also contained in the pair article. Hence, questions can be generated from the summary sentences by masking the Named Entities contained therein. The masked entities represent the answers, and thus a masked question should be answerable given the source article. So far, no multilingual training dataset has been proposed for Question Answering. This methodology could be thus applied on ML-SUM as a first step toward a large-scale multilingual Question Answering corpus. Incidentally, this would also allow progressing towards multilingual Question Generation, a crucial component to employ the neural summarization metrics mentioned in Section 5.
News Title Generation While the release of MLSUM hereby described covers only articlesummary pairs, the archived news articles also include the corresponding titles. The accompanying code for parsing the articles allows to easily retrieve the titles and thus use them for News Title Generation. Topic detection A topic/category can be associated with each article/summary pair, by simply parsing the corresponding URL. A natural application of this data for summarization would be for template based summarization  (Perez-Beltrachini et al. , 2019) , using it as additional features. However, it can also be a useful multilingual resource for topic detection. We presented MLSUM, the first large-scale Multi-Lingual SUMmarization dataset, comprising over 1. 5M article/summary pairs in French, German, Russian, Spanish, and Turkish.
Kryściński et al. (2019)(bottom) results obtainedby the models described in 4. 1 on the different proposed datasets. article. Further,Kryściński et al. (2019)proposed a discriminator trained to measure the factualness of the summary. While Böhm et al. (2019) learned a metric from human annotation. All these models were only trained on English datasets, preventing us to report them in this paper. The availability of MLSUM will enable future works to build such metrics in a multilingual fashion.
Based on Observation System Simulation Experiments (OSSE), we will use the the NATL60 high resolution deterministic ocean simulation of the North Atlantic to generate two types of pseudo altimetric observational dataset: along-track nadir data for the current capabilities of the observation system and wide-swath SWOT data in the context of the upcoming SWOT mission. We briefly introduce the analog data assimilation (AnDA), an up-to-date version of the DINEOF algorithm, and a new neural networks-based end-to-end learning framework for the representation of spatio-temporal irregularly-sampled data. The main objective of this paper consists in providing a thorough intercomparison exercise with appropriate benchmarking metrics to assess if these approaches helps to improve the SSH altimetric interpolation problem and to identify which one performs best in this context.
More specifically, promising preliminary results have been seen for the sea surface reconstruction and prediction from partial and noisy satellite observations. In this paper, we propose an intercomparison exercise of several data-driven and learning-based approaches to help for the reconstruction of altimetric fields. As a baseline the DUACS operational processing tool based on well established optimal interpolation (OI) techniques will be considered  [7]. In Section 2, we present the case study and its dataset, developed within the BOOST-SWOT project framework (https://meom-group. github. io/projects/boost-swot): the NATL60 high resolution deterministic ocean simulation of the North Atlantic  [8]  is used as reference to simulate Sea Surface Height (SSH) along-track observations collected by four nadir, which is typically representative of the current observational altimetric capabilities.
As an additional feature for the upcoming 2021 SWOT mission, pseudo-SWOT wide-swath observations also following realistic orbits are generated based on the NATL60 simulation. In Section 3, we present the data-driven approaches used in the intercomparison: 1) AnDA, a purely data-driven data assimilation scheme combining a patch-based analog forecasting operator with Kalman-based ensemble data assimilation, 2) VE-DINEOF, an EOF-based iterative method to interpolate in space and time the missing data, and 3) learning-based innovative end-to-end learning techniques that aims to learn jointly the Neural Network (NN) representation of the dynamics coupled with a NN-based solver of the targeted minimization problem. In Section 4, we provide a detailed evaluation of the results obtained over two small regions, GULFSTREAM and OSMOSIS, part of the North-Atlantic basin and labeled with very different energetic dynamics.
Nadir To provide the pseudo-nadir dataset, supposed to be representative of what is a current pre-SWOT observational altimetric dataset, the groundtracks of 4 altimetric missions (TOPEX/Poseidon, Geosat, Jason-1 and Envisat) picked up from the 2003 constellation, are used to interpolate the NATL60 A Gaussian white noise with variance σ 2 = 4 -9cm 2 is then added to the interpolated NATL60 simulation by the SWOTsimulator tool to mimic a noise with a spectrum of error consistent with global estimates from the Jason-2 altimeter  [9]. Because the space-time interpolations will focus on a daily-basis temporal resolution, we also build nadir pseudo-observations with an additional strategy by accumulating observations over a time window t k ± d days centered at time t k in order to increase the daily nadir spatial sampling.
SWOT In the same line, SWOT-like pseudo observations are also produced by the swotsimulator tool [10] in its swath mode with an along-track and across-track 2km spatial resolution, the same theoretical resolution the upcoming SWOT mission derived products should be able to provide. The nadir mode of the generator also provide pseudo-nadir along track observations though they are not used here. The simulator also adds instrumental noise on the idealized pseudo-SWOT dataset  [11, 12]. This noise potentially exhibits strong space-time correlations. Thus, the pseudo-SWOT observations are first preprocessed  [13]  to filter out these correlated components and avoid major issues in the assimilation and/or learning process of the interpolation methods. Let precise that over the low-latitude GULFSTREAM domain, the SWOT sampling is irregular leading to sequences of several days with only pseudo-nadir observations.
AnDA The Analog Data Assimilation (AnDA) is a purely data-driven data assimilation method introducing a statistical operator A as a substitute for the dynamical model M, leading to the following state-space formulation : dx k+1 = A k+1 (dx k ) + µ k dy k = H k (dx k ) + ε k (2) The analog forecasting operator A : dx a k-1 → dx f k , where superscripts a and f respectively relies to analysis and forecast, is built from the K most similar states to dx a k-1 in the available past state dynamics catalog, supposed to be large enough to describe the space-time evolution of the processes. More precisely, dx f k is sampled from the Gaussian prior dx f k |dx a k-1 ∼ N (µ k , Σ k ) , where the mean µ k and the covariance matrix Σ k are estimated using the so-called locally linear model  [2] , i. e. a weighted linear regression between the K nearest analogs and their successors.
End-to-end NN-learning An end-to-end learning representation has recently been introduced in  [15]  to deal with image sequences involving potentially large missing data rates. In this framework, an energy-based representation U θ to minimize is introduced : U ψ (dx) = dx -ψ (dx) 2 (3) where the operator ψ = ψ θ denotes a NN-based representation of the underlying processes and. 2 Ω refers to the L2 norm evaluated on subdomain Ω. Within a Bayesian framework, the interpolator I U ψ of the irregular space-time dataset {dy k (Ω k )}, referred ad the hidden state in a classic data assimilation framework, can be obtained by solving the minimization statement: dx k = I U ψ (dy k (Ω k )) = arg min dx U ψ (dx) (4) such that I U ψ (dy k (Ω k )) = dx k if no observational error are considered. Last, for a specific definition of interpolator I, the learning problem for optimizing parameters θ of the NN representation ψ can be stated as the minimization of the reconstruction error for the whole observed data time series: filters.
The two NN-architectures are then referred as FP-ConvAE and FP-GENN. Let note that additional improvements are expected when using an iterative gradient-based formulation of the solver, where the gradient of U ψ is replaced by a ConvNet or LSTM unit G(xψ(x)), thus enabling to solve jointly for the parametrization of ψ and G. Complementary results on SST datasets regarding this point can be found in  [15]. Let precise that during the learning phase, anomaly image time series dx k±dT = dx k-dT:k+dT are built with time window dT = 5, centered on time t k , leading to image time series of length 11. Last, the above-mentioned works are generalized to establish a connection between 4DVAR variational data assimilation and joint learning of models and solvers in  [16]. Table  1  provides all the formulas used to compute the above mentioned metrics used along Section 4.
The use of the OI as a covariate improve the FP-GENN behaviour but not systematically. SWOT data is available. We hope that a longer training period could help the network to learn from the masking periodicity of 2D wide-swath data. Based on these first results, the FP-GENN interpolator is used in its unsupervised configuration with OI used as a covariate. Because FP-ConvAE generally shows lower performance, probably because auto-encoders may not be relevant for the reconstruction of fine-scale processes, it will be used in the following in its mid-supervised configuration (FP-ConvAE-MNM) as a low-rated NN-scheme among the NN-based interpolators. Figure  6  presents the daily nRMSE of the different interpolators: it can be seen how FP-GENN significantly outperforms the conventional OI-based interpolator, but also the other data-driven algorithms used in the experiment.
In Table  2 , R/I/AE-scores are applied to both SSH (after application of a retrieving high-pass filter to keep only the small scales information) and its gradient (module). Regarding the R-scores, AnDA and VE-DINEOF are often the best way to keep track of the known areas, which is not surprising since these two methods makes an explicit use of the observational altimetric data into their mapping process. When looking at the I-scores, where no data is available, FP-GENN now clearly stands out from the other interpolators, which motivate its future use for irregularly-sampled data with large missing data rates. In addition, because its reconstruction scores remain overall satisfactory, in particular when considering the joint learning on nadir and SWOT data, these results are supplementary arguments on account of this markovian-related NN-based formulation.
Last, when computing the radially averaged power spectra as a spatial domain averaged over the 80-days validation period and the associated signal-to-noise ratio for joint use of along-track nadir with SWOT data (Figure  7b ), we observe that AnDA and FP-GENN lead to a better constraint of the SSH spectrum compared to the actual OI capabilities. In particular, FP-GENN produces a spectrum closer to the ground truth real spectrum, by catching up the submesoscale range up to 60km (when picking up signal-to-noise ratio equals to 0. 5) when considering a joint learning from along-track nadir and additional wide-swath SWOT data. Let note on Figure  7b  the importance of the patch-based AnDA post-processing on its performance which clearly appears on the spectra: its overestimation by the blocky patch-based AnDA rough outputs is partly mitigated thanks to the smoothing produced by averaging the patches overlapping areas.
This result may certainly be further improved, for instance by training a CNN rather than using a simple average-based smoothing. To further enhance the vizualisation of the improvements brought by the different interpolators,    It is especially noticeable on the 20-days long time series, see Figure  11. However, this result should be qualified because when replicating the same preliminary work to find the best FP-GENN configuration but with no observation errors, see Figure  A12  in Appendix B, the unsupervised configuration is again the best solution. Thus, on this less energetic OSMOSIS domain, but with more discernable fine scales, the observational errors seems to have much more consequences than when considering a domain mainly driven by mesoscale energies. In this Section, we then selected the supervised configuration FP-GENN-MNM + OI, in which the gap-free ground truth is used as target in the learning process, which does not prevent its use for future operational context, since the GENN inputs are still made of purely observational data: along this line, this type of configuration is here similar to the AnDA setup that needs both observation data and gap-free data to be operated.
Discussion In this study focusing on how data-driven and learning-based algorithms may help to improve the reconstruction performance of altimetric fields generally given by a state-of-the-art optimal interpolation (OI) baseline, here provided by the DUACS processing chain, we used two small areas with different energetic dynamics: the 10 • × 10 • GULFSTREAM domain mainly driven by mesoscale processes and the 8 • × 10 • OSMOSIS domain, less energetic but labelled with more small scale structures. Based on the NATL60 numerical simulations  [8] , some experiments were designed in which pseudo observational along-track nadir and wide-swath SWOT realistic datasets are generated. Because the DUACS OI  [7]  of these pseudo-observations is used as the reference, all the investigated methods are applied in a multi-scale decomposition framework where the anomaly dx is seen as the difference between the original field x and the large-scale component x provided by the OI.
Within this combined use of the two datasets, the spectral analysis indicates the new capability to reconstruct spatial scales up to 50-60km which is an important improvement compared to the scales that OI is handling by now; on the other hand, the temporal sampling being less important than nadir tracks, in particular on the GULFSTREAM domain where periods of several days without any SWOT information appears, the reconstruction on these specific periods is sometimes better when learning only with along-track nadir as inputs: we believe that a longer training period (not available here) should improve the behaviour of the NN on this specific issue; • The possibility of neural network methods to learn from the single observations, without requiring any numerical simulations, which is of particular interest on low latitude areas where the Rossby radius of deformation is large, thus requesting an important catalog to efficently retrieve the SSH dynamics over the year.
As it stands, the results obtained are very encouraging: FP-GENN is a "plug-and-play" algorithm whose conceptual use easily enables its implementation on new datasets. Many perspectives have to be considered in the short and medium terms. The configuration of FP-GENN used here aims at minimizing the difference between the true anomaly state of the system dx and its representation ψ(dx) through energy form ||dxψ(dx)|| 2. Alternate energy forms have to be investigated, considering extremes or more generally the whole pdf. In addition, the fixed-point solver used in the joint interpolation approach with GENN never goes too far from the observations, even though they are noisy, which can be an issue in the case of a strong noise including spatial and/or temporal correlations, which was already seen when using SWOT data without any preprocessing (not shown here).
Using a Deep Learning multi-GPU framework and build a pre-operational demonstrator should be of great interestest in the community, as are other SWOT use cases, e. g. using a pre-learning on SWOT data to produce a new interpolation of historical along-track nadir datasets, or taking advantage of the SWOT fast-sampling phase data as inputs for learning prior to its use with SWOT upcoming "operational" data. Last, because the 2D information brought by SWOT showed a significant gain in the reconstruction, a natural extension of this work would be to consider pseudo-observations SKIM datasets [20], whose swath width is more than twice larger (110km vs 270km), and also to propose multivariate analyses including complementary datasets (SST/SSS), already existing in other data-driven schemes like AnDA with an easy extension as additional channels in a neural networks framework.