dataesr/scientific-paragraphs-categorization huggingface dataset:
We focus on news article titles as they are shared on social networks, and aim to discover coherent clusters of events. We first address the problem of document representation and look into how algorithms interact with document features, formulating the hypothesis that they impact the results. Next, we experiment with news story tracking with article titles using two algorithms, one of them supervised and the other unsupervised. The latter is made relevant by the lack of annotated datasets in this research field. We also release the implementation of our tracking algorithms in Python Packages, as well as all the datasets and resources we used 3.
In 2005, the Europe Media Monitor project  [25]  enriched the field of study with new results and strategies to identify emergent topics from press articles. They proposed an approach to track real-world events, not only macro-topics. The newsBrief system is still running 4 and gives a view of trending topics mentioned in news articles. Later on, after 2010, the Event Registry project  [19, 28]  published a multilingual dataset of recent press articles. They used TF-IDF vectors with unsupervised algorithms to group articles related the same news events. The newsLens news tracking system  [17]  also benefits from TF-IDF vectors to cluster documents. The passage of time is materialised by time buckets: it assumes that articles close in time may relate to the same events  [31]. Hence they discovered that buckets of 6 days, with 50% overlap between them are the most suitable parameters to treat group of news articles.
Some time after, Miranda et al. [22]  introduced and described a supervised and streaming algorithm able to cluster press articles into coherent mono-lingual and multi-lingual news stories. A more recent study  [32]  analysed the impact of vectorisation for news tracking algorithms and concluded that TF-IDF vectors provide competitive results and outperform dense vectors computed with doc2vec  [18]. The code of implementations are rarely released but the algorithms are well described and datasets shared with the community to simplify the reproduction of experiments  [17, 22]. Miranda et al. [22]  shared their implementation of their algorithm. In this paper, we will enhance this implementation with a new API and we will implement a baseline proposed in other research articles  [21, 32, 28].
About the experiments performed on short messages, the propagation of the information on Twitter is studied since 2010  [13, 27, 33, 23]. Researchers focused on tweets to track events discussed on the network. Tweets are short messages published on this social network, originally 140 characters long, 280 since 2017  [29]. Most of them used vectorisation to represent documents  [24]  while others used Twitter specific features, such as hashtags, internal links, followers or retweets to characterise tweets  [4]. In this paper, we analyse two datasets from which we only keep news claims, that is to say, news article titles shared on the network, ignoring users reactions. The scientific literature lacks news articles annotated in clusters of topics or stories. In the context of the pandemic, we took advantage of news article titles published on Twitter that are linked to fact checking services.
Datasets Our task consists of building stories from documents written in natural language. We deal with article titles, which are short in size and contain a little amount of information. A suitable dataset for the task of tracking news stories has to provide events or clusters identifiers. In addition to generally used datasets  [28]  in this field, the emergence of Covid-19 datasets from Twitter with references to fact checking services such as PolitiFact 5  is the opportunity to carry out experiments with publicly available resources. Our experiments focus on three available datasets, upon which we present relevant statistics in Table  1  -Event Registry  [28, 10] : a widely known news tracking dataset that has been used in various recent researches  [22, 32, 20]   In the Event Registry dataset, each document is associated with a ground truth cluster identifier.
Not the two others. For them, each tweet is connected to a news claim URI and we use it as a label for clustering analysis. This way, tweets connected to the same URI are considered within the same cluster, so within the same news story. The number of clusters given in Table  1  is the number of distinct clusters, so news stories, given by the labels in the datasets. The issue with Covid-19 outbreak and the struggle against the infodemic resides in detecting false or misleading information in news as they emerge. We twist the purpose of these corpora to apply them to news story tracking.
To that extent, we use four representations to encode documents, two of them are sparse while the other two are Transformer-based dense vector representations. Sparse document representation Usually, TF-IDF weights are computed with the train part of the datasets. Here, we consider documents are handled in a stream and they are unknown before being processed. TF-IDF weighting models then have to be trained before processing data. We build them with huge sets of documents, independent from the data to weight. Different sizes of input documents used to feed the TF-IDF models give different weights. Hence, logically, they will produce different algorithm results. To evaluate the impact of vectorisation, we propose to use different sets of documents to fit TF-IDF models and compute document vectors.
We collected two: one with news articles, the other one with tweets  [12]. News articles come from the Deutsche Welle (DW) website  6  , which is scrapped to extract the title and body. DW is one of the only website that provides content in multiple languages and that is free to query and download. The other one is a collection of tweets, published in English from institutional press accounts. We manually chose press agencies or newspaper that publish on Twitter. There is an API limitation and we can only download 3200 tweets per source 7. To overcome this problem, we selected a high number of press accounts. With this paper, we share the news articles and tweets identifiers with the code that weights the documents of each dataset listed in Section 3. In addition to computing our own vectors, we use the pre-computed ones published by Miranda et al.
[22]  for the Event Registry dataset. We use them to have results comparable to previous works. In their paper, each document is characterised by several vectors of features associated to the text. This means for each document there are several TF-IDF vectors: one for the tokens, one for the lemmas and one for the named entities. For a news article which has a title and a body, there are at least six vectors: three for the title and three for the body. With titles we deal with only three vectors: the title tokens, lemmas and entities. To compute the respective weightings, we fit three different TF-IDF models. One will weight the tokens, another one the lemmas and the third one the entities. For all datasets of Section 3, we compute TF-IDF vectors using both sets of documents we collected: news and tweets.
To extract tokens, lemmas and entities from the titles, we use the spaCy software  8. We only keep GPE, ORG and PER entities, as in Miranda et al. works  [22]. To give an idea of the sizes of the TF-IDF datasets we used to weight the document features, some statistics are shown in Table  2. The software to compute weights is freely shared over the Internet  [6]. Dense Document Representation We use the Transformer architecture, especially the S-BERT  [26]  algorithms and models to encode the title texts into dense vectors. We select pre-trained models that focus on semantic similarity to compute title vectors. Among all the proposed models, we retained multilingual models with the highest scores in semantic search, available at the time of the experiments. They are distiluse-base-multilingual-cased-v1 and paraphrase-multilingual-mpnet-base-v2.
Streaming Algorithm to Build News Stories In this section, we provide more detailed explanations about the Miranda et al. algorithm we use in this article. This latter handles the documents of the dataset as a stream and each incoming document is compared to every existing cluster in the pool of already known clusters. Each existing cluster is a candidate in whom the document might be added if its similarity with the cluster is over a specific threshold T 1. If multiple candidates exist, the one with the highest similarity wins. On the contrary, if no candidate exists, a new cluster is created accordingly. The algorithm handles heterogeneous data with vectorised texts and with timestamps of documents and clusters. The similarity measure sim(d, C) between a document d and a cluster C is detailed below.
Unsupervised News Tracking with K-Means In accordance with suggestions of previous authors  [21, 32, 28]  we use the K-Means algorithm as an unsupervised method to create news stories, with cosine similarity as the distance measure. We propose this algorithm to counter the lack of training data, which are rare in this field of research. To simulate the time that passes, the dataset is split into buckets of sliding windows  [21, 17]. We use the newsLens optimal parameters given in Section 2 with a window of 6 days. We try different configurations, as K-Means is an unsupervised algorithm. An Fig. 1 : Description of the whole process, from the documents in the datasets without features to the clustering evaluation. Two key parts are noticeable: dataset features computation and algorithm evaluation.
Experiments With this paper, we release our implementation of the algorithms  [7]  and the training software  [8]  written in Python. It is a package with its own API. We intend to fulfil the lack of an end-to-end tool that builds news stories from documents. To the best of our knowledge, it does not exist yet. The experiments consist of applying the algorithms mentioned in Subsection 4. 1 with the datasets of Section 3, as described in Figure  1. To evaluate the cluster results and coherence, standard and BCubed  [3]  evaluation metrics are computed. The latter is a more accurate evaluation method of clustering performance. First, we run experiments with the algorithm developed by Miranda et al. [22]  and the results are shown in Table  3. The three datasets are tested with document vectors computed with the sparse and dense models described in Subsection 3.
1. CoAID and FibVid are not multilingual, so we only focus on the English language in the Event Registry dataset. For the news story tracking results, the precision is good with Event Registry and CoAID. The low recall is correlated with a high number of clusters. By creating more clusters, the system focuses on precision, Table  3 : Experimental results of the Miranda et al. [22]  algorithm on all the datasets described in Section 3. Sorted according to F1 BCubed score. hence decreases the recall. Results are constantly bad with Fibvid, which seems not to be a very suitable dataset for this task: it comprises a low number of news claims over the number of user reactions (220 K) we eliminated. We also tracked the necessary time to process the datasets that depends on two factors: the number of documents and the number of clusters found by the algorithm.
As we previously mentioned, each incoming document is compared to every existing cluster and as a consequence, a high number of clusters increases the processing time. When analysing the results of Miranda et al. algorithm on the three datasets, we first notice the F1 scores are all below 75% and the precision is always very high compared to the recall. There is no clear trend in favour of a specific document representation. For Event Registry, MPNet representation is way higher than its competitors, the second one being the TF-IDF document representation based on the News TF-IDF corpus. With CoAID, we have the right opposite, MPNet is the worst one while the TF-IDF representation based on the Tweets TF-IDF corpus, then on the News one give close results. In addition to this clear distinction between Event Registry and CoAID vectors, the algorithm on FibVid behaves differently.
The Silhou- ette coefficient process computes every possible clustering from 1 to the number of documents in the window. With this algorithm and Event Registry, the dense MPNet representation over-performs the sparse TF-IDF vectorisations (with between 9 and 17 points less of F1 compared to News TF-IDF, between 4 and 11 points for Tweets TF-IDF). The difference between the gold clustering results and those given with Silhouette are closer with dense vectors. With CoAID, sparse TF-IDF vectors are better for clustering the dataset, in this case also, the results are close between the gold and silhouette method. With this dataset, dense vectorisation is not an option. Finally, with FibVid, the algorithm behaves similarily to the other one, and the results are close to each other with a F1 score of about 35%.
The Issue of Short Message Similarities Results published in Table  3  and Table  4  are low in comparison to other studies that processed the Event Registry dataset for the same task  [22, 32, 20]. In these works, authors use the whole text and title of the article. Our approach focuses on news article titles, and we wonder how well it is possible to apply news tracking algorithms on this specific type of documents. We compare the document -cluster similarities described in Subsection 4. 1 with two types of datasets: one with titles (the one we are using in this paper) and one that also includes article content text. We display in Figure  2 , the cosine similarities mean and standard deviation for document -cluster pairs belonging to the same news story (in green) or not (in red). This means for the feature f entities all in Figure  2a , the mean similarity for this feature is around 0.
Discussion As we showed in our experiments, it is possible to obtain rather good clustering results with a very little of information contained in article titles. On the other hand, we wanted to evaluate the impact of the document representation, with sparse TF-IDF weightings and dense vectors. It is impossible to conclude on a general trend that would allow us to give a recommendation on whether using one instead of the other. Our results may be considered as illogical, as it is reasonable to state that dense vectors should perform better in any circumstance as they better capture the context, over TF-IDF vectors that are necessarily shorter because of the limited number of tokens in article titles. Our suggestion, when tracking news stories in short messages, especially articles titles is to always test multiple document representations on the dataset in order to select the one that performs best.
Conclusion In this paper, we published an analysis of news propagation with tweets and articles titles coming from public sources. We tackled the issue of applying news tracking algorithms on short documents: article titles. We took advantage of the Covid-19 infodemic to twist the purpose of datasets dedicated to true and false news detection. We proposed to use the supervised algorithm released by Miranda et al. [22]  to build stories from tweets when there are training data. For cases when they are missing, K-Means is a suitable unsupervised algorithm. We experimented the impact of document representation, with sparse TF-IDF vectorisations based on two corpora, and dense vectorisation with the Transformer architecture. We showed that the representation of a document is a major issue sometimes neglected in the literature.
With this article, we release all resources: the code of the algorithms and the sets of data we collected to vectorise documents. We share with the community our implementations to let anyone reproduce our results and experiment with private datasets. We showed one of the reason why clustering article titles works worse than when also taking the article content into account. Short messages do not contain enough discriminant data. We also lacked big datasets qualitatively annotated with events. By analysing Fibvid, we notice the quality of primary data could be the explanation of the rather bad results it produces. On another hand, we computed the dense vectors with multilingual models  [26]. These vectors are aligned in multiple languages, providing similar vectors for similar semantic in different languages.
Acknowledgments This work has been supported by the  European Union 's  Horizon 2020 research and innovation program  under grants  770299  (NewsEye) and by the  ANNA  project funded by the  Nouvelle-Aquitaine Region. The research work by  Paolo Rosso  was partially funded by the  Generalitat Valenciana under DeepPattern  ( PROMETEO/2019/121 ). The authors would like to thank the  Polytechnic University Of València (UPV)  ,  Spain , which made this work possible, and its  IT laboratory, DSIC.
We also reprocess the 20Hz along-track data from Jason and Sentinel-3a Geophysical Data Records (GDR) with the best correction parameters in the area. These two SSH time series (i. e. in situ and altimetry) allow us to compute the altimeter biases time series over the entire Jason and Sentinel-3a period. With our 3 weeks field campaign, we reanalyse about 20 years of altimetry observations and find inter-mission biases consistent with historical calibration sites, thus further increasing our knowledge of the local sea level rise in this region. This offers many opportunities to develop Cal/Val activities in the lagoon, which is also the subject of several experiments for the scientific calibration phase of the future large-swath altimetry mission SWOT. Short summary. Altimetry satellite are essential to monitor and understand sea level evolution around the world with rates accuracy of mm/year.
Noumea tide gauge long term measurements The French Hydrographic Service (Shom) provides sea level observations at Noumea through the Chaleix (operating from  and Numbo (2005 to present) tide gauges (Fig. 1a , blue dots). Before 1967, measurements were paper records, and electronic observations began in 1967. Thanks to a 6 months overlap of data collection, the old Chaleix site has been linked to the new Numbo site, located about 6 km away (Fig. 1a , blue dots). Aucan et al. , 2017  were thus able to reconstruct the whole time series by concatenating data from 1957 to 2018, making it one of the longest series available in the South Pacific. In this paper, we used the data available online (http://uhslc. soest. hawaii. edu/data/ -ID 019) and regularly updated with the latest measurements from Numbo tide gauge.
Jason and Sentinel-3a Geophysical Data Records (GDR) 210 There is a large diversity of altimetry products and sources. In order to have homogeneous data, we tried to use the most recent data with the same format over the whole comparison period. For the Jason track #162, we use the last Geophysical Data Records (GDR) delivered by the AVISO+ FTP, that integrate precise orbits and up-to-date corrections for 20 Hz measurements (Table  2 ). For Sentinel-3a dataset, the SRAL Level 2 Marine data were used to ensure a global coverage of the lagoon. These data are available on the EUMETSAT portals  (until September 2022). From 2016 to 2019, the Sentinel-3a data were 215 reprocessed using up-to-date standards of the Baseline Collection 004, used for Sentinel-3a products after 2019 (Table  1 ).
In the lagoon, the effect of the land contamination is visible when approaching the main island, but at our comparison point, the radiometer correction seems to be exploitable for both Jason and Sentinel-3a missions (more details in Appendix D). To confirm this hypothesis, we also test to other datasets: (1) a wet tropospheric delay provided by the European Center for Medium Range Weather Forecasting (ECMWF) and (2) a wet tropospheric correction computed from inland permanent GNSS stations (more details about this processing in Appendix D). When comparing with the in situ observations, we will be able to analyse the impact of these different solutions. Finally, altimetry satellites do not fly over the exact same point at each pass: it is therefore necessary to consider the height difference between the comparison point and the actual pass of the satellite track, which we approximate to the geoid height difference between the 2 points (∆ ï ).
Evaluation of the in situ SSH determination method To evaluate our methodology for the    reconstruction, we compared the mean bias estimated using the 2019x pressure sensor measurements with the one computed using our method (i. e. Eq. 1) over their common observation period (from October 2019 to November 2020). Figure  7  shows the evolution of the altimeter bias for the Jason and Sentinel tracks according to the in situ data considered. For the 3 tracks, the difference between the mean biases is a few millimetres (respectively +3/+5/+5 mm for the tracks #162/#359/#458). Despite these small differences, we can notice centimetric variations in the time series of differences (lower right panels, blue curve). This variability can be partly explained by differences in the lagoon hydrodynamic between the location of the tide gauge and the pressure sensor.
Long-term altimetric bias evolution Using the Noumea tide gauge data, we computed the Jason altimeter bias for all data from 2002 to 2021. The absolute bias estimates from our study are detailed in Fig. 9  for Jason 1 (+90. 2 ± 2. 6 mm), Jason 2 (+65. 2 ± 3. 0 mm) and Jason 3 (+39. 3 ± 3. 4 mm). As for Sentinel-3a, these values are higher than those observed at historical calibration sites (Table  3 ). When comparing to the Corsica and Bass Strait sites which are in really good agreement (see Table  3 ), our results show a mean difference of about +46mm (J1), +48mm (J2) and +46mm (J3). When regarding the resulting inter-mission biases, we find -25 mm for J2/J1 and -25. 9 mm for J3/J2, which is close to the resulting inter-mission bias for Corsica (resp. To the first order, the altimeter bias, differences between altimetry sea level variations and those seen by tide gauge (see Eq.
4), can be related to Vertical Land Motion (VLM) at the tide gauge site  (Wöppelmann and Marcos, 2016). We therefore analysed the linear trend estimated on our altimeter bias time series to compare with the vertical motions of nearby GNSS stations. In order to create the longest possible time series, we applied our inter-mission bias (i. e. -25 mm for J2/J1 and -25. 9 mm for J3/J2) to realign the datasets to Jason 3 measurements. Over the whole Jason period  (2002) (2003) (2004) (2005) (2006) (2007) (2008) (2009) (2010) (2011) (2012) (2013) (2014) (2015) (2016) (2017) (2018) (2019) (2020) (2021) , a linear trend of -0. 13± 0. 12 mm/year is estimated. To get a more robust estimate of this trend, we used a bootstrapping method, which consists in estimating the trend 200 times on a random sample of 85% of the original series.
It is important to note that this trend is sensitive to the inter-mission biases applied: for example, using Bass Strait inter-mission biases (i. e. -27/-26 mm instead of -25/-25. 9 mm), a zero linear trend is estimated. This being said, our results do not show any significant uplift in Noumea. This differs from the conclusions of  Aucan et al. (2017) , that find an uplift of +1. 41 ±0. 67 mm/y over the altimetric period  inferred from the difference between satellite altimetry and tide gauge. The difference likely originates in the method used by  Aucan et al. (2017) , where the satellite altimetry time series was extracted from a multi-mission gridded dataset at a point far outside the lagoon, before being compared to the tide gauge. Section 4. 2. 2 showed that, even being only a few km apart, there is SSH differences between the tide gauge and the pressure sensor: the difference with a point outside the lagoon can therefore be even greater.