dataesr/scientific-paragraphs-categorization huggingface dataset:
No prior knowledge of the nature of the coming data (format, structure, features, etc. ) 2. No prior knowledge of how many categories can the data be classified into (number of clusters is unforeseen) 3. The clusters probably will evolve with time (keep changing dynamically, i. e. , creating, removing, splitting and merging clusters). Since we are interested mainly in hierarchical clustering, we studied the machine learning libraries within the scope of this algorithm. Datumbox  [33]  is a robust framework that provides different functions like Sentiment Analysis, Twitter Sentiment Analysis, Subjectivity Analysis, Topic Classification, Language Detection, Keyword Extraction, Text Extraction and Document Similarity. At low-level, the basic machine learning algorithms such as K-means, hierarchical clustering, and classification algorithms perform the above functionalities.
SPMF  [35]  offers implementations of 120 data mining algorithms for association rule mining, item set mining, sequential pattern mining, and of course clustering and classification. It works only with numerical data and this was mentioned explicitly in the documentation. The input is a set of vectors containing double values only, a parameter "maxdistance" and a distance function. This implies the same limitation that Spark has. To the best of our understanding, this shortcoming is obvious because, the clustering is usually done according to Euclidean or Manhattan distance functions that need numerical data to be applied. WEKA  [36]  is a widely-known library. It is an integrated system which consists of a collection of machine learning algorithms for data mining tasks. The algorithms can either be employed directly for a dataset or called from within a Java code.
Datum box had a specific structure for storing and processing data and it lacks the ability to read data by lines. It can handle data as a batch that can be read in one go and thus it was not suitable for real time environments. WEKA developers extended it into MOA, the version that fits real time processing but it is still very recent and thus had some code bugs that did not allow us to benefit from it  5. Besides that, MOA is locked into a specific file format which is ARFF and hence it is unable to read other data format. An ARFF format needs to have attributes, types, and data explicitly mentioned within the file. Nevertheless, in our case the data streamed and fed into the algorithm do not necessarily have an attribute. Rather, data could be a set of records each of which is made up of different text words.
IBRIDIA -Solution Overview In this section, we describe the two main modules of IBRIDIA. In logistics system, we have data generated from internal system for stock, orders, shipments, etc. Also, there is a need to collect and analyze data from external sources in realtime especially to monitor the different statuses of the delivery. To address both needs, within IBRIDIA, we developed a batch style data processing engine that we called ProLod and a realtime data processing engine that we called RePLoD. We explain these two modules in the following subsections. Then, we describe the data processing model that IBRIDIA relies on for both realtime and batch style data processing. Figure  2  depicts a high-level architecture of IBRIDIA. In IBRIDIA there are four components: data streamer, the storage, batch style data processing engine, and real-time data processing engine.
It also fetches data from external sources such as Twitter, traffic sensors, weather sensors, Facebook and other social medias. In addition, IBRIDIA's processing components extract archived sensor data of completed logistics processes. In most of the cases, we found that data extraction from internal sources is more trivial than external ones. Additionally, internal data were transferred faster than the external ones. IBRIDIA can collect structured and unstructured data. For instance, it collects unstructured texts from Twitter, and Facebook and structured business process data from logistics information system. • Data Filtering: It refers to a wide range of strategies or solutions for refining data sets. Datasets are refined into simply what a user (or set of users) needs, without including other data that can be repetitive, irrelevant or even sensitive.
ProLoD and RePLoD aim to eliminate all possibilities of data overloading which can increase computational cost and effort during data processing and may jeopardy the analysis regarding accuracy. They collects data that are related to logistics and specifically the data chunks whose hashtags (the words prefixed by #) determine direct and indirect connections with transportation, delivery, logistics, shipment, etc. Consider the term "protest" which may be a political protest or else but can have a great impact on delivery of goods and hence can delay the delivery. However, consider a tweet "the New York stock prices are extremely high today" which will be removed by the data filter because it does not carry any information related to logistics processes. • Data Cleaning (i. e. data scrubbing): It is the process of detecting and correcting (or removing) corrupted or inaccurate records from a record set, table, or database.
ProLoD and RePLoD clean data from all unwanted symbols, numbers, stopping words, hashtags, and any other data items that might lead to noise and cause inaccuracy. Figure  3  shows an example of cleaning Twitter data using ProLoD. • Data Integration: In IBRIDIA, data integration is performed in two steps. In the first step, the data are transformed from source to target serialization format. Currently, the target format is CSV. The second step is merging the transformed data. • Data Storage: This step aims to deal with the storage of the integrated datasets. After preparing the integrated datasets, ProLoD and RePLoD store data into the storage.
We choose clustering for our model because the objective function is expected to produce a clustered dataset which facil-itates efficient analysis in prediction of delivery delay. Clustering is a process of grouping or segmenting data items that are similar between them in a cluster and dissimilar to the data items that belong to another cluster  [41]. There are different types of cluster models which are grouped into Connectivity models, Centroid model and Distribution models, Density models, Subspace model, Group model, and Graph-based models  [42]. We are interested in techniques used for building connectivity model which fits to our objective more than the others. Hierarchical clustering is a widely used approach for building connectivity model based on distance connectivity between the data items.
Definition 3. Average distance: d(C i , C j ) = x∈C i ,x ∈C j d(x,x ) |C i |•|C j |. Definition 4. Wards method d ij = d({X i }, {X j }) = X i -X j 2 is the sum of squared Euclidean distance is minimized. The iteration is continued by grouping data items until a cluster is formed. As mentioned earlier that the clusters are presented graphically by a dendo- • Step 6 : Repeat until ONLY one cluster remains. gram In  [3] , we reported several disadvantages of the basic agglomerative clustering algorithm. In particular, undoing is not allowed and the time complexity is O(n 2 log n) where n denotes the number of data points. For a large dataset, the performance with respect to processing time may not be satisfactory. Based on the type of distance matrix chosen for merging, different algorithms may have one or more of the following drawbacks: (i) sensitivity to noise and outliers, (ii) partitioning a large cluster, (iii) difficulty in handling different sizes of clusters and handling convex shapes.
Implementation of IBRIDIA We studied various technologies for implementing IBRIDIA. We investigated existing libraries for data extraction, filtering, and transformation. Our goal was to reuse existing ones instead of developing the new ones. Also, we studied machine learning libraries including DatumBox 6 , SPMF 7 , Massive Online Analysis (MOA), and Spark MLib 8 to implement our data processing model. From our study, we found that existing libraries could not be used to implement our model (discussed in the previous section). Therefore, we decided to implement the model on our own. For implementation, we used Java language on Eclipse. To sum up, IBRIDIA is a framework that integrates three APIs for extracting external data from different sources including Twitter API, Facebook API, and Open Weather API.
It uses an open source parser. Also, it includes tools for cleaning and transforming incoming data. The prototype of ProLoD is available in GitHub. We investigated different data processing frameworks including Apache Spark  9  , and Apache Storm 10  to develop RePloD module of IBRIDIA. To the best of our understanding Storm is more potential computation system for our ProLoD. It is fast, can process over a million of tuples per second. It is scalable, fault-tolerant, guarantees our data will be processed, and is easy to set up and operate. Storm integrates with the queuing and database technologies we already use. A Storm topology consumes streams of data and processes those streams in arbitrarily complex ways. However, repartitioning the streams between each stage of the running computation is needed.
Durability is provided by replicating messages to different brokers. Apache Kafka provides four different types of APIs. The Producer API allows RePLoD to publish a stream of records to one or more Kafka topics. The Consumer API allows RePLoD to subscribe to one or more topics and process the stream of records produced to them. The Streams API allows RePLoD to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams. The Connector API allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table. The data processing topology of RePLoD comprises the data cleaner, the data filter, the data transformer, and the data clustering component.
We implemented two different versions of our model: real-time and Batch style. We tested both versions with a test dataset. RePLoD reads data by records and clusters each incoming record. The clusters are mutable; a cluster may change when a new record is added in the cluster. However, since the algorithm is greedy, the execution time has a positive correlation with number of records, i. e. , the execution time increases as the number of records increases. This can be solved using the scalability of the system by inserting more nodes to the cluster for faster processing resources. Table  1  shows the result of our experiment with the realtime version "RePLoD". The batch-style ProLoD performs bulk reading and clusters batch data. The reading and processing occur only once per batch. Table  2  shows the results of batch style version "ProLoD".
We compared the performance of both versions. Figure  5  shows the comparison. According to our experiments, we observed that the batch-style version performs better than the real-time version of IBRIDIA. Our study reveals that performance of the real-time version was not satisfactory due to the centralized environment of the experiment consisting of only one node. We believe that performance will improved significantly in a distributed and scalable computation framework with multiple nodes. We believe that technologies which we used for implementation of our solution called Apache Storm has high computational power; hence, it can process a huge number of records within a unit of time e. g. , a millisecond. In addition, we assume that the performance of the batch-style version of ProLoD may decrease if the dataset is large.
Currently, the dataset is small; therefore, the size of each batch is small and faster in processing (clustering). We compared our model with the hierarchical clustering algorithm of WEKA. The WEKA algorithm produced 7 clusters with 93 records. Table  3  shows the result which is also visualized in Figure  6  in-which the distribution of the records is also presented as percentage for each cluster separately. Furthermore, we observed that both real-time and batch style versions of ProLoD produced consistent and representable clusters that will assist in exploring data, discovering insights, and supporting predictive analytics when data distribution is observed. Table  4  shows the result which is also shown in Figure  7  in-which the distribution of the records is also presented as percentage for each cluster separately.
According to our analysis, any labeling process using any of the attributes from the dataset will produce results that are representable and understandable (by representable we meant a reasonable number of clusters produced by the algorithm). Many of the clusters produced by WEKA intersect with other clusters while treated independently. For example cluster 1 and cluster 2 do not need to be represented as two different clusters and cluster 4 and cluster 7 intersect due to the fact that they have a common value called Major Event of the attribute type of event. According to our observation, the separation of clusters (if applicable) is more effective than clusters with intersection over common values such as Major Events. Some clusters are misclassified and not representable, according to our observation.
Conclusion In this paper, we presented a framework called IBRIDIA for processing logistics data. These data are stemming from various internal and external sources within the logistics domain. The proposed framework integrates several data preparation and processing functions. The output of such functions is a clustered dataset. Afterward, the analytical engine exploits this dataset to perform predictive analytics. In order to develop our data processing model, we studied different machine learning algorithms; eventually, we choose Johnson's hierarchical clustering algorithm. However, we modified the algorithm to become a stream clustering algorithm that supports incremental grouping of text messages according to their similar characteristics directly on the go. We presented the solution architecture of IBRIDIA which consists of components for performing data preparation tasks such as collection, filtering, cleaning, integration, and data storage.
e. , the Jacobians. Therefore, RTTOV-gb represents the forward model needed to assimilate ground-based MWR data into numerical weather prediction models, which is currently pursued internationally by several weather services. RTTOV-gb is fully described in a previous paper (De Angelis et al. , 2016), while several updates are described here. In particular, two new MWR types and a new parameterization for the atmospheric absorption model have been introduced since the first paper. In addition, estimates of the uncertainty associated with the absorption model and with the fast parameterization are given here. Brightness temperatures (T B ) computed with RTTOV-gb v1. 0 from radiosonde profiles have been compared with ground-based MWR observations in six channels (23. 8, 31. 4, 72. 5, 82. 5, 90.
This paper introduces several updates of RTTOV-gb since its first development  (De Angelis et al. , 2016). In Sect. 2 we introduce a new absorption model parameterization and two new sensors that have been added among the setting options. Section 3 presents the evaluation of RTTOV-gb against the reference line-by-line radiative transfer model and real radiometric ground-based observations. Section 4 summarizes the findings while the code and data availability section provides instructions for code and data access and use.
Absorption model Similar to RTTOV, RTTOV-gb is a parameterized atmospheric radiative transfer code. In the microwave region and for clear-sky conditions, the parameterization only affects the atmospheric gas absorption. This means that the optical depth of each layer is only due to absorption by atmospheric gases (mainly oxygen, water vapor, and nitrogen). The parameterization consists of the fact that the layer optical depth is not computed from a complex line-by-line (LBL) absorption model  (Clough et al. , 2005)  but rather from a simplified parameterized model. The simplified model consists of a linear regression, which relates the layer optical depth to predictors derived from the layer atmospheric thermodynamical properties (i. e. , pressure, temperature, and humidity). The regression coefficients are computed off-line from a diverse training dataset of atmospheric thermodynamical profiles and corresponding optical depths computed with an LBL model.
Radiosondes usually reach up to 10 hPa (∼ 30 km altitude), leaving the five uppermost RTTOV-gb levels to be covered with climatological profiles. This has negligible impact on ground-based radiance calculations. The LWP-U72-82 instrument has four channels (23. 84, 31. 4, 72. 5, and 82. 5 GHz), and it is mainly used for radio propagation studies. The available dataset extends for 1 month (from 16 June to 15 July 2018), corresponding to relatively moist midlatitude summer conditions. The dataset includes radiometric observations and pressure, temperature, and humidity profiles measured by radiosonde ascents launched twice daily from the Milan Linate airport (∼ 20 km from the Polytechnic University campus). Radiometric observations are collected at a fixed elevation angle (35. 3 • ), matching the direction of the Alphasat telecommunication link.
Absolute MWR calibrations were performed 9 months earlier and 4 months later than the period under study, showing no substantial change in the calibration coefficients. Thus, we assume the calibration was stable during the period under study. An example of data is shown in Fig. 3  for 3 consecutive days. Here, T B observed at the four channels is plotted together with RTTOV-gb simulations and their estimated uncertainty. It appears that simulations usually fit the observations within uncertainty, except for periods with clouds (at ∼ 167. 0, i. e. , 00:00 of 16 June) and rain (∼ 169. 0, i. e. , 00:00 of 18 June). This is expected as RTTOV-gb simulations are computed from radiosonde measurements, which do not provide hydrometeor content and thus do not take into account the radiative contribution of clouds and rain.
2 K. Thus, data identified as cloudy, by the standard deviation of T B (31. 4 GHz) over a 10 min pe-riod being larger than 0. 2 K, have been discarded. The cloud and rain screening reduced the dataset by ∼ 33 %, leaving 40 matchups between clear-sky radiosonde and radiometric observations (averaged within ±5 min from the radiosonde launch). Scatterplots of simulated vs. observed T B at 35. 3 • elevation for the four channels of LWP-U72-82 are shown in Fig. 4. Note that the correlation coefficient is 0. 98 for all four channels. The slope is within 5 % for all channels but 72. 5 GHz (∼ 8 %), for which the difference between observations and simulations tends to increase as T B decrease. This may be due to an issue with the instrument gain calibration as well as to increasing uncertainty for this channel at lower temperature and moisture conditions (see Fig.
2 ). Statistics at 23. 84 and 31. 4 GHz are of the same magnitude as those reported by De Angelis et al. (2017) at 30 • elevation (their Fig. 5c ). The LWP-90-150 instrument has two channels (90. 0 and 150. 0 GHz), and it is mainly used for the retrieval of total column cloud liquid water content. The instrument considered here has been running at the ARM SGP central facility between November 2006 and November 2013, performing calibrations regularly using the tip curve method  (Cadeddu et al. , 2013). Here we exploit a 2-month dataset of radiometric and radiosonde observations (ARM, 2018a, b) collected in January-February 2012. This dataset corresponds to relatively dry midlatitude winter conditions. An example of data is shown in Fig. 5  for 3 consecutive days, corresponding to a dry clear-sky period with intermittent thick clouds and rain.
Observations flagged by the rain sensor have been discarded. In addition, cloudy conditions have been identified with the same approach as described above, i. e. , setting a threshold on the 10 min standard deviation of T B at a window channel, here replacing the 31. 4 GHz with the 90. 0 GHz channel. However, since T B (90 GHz) has ∼ 6 times larger sensitivity to water vapor  (Cimini et al. , 2007) , the clear-sky thresh-old is increased by the same factor, i. e. , 1. 2 K. Thus, data with 10 min standard deviation of T B (90 GHz) larger than 1. 2 K have been discarded. The cloud and rain screening reduced the dataset by ∼ 26 %, leaving 173 matchups between clear-sky radiosonde and radiometric observations (averaged within ±5 min from the radiosonde launch). Scatterplots of simulated vs. observed T B at 90 • elevation for the two channels of LWP-90-150 are shown in Figure  6.
The experimental bias is generally larger than the simulation-estimated uncertainty, as one would expect since the observations are also affected by uncertainty. Except for the 72. 5 GHz channel, the estimated uncertainty and experimental bias are within 0. 5 K, which corresponds to the absolute T B accuracy claimed by the manufacturer for the LWP radiometer series. At 72. 5 GHz, as anticipated, observation-simulation differences tend to increase as T B decreases, possibly due to either conditiondependent uncertainty or an issue with the instrument gain calibration. This will be the subject of a future investigation. 4 Summary and future developments RTTOV-gb v1. 0 is now freely available, after website registration (see section code and data availability). The updates with respect to the original development  (described in De Angelis et al.
3 and 90 • ). Differences between simulated and measured T B are within uncertainty as expected from instrumental and simulation contributions. Future developments include additional sensors (e. g. , at 183 GHz) and characterization of liquid water absorption uncertainties. We expect this paper will provide a reference for the exploitation of RTTOV-gb for MWR data assimilation into NWP models, as has already started at some meteorological services in Europe as well as in other continents. Code and data availability. RTTOV-gb v1. 0 is available to licensed users free of charge. RTTOV-gb may be obtained by registering (https://www. nwpsaf. eu/site/register/, last access: 14 November 2018) with the NWP SAF website (https://www. nwpsaf. eu/, last access: 14 November 2018, NWP SAF, 2018) and then selecting RTTOV-gb in your software preferences.
Instructions for compiling and running RTTOV-gb are provided in the RTTOV-gb User Guide within the software package. The software package also includes scripts to verify the installation and FORTRAN code examples for running the RTTOV-gb forward and K (Jacobian) modules. RTTOV-gb is designed for UNIX/Linux systems. The software is now successfully tested on the following architectures and Fortran 90 compilers: Intel systems with gfortran, ifort, NAG, and pgf90, and Apple Mac systems with gfortran. The RTTOV-gb v1. 0 code is based on RTTOV v11. 2 and the programming interface is identical to that version of RTTOV, though some inputs and outputs are not used by RTTOV-gb. The original RTTOV v11. 2 can be obtained from the NWP SAF website (http://nwpsaf. eu/site/software/rttov/ rttov-v11/, last access: 29 April 2019, Hocking and Rundle, 2019).