dataesr/scientific-paragraphs-categorization huggingface dataset:
The increasing availability of big Automatic Identification Systems (AIS) sensor data offers great opportunities to track ship activities and mine spatial-temporal patterns of ship traffic worldwide. This research proposes a data integration approach to construct Global Shipping Networks (GSN) from massive historical ship AIS trajectories in a completely bottom-up way. First, a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm is applied to temporally identify relevant stop locations, such as marine terminals and their associated events. Second, the semantic meanings of these locations are obtained by mapping them to real ports as identified by the World Port Index (WPI). Stop events are leveraged to develop travel sequences of any ship between stop locations at multiple scales.
This research proposes a data integration bottom-up approach to construct shipping networks from massive dynamic AIS messages at a global scale. AIS messages contain rich ship movement information such as location, speed, and course  [10] , which together could reflect the global shipping network and cargo flows. Since ships usually stay at a given terminal for hours and send a large number of AIS messages, it would be possible to use some density-based unsupervised clustering method to identify terminal locations. In order to do so, we applied a DBSCASN (Density-Based Spatial Clustering of Applications with Noise) to identify terminal locations and ship stops. Terminal locations are further mapped to ports, cities, and countries as given by public datasets. Together with ship stops at each of these locations, we explore the travel sequences and shipping networks at multiple scale levels, and relate individual AIS locations to higher level locations such as terminals, ports, and countries.
Centrality analysis are performed at the terminal, port, and country levels and characterize the main maritime nodes and routes, while the most important connections between maritime nodes are also identified. The contributions of this research are as follows. First, it provides a novel approach to construct GSNs at multiple spatial levels, as well as a tree-based hierarchical structure to link bottom-level ship movements to high-level stop locations and shipping networks. Second, the approach develops a terminal level global shipping network based on AIS sensor data that provides a local view of the trajectory and stop patterns at the local port level. Third, this approach enables us to track the detailed movements and behaviors of each ship between any two terminals and ports, with this also being valuable for further applications evaluating ship emissions or extracting the main routes among terminals and ports.
Li and Zheng et al. [23]  developed a similar algorithm, but it is based on distance and time duration among trajectory points. DBSCAN has been widely used to identify stop points and stay locations detection. It performs relatively well to detect clusters with arbitrary shapes. It has proven to be computationally efficient for large datasets and also requires little inputs from domain knowledge  [24, 25]. Palma et al. [21]  use DBSCAN to detect stops by clustering low speed points into a single trajectory. The original DBSCAN algorithm has been also extended to take both the temporal and spatial features into consideration  [14, 26]. Gong et al. [27]  apply an improved version of DBSCAN (C-DBSCAN), which takes both time constraints and direction variations into account, and a Support Vector Machines (SVM) is applied to detect both stop status and stop types from trajectories.
Density-based clustering methods are also used to mine parking locations  [28]  and travel sequences of mobile objects  [15]. Although trajectory mining research is mainly applied to urban contexts  [13] , it appears with the increasing availability of ship trajectories, as extracting insights and knowledge from ship trajectories has drawn increasing attention. For example, Yan et al. [29]  detect stay locations and main routes within the Port of Singapore from ship trajectories based on a modified DBSCAN which takes the location, speed, and direction of a vessel into account. Zhang et al. [30]  investigate the spatial-temporal patterns of ship traffic in the port of Singapore based on large AIS datasets. De Souza et al. [31]  develop an approach to identify fishing activity from Satellite-based AIS.
Chen et al. [32]  estimates the throughput of major container ports from containers ship trajectories and other open data sources. Ship trajectories are also used to identify ship stationary events and container handling events for measuring the performance of major container ports  [33]. Although DBSCAN is indeed used in this study, the peculiarity of our approach is its multi-level component. In fact, most of current approaches are either applied at the port and local levels or at the global scale. For instance, several researches  [29, 30]  have studied ship behaviors at the port level, but did not extend the principles of their approach to larger scales, as the motivation and application of these works is different from our work. Conversely, while several recent works investigate global shipping and maritime logistics and attempt to generate global shipping networks of different ship types based on port records or services schedules of major carriers, these studies do not consider local and global terminal-level transportation patterns or specific ship behaviors at the local level.
We consider that many large ports, such as Shanghai, consist of several geographically separated terminals. This is an important property to analyze ship behaviors and activity at the local and regional levels. Generally, and despite the specific interest and contribution of these works, the full potential of AIS data at both the global, regional and local scales is not completely addressed. In fact, current AIS data-driven researches are mainly oriented toward the analysis of trajectory patterns at either the regional and local levels. However, they generally cover a relatively small number of world ship fleet. In addition, they apply the analysis of world fleet patterns or global shipping trends, as well as local patterns at the port and terminal levels. Although a few recent works investigate global shipping and maritime logistics and attempt to generate some global shipping networks of different ship types based on port records or services schedules of major carriers, these studies still do not model local and global terminal-level transportation patterns, as well as specific ship behaviors at the local level.
Data Developing a global shipping network requires a sound data integration approach from multiple datasets. The first dataset considered is given by a global AIS dataset that both includes coastal-based AIS and satellite AIS messages. This dataset contains the detailed and complete movement records of ships worldwide in 2015. Each record contains detailed information about its associated ship and journey. We mainly consider dynamic and geographical data from AIS dataset, including ship's Maritime Mobile Service Identity (MMSI), a unique identification number, navigation status, Speed over Ground (SOG), Longitude, Latitude, and Timestamp-Coordinated Universal Time (UTC) second of the AIS message generated. Ships usually send AIS messages every 2 to 10 seconds when they are at underway (the faster the speed and course, the higher report frequency) and every 3 min while they are at anchor  [10].
Ships over 300 Gross Tonnage (GT) engaged in international voyages, cargo ships more than 500 GT in inland voyages, and all passenger ships regardless of size are required to install an AIS transponder on shipboard by International Maritime Organization  [34]. The second dataset is a global ship database which contains the detail information of more than 100,000 vessels all over the world, such as ship MMSI, Call Sign, name, service speed, dimensions, Dead Weight Tons (DWT), GT, and so on. Based on these MMSI, the ship characteristics can be related to the AIS database and then detailed movement and the physical characteristics of each ship. The third dataset is the World Port Index (WPI), which is published by the National Geospatial-Intelligence Agency (NGA) and publicly available. WPI contains the geolocation, physical characteristics, and facilities and services offered by approximately 3700 major ports worldwide  [35].
Many ports, for instance, the Port of Shanghai, consist of several terminals, and the center location of its host city is typically used to represent the port location. In addition, WPI also provides the associated regions and countries of these ports. The last dataset is the Global Administrative Areas (GADM) database, which is freely available for academic and other noncommercial use, and is mainly maintained by the University of California. The GADM is a spatial database of the geolocation of the administrative boundaries worldwide and contains countries and lowers level subdivisions, such as provinces, cities, and counties. The current versions of GADM includes 386,735 administrative areas. WPI and GADM datasets are mainly leveraged to add semantic meanings to ship stay locations derived from AIS database in order to form a multi-level structure of these locations.
Methodology The overall process applied to derive a GSN is shown in Figure  1. It mainly consists of five steps. The first three steps generate and construct the network nodes, while the last two steps derive the network links. Specifically, the first step identifies the terminal candidates based on the geographic locations of the ships extracted from AIS datasets. The second step is to detect the stop events of each ship at each terminal candidate according to the temporal characteristics of the AIS data. The next step is to obtain the semantic information of the selected terminals by mapping these terminals to its related ports and countries based on the WPI and GADM reference datasets, which are publicly accessible. The fourth step gives a label to these candidates based on their traffic distribution derived from stop events and satellite imageries and maps offered by service providers.
Terminal Candidates Identification Marine terminals are important stay locations that could be extracted from massive historical AIS messages. A first step is to reduce the total number of data points by removing nonsignificant trajectory points. Ships are often static when stopped at terminals, which means that not all AIS points are equally useful. Therefore, we first consider AIS locations whose speed is null and navigation status is equal to "moored" as inputs. Abnormal locations mainly refer to locations with either longitude and latitude irrational values, and implausible locations (i. e. , ground locations). They are all detected by the DBSCAN algorithm. Next, the overall computing time should be minimized. Gridding approaches have been notably used to downsize a given dataset for clustering.
This approach is similar to a quadtree algorithm. In order to control the number of points in each grid, the size of the grid can be adapted according to the application context. Specifically, AIS points in the dataset are divided into a set of nonoverlapping geographic grids, so that the total number of points in each grid does not go beyond a given threshold (thre_n). The ocean surface is initially broken into a set of 1x1 degree grids, then each AIS point is grouped into a grid based on its geolocation. Next, the total number of points of each grid (grid_n) is calculated. When a grid_n of a given grid is higher than the thre_n, this grid is broken into 10 sub-grids. This process does not stop until all grids satisfy the constraint of the thre_n. Consequently, all grids will contain a relatively small number of points, with this being an asset for further query manipulations.
This approach is similar to a quadtree algorithm. In order to control the number of points in each grid, the size of the grid can be adapted according to the application context. Specifically, AIS points in the dataset are divided into a set of non-overlapping geographic grids, so that the total number of points in each grid does not go beyond a given threshold (thre_n). The ocean surface is initially broken into a set of 1x1 degree grids, then each AIS point is grouped into a grid based on its geolocation. Next, the total number of points of each grid (grid_n) is calculated. When a grid_n of a given grid is higher than the thre_n, this grid is broken into 10 sub-grids. This process does not stop until all grids satisfy the constraint of the thre_n. Consequently, all grids will contain a relatively small number of points, with this being an asset for further query manipulations.
For example, time can be modelled as a series of time slots with equal time length, and each point at these locations is grouped into a specific time slot based on its timestamp, while only one point for each slot is retained. The third step is to identify the terminal candidates at each grid. A typical terminal usually contains several berths and is frequently visited by many ships. Each ship may visit and stay at a terminal for at least several hours to unload or upload cargos, which results in a considerably large amount of AIS messages. Consequently, the density of AIS points at the terminal is likely to be higher, as well as much higher than nearby locations, as illustrated in Figure  2. DBSCAN is an unsupervised, density-based clustering method particularly developed to detect arbitrary shape from big datasets with noises.
It is applied here for terminal identification. The DBSCAN algorithm is fast, robust, and suitable for the mining tasks considered. The main peculiarity of our work lies not only in the multi-level application, but also in the way the DBSCAN parameters are selected. This reflects the fact that the domain knowledge is taken into consideration. These properties are crucial to detect terminal locations, as terminal shapes largely vary from port to port. Moreover, since AIS data are often not 100% reliable and noisy points that can be suppressed, this is a secure property to further develop our project. It has been observed that AIS data might be not completely reliable when considered at the local and individual levels, and this can happen for many reasons (e. g. , communication errors, AIS data not always emitted or received).
However, the advantage of our framework is that the large trajectory datasets considered are much more reliable for statistically analyzing the patterns that appear at both the global, regional, and even local scales. Furthermore, the application of the DBSCAN algorithms is noticeably efficient in dealing with large datasets with local noisy data DBSCAN requires two parameters. The first one denotes the search radius (eps) to search for nearby neighbors. The second one refers to the minimum amount of points required (minPts) to form a core point. The clustering process of DBSCAN starts with an arbitrary point (p) and then searches for points located within the eps-neighborhood of p, denoted by Neps(p). If the number of points |Neps (p)| >= minPts, then p will be qualified as a core point and a cluster is identified.
The algorithm then evaluates the next point and further expands the cluster or develops new clusters until all the points are visited. Terminal candidates of each grid should be also detected by DBSCAN. Each candidate location is represented by the median location of all the points contained. Although gridding could significantly reduce the total points of each grid, a real terminal may be broken into multiple terminal candidates associated to different grids. These candidates need to be merged back. DBSCAN is employed to merge them back with eps = 0. 01 (about 1 km) and minPts = 1. This means that any two candidates with a distance less than eps will be merged. DBSCAN is an unsupervised, density-based clustering method particularly developed to detect arbitrary shape from big datasets with noises.
However, the advantage of our framework is that the large trajectory datasets considered are much more reliable for statistically analyzing the patterns that appear at both the global, regional, and even local scales. Furthermore, the application of the DBSCAN algorithms is noticeably efficient in dealing with large datasets with local noisy data DBSCAN requires two parameters. The first one denotes the search radius (eps) to search for nearby neighbors. The second one refers to the minimum amount of points required (minPts) to form a core point. The clustering process of DBSCAN starts with an arbitrary point (p) and then searches for points located within the eps-neighborhood of p, denoted by N eps (p). If the number of points |N eps (p)| >= minPts, then p will be qualified as a core point and a cluster is identified.
Terminal Mapping A terminal candidate identified in the previous step can be simply considered as a set of data points, which are difficult to consider for further analysis due to lack of semantic meanings. Our approach associates these terminals with real ports and countries as identified by WPI and GADM. It would be reasonable to map these terminals with their nearest port based on port locations given by WPI. However, this is not an error-prone approach. For example, locations of the Port of Shanghai and Zhoushan are located at the center of their host cities according to WPI, while the location of the Yangshan, terminal one of the main terminals of the Port of Shanghai, is much closer to the Port of Zhoushan, which leads to several potential mismatches. In order to appropriately identify terminal, a three-set process is applied using GADM city boundaries and WPI port terminal locations.
Results and Discussions The proposed methodology has been applied to different type of ships. Container ships with capacity of more than 1000 TEUs have been considered as an example to develop a Global Container-Shipping Network (GCSN) based on massive ship trajectories in 2015. The global AIS database and ship characteristic database are mainly provided by www. hifleet. com, which is one of the leading maritime data providers in China. At the current stage, a manual approach is applied to classify the terminal candidates. The main process is as follows. First, a satellite image of a terminal candidate is downloaded from map service providers. Second, traffic statistics are derived from stop events at each candidate terminal. Key statistics of a typical terminal candidate include the total number of ships visited, total number of stops, and the duration of these stops.
Data Preparation AIS data of container ships is extracted from the global AIS database by matching the MMSIs of AIS data with the MMSIs of container ships from the global ship database. Overall, this gives 4333 container ships with 349,759,314 AIS points distributed in 23,111 1x1 degree geographic grid cells. Of them, 3593 have a capacity equal to or great than 1000 TEUs, accounting for approximately 84% of the total capacity of the world container ship fleet. With the purpose of mining marine terminals, data points with a speed of 0 and status of "moored" have been selected, which results in 27,106,564 AIS points located in 635 1x1 degree geo-grids. Although the size of the original dataset has been considerably reduced by the above process, two tasks are taken to facilitate the DBSCAN data analysis.
The first one is a time sampling operation. It appears that 147 geo-points have more than 2000 duplicated points (i. e. , with same latitude and longitude). Time sampling is therefore used to reduce these duplicated points. Only one point is retained for every 5 min for each of these geo-points for each ship. Finally, 26,881,581 data points were generated after this data filtering process. Therefore, all data points were grouped into appropriate grids. These grids are initialized with different sizes depending on their geo-locations. The value of thre_n depends on the size of the host computer's RAM and speed required to run DBSCAN. Computing the distance matrix for unsupervised clustering like DBSCAN is time-consuming for large datasets and requires very large RAM for storage, as the distance matrix is square the number of points.
Interactive Database Exploration (IDE) is the process of exploring a database by means of a sequence of queries aiming at answering an often imprecise user information need. In this paper, we are interested in the following problem: how to automatically assess the quality of such an exploration. We study this problem under the following angles. First, we formulate the hypothesis that the quality of the exploration can be measured by evaluating the improvement of the skill of writing queries that contribute to the exploration. Second, we restrict to a particular use case of database exploration, namely OLAP explorations of data cubes. Third, we propose to use simple query features to model its contribution to an exploration. The first hypothesis allows to use the Knowledge Tracing, a popular model for skill acquisition, to measure the evolution of the ability to write contributive queries.
Introduction Interactive Data Exploration support addresses the development of techniques that allow users to interactively explore their data and help them to better gain insights. Many approaches have recently been developed to support IDE, as illustrated by a recent survey of the topic  [1]. Typically, an exploration includes several queries where the result of each query triggers the formulation of the next one. OLAP analysis of data cubes is a particular case of IDE, that takes advantage of simple primitives like drill-down or slice-and-dice for the navigation. Given the exploratory nature of OLAP analysis of multidimensional data (see e. g. ,  [2, 3] ), many exploration techniques have been specifically developed in the context of interactive OLAP exploration of data cubes. However, while there exists many benchmarks recognized by the database community as relevant for evaluation and comparison of performance of database systems, such as the benchmarks from TPC organization, there is yet no commonly agreed upon method for evaluating to what extent interactive explorations conducted with such systems are indeed successful.
We note that a similar principle has been used recently to score sequences of book reviews  [6]. Our approach builds upon our previous work that aimed at characterizing focus in OLAP explorations  [7]. In addition to using a model of skill acquisition for assessing explorations, we develop a robust method to produce the model of query contribution, adding extra features to the feature set defined in  [7] , studying feature correlation, and balancing datasets. We evaluate our approach on three different datasets, two real ones and an artificial one. Many practical benefits of the proposed assessment technique can be envisioned. As it puts the user and their skills in the center of the data analysis activity, it can be seen as an important driver in the design of systems supporting Interactive Data Exploration  [1] , as well as the corner stone of the development of benchmarks for such systems  [8, 9].
We conclude this subsection by noting that other models exist for predicting a learner's skill. Specifically, Performance Factor Analysis  [13]  uses standard logistic regression with the student performance as dependent variable. Interestingly, it is shown in  [14]  that Knowledge Tracing can achieve comparable predictive accuracy as Performance Factor analysis. Finally, Deep Knowledge Tracing  [15]  uses Recurrent Neural Networks to model student learning, with the advantage of not having to set explicit probabilities for slip and guess. However these models need very large datasets to learn the latent state from sequences, and most importantly, the encoding of the input vectors depends on an upper bound on the number of exercises which does not directly fit our context.