dataesr/scientific-paragraphs-categorization huggingface dataset:
• Data representation conflicts: It can happen that semantically equivalent attributes with different data types are mapped in the database source. The target database must have a data type that can represent both. The data sources are migrated to the target database using a conversion function. If different value ranges are specified in the source database for the same attributes, the union must be selected in the target database. If null values are allowed in any of the source databases for an attribute, null values must also be allowed in the target database for that attribute. Default values cannot be taken from the source databases if the same default was not used for all semantically equivalent attributes. • With regard to the data conflicts, when research information is stored in the source databases according to different conventions, the source databases may contain redundant data.
Such can be largely corrected by machine. If the conflicting data content is the result of misspellings or different replication of existing research information, then in most cases manual intervention is required. Figure  3  explicates these issues and solves them using RCD Standard during their integration into RIS. Standardization of research information would pave the way for sharing comparable data between research institutions. Research information must be maintained in a RIS suitable for both the collection and the preparation and provision of the data. The RCD standard provides the possibility for a simplified comparative presentation. RCD standard exists as XML and OWL (Web Ontology Language) as a data exchange format. In addition, the RCD team also provides an Entity Relationship Model (ERM) that can be used to understand the areas of RCD.
However, as the core dataset's goal is to simplify the exchange of data and explicitly not build a centralized database system with research information from various research organizations, the core dataset project does not develop a schema for a database system. On the basis of the RCD-XML schema, transformation rules (e. g. with XQuery or XSLT) can be defined, which convert existing data models (e. g. CERIF) into the data model of the core data record. RCD Standard creates several advantages for research institutions. This can be listed as follows: • supports the interoperability of data of different formats • increases the flexibility of different systems • facilitates the understanding of information structures • simplifies data exchange, communication and networking • minimizes misinterpretations and resulting in additional expenses • supports the harmonization of business processes within and between research institutions • increases data quality and thus significantly strengthens transparency and comparability between research institutions • reduces the administrative effort • enables the development of systems that work with these structures.
Conclusion Research information systems provide information about research activities and their results. This information is essential for insight knowledge of research activities, for the evaluation of research performance and for well-informed decision-making. But as we stated above, the utility and the perceived usefulness of a RIS is related to the quality of ingested and processed data. Data quality problems in research information can lead to wrong decisions and in some cases even jeopardize the existence of the institutions. The workflow presented in our paper offers institutions the opportunity to assess and improve the quality of research information prior to integration in RIS. In this context, the implementation of the Research Core Dataset standard can contribute to better data quality and better data processing.
The RCD specification describes data grouping by content and formats with a flexible model. The standard can be implemented as part of the RIS integration layer (input) as well as part of the presentation layer (export). The RCD contributes as well to the uniform definition and structuring of research information as for ensuring the data quality and comparability of the data. In our paper, we focus on integrating RCD into the RIS because universities and research institutions integrate many different heterogeneous data sources into RIS. Collecting and integrating data is a key task in the decision-making process; increasing the data quality ahead of further data processing is a top priority for RIS data management. As we mentioned at the beginning, this paper is part of a larger research project and should be considered in the context of other related topics, like user acceptance, text and data mining approaches and other, more traditional data quality management tools.
Regarding RCD, we would suggest at least three further research perspectives: 1. Data export: We put the focus on the input, ahead of the RIS data processing, as this appears to be a priority task for RIS data management. However, additional attention should be paid to the potential usefulness of the RCD standard downstream, as part of the RIS presentation layer and the data export. 2. Comparative studies: We highlighted the potential interest of the RCD for non-German speaking countries and institutions. Other countries have developed similar terminological standards, and it may be helpful for further development, harmonization and interoperability to identify other national standards and to compare them with the German RCD. 3. Use cases: This paper provides general information about the RCD and its potential advantages for data quality management in RIS.
Academic and research institutions start to make use of the RCD, and it would be interesting, in one year or two, to conduct a survey on the experience with and the impact of the implementation of the RCD in different institutional and technological environments. Further investigation should also consider new approaches to data quality issues in the larger context of big data  [18]. In particular, a recent study on data collaboratives  [22]  may be helpful for a better understanding and assessment of RIS data quality insofar it provides a taxonomy of some main specific coordination problems, such as matching potential data providers and data users, maintaining control over the data and its unforeseen uses or matching a particular problem with the attributes of the data, and describes potential mechanisms to deal with these problems in a given organizational environment, including process standardization, transfer of knowledge and negotiation.
Pervasive data have become a key source of information for mobility and transportation analyses. However, as a secondary source, it has a different methodological origin than travel survey data, usually relying on unsupervised algorithms, and so it requires to be assessed as a dataset. This assessment is challenging, because, in general, there is not a benchmark dataset or a ground truth scenario available, as travel surveys only represent a partial view of the phenomenon and suffer from their own biases. For this critical task, which involves urban planners and data scientists, we study the design space of the visualization of cross-origin, multivariate flow datasets. For this purpose, we introduce the Modalflow system, which incorporates and adapts different visualization techniques in a notebook-like setting, presenting novel visual encodings and interactions for flows with modal partition into scatterplots, flow maps, origin-destination matrices, and ternary plots.
Owing to its complexity and massiveness, no data source alone can account for the whole phenomenon of mobility within a city. Different ways of measuring and approximating this phenomenon have been developed in order to understand and shape the city. Travel surveys have been a key tool, relying on a small, but carefully orchestrated, sample, access to census data and statistical craftsmanship. They are expensive and time consuming to produce  [2]  and they show heavy under-reporting  [3]. This traditional source is contrasted with the information that can be extracted from available and emerging pervasive data sources, such as mobile phone  [4] , Wi-Fi  [5] , and social network data  [6]. This type of data are inexpensive and could potentially be used for real-time analysis, but, as a secondary data source for urban planning, it has its different methodological origin, which makes a comparison between datasets a challenge for both domain experts and data scientists.
For example, as network data does not rely on self-report, trips, and mode of transport need to be inferred from waypoint traces, which is not an easy task, because of their relatively inaccurate temporal and spatial sampling, which introduces an uncertainty that was not present in this form in survey data (or at least, it does not have the same origin). We call datasets stemming from different methodologies "cross-origin". Especially at the pilot stage, where a new dataset is being evaluated for its possible adoption and use, urbanists need to qualitatively assess the new data source. From what we have collected in interviews with experts, there are no protocols for this kind of assessment in use yet. Because official information on mobility is scarce, the travel survey is the main benchmark to which compare a dataset, which suffers from its own structural biases, and it is updated in the lapse of years.
In this unprecedented scenario, we developed Modalflow as a system for visualizing and comparing cross-origin datasets of urban mobility. Its name is derived from the two concepts that define its data workflow: flows and modality (as in modal split). Using juxtaposed and linked views in a notebook-style layout, it incorporates and adapts different visualization techniques for flow data including scatterplots, origin-destination matrices, edge bundling, flow maps (Figure  1 ), and ternary plots. Modalflow allows users to look into the phenomena of transport and, more importantly, to compare different sources on these phenomena, identifying each source by its methodological features. The main contributions of our work are:.
• A validation of our approach with a case study using real-world cross-origin datasets. We tested Modalflow while using two datasets from the Santiago metropolitan area: the last travel survey held in the city and a mobile phone network-inferred dataset, drawing insights on their differential qualities, strengths and flaws. We hope this system will foster critical evaluation of new, non-traditional data sources and their adoption by domain experts, and also improve the communication with the field of data scientists, which produces and makes use of these datasets.
In spite of these improvements, OD matrices have not been used for visualizing networks of more than around a hundred nodes. Graph. Graph-based flow visualization literature is greatly centered around reducing the hairball problem, i. e. , visual clutter. We can divide them into those that operate some kind of simplification over the graph and then render the result with a linear technique  [11] [12] [13] , and those that make use of non-linear or iterative techniques for path calculation and rendering  [14, 15]. The techniques that were developed for this purpose rely on heavily distorting flow paths and edge appearance in order to improve readability. However, these developments have generally taken place without taking into account the particular characteristics of datasets and how they might behave on different data scenarios (e.
Domain Problem In the target domain of application, there is not a standard methodology that can embrace current pervasive data sources, as these are mostly secondary sources, i. e. , data that were not originally captured for the purpose of transport analysis. We consider pervasive data to be mode-unspecific and stemming from networks, such as social networks, Wi-Fi networks, and mobile phone networks. Primary sources, on the other hand, are datasets whose raw form and original purpose are inherently related to mobility (and not only as the consequence of an additional processing, as in XDR-inferred trips). There are other data sources that also enable mode-specific analysis, such as automatic passenger counting mechanisms in public transport  [5] , which, although highly granular, lose, to a certain extent, the origin and destination aspect of flows.
Considerations on Cross-Origin Dataset Comparison Pervasive data as a secondary data source has its own qualities that sets it apart from primary source data. Here, we try to name and account for the aspects that make difficult the comparison between these sources, in the case of mobility flow data. Trip closure. The definition of one trip does not share its origin in self-reported and predicted sources. For quantitative methodologies, what defines a trip as a unit is not part of any of these dimensions but a discursive element: the motivation of the sampled person (e. g. , go to work, buy food, go for a walk, return home, etc. ). Because tracking does not deal with persons but with devices, a trip is not a given entity, but it must be inferred from the data itself, usually from finding stay patterns that then become origin-destination features.
Requirements When considering the above synthesis, we extract design requirements that support a comparison of cross-origin flows given these aspects of conflict. (R1) Comparing flows as multidimensional entities. Users should be able to visually compare different datasets in an origin-independent way, i. e. , having the recorded trips aggregated into the common language of flows as multidimensional points, where different capture methods are comparable without hiding their inherent differences. (R2) Comparing flows at different aggregation levels and ranges. Users should be able to filter and select flows according to spatial dimensions, origins and destinations, quantitative dimensions, time period, and visualize flows that are aggregated at different granularities. (R3) Comparing mode split distributions.
Data Our cross-origin datasets comprise trips in Santiago, the capital of Chile, a city with almost 8 million inhabitants in 35 administrative units denoted municipalities within its urban area, further subdivided into 735 traffic analysis zones. We define a flow as a unique origin-destination pair within a certain period (a partition of time), which has a magnitude (number of trips observed or predicted from origin to destination for the corresponding period) and a modal partition (the proportion of the magnitude corresponding to each mode of transport). Understood as a network where nodes are georeferenced objects, a flow is a weighted directed link with a modal partition uniquely defined by an origin-destination-period tuple. We also calculate a distance feature for each flow, as the linear geographical distance between origin and destination.
Note that the traffic analysis zones of the city are defined by this survey. Mobile Phone Network Data. As input data for Modalflow, here we work with a dataset of inferred commuting trips from mobile phone network data. Specifically, we use Extended Data Detail Records (XDR), which is a type of passive data that are used by telecommunication companies for billing Internet usage from mobile phones  [4]. The dataset used in this paper was provided by the telecommunications company Telefónica Movistar, which had a market share of 30% in April 2017. The dataset contains trips during April 2017 for approximately 600K devices, which were extracted while using a previously published method  [42]. The trip distribution inferred from this dataset has been validated with experts and through comparison with a travel survey, which implied that the market share covered by the dataset is enough to measure mobility patterns in Santiago  [39].
General Layout Modalflow works as a set of coordinated visualizations displayed in a notebook-style layout  [43] , i. e. , visualizations are stacked in display and usage order from top to bottom, without constraining the layout to fit into a screen. Gleichner categorizes comparison layout strategies as juxtaposition, superposition, and explicit encoding  [44]. We chose to use juxtaposition, because using overlay would have resulted in a cluttered view, and explicit encoding would have removed the data context  [45]. Each layout element is a view from the system where the different datasets are visualized in juxtaposition to ease comparison across views. Each visualization shows different dimensions of the data and acts in coordination with the rest: when a selection is created in one view of the system, it extends to the other dataset and views of the system.
While testing the scatterplot, we discovered that normal range selection proved to be not effective for comparing different datasets, as range selections in one dataset would probably lead to an empty selection in the other one due to the very different distributions of data. Thus, to comply with R1, we developed a distribution-aware selection tool. As the distribution of the examined datasets proved to be different over this space, selections that were made over one dataset usually ended in an empty selection over the other one, making a the comparative approach ineffective. For this reason, we developed a distribution-aware selection tool that translates a selection over one distribution into a more alike selection over another distribution, as shown in Figure  3. It is a linear transformation of the XY space based on data distribution.
This technique can be generalized to different datasets; however, a decision has to be made over the formal nature of their distribution to define the linear mapping. In this case, preliminary visualization experiments showed a power law-type distribution; thus, a measure of its powerlaw had to be taken into account (for a normal distribution, for example, its median and variance should determine the outcome mapping). In order to determine the mapping for the distribution-aware tool, the method is as follows: (1) for each dimension, sort the data according to that dimension and determine the threshold at which the highest ranking elements are equal in value to all the others; and, (2) this threshold is then the value of the scale factor k i for dataset i. The mapping of coordinate x j from coordinate x i is given by For the particular case of mobility flows, we deemed that using this technique only in the magnitude dimension was the most sensible choice, because using distribution-aware selection over distance would change the original meaning of a selection.
Edge Bundling An edge bundled view of the flows is also available  [48]. This encoding for the flow map is used in order to produce visual structures with coarser features to show major trends in the data, particularly for mode-filtered dataset comparison (R1), and so, this is the only encoding where the modal color scale is not used, but a radial color scale here is used to encode bundled flow direction. Taking into account our data-set specificity, we selected the most appropriate edge bundling technique, which allies computation simplicity and a sufficient visual aggregation results, which balances data distortion and visual accuracy  [16]. For this reason, we applied the KDEEB technique  [49].
Results Here, we describe the insights that we, as researchers and users of the system, learned by comparing pervasive data to survey data, in a general sense, and about the particular inference model behind the XDR dataset. The scatterplot reveals that pervasive data capture shorter and longer flows better than the survey, which has a slightly narrower distance range and is also biased towards mid-distance flows, as shown on Figure  5. This difference can have a rather important effect on the interpretation of data: a steeper, long-tailed (powerlaw) distribution of flows suggests a scale-invariant network phenomenon, while the flatter survey distribution does not, and thus surveys do not capture the full complexity of mobility. XDR has almost total municipal origin-destination pair sampling coverage for a given period, whereas survey data do not, as shown through the OD Matrix in Figure  6.
Additionally, there is an observable pattern of sampling in survey data: short active flows are unintentionally oversampled (showing in the bright diagonal). However, when flows are expanded, these flows are downplayed and overshadowed by expanded public and private mode longer-distance flows. The geographic view of the flows is quite dissimilar for both datasets, exhibiting rather different features, as can be observed in Figure  7. Pervasive data show notorious change in behaviour during the day: in the morning, peripheral flows are directed towards the center and northeastern part of the city (the small peaks appearing on these nodes); flows in the afternoon show roughly the opposite pattern, but also an emergence of active flows around certain parts of the city; at night, short flows become totally predominant across the city.
The distribution-aware selector tool in the scatterplot was received as an interesting feature and its implications could be further developed, attending to the idea of an "equivalence principle" between flows of different datasets. Such an endeavour could be an interesting area of research for modelling and mixed data analysis. This also poses new questions regarding the layout of the system. Modalflow was designed with linked, juxtaposed visualizations, where the selector tool provided a way to link data semantics. However, there exist other ways of composing visualizations in layouts, such as superimposition, overloading and nesting  [51] , which would imply a different implementation of the selection, while maintaining the data-semantic approach. These layouts will be studied in future work.
Because of the notebook layout, they are not exclusive-for instance, the OD matrices may be juxtaposed, whereas the flow maps may be partitioned or nested, enabling an advanced comparison and interaction with cross-origin datasets. Other shortcomings of the design that were noted by experts were that the color encoding used is not color-blind safe (making a three-component color scale to be inclusive is a challenge by itself, which we propose as a future line of research) and that defining a three-fold categorical variable as basis for the system seems like a limited choice, as some urban planners may want to disaggregate the modes of transportation into four, or the usage of another categorical variable may need more categories (for instance, income quintiles). From a technical standpoint, our choice presents a trade-off between an overview and a detailed view of the datasets.
Conclusions In this paper, we have presented the Modalflow system for visualizing modal flow data in mobility, with an emphasis comparing cross-origin flow data sources. The research contribution of our paper consists of two main components: a set of considerations regarding the nature of pervasive and traditional data sources, which we then used to determine the requirements for our system design, and a set novel visualization techniques to fulfill those requirements, including a distribution-aware selection tooltip and a bidirectional flow encoding. We validated our approach with a case study while using real-world cross-origin datasets: XDR-inferred and traditional survey data of the city of Santiago. Our aim was to create a tool that could help domain experts in urbanism to assess new datasets from non-traditional sources and hopefully incorporate them into their workflow, at the same time fostering interaction between data science and urbanism, as we saw this as an important gap to be filled by the visualization community.