dataesr/scientific-paragraphs-categorization huggingface dataset:
Methodology To compare IGN and OSM POIs, we first identify the types of features that are represented in both datasets. As shown in Figure  2 , stations and subway entrances are types of objects that have relevant changes regarding names or positions and they are represented in both datasets. Thus, we focus the data matching on stations representing subway stations and subway entrances. Figure  9  illustrates the data model for subway station representation and defines the links between the real world and the two representations from IGN and OSM. Thus, a subway station in the real world (represented in the blue box) is characterized by a name and a certain number of subway entrances and exits. Regarding the representation of subway stations in the two datasets, two main differences can be noticed.
In the IGN dataset (green box), a subway station from the real world is represented by n features. Thus, with respect to the IGN data specification, two types of mapping for each subway station are possible: (i) for subway stations without a connection between several lines, only one point feature is represented in the dataset; (ii) for subway stations that connect to at least two subway lines, at least two feature points are represented in the dataset. Even if the concept of entrance/exit does not exist in the IGN dataset, the features representing the subway station are frequently located close to an entrance. On the other hand, in OSM (purple box), a subway station in the real world is represented by m features belonging to the OSM type station. The subway station is composed of m' entrances/exists represented by a point feature, each entrance/exit having a name.
In Figure  9 , dotted lines illustrate matching links (i. e. , features representing the same object in the real world and numbers illustrate the cardinality of the matching links (e. g. , n IGN subway stations should be matched with m OSM subway stations). Given the complexity of the representation of the two datasets, many types of data matching links are possible but we are interested in two types of data matching links in this study. First, n:m links are defined for the concept of a subway station (i. e. , n IGN subway points can be matched with m OSM subway points representing the same subway station). Second, m:m' links are defined for a subway station and its entrances/exits (m OSM subway stations are matched with m' OSM entrances/exits). Finally, 1:1 links are defined for entrances/exits (1 IGN subway station is matched to 1 OSM entrance/exist).
ISPRS Int. J. Geo-Inf. 2017, 6, 80 13 of 29 OSM entrances/exits). Finally, 1:1 links are defined for entrances/exits (1 IGN subway station is matched to 1 OSM entrance/exist). Figure  10  shows an example of two different representations in the IGN and OSM datasets. For example, Duroc subway station (Figure  10a ) has one point in OSM data and two points in the IGN dataset that are closer to two subway entrances in OSM ('Duroc-sortie 3 place Léon-Paul Fargue and 'Duroc-sortie 1 boulevard des Invalides'). On the contrary, for Saint-Sulpice subway station, one point is mapped in both IGN and OSM data but there is no link between the IGN subway station and the OSM subway entrance. In Figure  10b  two OSM entrances can be seen near the OSM subway station; one of them can be considered a duplicate entrance and should be removed.
The data matching algorithm works as follows: for each reference feature from a dataset, the algorithm looks for candidates (from the comparison dataset) to match using a buffer. Then different similarity factors are compared between the reference feature and all selected candidates. Knowing the characteristics of our datasets (see Sections 2. 1 and 2. 2), three criteria were used: position, toponym, and semantic criteria. The position criterion is based on the distance between the reference feature and a candidate. The toponym criterion compares the name of the reference feature with the name of the candidate. Among the different measures tested to compare names, the normalized Levenshtein distance was used since it was considered the most suitable for our dataset. Finally, the semantic criterion, which compares feature types, was used to select features in both datasets that  OSM entrances/exits).
Finally, 1:1 links are defined for entrances/exits (1 IGN subway station is matched to 1 OSM entrance/exist). Figure  10  shows an example of two different representations in the IGN and OSM datasets. For example, Duroc subway station (Figure  10a ) has one point in OSM data and two points in the IGN dataset that are closer to two subway entrances in OSM ('Duroc-sortie 3 place Léon-Paul Fargue and 'Duroc-sortie 1 boulevard des Invalides'). On the contrary, for Saint-Sulpice subway station, one point is mapped in both IGN and OSM data but there is no link between the IGN subway station and the OSM subway entrance. In Figure  10b  two OSM entrances can be seen near the OSM subway station; one of them can be considered a duplicate entrance and should be removed. The data matching algorithm works as follows: for each reference feature from a dataset, the algorithm looks for candidates (from the comparison dataset) to match using a buffer.
Then different similarity factors are compared between the reference feature and all selected candidates. Knowing the characteristics of our datasets (see Sections 2. 1 and 2. 2), three criteria were used: position, toponym, and semantic criteria. The position criterion is based on the distance between the reference feature and a candidate. The toponym criterion compares the name of the reference feature with the name of the candidate. Among the different measures tested to compare names, the normalized Levenshtein distance was used since it was considered the most suitable for our dataset. Finally, the semantic criterion, which compares feature types, was used to select features in both datasets that The data matching algorithm works as follows: for each reference feature from a dataset, the algorithm looks for candidates (from the comparison dataset) to match using a buffer.
Then different similarity factors are compared between the reference feature and all selected candidates. Knowing the characteristics of our datasets (see Sections 2. 1 and 2. 2), three criteria were used: position, toponym, and semantic criteria. The position criterion is based on the distance between the reference feature and a candidate. The toponym criterion compares the name of the reference feature with the name of the candidate. Among the different measures tested to compare names, the normalized Levenshtein distance was used since it was considered the most suitable for our dataset. Finally, the semantic criterion, which compares feature types, was used to select features in both datasets that represent the same type of objects in the real world, in our case: subway stations, stations, and subway entrances.
The data matching described above was applied on different datasets: (i) subway stations from IGN with OSM stations (link * in Figure  7 , n:m cardinality) and (ii) public stations from OSM with subway entrances from OSM (link ** in Figure  7 , m:m' cardinality). In the first case, the reference feature is IGN and the candidate features are from OSM, while for the second case the reference feature is OSM subway entrances and the candidate features are OSM stations. The thresholds used in the matching algorithm were defined with respect to data characteristics obtained by analyzing the distributions of the distances for each criterion. Thus, the threshold for the position criterion (Euclidian distance) and the name similarity criterion (Levenshtein distance), which indicates that it is unlikely for two features to represent the same object in the real world, are equal to 220 m and 0.
We can see that the homologous subway stations are quite far away from one another, i. e. , 140 m and 180 m from the IGN subway station. ISPRS Int. J. Geo-Inf. 2017, 6, 80 14 of 29 represent the same type of objects in the real world, in our case: subway stations, stations, and subway entrances. The data matching described above was applied on different datasets: (i) subway stations from IGN with OSM stations (link * in Figure  7 , n:m cardinality) and (ii) public stations from OSM with subway entrances from OSM (link ** in Figure  7 , m:m' cardinality). In the first case, the reference feature is IGN and the candidate features are from OSM, while for the second case the reference feature is OSM subway entrances and the candidate features are OSM stations. The thresholds used in the matching algorithm were defined with respect to data characteristics obtained by analyzing the distributions of the distances for each criterion.
When the IGN subway station is closer to a subway entrance, then distances are around 80 m. Finally, when an IGN subway station is closer to an OSM subway station, then the distances are equally distributed across a 50-m interval. In addition, the data matching and the evaluation of results allowed us to identify the following omissions: • one OSM subway station ('Pont de Neuilly') is not mapped. It is important to note that this station was added during 2016, after we had extracted our OSM dataset.
• Different nomenclature is used: 'Palais Royal (Musée du Louvre)' for IGN versus 'Palais Royal -Musée du Louvre' for OSM. In addition, the data matching and the evaluation of results allowed us to identify the following omissions: • one OSM subway station ('Pont de Neuilly') is not mapped. It is important to note that this station was added during 2016, after we had extracted our OSM dataset.
• case 3: OSM subway entrances for which the name contains 'exit xx' or '-xx'. The results for these three cases are given in Table  9. Table  9. Data quality analysis with respect to the observed naming standard for OSM subway entrances. For an OSM subway station that has at least two corresponding OSM subway entrances, 463 OSM subway entrances have a name contained in the name of the OSM subway station Among the 463 OSM subway entrances, 306 (66%) have exactly the same name as the corresponding OSM subway station case 3.
Name of OSM Subway Entrances Versus Name of Subway Stations 168 OSM subway entrances contain the word exit. Among those, 22 do not contain the name of the OSM subway station (e. g. , 'Pernéty' versus 'Pernety-sortie 2-rue Niepce'. Adding the sign '-' between the name of the subway station and the word exit has been noticed 142 OSM subway entrances that contain the name of the subway stations follow the standard: Name of the subway station-Exit nn-street xxx (e. g. , 'Bastille-sortie 1-rue de la Roquette') 4 OSM subway entrances start with the word Exit (e. g. , 'Sortie 3-Rue Goscinny') 148 OSM subway entrances have a number after the word Exit In addition to the previous analyses, the data matching and the evaluation of the results allowed us to identify the following omissions:.
Assessment by Checking Complementary VGI Sources As discussed in the previous section, a common approach to assessing the quality of VGI data is comparing it against authoritative datasets. This method enables the use of existing quality evaluation methods and has a number of advantages that can provide a comprehensive understanding of VGI, it can reveal any patterns or biases that might exist in the data (e. g. , number of contributions and level of participation in rural vs. urban areas) or it can document, in a tangible way, the agreement of VGI data with authoritative, and thus asserted and trusted, spatial data. However, there are some equally important disadvantages. First, authoritative data are not always freely available or come with restrictive licensing agreements. Secondly, in some cases, VGI datasets are more current than the authoritative ones and thus omission and commission errors are hard to assess.
Methodology In order to cover this gap in quality assessment, we used data from other sources of implicit VGI content to evaluate the validity and quality of OSM POIs. For this role we chose the geo-tagged photographs available from Flickr. The combined view of OSM POIs and geo-tagged photographs can help to evaluate the quality of the OSM POIs and disambiguate inconsistencies or contradictory contributions, especially when the time-stamp available in both VGI datasets are taken into consideration. This evaluation was made manually for OSM POIs that are not directly visible from satellite imagery (e. g. , POIs that are under wooded areas).
Results We used the IGN dataset for the wooded areas of Paris in order to spot the OSM POIs that are not visible using satellite imagery. Those POIs cannot be evaluated or corrected by contributors unless they physically visit the location. However, taking into consideration the Pareto principle, which also applies to VGI contributions  [36]  and dictates that a small percentage of volunteers generate the vast majority of content, as well as the participation and spatial biases of the contributors, it is understandable that quality evaluation or update of "hidden" elements is a challenging issue. In this context, we manually examined whether the use of geo-tagged photographs can provide additional information for POIs that are not directly visible from remote sensing. Figure  15a  shows the OSM map for an area in Paris ('Musée Rodin Gardens') where a number of POIs are under trees.
Motorway Junctions Motorway junctions are the features that moved the most, according to historical analysis. Their quality can be assessed by measuring their topological relations with the roads, e. g. , motorway junction POIs should be located at the intersection of motorways and a ramp. The study of the 76 points of the dataset showed a perfect 100% consistency with roads. The history of these points shows that most of them (around 70%) have been displaced from their initial position where there was no consistency with the roads, with often large displacements (Figure  16 ). analyzed whether the quality of the POIs improved over time in terms of their spatial consistency with the surrounding features.
Motorway Junctions Motorway junctions are the features that moved the most, according to historical analysis. Their quality can be assessed by measuring their topological relations with the roads, e. g. , motorway junction POIs should be located at the intersection of motorways and a ramp. The study of the 76 points of the dataset showed a perfect 100% consistency with roads. The history of these points shows that most of them (around 70%) have been displaced from their initial position where there was no consistency with the roads, with often large displacements (Figure  16 ). In our case study, only the history of points were collected, so we cannot be sure that the displacement improved the topological consistency of motorway junctions, because they might just have followed the displacement of the roads.
IGN Subway Station Evolution As mentioned in Section 2. 2, a change in IGN data is possible when the following situations occur: a change in the real world, error corrections, a change in data specifications, or new partnership agreements for sharing open data arise for some layers. Whatever the source of information, all features used to update IGN POI datasets are checked and follow the data quality validation. Figure  19a  illustrates the changes in subway stations in the Paris region from 2006 to the present with respect to the source of information. We can see that, from 2006 to 2011, position changes are mostly due to updates and they are coming from another IGN database (blue line). Some thematic changes (the cumulative distance is 0) coming from paper maps (red line) can also be observed from 2010 and 2013.
The interesting point to highlight is that starting with 2012 (line light blue), the change in the subway station location is high, the cumulative distance being more than 1600 m in 2014, followed by 1500 m in 2015, and tending towards stability (e. g. , less than 500 m). This is due to a new partnership agreement between IGN and the RATP. Indeed, we have noticed that 93% of the subway stations that changed in 2014 are from the RATP database. Figure  19b  illustrates the variation of distance between consecutive changes with respect to the number of changes, which shows that the number of changes correlates positively with the median distance between consecutive changes. For example, for subway stations having two location changes over time, the median distance is around 50 m and more than 300 m for subway stations having four location changes.
As mentioned in Section 2. 2, a change in IGN data is possible when the following situations occur: a change in the real world, error corrections, a change in data specifications, or new partnership agreements for sharing open data arise for some layers. Whatever the source of information, all features used to update IGN POI datasets are checked and follow the data quality validation. Figure  19a  illustrates the changes in subway stations in the Paris region from 2006 to the present with respect to the source of information. We can see that, from 2006 to 2011, position changes are mostly due to updates and they are coming from another IGN database (blue line). Some thematic changes (the cumulative distance is 0) coming from paper maps (red line) can also be observed from 2010 and 2013. The interesting point to highlight is that starting with 2012 (line light blue), the change in the subway station location is high, the cumulative distance being more than 1600 m in 2014, followed by 1500 m in 2015, and tending towards stability (e.
g. , less than 500 m). This is due to a new partnership agreement between IGN and the RATP. Indeed, we have noticed that 93% of the subway stations that changed in 2014 are from the RATP database. Figure  19b  illustrates the variation of distance between consecutive changes with respect to the number of changes, which shows that the number of changes correlates positively with the median distance between consecutive changes. For example, for subway stations having two location changes over time, the median distance is around 50 m and more than 300 m for subway stations having four location changes.
In this last study only a few integrity constraints are defined and the approach was tested on a small dataset. Thus, more research is needed to define and combine many integrity constraints. The four methods presented here can all be improved. Regarding the data matching approach, the semantic schema matching, which has been done manually in this paper, can be improved by using a schema matching based on ontology. For that, more research is necessary in order to align the IGN topographic ontology  [41]  with OSM ontology  [42]. Regarding the spatial relations method, the main problem is to interpret what the spatial relations mean in terms of quality. It would be helpful to use machine learning techniques to infer the quality of the POI given a set of spatial relations. Although the paper urges the use of holistic approaches for VGI quality assessment, the combination of assessment methods experimented with in this paper used only two of the four presented methods: history with spatial relations, and history with matching to NMA reference data.
Guidelines for Typical Use Cases A first typical use case is a NMA that controls a photo sharing web app and/or a vector (i. e. , point feature) contribution platform. How can the crowdsourced POIs be integrated with the authoritative datasets of the NMA? Data matching can be used in this situation to link both datasets in order to identify homologous features or crowdsourced POIs that do not exist in the authoritative dataset. There can be three types of results in the matching process. First, if the crowdsourced POI is not matched, a joint analysis of its spatial relations with the authoritative dataset and of its past versions can help us to assess if it can be integrated into the authoritative dataset (i. e. , the quality is good or not). If the crowdsourced POI is matched and very close to its counterpart in the authoritative dataset, then the two features can be merged, which might semantically enrich the NMA dataset.
A second use case is the derivation of maps with the cartographic background from a NMA, and the OSM POIs on top to create an enhanced map. As described in  [32] , the problem here is the consistency between the POIs and the authoritative background. In this use case, no matching is needed because the POIs are taken from OSM as an additional dataset. The first step is to analyze the consistency in the spatial relation between the crowdsourced POI and the map background. If consistency is poor, or inconclusive, a joint analysis of past versions and a spatial relation consistency analysis is performed with OSM data only, as outlined in Section 7. 1, which can remove ambiguity and again aid the decision-making process. For example, there may be a case where a store POI has been moved through different versions to a consistent position inside the building in OSM, but the inconsistency has come from a shift in the building's position.
Conclusions In this paper we focused on a quality evaluation of the POIs of the most successful VGI project: OSM. The first contribution of the paper has been to propose four different methods to assess VGI POI quality. Two of these methods include a comparison to a reference dataset approach, i. e. , multi-criteria matching to NMA data and the use of Flickr photographs to check POI validity. Another method follows Goodchild & Li's  [28]  crowdsourcing approach by analyzing feature history. The last method follows Goodchild and Li's  [28]  geographic approach by analyzing the spatial relations of the POIs with their neighboring features. On the one hand, the application of each method on the same benchmark dataset shows that the results obtained by a method can highlight a phenomenon and can be used as an input for another method to better analyze the reasons for this phenomenon.
For example, the analysis of history showed that some feature types change more than other types. In order to better understand the reason, the comparison with other sources of information can be carried out on these types of features. On the other hand, each method can provide unique insights on data quality, but none is sufficient to clearly evaluate the quality of POIs from VGI. Then based on the lessons learned from the strong and weak points of each method, we tried to combine them so as to gain a better understanding of the data quality. For instance, the combination of history and spatial relations shows that multiple edits of a feature mainly tend to improve the logical consistency with other features; combining matching to NMA data and history highlights several patterns of improvement or disagreement when several contributors edit the same feature.
The paper shows that a holistic approach is needed for the quality evaluation, so much so that totally new ways need to be developed compared to the existing spatial data quality evaluation methods. Thus, this analysis helped us to understand what the next steps in building a solid VGI quality evaluation framework are, how it should behave, and where the challenges lie. For example, when it comes to the use of geo-tagged images, multiple sources (not just Flickr) can be included in the evaluation process and the method can also make use of other pieces of data such as titles, tags, or descriptions. Furthermore, new algorithms should be developed in order to filter out much of the noise that is present in social networking content. Automatic image analysis and classification, as presented in  [43] , will greatly improve the process.