A hidden Markov model for matching spatial networks

Datasets of the same geographic space at different scales and temporalities are increasingly abundant, paving the way for new scientific research. These datasets require data integration, which implies linking homologous entities in a process called data matching that remains a challenging task, despite a quite substantial literature, because of data imperfections and heterogeneities. In this paper, we present an approach for matching spatial networks based on a hidden Markov model (HMM) that takes full benefit of the underlying topology of networks. The approach is assessed using four heterogeneous datasets (streets, roads, railway, and hydrographic networks), showing that the HMM algorithm is robust in regards to data heterogeneities and imperfections (geometric discrepancies and differences in level of details) and adaptable to match any type of spatial networks. It also has the advantage of requiring no mandatory parameters, as proven by a sensitivity exploration, except a distance threshold that filters potential matching candidates in order to speed-up the process. Finally, a comparison with a commonly cited approach highlights good matching accuracy and completeness.

Related papers:

title: A New Merging Process for Data Integration Based on the Discrete Fréchet Distance
abstract: The overlay process is currently one of the main computational solutions used to integrate several data layers from different sources. Unfortunately, it is problematic when trying to overlay many layers. This leads to several geometric problems such as the management of sliver polygons. This paper proposes a new merging process to complement the vector overlay for data integration of several layers. This process, based on measures derived from the Frechet distance, matches common points (either lines or polygons). It also merges an ordered set of pairs of matching points (vertices) into a single geometry.

title: Development of Conflation Components
abstract: Data updating, integrating and sharing has been one of central issues in GIS. Geodata conflation technology that involves matching the common objects in different data sets, eliminating data discrepancy and combining multiple data sources has been proposed to tackle the problem. During recent decades, many conflation systems have been developed. Due to the complexity of the conflation problem, it has been found that one system is difficult to solve the various types of problems and take all different kinds of cases into account. In this paper, following a review of the status of conflation methods, a system development strategy based on the componentware technology is proposed. This componentware technology makes it possible to develop independent but interoperable components that can be assembled to solve any specific conflation problem. As an example, a conflation component for attribute transfer is developed. This component can be used for transferring attributes from one map with lower spatial accuracy to another with higher spatial accuracy. The algorithm developed for the implementation of the conflation component is presented and the evaluation of results is discussed.

title: Multi-representations and multipleresolutions in geographic databases
abstract: spatio-temporal databases Reference LBD-CONF-1999-009 Record created on 2007-01-16, modified on 2016-08-08

title: Matching spatial data sets: a statistical approach
abstract: Although the acquisition and maintenance of spatial data is very costly and time consuming very often the same objects of the real world are captured in many different data models, at different acquisition times, with different quality characteristics or at different scales. This situation will become intensified when more and more digital spatial data are offered by using internet technologies. Integration methods are needed to take advantage of the characteristics of more than one data set. These advantages could be, for example, new applications for which the data models had not been originally designed, higher reusability, improvement of the quality, or cost minimization of data acquisition. In this paper a relational matching approach for integration of spatial data from different sources is introduced. The research work was performed on street centrelines which were captured in different data models. The approach is based on statistical investigations between the data of two data models, and can be ...

title: A generic matching algorithm for line networks of different resolutions
abstract: 1  Many cartographic applications require the interoperation between datasets of different spatial and semantic resolutions covering the same geographic space. One of the approaches to realize the interoperation is to build a multiple representation database (MRDB) in which corresponding objects at different resolutions are connected to each other by means of automatic matching techniques methods reported in the literature hitherto have revealed high matching rate and efficiency on certain data types of selected test areas (Walter 1996, Meng & Töllner 2004, Ochieng Quddus & Noland 2003). However, these methods were mostly developed as specialized solutions with limited reusability. Their matching performance drops dramatically as soon as the data type or the test area changes. In order to overcome this drawback, a generic matching algorithm is proposed in this paper and implemented in ArcGIS. Being characterized by its self-learning ability and adaptive buffer growing, this algorithm has been successfully tested in a case study on postal data integration.  1 depicts the key components of our generic matching approach. The preprocessing block serves the purpose of reducing the noise or irrelevant details in the datasets to be matched. The matching block is driven by geometric, topologic and semantic parameters with values that are dynamically determined or refined during the matching process. Finally the connections between matched objects are established in the linkage block and stored as a link file which can be fed into ArcGIS for the interactive visualization and data integration based on the matching results. Fig.1 The key components of the generic matching approach 2.1 Data preprocessing By comparing the number of object classes, the overall distribution density which can be described by the total length of line objects as well as the number of intersection points per unit area, the relative distribution density of each object class, and the number of attributes existing in the two datasets, i.e. the reference dataset and the target dataset for the matching, it is possible to exclude those classes and attributes that exist only in one of the datasets. The distribution density and attribute information of the remaining objects are then used as important indicators to the resolution difference between the reference dataset and target dataset. Further, geometries, topological relationships and semantic details of the dataset at higher resolution are simplified until its level of details approaches that of the other dataset. This simplified dataset is stored in a temporary file …

title: Matching road data of scales with an order of magnitude difference
abstract: Nowadays often a multiplicity of different spatial datasets exists for the same area. In order to use and maintain these datasets effectively they have to be integrated into a multiple representation database (MRDB). In an MRDB, the objects within the different datasets that model the same real-world phenomenon can be linked with each other. When integrating existing datasets these links have to be created by means of automatic matching techniques. In this paper we describe such a matching technique for road data of scales with an order of magnitude difference. The process starts by selecting possible candidates for roads and nodes using a buffer. The candidates are filtered using semantic, geometric and topological information. The remaining node candidates are compared by means of geometric measures, and 1 : 1 links between nodes are created. The node links are converted into road links by a shortest path algorithm. The method has been implemented and successfully tested with data at the scales of 1 : 25.000 and 1 : 200.000.

title: Matching Networks with Different Levels of Detail
abstract: This paper deals with the issue of automatically matching networks with different levels of details. We first present why this issue is complex through an analysis of the differences that can be encountered between networks. We also present different criteria, tools and approaches used for network matching. We then propose a matching process, named NetMatcher. This process is a several steps process looking for potential candidates and then analysing them in order to determine the final results. It relies on the comparison of geometrical, attributive, and topological properties of objects. It determines one-to-many links between networks: in particular a node of the less detailed network can be matched to several arcs and nodes forming a complex junction in the most detailed network. An important strength of the process is to self-evaluate its results through the comparison of topological organisation of networks. This paves the way to an interactive editing of the results. The NetMatcher process has been intensively tested on a wide range of actual datasets, thus highlighting its effectiveness as well as its limits.

title: A feature-based approach to conflation of geospatial sources
abstract: A Geographic Information System (GIS) populated with disparate data sources has multiple and different representations of the same real-world object. Often, the type of information in these sources is different, and combining them to generate one composite representation has many benefits. The first step in this conflation process is to identify the features in different sources that represent the same real-world entity. The matching process is not simple, since the identified features from different sources do not always match in their location, extent, and description. We present a new approach to matching GIS features from disparate sources. A graph theoretic approach is used to model the geographic context and to determine the matching features from multiple sources. Experiments on implementation of this approach demonstrate its viability.

title: Data Matching - a Matter of Belief
abstract: Nowadays, it is often that a geographic area is described by several independent geographic databases. Yet users need to fusion various information coming from these databases. In order to integrate databases, redundancy and inconsistency between data should be identified. Many steps are required to finalise the databases integration, in particular automatic data matching. In this paper, one approach of matching geographic data bearing on the belief theory is presented. This approach consists in combining criteria from knowledge such as geometry, orientation, nature of roads, names and topology. Then it is tested on heterogeneous network representing roads.

title: A probability-based multi-measure feature matching method in map conflation
abstract: This paper presents a probability-based multi-measure feature matching method in map conflation. Feature matching is used to determine the corresponding features in different datasets that represent analogous entities in the real world. In the proposed method, a total matching probability is computed by the weighted average of multiple measures, including positional measure, shape measure, directional measure and topological measure. The matching strategies for point features, linear features and areal features are also provided. The proposed method is implemented in a prototype for matching features from two different data sources, and is compared with traditional methods. The results demonstrate not only the practicability of using the proposed method to resolve feature matching issues in map conflation, but also its advantages compared with traditional methods in terms of matching effects.

title: AUTOMATICALLY AND ACCURATELY MATCHING OBJECTS IN GEOSPATIAL DATASETS
abstract: Identification of the same object represented in diverse geospatial datasets is a fundamental problem in spatial data handling and a variety of its applications. This need is becoming increasingly important as extraordinary amounts of geospatial data are collected and shared every day. Numerous difficulties exist in gathering information about objects of interest from diverse datasets, including different reference systems, distinct generalizations, and different levels of detail. Many research efforts have been made to select proper measures for matching objects according to the characteristics of involved datasets, though there appear to have been few if any previous attempts to improve the matching strategy given a certain criterion. This paper presents a new strategy to automatically and simultaneously match geographical objects in diverse datasets using linear programming, rather than identifying corresponding objects one after another. Based on a modified assignment problem model, we formulate an objective function that can be solved by an optimization model that takes into account all potentially matched pairs simultaneously by minimizing the total distance of all pairs in a similarity space. This strategy and widely used sequential approaches using the same matching criteria are applied to a series of hypothetical point datasets and real street network datasets. As a result, our strategy consistently improves global matching accuracy in all experiments.  Corresponding author.

title: An optimisation model for linear feature matching in geographical data conflation
abstract: Issues of heterogeneity and incompatibility in geospatial data become increasingly important as data sources become more abundant. Scientific research and decision-making usually require geospatial data from a variety of sources, since it is not realistic to collect all data directly; therefore, it is important to effectively utilise data created by various agencies using different methodologies under different circumstances. The term conflation here refers to the problem of combining incompatible geospatial data. One crucial component in conflation is feature matching, which is a prerequisite for the subsequent steps such as feature transformation. Although previous research has provided different methods of feature matching for specific applications, most of them have relied on a greedy strategy to execute the matching process. This article develops a new optimisation model to improve linear feature matching in situations with one-to-one, one-to-many and one-to-none correspondences by extending the optimised feature matching method proposed by Li and Goodchild (Automatically and accurately matching objects in geospatial datasets.  2010). Considering all possible matched pairs simultaneously, this new model achieves a high percentage of correctly matched features by maximising the total similarity between all matched pairs. When autocorrelated distortions exist in the datasets, an affine transformation can be integrated into the feature matching to improve the matching results. In addition, this study takes advantage of the asymmetry of a dissimilarity metric – directed Hausdorff distance – to address one-to-many correspondences.

title: A linear road object matching method for conflation based on optimization and logistic regression
abstract: The purpose of object matching in conflation is to identify corresponding objects in different data sets that represent the same real-world entity. This article presents an improved linear object matching approach, named the optimization and iterative logistic regression matching (OILRM) method, which combines the optimization model and logistic regression model to obtain a better matching result by detecting incorrect matches and missed matches that are included in the result obtained from the optimization (Opt) method for object matching in conflation. The implementation of the proposed OILRM method was demonstrated in a comprehensive case study of Shanghai, China. The experimental results showed the following. (1) The Opt method can determine most of the optimal one-to-one matching pairs under the condition of minimizing the total distance of all matching pairs without setting empirical thresholds. However, the matching accuracy and recall need to be further improved. (2) The proposed OILRM method can detect incorrect matches and missed matches and resolve the issues of one-to-many and many-to-many matching relationships with a higher matching recall. (3) In the case where the source data sets become more complicated, the matching accuracy and recall based on the proposed OILRM method are much better than those based on the Opt method.

title: A polygon-based approach for matching OpenStreetMap road networks with regional transit authority data
abstract: Matching road networks is an essential step for data enrichment and data quality assessment, among other processes. Conventionally, road networks from two datasets are matched using a line-based approach that checks for the similarity of properties of line segments. In this article, a polygon-based approach is proposed to match the OpenStreetMap road network with authority data. The algorithm first extracts urban blocks that are central elements of urban planning and are represented by polygons surrounded by their surrounding streets, and it then assigns road lines to edges of urban blocks by checking their topologies. In the matching process, polygons of urban blocks are matched in the first step by checking for overlapping areas. In the second step, edges of a matched urban block pair are further matched with each other. Road lines that are assigned to the same matched pair of urban block edges are then matched with each other. The computational cost is substantially reduced because the proposed approach matches polygons instead of road lines, and thus, the process of matching is accelerated. Experiments on Heidelberg and Shanghai datasets show that the proposed approach achieves good and robust matching results, with a precision higher than 96% and a F1-score better than 90%.

title: Aggregation and Fusion of Imperfect Information
abstract: Aggregation Operators for Fusion under Fuzziness: S. Ovchinnikov: On robust aggregation procedures R. Mesiar, M. Komornikova: Triangular norm-based aggregation of evidence under fuzziness J. Fodor, T. Calvo: Aggregation functions defined by t-norms and t-conorms.- Aggregation in Decision Making and Control: M. Grabisch: Fuzzy integral as a flexible and interpretable tool of aggregation A. Kelman,  Using priorities in aggregation connectives V. Cutello,  Aggregation operators for fuzzy rationality measures M.T. Lamata: Aggregation in decision making with belief structures  Multistage fuzzy control with a soft aggregation of stage scores.- Fusion of Complementary Information: S. Benferhat, D. Dubois,  From semantic to syntactic approaches to information combination in possibilistic logic S. Moral, J. del Sagrado: Aggregation of imprecise probabilities I. Bloch, H. Maitre: Fusion of image information under imprecision G. Mauris, E. Benoit, L. Foulloy: Fuzzy linguistic methods for the aggregation of complementary sensor information A. Appriou: Uncertain data aggregation in classification and tracking processes M. Sato, Y. Sato: A generalized fuzzy clustering model based on aggregation operators and its applications.

title: Error detecting and error correcting codes
abstract: The author was led to the study given in this paper from a consideration of large scale computing machines in which a large number of operations must be performed without a single error in the end result. This problem of “doing things right” on a large scale is not essentially new; in a telephone central office, for example, a very large number of operations are performed while the errors leading to wrong numbers are kept well under control, though they have not been completely eliminated. This has been achieved, in part, through the use of self-checking circuits. The occasional failure that escapes routine checking is still detected by the customer and will, if it persists, result in customer complaint, while if it is transient it will produce only occasional wrong numbers. At the same time the rest of the central office functions satisfactorily. In a digital computer, on the other hand, a single failure usually means the complete failure, in the sense that if it is detected no more computing can be done until the failure is located and corrected, while if it escapes detection then it invalidates all subsequent operations of the machine. Put in other words, in a telephone central office there are a number of parallel paths which are more or less independent of each other; in a digital machine there is usually a single long path which passes through the same piece of equipment many, many times before the answer is obtained.

title: Verb Semantics and Lexical Selection
abstract: This paper will focus on the semantic representation of verbs in computer systems and its impact on lexical selection problems in machine translation (MT). Two groups of English and Chinese verbs are examined to show that lexical selection must be based on interpretation of the sentences as well as selection restrictions placed on the verb arguments. A novel representation scheme is suggested, and is compared to representations with selection restrictions used in transfer-based MT. We see our approach as closely aligned with knowledge-based MT approaches (KBMT), and as a separate component that could be incorporated into existing systems. Examples and experimental results will show that, using this scheme, inexact matches can achieve correct lexical selection.

title: Introduction to Graph Theory
abstract: 1. Fundamental Concepts. What Is a Graph? Paths, Cycles, and Trails. Vertex Degrees and Counting. Directed Graphs. 2. Trees and Distance. Basic Properties. Spanning Trees and Enumeration. Optimization and Trees. 3. Matchings and Factors. Matchings and Covers. Algorithms and Applications. Matchings in General Graphs. 4. Connectivity and Paths. Cuts and Connectivity. k-connected Graphs. Network Flow Problems. 5. Coloring of Graphs. Vertex Colorings and Upper Bounds. Structure of k-chromatic Graphs.  Planar Graphs. Embeddings and Euler's Formula. Characterization of Planar Graphs. Parameters of Planarity. 7. Edges and Cycles. Line Graphs and Edge-Coloring. Hamiltonian Cycles. Planarity, Coloring, and Cycles. 8. Additional Topics (Optional). Perfect Graphs. Matroids. Ramsey Theory. More Extremal Problems. Random Graphs. Eigenvalues of Graphs.

title: Online map-matching based on Hidden Markov model for real-time traffic sensing applications
abstract: In many Intelligent Transportation System (ITS) applications that crowd-source data from probe vehicles, a crucial step is to accurately map the GPS trajectories to the road network in real time. This process, known as map-matching, often needs to account for noise and sparseness of the data because (1) highly precise GPS traces are rarely available, and (2) dense trajectories are costly for live transmission and storage. We propose an online map-matching algorithm based on the Hidden Markov Model (HMM) that is robust to noise and sparseness. We focused on two improvements over existing HMM-based algorithms: (1) the use of an optimal localizing strategy, the variable sliding window (VSW) method, that guarantees the online solution quality under uncertain future inputs, and (2) the novel combination of spatial, temporal and topological information using machine learning. We evaluated the accuracy of our algorithm using field test data collected on bus routes covering urban and rural areas. Furthermore, we also investigated the relationships between accuracy and output delays in processing live input streams. In our tests on field test data, VSW outperformed the traditional localizing method in terms of both accuracy and output delay. Our results suggest that it is viable for low latency applications such as traffic sensing.

title: Hidden Markov map matching through noise and sparseness
abstract: The problem of matching measured latitude/longitude points to roads is becoming increasingly important. This paper describes a novel, principled map matching algorithm that uses a Hidden Markov Model (HMM) to find the most likely road route represented by a time-stamped sequence of latitude/longitude pairs. The HMM elegantly accounts for measurement noise and the layout of the road network. We test our algorithm on ground truth data collected from a GPS receiver in a vehicle. Our test shows how the algorithm breaks down as the sampling rate of the GPS is reduced. We also test the effect of increasing amounts of additional measurement noise in order to assess how well our algorithm could deal with the inaccuracies of other location measurement systems, such as those based on WiFi and cell tower multilateration. We provide our GPS data and road network representation as a standard test set for other researchers to use in their map matching work.

title: Error bounds for convolutional codes and an asymptotically optimum decoding algorithm
abstract: The probability of error in decoding an optimal convolutional code transmitted over a memoryless channel is bounded from above and below as a function of the constraint length of the code. For all but pathological channels the bounds are asymptotically (exponentially) tight for rates above R_{0} , the computational cutoff rate of sequential decoding. As a function of constraint length the performance of optimal convolutional codes is shown to be superior to that of block codes of the same length, the relative improvement increasing with rate. The upper bound is obtained for a specific probabilistic nonsequential decoding algorithm which is shown to be asymptotically optimum for rates above R_{0} and whose performance bears certain similarities to that of sequential decoding algorithms.

title: Inferring the Scale of OpenStreetMap Features
abstract: Traditionally, national mapping agencies produced datasets and map products for a low number of specified and internally consistent scales, i.e. at a common level of detail (LoD). With the advent of projects like OpenStreetMap, data users are increasingly confronted with the task of dealing with heterogeneously detailed and scaled geodata. Knowing the scale of geodata is very important for mapping processes such as for generalization of label placement or land-cover studies for instance. In the following chapter, we review and compare two concurrent approaches at automatically assigning scale to OSM objects. The first approach is based on a multi-criteria decision making model, with a rationalist approach for defining and parameterizing the respective criteria, yielding five broad LoD classes. The second approach attempts to identify a single metric from an analysis process, which is then used to interpolate a scale equivalence. Both approaches are combined and tested against well-known Corine data, resulting in an improvement of the scale inference process. The chapter closes with a presentation of the most pressing open problems.

title: Self-organized natural roads for predicting traffic flow: a sensitivity study
abstract: In this paper, we extended road-based topological analysis to both nationwide and urban road networks, and concentrated on a sensitivity study with respect to the formation of self-organized natural roads based on the Gestalt principle of good continuity. Both annual average daily traffic (AADT) and global positioning system (GPS) data were used to correlate with a series of ranking metrics including five centrality-based metrics and two PageRank metrics. It was found that there exists a tipping point from segment-based to road-based network topology in terms of correlation between ranking metrics and their traffic. To our great surprise, (1) this correlation is significantly improved if a selfish rather than utopian strategy is adopted in forming the self-organized natural roads, and (2) point-based metrics assigned by summation into individual roads tend to have a much better correlation with traffic flow than line-based metrics. These counter-intuitive surprising findings constitute emergent properties of self-organized natural roads, which are intelligent enough for predicting traffic flow, thus shedding substantial light on the understanding of road networks and their traffic from the perspective of complex networks.

title: Declarative task delegation in OpenMOLE
abstract: In this paper we present OpenMOLE, a scientific framework providing a virtualized runtime environment for distributed computing. Current distributed execution systems do not hide the hardware and software heterogeneity of computing and data resources whereas OpenMOLE provides generic services to develop distributed scientific algorithms independently from the execution environment architecture. OpenMOLE uses abstraction layers to delegate computing tasks with the same high level interface for the major underlying architectures: local processors, batch systems, computational grids, Internet computing and cloud computing. The file access abstraction layer is another key feature helping a generic usage of the computation power provided by grids and clusters. The OpenMOLE framework has been tested with the exploration of a bacterial biofilm simulation with an individual-based model.

title: OpenMOLE, a workflow engine specifically tailored for the distributed exploration of simulation models
abstract: Complex-systems describe multiple levels of collective structure and organization. In such systems, the emergence of global behaviour from local interactions is generally studied through large scale experiments on numerical models. This analysis generates important computation loads which require the use of multi-core servers, clusters or grid computing. Dealing with such large scale executions is especially challenging for modellers who do not possess the theoretical and methodological skills required to take advantage of high performance computing environments. That is why we have designed a cloud approach for model experimentation. This approach has been implemented in OpenMOLE (Open MOdeL Experiment) as a Domain Specific Language (DSL) that leverages the naturally parallel aspect of model experiments. The OpenMOLE DSL has been designed to explore user-supplied models. It delegates transparently their numerous executions to remote execution environment. From a user perspective, those environments are viewed as services providing computing power, therefore no technical detail is ever exposed. This paper presents the OpenMOLE DSL through the example of a toy model exploration and through the automated calibration of a real-world complex-system model in the field of geography.

title: Half a Billion Simulations: Evolutionary Algorithms and Distributed Computing for Calibrating the Simpoplocal Geographical Model
abstract: Multiagent geographical models integrate very large numbers of spatial interactions. In order to validate these models a large amount of computing is necessary for their simulation and calibration. Here a new data-processing chain, including an automated calibration procedure, is tested on a computational grid using evolutionary algorithms. This is applied for the first time to a geographical model designed to simulate the evolution of an early urban settlement system. The method enables us to reduce the computing time and provides robust results. Using this method, we identify several parameter settings that minimize three objective functions that quantify how closely the model results match a reference pattern. As the values of each parameter in different settings are very close, this estimation considerably reduces the initial possible domain of variation of the parameters. Thus the model is a useful tool for further multiple applications in empirical historical situations.

title: Growing Models from the Bottom Up. an Evaluation-Based Incremental Modelling Method (EBIMM) Applied to the Simulation of Systems of Cities
abstract: This paper presents an incremental method of parsimonious modelling using intensive and quantitative evaluation. It is applied to a research question in urban geography, namely how well a simple and generic model of a system of cities can reproduce the evolution of Soviet urbanisation. We compared the ability of two models with different levels of complexity to satisfy goals at two levels. The macro-goal is to simulate the evolution of the systemâ€™s hierarchical structure. The micro-goal is to simulate its micro-dynamics in a realistic way. The evaluation of the models is based on empirical data through a calibration that includes sensitivity analysis using genetic algorithms and distributed computing. We show that a simple model of spatial interactions cannot fully reproduce the observed evolution of Soviet urbanisation from 1959 to 1989. A better fit was achieved when the modelâ€™s structure was complexified with two mechanisms. Our evaluation goals were assessed through intensive sensitivity analysis. The complexified model allowed us to simulate the evolution of the Soviet urban hierarchy.

title: A New Method to Evaluate Simulation Models: The Calibration Profile (CP) Algorithm
abstract: Models of social systems generally contain free parameters that cannot be evaluated directly from data. A calibration phase is therefore necessary to assess the capacity of the model to produce the expected dynamics. However, despite the high computational cost of this calibration it doesn't produce a global picture of the relationship between the parameter space and the behaviour space of the model. The Calibration Profile (CP) algorithm is an innovative method extending the concept of automated calibration processes. It computes a profile that depicts the effect of each single parameter on the model behaviour, independently from the others. A 2-dimensional graph is thus produced exposing the impact of the parameter under study on the capacity of the model to produce expected dynamics. The first part of this paper is devoted to the formal description of the CP algorithm. In the second part,we apply it to an agent based geographical model (SimpopLocal). The analysis of the results brings to light novel insights on the model.

title: Distributed computing in practice: the Condor experience
abstract: Since 1984, the Condor project has enabled ordinary users to do extraordinary computing. Today, the project continues to explore the social and technical problems of cooperative computing on scales ranging from the desktop to the world‐wide computational Grid. In this paper, we provide the history and philosophy of the Condor project and describe how it has interacted with other projects and evolved along with the field of distributed computing. We outline the core components of the Condor system and describe how the technology of computing must correspond to social structures. Throughout, we reflect on the lessons of experience and chart the course travelled by research ideas as they grow into production systems.

title: The Open Science Commons for the European Research Area
abstract: Nowadays, research practice in all scientific disciplines is increasingly, and in many cases exclusively, data driven. Knowledge of how to use tools to manipulate research data, and the availability of e-Infrastructures to support them for data storage, processing, analysis and preservation, is fundamental. In parallel, new types of communities are forming around interests in digital tools, computing facilities and data repositories. By making infrastructure services, community engagement and training inseparable, existing communities can be empowered by new ways of doing research, and new communities can be created around tools and data. Europe is ideally positioned to become a world leader as provider of research data for the benefit of research communities and the wider economy and society. Europe would benefit from an integrated infrastructure where data and computing services for big data can be easily shared and reused. This is particularly challenging in EO given the volumes and variety of the data that make scalable access difficult, if not impossible, to individual researchers and small groups (i.e. to the so-called long tail of science). To overcome this limitation, as part of the European Commission Digital Single Market strategy, the European Open Science Cloud (EOSC) initiative was launched in April 2016, with the final aim to realise the European Research Area (ERA) and raise research to the next level. It promotes not only scientific excellence and data reuse, but also job growth and increased competitiveness in Europe, and results in Europe-wide cost efficiencies in scientific infrastructure through the promotion of interoperability on an unprecedented scale. This chapter analyses existing barriers to achieve this aim and proposes the Open Science Commons as the fundamental principles to create an EOSC able to offer an integrated infrastructure for the depositing, sharing and reuse of big data, including Earth Observation (EO) data, leveraging and enhancing the current e-Infrastructure landscape, through standardization, interoperability, policy and governance. Finally, it is shown how an EOSC built on e-Infrastructures can improve the discovery, retrieval and processing capabilities of EO data, offering virtualised access to geographically distributed data and the computing necessary to manipulate and manage large volumes. Well-established e-Infrastructure services could provide a set of reusable components to accelerate the development of exploitation platforms for satellite data solving common problems, such as user authentication and authorisation, monitoring or accounting.