Ground Truth Free Evaluation of Segment Based Maps

For ground truth free map evaluation, map intrinsic properties have to be utilized in order to define a measure of quality. This paper suggests the property of map consistency, based on the similarity of different model ele ments describing a single feature in the environment. The evaluat ion algorithm is tailored for segment based maps. Respecting th e segment representation, it inherits the properties of memo ryand runtime efficiency. In comparison to classic grid based evaluation models, which are mostly used for evaluation of maps represented by points, it is also able to detect structural inconsistencies of maps, even if the model elements (t he segments) overlap. The paper explains the core technique, hierarchical clustering of segments, illustrates propert ies of the underlying segment similarity measures and cluster confide nce measure. Experiments on different segment based maps show its consistency with visual inspection. I. I NTRODUCTION AND APPROACH Robot maps are the basis for nearly all autonomous navigational tasks in robotics. It is therefore an importan t sk to evaluate the performance of robot mapping systems with respect to different parameters, e.g. precision, complete ness and consistency. Mapping, in general, is spatial analysis o f environmental features of interest. Inherent to this proce ss is its task dependency, hence there is no ’optimal general mapping’. As a direct consequence, there is no single best strategy to evaluate maps. Completeness might be preferred to precision in path planning tasks, while precision might b e of higher importance to locally bounded tasks. In fact, map evaluation turns out to be a field with many different aspects . The showcases of precision and completeness are parameters which can only be evaluated if a trusted model, the ground truth, is available. The field of robot map evaluation opens up even wider if ground truth independent techniques are taken into account as well. The parameter of consistency, i.e. a parameter that describes a map intrinsic property, replaces the aforementioned examples in such cases: precision and completeness can only be estimated based on the consistency of features. Both, ground truth based and non ground truth based evaluation deal with aspects of distance; either distance to an external reference, or an intra-distance between elements of a single map. The distance aspect is expressed by the choice of a particular distance-measure (which is not necessarily distance metric). The most common choices, Euclidean distance of map-features or angular/translatio n l (pose) distance, lead togrid-based evaluation, andposebased evaluation respectively. Rolf Lakaemper is with the Department of Computer and Information Sciences, Temple University, Philadelphia, P A, USA lakamper@temple.edu Additionally, design of map a evaluation strategy is an engineering process, in the sense that it should respect advantages/disadvantages of different data representati o s. In practice, the way of robot internal representation of the physical world is carefully chosen; not following the speci fic data representation, but transforming the model, might ren der the evaluation useless (an example is given in section VIIA, where map evaluation is suggested to provide a break condition for an iterative mapping process). An example for such a case is the evaluation described in this paper: it respects the underlying data structure of lin esegments. One argument for line-segment based maps is its efficiency. A transformation to point based map, e.g. for gri d based evaluation, could be too costly in cases where online, real time evaluation is needed. Using the aforementioned categories, the presented evaluation • is not ground truth based • measures the map consistency • works on line segments as underlying data structure