Validation solves this issue by providing a means to distinguish between artefacts and true cell behaviour. Increasing the trustworthiness of tracking results requires a conservative validation strategy. It must not aim to maximize the success rate (i.e. the number of complete paths and genealogic trees), but to minimize the risk of erroneous results, even at the expense of losing many paths. The detection of errors is more important than their rectification, as this may introduce new errors. Table 2 shows that our validation method rejects the majority of paths as untrustworthy; only 377 complete paths (23%) out of 1635 are returned as tracking results. However, the trustworthiness of these paths exceeds 95%, which is a huge improvement over our unvalidated tracking data (naive tracking) with a trustworthiness of 56.4% only (see table 2)and therefore an indispensable precondition for the subsequent derivation of statistical measures. The argument could be raised, that validation may introduce a bias by accepting specific cell types only. In fact, every detection strategy, be it cell or error detection, relies on basic assumptions about the detected subject and thus, is generally threated by the danger of being biased. However, our validation algorithm implies only very general assumptions about the spatiotemporal contiguity of cell paths. In particular, we expect that cells do neither emerge ‘‘out of nothing’’ nor vanish (cf. fig 5). Compared to the assumptions, which need to be made e.g. for cell detection – e.g. about cell shape and cell behaviour – our validation algorithm appears to be much less critical and less prone to introduce bias. The only cellular event, which might be erroneously not taken into account, is cell fusion. Putting aside that cell fusion appears to be rather rare in vitro, the validation algorithm could be easily adjusted to include those as well: The already existing merged-cell detection would have be augmented with the condition that – in contrast to merged cells – fused cells are not allowed to depart later. Besides this reasoning, we have also experimentally checked, whether the validation algorithm introduces a bias. Comparing the validated tracking results with the manually created reference data reveals that the validated/accepted paths do not constitute a particularly selected subset, but instead represent the true distribution of properties (comparison of life time distributions, see fig. 13a). In contrast, waiving validation heavily distorts the distribution function (green data points in fig 13a). Also, we have tested the reproducibility of our validated tracking through an additional experiment, in which we observed pancreatic stem cells in passage 31 at 10 different coordinates, i.e. we made 10 movies by manually selecting different positions in one cell dish (see Figure S1 and Video S5, S6, S7, S8, S9, S10, S11, S12, S13, S14). Afterwards, we compared those to one another (Figure 13b) and to the reference data sets (Figure 13c). It is apparent that the variances are in both cases small. Additionally, we have tested our tracking/validation approach with another cell type (human dermal fibroblasts in passage 7, data set D). These cells, although quite different from pancreatic stem cells (e.g. more elongated form, much slower doubling time), can be handled equally well by our algorithm (see Figure S2 and Video S15, S16, S17, S18). One important result of of validation in automated cell tracking is the fact that complete paths and statistical distributions of cell properties are the natural outcome of such an algorithm, rather than single cell information and genealogic trees. This is a consequence of the conservative validation approach, which accepts only error-free, complete, but often non-connected paths. In our case, we found 23% of all complete paths and less than 8% of all possible genealogic assignments in the last frame. However, the statistical parameters that can be derived from validated Automated Tracking Turns Cell Culture into Numbers tracking data are highly significant and a logical way to describe the properties of large cell populations. Should the validation process leave too small a sample size, the number of paths entering the analysis can always be increased by enlarging the observed area. Thus, validation turns the qualitative problem of achieving reliable tracking results into a quantitative problem of data acquisition and computing power. This turn substantially extends the possible applications of cell tracking techniques, because it allows for the non-invasive, quantitative description of cell cultures in a fully automated manner. Figures like life-time distributions, growth curves, sibling symmetry, motion analyses etc. can be used to parameterize the conventional cell culture, i.e. turn its qualitative nature into biologically meaningful numbers. In particular, it may become possible to assess the heterogeneity of a cell population merely by "looking at the cells" through its life time distribution. To the best of our knowledge, single cell tracking is the only method in existence to non-invasively analyze life time distributions and sibling symmetries. Every other method, e.g. genetically engineered reporter systems, BrdU or EdU assembly, immunofluorescence etc. either kills or at least heavily disturbs the cells. II) Evaluation Although quite a number of methodical work on cell tracking has been published within recent years, most papers stood rather isolated and could not relate to one another in a constructive scientific discourse. This was mainly due to a lack of both, common evaluation criteria and common reference data. Among the methods previously used to construct reference data were manual lineage construction from real time-lapse data [8,12,14,28], often performed on smaller subsets [10,19,20], artificially created data [16,22] as well as manual mitosis detection [39]. Unfortunately, these reference data are all afflicted with one or another disadvantage; they are either not publicly available, too small for statistical measures, contain too few mitoses, or are based on labelling/life staining. Here, we publish a data set with a significantly increased number of tracked cells (.240.000 cell in reference set A, .80.000 cells in set B; 6120/573 paths; 2019/573 mitoses); for unlabeled cells this increase amounts to roughly two orders of magnitude (for a comparison of different tracking work refer to Table S1). These numbers along with their ‘‘real life nature’’ make the reference data sets suitable as a benchmark for testing future cell tracking algorithms. Diversity can be also found among the previous evaluation criteria; mostly, they have been adapted to the particular tracking approach and are not generally suitable for comparing different tracking algorithms. A notable exception is the work of Hand et al., who undertook the effort of comparing six different tracking methods [16]. For this purpose, they used criteria, which are very similar to ours (cell detection rates, complete path detection rates). Yet, they have worked with an artificially created data set only, which contained no mitoses. This prevented them from deriving measures for evaluating the genealogic nexus of the data. For the evaluation of real world data (with mitoses, imaging errors etc), we therefore propose to use our enhancement to their evaluation criteria as a starting point for scoring and comparing different tracking approaches. Conclusions Based on the notion that no measurement is error-free, and error propagation in cell-tracking is particularly bad, we demonstrate, that validation is obligatory to achieve fully PLoS ONE | www.plosone.org 13 November 2011 | Volume 6 | Issue 11 | e27315