And the evidence is that they are far from perfect. Correlations apparent links between two variables are another much-prized outcome of DM that are fraught with pitfalls. Statisticians have known for years the dangers of confusing correlation with causation, especially when the apparent link makes little sense. For example, there s a statistically highly significant correlation between the stork populations of 17 European countries and the corresponding human birth rates. Well, probably not.
Common sense is usually enough to demolish many such spurious correlations.
But mining commercial data can easily throw up cases that are not so easily dismissed. It would be a brave supermarket that puts its cat food between the tinned vegetables and instant mash. The apparently harmless pursuit of repeatedly scouring data in search of interesting findings also undermines the protection offered by these tests. To be fair, many data miners fully accept the dangers of their nugget-hunting, and some of its advocates are calling for more work to be done on developing better safeguards. One promising way forward lies in so-called Bayesian inference, which allows the crucial concept of plausibility to be built into the data mining process.