A number of competitors in the conference's e-mining contest were so aghast at the time they spent in data preparation that they halved the numbers before reporting them. More is gained in data cleaning and preparation than is often assumed. In the process, you gain essential pedestrian knowledge about the application domain and discover particulars that any model must address in some way. This task requires a lot more sophistication than just about everyone expects. He also offers software that uses intelligent extrapolation to speed up your efforts.
Automated methods can help.
One paper presented a knowledge-based intelligent data cleaner. Rule-based approaches can be complicated to build and are often application specific, but are particularly valuable when a great deal of data of the same ilk keeps arriving. In many areas, including e-commerce, text and knowledge management, algorithms, and commercial software, a leap to the next level of principles and generalizations is waiting to happen. Here are some examples. Although technology and business models seem to be evolving at lightning speed, customer satisfaction is not.