The data cleansing process entails several steps to identify and fix problem entries. The first step is to analyze the data to identify errors. This may involve using qualitative analysis tools that use rules, patterns, and constraints to identify invalid values. The next step is to remove or correct errors.
Common data cleaning steps include remediating:
Duplicate data: Drop duplicate information
Irrelevant data: Identify critical fields for the particular analysis and drop irrelevant data from the analysis
Outliers: Outliers can dramatically affect model performance, so identify outliers and determine appropriate action
Missing data: Flag and drop or impute missing data
Structural errors: Correct typographical errors and other inconsistencies, and make data conform to a common pattern or convention