improved data integrity, validations and tests