removed inconsistent and duplicated records