new final commit, added bad data checks