ky applied to be nicer to the db
moved dupe checking to the parser
moved dupe checking to parser