host of changes to prepare for auto dupe check, probably some bug fixes thrown in for good mesasure