refactored dedup code, created more options, tried getting rid of at least one edge scoring bug