#154 Improvements to website deduplication logic

as discussed with @cantino