updated to better calculate ranking
added code to pull out url parts for dedup