updated README and refactored code to deal with non-valid urls, blank lines, and duplicates