Added code to remove duplicate links and retrieve all pages