restarting crawler in spider, using urljoin and beautiful soup to format urls