Crawling continuously, added 30 second timeout on bad request.
Everything works until a book with no recommended books is reached. Book selection from recommended books is more automated.