I would guard with a "if '---' in response.text" otherwise this will bomb with an IndexError if the URL is OK, but the page is malformed (no header).