Have you considered using Elementree directly on the returned html instead of having to depend on BeautifulSoup ?