fixed the worker not writing down that it had processed a url that could not be successfully got, this was causing workers to endlessly try and get broken urls, fixed the worker not catching an error thrown by lxml when parsing a html page.

Signed-off-by: mep <miethpo@gmail.com>