scanner: better error handling

Signed-off-by: zeljko <zeljko@zwr.fi>
more robust scanner

This commit fixes several interconnected issues (which could
have been handled separately, but they were not):

1. Each scan is now a separate sidekiq task, which is more
how sidekiq likes it.

2. Long urls (over 800 characters) are not saves as such, but
their hash is saved. This is because MongoDB does not like such
strings to index. Of course separatelly and non-indexed URl is saved.
This will influence and potentially break some filters (but only for
sites which do have long URLs, which is currently only bestseller.com).

3. Error handling improved and logged. Also, cookies are restarted
on each error.

Signed-off-by: zeljko <zeljko@zwr.fi>