stewartmckee/cobweb — pull request #5 by rojotek (2012-04-24 13:30:54+00:00)
Title: Change to ensure that utf8 content can be crawled.

...ncode content before it is given to resque, wrapping the user specified processing_queue in a special queue named "encoding_safe_process_job", which will decode the content before giving it back to the processing_queue to handle. Ensure that there is a encoding_safe_process_job worker started.