Resolution for defect GALAXY-671.  When character sets dont match to expected, convert to UTF8.
Adding fixes for tests
Adding temrination checking support so that jobs can be requeued when
terminated.
There is a decent amount of coupling here between the library and the
job doing the processing, however there really isnt any other way
currently
Fixing the problem with gems Until I can get charlock_holmes to work on heroku
Removing depeedency on charlock_holmes since it doesnt work on
heroku. Until the issues with charlock_holmes can be resolved, we are
reverting to very naieve character detection, and thus the bug raised
in GALAXY-671 will still be present
Changing signal processing to use Resque::TermException. We are tied
to resque anyway so making the exception catching generic is kind of
pointless
Also changed some logging statements to be in debug mode only
Removing generic exception handling to make sure errors propogate correctly
Bunping version numbers
Updated to version of charlock_holmes that works, and remove rchardet
which is broken
fixing bug that occurs for certain sites when the text content options is passed but is subsequently changed by the page headers
Merge branch 'master' into 784-requeue-on-termination-signal