Merge branch 'master' of https://github.com/stewartmckee/cobweb
Adding Makefile so that we can run the build using Jenkins in the same
manner that we run ganymede
Merge branch 'master' of github.com:ephox/cobweb
Fixing makefile to be correct dependency chain
Resolution for defect GALAXY-671.  When character sets dont match to expected, convert to UTF8.
Adding fixes for tests
Adding temrination checking support so that jobs can be requeued when
terminated.
There is a decent amount of coupling here between the library and the
job doing the processing, however there really isnt any other way
currently
Merge branch 'cobweb2'
Simplifying the crawling mechanisms so that link finding and processing are distinct entities. This means that the processing doesnt need to know anything about
the link finding and makes each part a lot conceptually simpler. Also makes it much easier to reason about since we are not dealing with mutliple
redis locks.
Fixing the problem with gems Until I can get charlock_holmes to work on heroku
Removing depeedency on charlock_holmes since it doesnt work on
heroku. Until the issues with charlock_holmes can be resolved, we are
reverting to very naieve character detection, and thus the bug raised
in GALAXY-671 will still be present
Changing signal processing to use Resque::TermException. We are tied
to resque anyway so making the exception catching generic is kind of
pointless
Also changed some logging statements to be in debug mode only
Removing generic exception handling to make sure errors propogate correctly
Bunping version numbers
Updated to version of charlock_holmes that works, and remove rchardet
which is broken
fixing bug that occurs for certain sites when the text content options is passed but is subsequently changed by the page headers
Merge branch 'master' into 784-requeue-on-termination-signal
Merge branch 'master' into cobweb2