Recovery: cleaner interrupt handling during cancellation

RecoveryTarget initiates the recovery by sending a start recovery request to the source node and then waits for the recovery to complete. During recovery cancellation, we interrupt the thread so it will wake up and clean the recovery. Depending on timing, this can leave an unneeded interrupted thread status causing future IO commands to fail unneeded.

RecoverySource already had a handy utility called CancelableThreads. This extracts it to a top level class, and uses it in RecoveryTarget as well.

This caused the failure in http://build-us-00.elasticsearch.org/job/es_core_master_strong/1755/
cancel doesn't really do much (it interrupts the threads) so you have no guarantee those will wake up before decRef() is called. They should hold their own reference. This way it just saves on a try finally within a try finally. Now that I think about it , the same argument holds for the decRef() . I'll change...