Recovery: refactor RecoveryTarget state management

The PR rewrites the state controls in the RecoveryTarget family classes to make it easier to guarantee that:
- recovery resources are only cleared once there are no ongoing requests
- recovery is automatically canceled when the target shard is closed/removed
- canceled recoveries do not leave temp files behind when canceled. 

Highlights of the change:
1) All temporary files are cleared upon failure/cancel (see #7315 )
2) All newly created files are always temporary 
3) Doesn't list local files on the cluster state update thread (which throw unwanted exception)
4) Better resiliency to errors while trying to extract checksums (like missing files / corrupted shards)
5) Recoveries are canceled by a listener to IndicesLifecycle.beforeIndexShardClosed, so we don't need to explicitly call it.
6) Simplifies RecoveryListener to only notify when a recovery is done or failed. Removed subtleties like ignore and retry (they are dealt with internally)
I can definitely return the value. I'm a bit conflicted regarding the assert as strictly speaking we can't guarantee it will work due to the retry logic when may set the thread before the clear command of the previous thread run. In practice it shouldn't be a problem because it only kicks in after 500ms. But still, I'm not sure it adds value to assert here?