Fix cherrypick cancellation properly