Ensure we don't stop twice if we encounter an exception for ticking threads