Optimize unfollow of dormant threads

We can detect the case where we followed a thread while it was asleep and
later try to unfollow before the thread woke up. Without this optimization
we may run out of memory if we accumulate dormant threads at a higher rate
than the rate at which they wake up.