I added a unit test for this scenario and the fix with it. I'm still not going through the list of timeouts because I sense it may be overkill.
Based on the monitoring metrics, we should not have many client flipping which could create too many timeouts. If we do, they will be cleared at most every second (based on the above comment on the min timeout).