Ignore the selection clear event. This should stop follow mode from jumping around a lot.