Fix the timeout too long when min/max latency set to 0