improve perf of InMemoryTimeoutPersister

so this gives a ~4x improvement in the perf of InMemoryTimeoutPersister

perhaps with this then #2406 might not worth fixing ??