Store fake timers in Set instead of Map