[server][Helpers] Fix a problem that tries to remove the timer that has already been removed.