[server][ActorCollector] Fix a bug that potentially calls g_source_remove with an invalid ID.