Use the new g_source_set_name_by_id() functionality, and generally be more paranoid removing sources