Never remove invalid GSource ids

Calling `g_source_remove` with an invalid GSource id is considered a
programming error since GLib 2.39 so we have to check that the id is
still valid before trying to remove it.

Signed-off-by: Niklas Koep <niklas.koep@gmail.com>