daemon: Don't try to remove non-existent GSource

The GSource would have already been removed when we quit the GMainLoop.

Fixes: https://bugzilla.gnome.org/731531