client: Don't user g_source_remove() on non-default main context

We created a GSource, attached it to a non-default main context and
then removed it via g_source_remove, which removed the source with the
same id on the main context, which typically was the X fd, causing
everything to stop working!

Non-default main contexts must use g_source_destroy()

https://bugzilla.gnome.org/show_bug.cgi?id=708744