build: Fix conditional build of GObject

In case we don't have GObject but we do have GLib, for some weird
reason, we don't want to forcibly depend on the former.