Don't depend on gmodule

Since we no longer have a extension system, we don't need this anymore,
right?

https://bugzilla.gnome.org/show_bug.cgi?id=735171
Don't link with X11

https://bugzilla.gnome.org/show_bug.cgi?id=735171