Fix build system a bit

We should use $(GST_PLUGIN_LDFLAGS) everywhere and also make sure
to pass all relevant libs and cflags and everything around.

Some libs might still be missing.