build: Use LDADD for libraries, not LDFLAGS

https://bugzilla.gnome.org/show_bug.cgi?id=724853