Set LDFLAGS, not LIBRARY_PATH, when not building libs.