Fix link order of test binary to resolve build failure

Thanks to Daniel T Chen <crimsun@ubuntu.com> for providing the patch via Ubuntu.
Remove unneeded linker flags in compile-only make target