Slightly refactor CFLAGS/LDFLAGS/LIBS

This way, we can avoid -rdynamic and -pthread warnings on darwin.
Spaces to tabs
Inherit CC for Lua
make check is a common naming convention for tests
Remove extra spaces
Only pass -rdynamic as linker option
Silence mkdir output
use `install`  as default installer (except on SunOS)