RF: don't use -fopenmp flag if it doesn't work

Fairly general refactor of setup to try candidate compile flags, and
remove flags from compile that cause a compile or link error.

MinRK's pyzmq zeromq build machinery was the main inspiration for this
way of doing it.