RF: don't use -fopenmp flag if it doesn't work

Fairly general refactor of setup to try candidate compile flags, and remove
flags from compile that cause a compile or link error.

MinRK's pyzmq zeromq build machinery was the main inspiration for this way of
doing it.