RF: don't use -fopenmp flag if it doesn't work

Fairly general refactor of setup to try candidate compile flags, and
remove flags from compile that cause a compile or link error.

MinRK was the main inspiration for this way of doing it.