go: fails_with :clang when cross-compiling

These fails_with conditions are getting far too complex and probably
should be simplified.

Fixes #22610.