Only use Clang-specific pragma if compiler is Clang

Otherwise the compiler will probably give us a warning that these pragmas are unknown. By default that warning is disabled with our own build system, but we also want to be able to link to our library without incorporating the entire build system too.

This shouldn't have any repercussions; I suggest testing it together with some other pull request to not spend too much time on that.