build: Set clang-specific build options only for clang

Fixes #172