Avoid diagnosing twice on non-x86 targets

The PowerPC and ARM native builders spotted this.