build: Avoid poisoning the host binaries with bad ABIs