build: Avoid poisoning the host binaries with bad ABIs

Change-Id: I5fa78a7a8d35cbf4d6f19f92863ee2194f2c30e1