build: Avoid poisoning the host binaries with bad ABIs

the vector class symbols are only needed on the target. Don't inject
(or expect) them into the host binaries to avoid multi-device build
conflicts and ABI breakage

Change-Id: Ic3fce3d00e41e74d49afd753268223efd1541ae5