cmake: support building 32 bit binaries on multilib host

This tries to ensure cmake won't find 64 bit libraries when trying to
build 32 bit binaries on a multilib system by passing -DBUILD_X64=Off

This also passes -m32 as a linker flag

This avoids needing to manually pass a toolchain.cmake file via
-DCMAKE_TOOLCHAIN_FILE=