Simplify gox toolchain building

We only build for 32-bit and 64-bit linux and 64-bit darwin so no need to build all 3 cross-compile toolchains.