Fix 32bit build

Similar to the fix for #2291.

Twitter conversation: https://twitter.com/chuhnk/status/535906099351003136

To test this out, add `linux/arm` to `KUBE_SERVER_PLATFORMS` in `hack/lib/golang.sh`.
If we want a hack, we could min with maxint.