runtime: fix integer overflow in hashmap
The test is problematic, because it requires 8GB+ of RAM.
Fixes issue 5239.

R=golang-dev, bradfitz
CC=golang-dev
https://codereview.appspot.com/8550043