Performance improvement: avoid resizing the hashset that often

Signed-off-by: Irmo van den Berge <bergerkiller@gmail.com>