core: use mask instead of modulo in hash table

Link: https://github.com/GeneAssembly/biosal/issues/724
Signed-off-by: Sébastien Boisvert <boisvert@anl.gov>