test for Float64HashTable with nan