Even after setting the memory parameters, we still need to insert a lot into the map to induce spilling. I have been able to trigger the exception that you found with the following:

(1 until 100000).foreach { i => map.insert(i, i) }
map.insert(Int.MaxValue, Int.MaxValue)

val it = map.iterator
while (it.hasNext) {
  it.next()
}