Don't check for when (x,y) is in the dict already