Neighbors is now a map to the actual neighboring point