Have the gems fall slower and draw them based on the actual position of the gem objects rather than using the grid values
Make sure holes aren't matched with each other and have the function that drops gems do it for any hole, not just -1