Don't fill new grids unless we need it.

Also update places where it isn't needed.