Only iterate over the necessary grid points