Fix #428

This only checks for NaNs in the x and y data, but that is all that
goes in the index, so should be sufficient.